In this section we describe how to create a delivery. A delivery is required to store facts in the Central Facts Layer.
There are two kinds of deliveries:
There are two ways to create a delivery. In most cases the delivery is created with an API. For developmental purposes, it is also possible to create and process a delivery manually:
External batch deliveries can be created through an API or via the i-refactory web app and will trigger automatic processing by i-refactory. Before creating a delivery, make sure the data is already in the entities in the Technical Staging Layer (TSL).
Create a delivery:
Menu > Supply > Delivery
.New
to create a new delivery. A form appears.
Delivery Agreement
. The delivery will inherit the relevant metadata of the Layer, Interface, Entities and Constraints.Received Date
. Submit the date on which the delivery was actually received.{warning} The
Received Date
you enter must always be later than theReceived Date
of previous deliveries.
Sent Date
. Submit the date on which delivery was actually sent.
Constraint Validation
needs to be executed. If you want to cancel the selection, click on No selection
.
None
: executes minimum technical prerequisite. The checks are fully based on database processing and are not part of the validation process. If you select this option and there are data errors, this may result in a database error or an other technical error.All
: all constraints are checked during the validation proces.Default
: all delivery agreements constraints in the default
group are checked during the validation process.Mandatory
: constraints that are inherent (mandatory) to the structure of a data model are checked during the validation process. The mandatory constraints to check are:
Deletes
in the dataset will be treated. Click here for more information about removing records and the differences between logical deletes and physical deletes.
Logical
: the deleted records will be flagged as deleted in the CFL. The logical deleted records are shown in the 'hist' views in the GDAL but not shown in the 'current view'. By default, this option is selected.Physical
: the context information of the deleted records is deleted in the CFL.Violated Constraint Max Sample Size
: maximum sample size of violated constraints that are logged. By default, this is 1000. A lower sample size can improve the throughput of the validation process and lowers the storage cost in case of a high volume of constraint violations. Sample size value is per constraint. So for each constraint a maximum of sample size records will be registered, including the record on which the constraint violated.
3.7. For delivery's on the Logical Validation Layer you can override the inherited includeExistingDataForSkippedRowsInSetValidation value from Delivery Agreement. includeExistingDataForSkippedRowsInSetValidationConclusion shows the computed value for this Delivery.
This setting prompts the user to decide whether to include rows that there rejected for basic constraints in previous deliveries by default the i-refactory tries to fetch the known data of these rejected rows from previous deliveries. This data is then used to perform set validations on rows in the current delivery that did not get rejected.
The table below shows how includeExistingDataForSkippedRowsInSetValidationConclusion
is computed for a Delivery.
Delivery value | Delivery Agreement value | Delivery Conclusion |
---|---|---|
DEFAULT | YES | YES |
DEFAULT | NO | NO |
YES | NO | YES |
NO | YES | NO |
Context Related Entities Contain New Rows
(for context-based deliveries): select this option if you are sure the context-related entities contain only new records. If this option is selected, i-refactory handles the delivery as a full delivery for every entity the context filter is set to True.{info} Selecting
Context Related Entities Contain New Rows
results in a quicker writing proces, because no difference is computed between a full and delta delivery set. The delivery won't delete any data, only save and update.
Message Number
: you can use this field to assign your own reference number to identify the deliveryMessage Text
: you can use this field to add additional information to identify the deliveryIn the 'Delivered entities' section, you can select which entities will be delivered and submit their snapshot date.
The following delivery loading strategies are available:
Complete
Delivery.Partial
Delivery.Full
Delivery of the related Entity.
Delta
Delivery of the related Entity.
Context Based Delivery
, you need to classify a foreign key relation between entities in your data model as use as context filter and select the driving entity as Delta
and the dependent entity as Full
.To customize the options:
Entities
are included in the delivery. You can unselect the checkbox to exclude entities, for example, for partial deliveries.Entities
are marked as Complete
. Unselect the checkbox to mark the entity as a delta delivery.Optionally set a Snapshot datetime
per entity. The Snapshot datetime
will be used as transaction datetime when registering facts in the Central Facts Layer. If you do not enter a specific Snapshot datetime, the transaction time will set to the system datetime of the i-refactory server.
{info} When to use Snapshot Datetime:
If you want strict control over the transaction timelines that manage the validity window of the provided records, you have the option to override the default behaviour of using the server datetime as the timestamp that opens new records and enddate deleted or updated records.
Use cases to do this is to align the target system managed by i-Refactory with the data validity timelines of the source system or even the source system entity. Another use could be to load historical datawarehouse data into the system managed by i-refactory and keep the already existing timelines available.
In all cases the snapshot datetime for a delivered entity must be after the last provided snapshot datetime.
ExampleSuppose there is already a delivery at Time 1 and a delivery at Time 3.You want to add a delivery at transaction Time 2. You cannot submit Time 2 as
Received Date
becauseReceived Date
needs to be later than Time 3 (this delivery is already in the database). However, you can enter Time 2 asSnapshot Datetime
and thetransaction datetime
in the database will correspond to Time 2.
If you choose the constraint validation level, All, Default, or Mandatory, you can set the Violated Constraint Max Sample Size
.
Enter a sample size to limit the logging of validation errors per validation. This will significantly improve the throughput of the validation process and lowers the storage cost in case of a high volume of constraint violations. If you do not set a value for Violated Constraint Max Sample Size
, 1000 validation error records are logged by default.
Sample size value is per constraint. So, for each constraint a maximum of sample size records will be registered, including the record on which the constraint violated.
{tip} A good practice is to chose the validation level
Mandatory
as a minimum option, since this controls minimum set of validations when loading data into the Central Facts Layer in a controlled manner.
The i-Refactory has built in logic to identify deleted records (records that do no longer exist).
For entities that are provided as full
, all records that already exist but are no longer available are considered 'relevant for deletion'. For entities delivered as delta
, the deletion indicator provided in the record will indicate 'relevant for deletion'.
The delete type
indicator, which you set when creating a delivery, affects how deletions are handled. The default scenario is a logical delete. This means the existing records will be end-dated and marked as deleted in the Logical Validation Layer and Central Fact Layer.
Physical delete means that the 'deleted records' will be physically removed from the Logical Validation Layer. In the Central Fact Layer, the Context data of the records will be physically deleted and the relevant key in the Anchor will be marked as physically removed.
For testing purposes you can process a delivery by manually switching the state transitions. This simulates API processing of external deliveries.
menu > Supply > Delivery
.Technical Staging Entity Updates
. PROCESSING
first, before you can switch to SUCCEEDED
.PROCESSING
first, before you can switch to FAILED
.PROCESSING
and press Apply to all selected rows. SUCCEEDED
and press Apply to all selected rows.menu > Supply > Monitor
, you can see the delivery process in the monitor.menu > Supply > Delivery Statistics
, you can analyze the delivery statistics.FAILED
and press Apply to all selected rows.menu > Supply > Monitor
, you can see the failed delivery in the monitor. {warning} Cancelling Deliveries could lead to an unwanted intermediate state of your data if the cancel happens after the Technical Staging Layer (TSL). A cancel is not a rollback to a previous state. If you cancel the current Delivery, the processed and successful entity updates in these delivery are stored in the Central Facts Layer, while the idle and scheduled tasks are stopped.
{info} i-refactory assumes that you have (manually) filled the associated tables in the TSL, as this is not part of the i-refactory application.
When creating a delivery on a Generic Data Access interface model you select the appropriate Delivery Agreement and indicate if physical deletes are allowed.
After submitting this delivery, the i-refactory engine will activate the required processes after which it will be possible to insert
, update
, and delete
the facts for the entities in the given Generic Data Access model. The Generic Data Access delivery will be active until stopped.
{info} If a delivery to the Logical Validation Layer (LVL) for the same Central Facts Layer is running, the GDAL Delivery will be blocked and only released when the LVL Delivery has reached an end-state.
You can find the API documentation on: <hostname>:3001
. For example: https://localhost:3001/
Example code for creating a delivery:
[
{
"id": 0,
"deliveryAgreementId": 0,
"receivedDt": "2022-02-17T10:22:48.472Z",
"sentDt": "2022-02-17T10:22:48.472Z",
"statusCode": "RECEIVED",
"messageNbr": 0,
"messageText": "string",
"constraintsToValidate": "'NONE'",
"violatedConstraintMaxSampleSize": 0,
"logicalValidationDeliveryDeleteType": "LOGICAL",
"contextRelatedEntitiesContainNewRows": true,
"includeExistingDataForSkippedRowsInSetValidation": "DEFAULT",
"includeExistingDataForSkippedRowsInSetValidationConclusion": "YES",
"genericDataAccessDeliveryPhysicalDeleteAllowed": true,
"acmExistsInd": true,
"acmLastCreatedDt": "2022-02-17T10:22:48.472Z",
"acmStartDt": "2022-02-17T10:22:48.472Z",
"acmEndDt": "2022-02-17T10:22:48.472Z",
"acmRecordInd": "N",
"acmModifierId": "string",
"acmEntityId": 0,
"architectureLayerCode": "string",
"interfaceCode": "string",
"tenantCode": "string",
"deliveredEntity": [
{
"id": 0,
"deliveryId": 0,
"entityId": 0,
"snapshotDt": "2022-02-17T10:22:48.472Z",
"isComplete": true,
"acmExistsInd": true,
"acmLastCreatedDt": "2022-02-17T10:22:48.472Z",
"acmStartDt": "2022-02-17T10:22:48.472Z",
"acmEndDt": "2022-02-17T10:22:48.472Z",
"acmRecordInd": "N",
"acmModifierId": "string",
"acmEntityId": 0,
"architectureLayerCode": "string",
"interfaceCode": "string",
"namespaceCode": "string",
"entityCode": "string",
"tenantCode": "string",
"deliveryReceivedDt": "2022-02-17T10:22:48.472Z"
}
]
}
]
The main purpose of the Logical Validation Layer (LVL) is to transform the data received from external data sources to fit into the logical data model structure. It is also responsible for validating deliveries. The Logical Validation Layer is also known as the Historical Staging In (HSTGIN) Layer.
A schema is a set of database objects, such as tables, views, triggers, stored procedures, etc. In some databases a schema is called a namespace
. A schema always belongs to one database. However, a database may have one or multiple schema's. A database administrator (DBA) can set different user permissions for each schema.
Each database represents tables internally as <schema_name>.<table_name>
, for example tpc_h.customer
. A schema helps to distinguish between tables belonging to different data sources. For example, two tables in two schema's can share the same name: tpc_h.customer
and complaints.customer
.
A Physical Data Model (PDM) represents how data will be implemented in a specific database.
{note} The i-refactory uses four PDMs: technical staging model, logical validation model, central facts model and generic access model. Each one of these models is implemented as an additional database, which is used to store data from external and internal data sources.
After you have changed the data model, you need to update the data model version. This way, you can keep track of the model releases.
Model > Properties
. Model Properties
. Model Properties
window opens. Change the version number next to Version
. OK
.It is a good practice to change the model version of every physical data model. The standard for versioning depends on the client. In most cases a version number such as 1.0.0 is used, where the third digit is updated during minor changes and the second digit by major changes.
{info} The model version of the logical validation model is displayed in the i-refactory monitor.
Similar to the valid timeline, a transaction timeline consists of a start date
and end date
.
It is the period during which a fact is valid in the system.
In the i-refactory the transaction time is tracked with acm_start_dt
and acm_end_date
.
The transaction time in i-refactory is different from what is commonly understood by transaction time. Transaction time is usually seen as the moment when a fact was stored in the database. In i-refactory the transaction time is as dictated by the source system, not by the i-refactory database.
The delivery tenant can specify with the snap_shot_date what the transaction start date (acm_start_dt
) has to be.
{warning} The transaction start date of a new delivery has to be later than the transaction start dates of earlier deliveries.