The i-refactory server processes deliveries first in first out. Which means that transformation tasks of the earliest delivery will always have a higher priority than subsequent deliveries. Further, the number of transformation tasks the i-refactory processes in parallel is limited by the max concurrent entity updates
setting on the Connection
. If for example the first delivery in total contains 100 tasks to complete the delivery the scheduler will assign as many process slots as possible to finish this delivery as soon as possible. If however dependencies between processing tasks exist another delivery will be assigned one or more process slots until the max concurrenty entity updates
slots are all granted.
To prevent deliveries from not getting any process slots you have the ability to set a maximum number of process slots on an agreement. Second, it is also possible to make a reservation for processing slots. When a reservation is made this number of processing slots is always available for processing a delivery. Making reservations therefore guarantees resources but at the cost of limiting other deliveries of consuming all the resources.
In total 16 processing tasks are created, 8 for delivery A1 and 8 for delivery B1. Delivery A1 is created before delivery B1. The maximum number of available slots is 5. Delivery A1 now tries to claim as much resources as possible. In this case there are no dependencies between the tasks for delivery A1. So 5 processing slots are assigned which will all be executed simultaneously. Once a task is finished the scheduler will check if it can run a task for delivery A1. If true again a slot is assigned for this new task. If however all the remaining tasks for delivery A1 have a dependency on not yet completed tasks the scheduler will check if a task for delivery B1 can be processed.
You can imagine that if delivery B1 somehow needs some priority in processing you want to limit the resources for delivery A1 or you want to make sure delivery B1 always has some available processing slots.
You can restrict the number of concurrent entity updates to prevent deliveries from using all the available entity updates. You can set a maximum number of concurrent entity updates:
On an agreement you can set the maximum number of entity updates and you can set a default maximum number of entity updates. The latter setting will be used if you create a new delivery for this agreement and did not specify a maximum number of entity updates on the delivery. However, the maximum number of tasks for all the not yet completed deliveries can never exceed the maximum number of tasks as set on the connection or the agreement.
The i-refactory allows you to reserve entity updates for deliveries and delivery agreements. By making reservations processing slots are always available for these deliveries. On a agreement you can set the maximum number of reservations and a default number of reservations. This behaviour works exactly the same as described above for the maximum number of entity updates. If a default is not set on the agreement nor on the delivery no processing slots will be reserved for the delivery.
Example 1:
The example above shows 2 active deliveries. On the agreement nothing is set. Delivery A1 is created without reservations. Delivery B1 is created with 2 reservations. Delivery A1 will be limited to 3 processing slots (max of 5 on connections minus the 2 reserved sessions for delivery B).
The i-refactory starts to process delivery B, despite the oldest delivery not being finished yet.
Example 2:
Delivery A is created without any reservations. The scheduler immediately assigns 4 processing slots to delivery A. Delivery B is created with 2 reservations. Delivery B now only gets one processing slot as only a max of 5 is available on the connection. When a task for delivery A1 completes these free slots will now be assigned for the remaining entities of delivery A1.
Example 3:
In this example 3 delivery agreements exist: A, B and C. Agreement A has no settings, B has set the max reserved entity updates
to 1, and C has a default reservered entity updates
of 4.
Delivery B1 is created with the delivery specific setting of reserved entity updates
set to 2. The max however on agreement is set to 1. So delivery B will only get one slot reserved.
Deliver A1 is created without any specific settings. Although tasks exist for delivery A1 the scheduler will not grant any processing slots to delivery A1. This is due to the fact that agreement C has a default reservered entity updates
of 4. These slots will be reserved for future deliveries.
If too many reserverations are made this might limit the throughput of the system. Not all processing slots will start processing tasks because the scheduler needs to reserve capacity for potential future deliveries.
On a connection and/or agreement a max reserved entity updates
can be set.
If the sum of reservations exceeds the maximum, the i-refactory distributes the reservations according to the FIFO principle. Thus, it assigns reservations to deliveries in ascending order.
Example 4:
Although delivery B1 and C1 requested for 7 reserverd slots only 3 will be granted due the limitation set on connection.
The main purpose of the Logical Validation Layer (LVL) is to transform the data received from external data sources to fit into the logical data model structure. It is also responsible for validating deliveries. The Logical Validation Layer is also known as the Historical Staging In (HSTGIN) Layer.
A schema is a set of database objects, such as tables, views, triggers, stored procedures, etc. In some databases a schema is called a namespace
. A schema always belongs to one database. However, a database may have one or multiple schema's. A database administrator (DBA) can set different user permissions for each schema.
Each database represents tables internally as <schema_name>.<table_name>
, for example tpc_h.customer
. A schema helps to distinguish between tables belonging to different data sources. For example, two tables in two schema's can share the same name: tpc_h.customer
and complaints.customer
.
A Physical Data Model (PDM) represents how data will be implemented in a specific database.
{note} The i-refactory uses four PDMs: technical staging model, logical validation model, central facts model and generic access model. Each one of these models is implemented as an additional database, which is used to store data from external and internal data sources.
After you have changed the data model, you need to update the data model version. This way, you can keep track of the model releases.
Model > Properties
. Model Properties
. Model Properties
window opens. Change the version number next to Version
. OK
.It is a good practice to change the model version of every physical data model. The standard for versioning depends on the client. In most cases a version number such as 1.0.0 is used, where the third digit is updated during minor changes and the second digit by major changes.
{info} The model version of the logical validation model is displayed in the i-refactory monitor.
Similar to the valid timeline, a transaction timeline consists of a start date
and end date
.
It is the period during which a fact is valid in the system.
In the i-refactory the transaction time is tracked with acm_start_dt
and acm_end_date
.
The transaction time in i-refactory is different from what is commonly understood by transaction time. Transaction time is usually seen as the moment when a fact was stored in the database. In i-refactory the transaction time is as dictated by the source system, not by the i-refactory database.
The delivery tenant can specify with the snap_shot_date what the transaction start date (acm_start_dt
) has to be.
{warning} The transaction start date of a new delivery has to be later than the transaction start dates of earlier deliveries.