The Data Logistics Register
stores operational metadata about loading data between the different interfaces. It is closely related to the delivery register and it provides auditing information on entities on row level.
To retrieve the current state of each element, i.e., all the metadata related to that element, you can use the i-refactory REST API or the views available in the Data Logistics Register
.
Every delivery can update one or more interfaces. For batch deliveries, this means updating from the Technical Staging all the way towards the Generic Data Access Layer. For CRUD deliveries, that means from the Generic Data Access Layer to the Central Fact Layer.
View: [acm_datalog].[interface_update]
id
: iddelivery_id
: delivery idinterface_mapping_id
: this is shows which is the source an target interface. It needs to be joined with the id of [acm_datadefmap].[interface_mapping]
status_code
: the status of the updating the interfaceView: [acm_datalog].[entity_update]
id
: this id found as acm_audit_id
when querying an entityinterface_update_id
: interface update identity_mapping_id
:load_type_code
: DL
or FL
for Delta and Full loadexecuted_update_script
: provides the SQL code executed when is availablerows_before
: count before loadingrows_after
: count after loadingstatus_code
: SCHEDULED
, FAILED
, IDLE
, PROCESSING
and SUCCEEDED
start_dt
: start timeend_dt
: end timemessage_nbr
: provides number type of messagemessage_text
: provides information on entity updates and error messagesThe task run stores all the actions performed in a delivery. For each task run, we can see its parent task run (if any), the status, the entity that was updated, its type, description as well the code executed.
View: [acm_datalog].[task_run]
id
: task run iduuid
: universally unique identifier of task rundelivery_id
: delivery idtask_run_id_parent
: the id
of the parent task run (if exists)ordinal_nbr
: the sequence of executing task runs per deliverytype
: type of task rundescription
: description of taskcontext
: refers to the architecture layer a task run belongs tostatus_code
: SCHEDULED
, FAILED
, IDLE
, PROCESSING
and SUCCEEDED
start_dt
: start timestamp of task runend_dt
: end timestamp of task runentity_update_id
: this can be found on the acm_audit_id on the entityBy looking at the task run id, task_run_id_successor
and task_run_id_predecessor
, we can find the dependency between task runs.
View: [acm_datalog].[task_run_dependency]
id
: Id of task run dependencytask_run_id_successor
: the id of the task runtask_run_id_predecessor
: the id of task runView: [acm_datalog].[violated_record]
id
: id of violated recorddelivery_id
: delivery identity_id
: the entity idrecord_nbr
: line number of the violated recordrecord
: log of the violated recordView: [acm_datalog].[violated_constraint]
id
: idviolated_record_id
: the id
of the violated recordcontraint_id
: the id of the constraint that is violated-- Select details of violated records
SELECT id
,delivery_id
,entity_id
,record_nbr
,record
,acm_entity_id
,acm_exists_ind
,acm_last_created_dt
,acm_start_dt
,acm_end_dt
,acm_record_ind
,acm_modifier_id
FROM ACME.acm_datalog.violated_record
;
The main purpose of the Logical Validation Layer (LVL) is to transform the data received from external data sources to fit into the logical data model structure. It is also responsible for validating deliveries. The Logical Validation Layer is also known as the Historical Staging In (HSTGIN) Layer.
A schema is a set of database objects, such as tables, views, triggers, stored procedures, etc. In some databases a schema is called a namespace
. A schema always belongs to one database. However, a database may have one or multiple schema's. A database administrator (DBA) can set different user permissions for each schema.
Each database represents tables internally as <schema_name>.<table_name>
, for example tpc_h.customer
. A schema helps to distinguish between tables belonging to different data sources. For example, two tables in two schema's can share the same name: tpc_h.customer
and complaints.customer
.
A Physical Data Model (PDM) represents how data will be implemented in a specific database.
{note} The i-refactory uses four PDMs: technical staging model, logical validation model, central facts model and generic access model. Each one of these models is implemented as an additional database, which is used to store data from external and internal data sources.