Pipeline monitoring and Documentation
Dashboard
The status of the pipelines can be monitored on the dashboard.
In addition users can view the filtered upcoming and failed executions too.
For failed executions, the reason is listed in the dashboard itself and the pipelines can be retried from the dashboard itself.
Data QA
ELT Data will run the following test cases on all the pipelines:
- Uniqueness test: Check if there are no duplicates in the primary key column/s.
- Not null checks: Check if the primary keys are not blank.
- Data freshness checks: Check if the pipeline is fetching the data as per the schedule.
Schema Evolution
The schema evolution is tracked and the users can view the schema definition under the "Silver" folder. Along with the evolution, the current schema too is listed in the folder.
Error Notifications
To get notified about the pipeline failures, navigate to the Notifications section and add the required emails.
Data rollback
Delta file format supports versioning. If required, the data can be rolled back to one of the previous versions.
Please write to contact@eltdata.io to rollback the data.
Pipeline Documentation
ELT Data creates automated pipeline documentation from the configured pipelines. The documentation lists the source, destination, schedule, dedupe logic, schema, destination and db details.
Navigate to the "Documentation" tab under settings.