Back to Assessments & Documents
Data and Analytics Active

Data Engineering, Platforms and Observability Maturity Assessment

A 49-question diagnostic across 8 domains that evaluates how data pipelines and data platforms are built, tested, run, secured, costed and improved, including the data they supply to AI. Written for the head of data engi...

Start Assessment

Overview

A 49-question diagnostic across 8 domains that evaluates how data pipelines and data platforms are built, tested, run, secured, costed and improved, including the data they supply to AI. Written for the head of data engineering or data platforms who needs to answer: do our data pipelines and platforms deliver correct data on time, and do we know quickly when they do not?

Detailed Description

Data teams often find out that a pipeline failed when a business user spots a wrong number. Pipelines are hand-scheduled, SQL changes go live without review or tests, schema changes break reports, backfills overwrite history and nobody can say how fresh a critical data set is.

This assessment follows the order in which a dependable data platform is built: an approved architecture, environments and patterns, a platform run as a product with planned capacity and visible debt; then pipeline engineering rules for version control, orchestration, schema change, templates and AI-assisted code; automated data tests, safe test data, contract tests, controlled release, rollback and backfills, and quality rules running inside pipelines; monitored data signals with automated lineage, data service level objectives, alerting, incident triage, recovery, review, anomaly detection and automated remediation; platform access, identities, logging, encryption and disaster recovery; cost by data product, storage lifecycle, guardrails and carbon; the data supplied to AI models, retrieval indexes and agents, including unstructured content; and measurement, benchmarking and improvement.

It tests data-specific practice and the hand-off to each neighboring model through named Input documents: Data Governance and Management, Metadata, Catalog and Lineage, Master Data and Data Quality Management, Data Classification and Handling, Data Products and Data Sharing, Privacy Program Operations, Records and Information Management, DevOps Engineering, SRE, DevOps and ITSM Convergence, Non-Human Identity and Secrets Management, IT FinOps and Cloud Financial Management, Business Continuity and Disaster Recovery Management and AI Pipeline and MLOps. Each question is scored on whether the practice is documented, implemented and effective.

Assessment Details

Audience

Head of data engineering, head of data platforms, data platform product owner. Secondary: data engineers and analytics engineers, DataOps and SRE leads, data security, FinOps partners, data product owners and the chief data officer.

Purpose

Shows where data pipelines still depend on manual steps and individual engineers, where bad data can reach consumers untested or unnoticed, where platforms are exposed, unrecoverable or overspending, and where AI systems receive data nobody controls. Gives each gap a concrete next step and a measurable target.

Effort

Initial assessment: 4 to 6 hours of facilitated sessions with data platform, data engineering, data operations, data security and data product owners. Evidence gathering: architecture and pipeline standards, repositories and orchestration history, test and release records, monitoring and incident records, access reviews, recovery test results and cost reports.

Cadence

Every six months, and after a major platform migration, a serious data incident or a significant change in AI data use.

Framework Alignment

DAMA-DMBOK2 Revised ISO/IEC 20547-3:2020 ISO/IEC 27001:2022 ISO/IEC 5259-3:2024 ISO/IEC 21031:2024 ISO 8000-61:2016 NIST SP 800-218 NIST AI RMF 1.0 The Site Reliability Workbook Ch. 5 The Site Reliability Workbook Ch. 6 The Site Reliability Workbook Ch. 13 OWASP Top 10 for LLM Applications 2025 OWASP Top 10 for Agentic Applications 2026 FinOps Framework 2026 APQC PCF COBIT 2019

KPIs

  • Consumer-Detected Incidents - Share of data incidents first reported by consumers rather than by monitoring. Target: 10% or fewer.
  • Data Objectives Met - Share of critical data sets with agreed objectives that met them over the last quarter. Target: 90% or higher.
  • Critical Pipelines with Blocking Tests - Share of critical pipelines whose automated data tests run on every change and block release on failure. Target: 100%.
  • First-Run Pipeline Success - Share of scheduled production pipeline runs completing on the first attempt, before any automatic or manual retry, over the month. Target: 98% or higher (model target).
  • Data Change Lead Time - Median days from first committed pipeline change to production. Target: 7 days or fewer, aligned with the DORA high performance band (one day to one week).

Dependencies

Data Governance and Management (v1.9.6): architecture principles, data roadmap and enterprise data KPIs (1.1, 1.6, 8.7). Metadata, Catalog and Lineage (v1.9.6): automated lineage in the Data Catalog (4.1); Master Data and Data Quality Management (v6.9.6): quality rules, quarantine and data issues (3.5, 3.6, 4.5). Privacy Program Operations (v1.9.6): rules for personal data in test and non-production environments (3.2); Enterprise Architecture (v7.9.6): reference architectures and technology radar (1.1, 8.5). Capacity Performance Management (v6.9.6): load and stress tests (1.5); API and Integration Lifecycle (v6.9.6): event streams and delivery guarantees (1.4); Continual Service Improvement (v8.9.6): improvement knowledge base and register (8.6, 8.7). Data Classification and Handling (v9.9.6): access, logging, encryption and permitted AI use by level and AI data copies (5.1, 5.3, 5.4, 7.4, 7.5); Data Products and Data Sharing (v1.9.6): contracts and the product register (2.4, 3.3, 4.3). DevOps Engineering (v1.9.6): software engineering, golden paths, AI coding rules, delivery pipeline, test strategy and metrics (2.1, 2.2, 2.5, 2.6, 3.1, 3.2, 3.4, 8.2, 8.4); SRE, DevOps and ITSM Convergence (v4.9.6): service record, infrastructure as code, capacity feed, change registration, observability, SLOs and runbooks (1.2, 1.3, 1.5, 3.4, 4.1, 4.2, 4.4, 4.8, 4.9). Incident Management (v9.9.6), Post-Incident Review and RCA Quality (v4.9.6) and Business Continuity and Disaster Recovery Management (v6.9.6): priority, reviews, backup and recovery (4.5, 4.6, 4.7, 5.5). Non-Human Identity and Secrets Management (v4.9.6): vault, workload authentication, key requirements and agent identities (5.2, 5.4, 7.3); DevSecOps (v6.9.6): supply chain controls for connectors and pipeline packages (2.1); IT FinOps and Cloud Financial Management (v5.9.6): attribution, guardrails, AI unit costs and carbon reporting (6.1, 6.3, 6.4, 6.5). AI Pipeline and MLOps (v11.9.6): dataset versions, retrieval indexes and agent runtime controls (7.1, 7.2, 7.3, 7.5); Records and Information Management (v1.9.6): retention and AI copy disposal (6.2, 7.4).

Assessment Outline

(8 domains · 49 questions)
  1. 1

    1 Data Platform Strategy and Architecture

    Reference architecture, platform service model, environment separation, approved batch, streaming and lakehouse patterns, capacity and performance planning, roadmap and technical debt

    • 1.1 Platform Reference Architecture 1 questions
    • 1.2 Platform Service Model 1 questions
    • 1.3 Environment Separation 1 questions
    • 1.4 Approved Processing Patterns 1 questions
    • 1.5 Capacity and Performance Planning 1 questions
    • 1.6 Roadmap and Technical Debt 1 questions
  2. 2

    2 Pipeline Engineering Standards

    Pipeline engineering standard, version control and review, managed orchestration, schema change control, reusable templates and AI-assisted pipeline code

    • 2.1 Pipeline Engineering Standard 1 questions
    • 2.2 Version Control and Review 1 questions
    • 2.3 Managed Orchestration 1 questions
    • 2.4 Schema Change Control 1 questions
    • 2.5 Reusable Templates 1 questions
    • 2.6 AI-Assisted Pipeline Code 1 questions
  3. 3

    3 Data Testing and Release

    Automated data tests, safe test data, contract tests at publish, release and rollback, backfill controls and business quality rules running in pipelines

    • 3.1 Automated Data Tests 1 questions
    • 3.2 Safe Test Data 1 questions
    • 3.3 Contract Tests at Publish 1 questions
    • 3.4 Release and Rollback 1 questions
    • 3.5 Backfill and Reprocessing Controls 1 questions
    • 3.6 Quality Gates in Pipelines 1 questions
  4. 4

    4 Data Observability and Reliability

    Monitored signals with automated lineage, data service level objectives, monitoring coverage, alert routing, incident triage, recovery and replay, post-incident review, anomaly detection and automated remediation

    • 4.1 Monitored Signals and Automated Lineage 1 questions
    • 4.2 Data Service Level Objectives 1 questions
    • 4.3 Monitoring Coverage 1 questions
    • 4.4 Alert Routing and On-Call 1 questions
    • 4.5 Data Incident Triage 1 questions
    • 4.6 Recovery and Replay 1 questions
    • 4.7 Post-Incident Review 1 questions
    • 4.8 Anomaly Detection on Data Signals 1 questions
    • 4.9 Automated Remediation 1 questions
  5. 5

    5 Platform Security and Identity Hand-offs

    Access by role and data level, pipeline service identities, data plane logging, encryption by data level, and platform backup and disaster recovery

    • 5.1 Access by Role and Data Level 1 questions
    • 5.2 Pipeline Service Identities 1 questions
    • 5.3 Data Plane Logging 1 questions
    • 5.4 Encryption by Data Level 1 questions
    • 5.5 Platform Backup and Disaster Recovery 1 questions
  6. 6

    6 Platform Cost and Sustainability

    Cost allocation to data products, storage tiering and lifecycle, compute and query guardrails, AI data workload costs, and energy and carbon of data workloads

    • 6.1 Cost Allocation to Data Products 1 questions
    • 6.2 Storage Tiering and Lifecycle 1 questions
    • 6.3 Compute and Query Guardrails 1 questions
    • 6.4 AI Data Workload Costs 1 questions
    • 6.5 Energy and Carbon of Data Workloads 1 questions
  7. 7

    7 AI and ML Data Supply

    Training data snapshots, retrieval data pipelines, agent-run data operations, label and deletion propagation into AI copies, and unstructured content ingestion

    • 7.1 Training Data Snapshots 1 questions
    • 7.2 Retrieval Data Pipelines 1 questions
    • 7.3 Agent-Run Data Operations 1 questions
    • 7.4 Label and Deletion Propagation 1 questions
    • 7.5 Unstructured Content Ingestion 1 questions
  8. 8

    8 Performance and Improvement

    Reliability measures, delivery speed and debt trend, consumer feedback, external benchmarking, technology evaluation, engineering knowledge reuse, and KPIs and improvement

    • 8.1 Pipeline Reliability Measures 1 questions
    • 8.2 Delivery Speed and Debt Trend 1 questions
    • 8.3 Consumer Feedback 1 questions
    • 8.4 External Benchmarking 1 questions
    • 8.5 Technology Evaluation and Adoption 1 questions
    • 8.6 Engineering Knowledge Reuse 1 questions
    • 8.7 KPIs and Improvement 1 questions

At a Glance

Category Data and Analytics
Type Free
Roles 6
Domains 8
Questions 49
Start Assessment

Still have questions? Get in touch

ITSGovernance
  • Twitter

Contact us

ITSGovernance