DataQ

Data quality monitoring platform built around Great Expectations — Snowflake (DEV/QA/UAT), ADLS Gen2, S3, Unity Catalog (Databricks), Apache Iceberg (native read), with ADF + Airflow + dbt orchestration integrations.

📖 Documentation site: https://theurgicduke771.github.io/DataQ/ (MkDocs Material — quickstart, concepts, architecture, guides).

Status: v1.0.0 released (2026-07-04) — the 8-week roadmap is complete (187/189; 2 items consciously re-scoped to post-v1). v1 is deployed to Azure Container Apps — API + worker + a runtime-configured frontend Container App (the sole public surface; the api runs on internal ingress behind it), with Key Vault, App Insights, and orchestration polling live. Auth is a generic OIDC client (validated against Azure AD; ADR 0028). The completed v1 ledger is archived at docs/progress-v1.md; live post-v1 progress at docs/progress.md.

What it does

• Checks on 5 datasources — Snowflake, Unity Catalog (Databricks), ADLS Gen2 + S3 flat files (CSV/Parquet, batch patterns), and Apache Iceberg (native pyiceberg read). Five check styles: GX expectations, custom SQL (rows returned = failures), freshness / volume monitors, and comparison (reconcile two datasets across connections — ADR 0015); plus a column profiler and dry-run preview on every datasource. Feature matrix →
• Assets, lineage & incidents — the table/file is a first-class entity: health rolled up across every suite that targets it, table-level lineage (a left-to-right graph of provenance and blast radius, from dbt's manifest or an OpenLineage catalog), and open incidents. Assets are the primary lens — the dashboard and sidebar lead with them (ADR 0034). Concepts →
• Quality by dimension — every check is classified (accuracy, completeness, consistency, integrity, timeliness, uniqueness, validity), filled in automatically from the check type. The asset scorecard turns that into the question people actually ask: not just "are the checks passing" but "what isn't being watched at all" — an asset with no Timeliness checks says so, and never shows a green tick for it. Dimensions →
• Three run modes — run now (live progress + cancel), cron schedules (timezone/DST-aware), and pipeline triggers: ADF, Airflow and dbt runs are monitored, and a successful pipeline can trigger the bound suite. Scheduling → · Orchestration →
• Severity + alerting — warn/fail/critical tiers band each check's unexpected-%; alerts to Teams / Slack / email with severity-aware routing, first-failure dedup, and per-check snooze. Notifications →
• Results you can share — dashboard health score + trends, per-run drill-down with PII-redacted failing-row samples, suite-level sharing (view/edit), admin control centre. Best practices →

Stack

Layer	Tech
Backend	FastAPI · Celery · Great Expectations · SQLAlchemy + Alembic · PostgreSQL · Redis
Frontend	React · Vite · Ant Design · generic OIDC (oidc-client-ts)
Auth / secrets	OIDC — Azure AD validated (AUTH_* contract, provider-neutral) · Azure Key Vault
Hosting	Azure Container Apps (API · worker · frontend) + Application Insights (deployed)
AI integration	FastMCP — 8 curated MCP tools at /mcp for Claude Desktop / Copilot / Cursor

Quick start

Run DataQ — prebuilt images (recommended)