Databricks is the platform we pick when data engineering, ML and GenAI need to live in one governed lakehouse. The pattern below is what we ship at regulated enterprises — multi-workspace, Unity Catalog as the single source of truth, Delta Live Tables for reliability, MLflow for the model lifecycle, and Mosaic AI for production GenAI.
Start with workspace topology. Most regulated estates need at least four: a development workspace, a non-production workspace for UAT / staging, a production workspace per business unit, and a sandbox workspace for data science. Account-level Unity Catalog links them. We isolate by VNet injection per workspace and use a single identity plane (Entra ID) for SSO.
Next, Unity Catalog. Three-level namespace (catalog.schema.table), with catalogs per business domain (finance, risk, customer, product). Tag-based masking policies and row-access policies get applied at catalog-creation time. We never create a table outside of Unity Catalog — the account-level ‘Default catalog isolation’ enforces it.
Delta Live Tables is the right abstraction for the bronze / silver / gold zones. We use Auto Loader for streaming ingestion from Kafka / Event Hubs / S3, and Python UDFs for transformation logic. Quality expectations are declared inline; we never test downstream of the pipeline.
MLflow sits on top of the gold layer for model lifecycle. Feature engineering happens in Databricks Feature Store, with online serving via Databricks Model Serving. Champion-challenger frameworks are standard, and we wire drift detection back to the training pipeline.
Mosaic AI is the GenAI layer. Vector Search over Delta tables, AI Playground for prompt iteration, RAG Studio for production RAG, Agent Framework for tool-using agents. Unity Catalog extends to models, so the same governance plane covers data and AI.
FinOps: cluster policies per workspace, serverless SQL where possible, DBSQL warehouses sized for concurrency, tag-based cost allocation, monthly FinOps reviews with finance.
IaC: Databricks Asset Bundles (DABs) for the application layer, Terraform for cloud infrastructure. We never click-ops a Databricks resource. The full setup — workspace, networking, IAM, Unity Catalog, storage, DABs — is in a Git repo.
Observability: MLflow traces, Databricks system tables, Unity Catalog audit logs, lakehouse monitoring for drift. We wire it all into Datadog or Grafana for one pane of glass.
Security: customer-managed keys (CMEK), IP access lists, private connectivity (PrivateLink / VNet injection), audit log shipping to SIEM, table-level grants reviewed quarterly.
The outcome is a lakehouse your security team trusts, your data team owns, your finance team can audit, and your ML / GenAI teams can ship on without re-platforming.