Azure Databricks Reference Architecture for Mid-Market Teams

What You Need to Know

An Azure Databricks architecture for a mid-market team is one Unity Catalog metastore in your primary region, separate dev, test and prod workspaces, ADLS Gen2 storage isolated at the catalog level, Lakeflow for ingestion and pipelines, a bronze-silver-gold medallion model, and serverless SQL warehouses serving Power BI. Decide identity, networking and storage first, because those are the parts that are costly to change.

• Cost: Databricks bills compute in DBUs while Azure bills storage, networking and any classic VMs separately, so plan both.
• Effort: the foundation (identity, network, metastore, storage) is a few weeks of focused work; pipelines come after.
• Risk: storage shared outside Unity Catalog and production jobs owned by named users are the two most common self-inflicted problems.
• Fit: strongest for Microsoft-centred teams that already run Entra ID, Power BI and Azure networking.

Enterprises are moving analytics onto this pattern at scale. In its December 2025 funding announcement, Databricks reported a $4.8 billion revenue run-rate growing more than 55% a year, with its data warehousing product alone above $1 billion in annualised revenue. Buyers are standardising on the lakehouse, and Azure is where many Microsoft-centred mid-market companies run it.

Scale is not the problem for most of those buyers. Maintenance is. A Fivetran benchmark of 500 senior data leaders found 53% of engineering capacity going to pipeline maintenance rather than new work. On Azure, that maintenance burden usually traces back to an architecture assembled one service at a time.

The symptoms are familiar. Three workspaces with three different permission models. Storage accounts mounted to DBFS and readable by anyone with a key. Azure Data Factory and Databricks both scheduling the same pipeline. A reference architecture exists to stop that drift before it starts, and the Microsoft Learn Azure Databricks reference architecture is the right starting map. What follows is how we cut it down for a mid-market team.

The Architecture at a Glance

Microsoft organises the reference architecture into swim lanes: source, ingest, transform, query and process, serve, analysis and storage, with governance and automation spanning all of them. Here is how each lane maps to Azure services and Databricks components, with the default we recommend for a team of five to twenty data people.

Foundation: Identity, Network and Workspaces

Start with identity. Unity Catalog grants privileges to principals defined at the Azure Databricks account level, and the Unity Catalog best practices page is explicit: provision users and groups from your identity provider at the account level, avoid workspace-level SCIM, and avoid direct grants to users wherever possible. Define groups in Entra ID so they match how the business is organised, then use them for both access and object ownership.

Networking comes next, because some choices are made when a workspace is created and cannot be flipped later. Decide whether workspaces deploy into your own virtual network, whether you need private connectivity to the control plane, and how egress is controlled. A mid-market team with an existing Azure landing zone should plug Databricks into it rather than invent a parallel network.

For a mid-market lakehouse, three workspaces is the common answer: dev, test and prod in the same region. They share one metastore, which keeps lineage and access control in one place, while catalog bindings keep prod data out of dev notebooks.

Storage: ADLS Gen2 Behind Unity Catalog

Every table and volume in the lakehouse sits in ADLS Gen2, but nobody should reach it except through Unity Catalog. Microsoft’s guidance prefers catalog-level managed storage as the primary unit of isolation, warns against using a container that anything outside Unity Catalog can access, and says not to reuse a container that was ever your DBFS root. Unity Catalog reaches storage through a storage credential backed by an Azure managed identity, so no account keys sit in notebooks.

One limit catches storage-heavy teams. The same guidance notes that an ADLS account supports 20,000 requests per second by default, and that multiple containers in one account do not raise that ceiling. If ingestion volume is high, stripe managed storage across several storage accounts. Users never see the difference.

Use managed tables for new work. They get Databricks’ automatic optimisations, and new platform features favour them. Keep external tables for data that other engines must read in place, or as a staging step when upgrading from the Hive metastore, and route writes through Databricks.

Ingest and Transform: Lakeflow and the Medallion Model

For SaaS applications and databases, start with Lakeflow Connect. Its pipelines are governed by Unity Catalog and run on serverless compute. Files landing in storage go through Auto Loader. Event streams from Event Hubs go through Structured Streaming. Keep Azure Data Factory for the sources it reaches that Databricks does not, such as on-premises systems behind a self-hosted integration runtime, and let it land files rather than orchestrate transformations.

Transformation follows the bronze, silver, gold pattern described in our medallion architecture guide. Build silver and gold in Lakeflow Declarative Pipelines (formerly Delta Live Tables) so dependencies, retries and data quality expectations are declared rather than hand-coded. Use AUTO CDC for change data from operational sources. Orchestrate the whole run in Lakeflow Jobs, which handles schedules, file arrival triggers and branching.

Serve: SQL Warehouses, Power BI and Operational Data

For BI, serverless SQL warehouses feed Power BI and Databricks dashboards, with Unity Catalog controlling what each report can see. Serverless removes idle cluster time from the bill, which matters more to a mid-market team than peak throughput. Publish metric definitions as Unity Catalog metric views so revenue means the same thing in Power BI, in a dashboard and in any AI assistant that queries the lakehouse.

Operational serving is a later decision. Lakebase, Databricks’ managed Postgres, is generally available on Azure and suits an application that needs low-latency reads of gold data. Most mid-market teams do not need it in the first release.

Governance and Operations Across Every Lane

Unity Catalog is the spine: one metastore per region, catalogs mapped to environment or business unit, schemas per team, ownership held by groups. Grant USE CATALOG and USE SCHEMA only to people who should see the data. Reserve MODIFY on production tables for service principals, and run production jobs as service principals rather than as named users. If the company already runs Azure Databricks alongside Microsoft Purview, Unity Catalog can export schema and lineage to it so the enterprise catalog stays complete.

Operations lives in system tables: billing usage, job runs and audit events you can query like any other table. Tag clusters, jobs and warehouses by team and project from day one, then build chargeback on top. Our Databricks cost optimization playbook covers what to do once the tags exist. Deployment runs through Declarative Automation Bundles (renamed from Databricks Asset Bundles in March 2026, with no breaking change), so jobs and pipelines move from dev to prod through CI/CD, not copy and paste.

Where Azure Builds Go Sideways

These are the problems we find most often when we assess an existing Azure Databricks estate.

• Storage mounted to DBFS and also registered as an external location, so the same data has two access paths and only one of them is governed.
• One storage account carrying every workload, which throttles ingestion at month-end when volume peaks.
• Production jobs owned by a named engineer who then leaves, taking the credentials and the schedule with them.
• Data Factory and Lakeflow Jobs both scheduling the same flow, so a failure alerts in one tool and hides in the other.
• No tags on compute, which leaves finance with one Databricks line and no way to charge it back.
• Workspaces created before networking was agreed, forcing a rebuild when security reviews the design.

None of these is exotic. Each one comes from building services before agreeing the operating model, which is why we run the Assess step of AIM-IT before provisioning anything.

Your Order of Work for This Quarter

Sequence matters more than speed. This is the order we use with mid-market Azure teams.

1. Agree the operating model: who owns data, who approves access, which environments exist, and how cost gets charged back.
2. Set up identity: account-level provisioning from Entra ID, groups for every role, service principals for automation.
3. Build the network and workspaces inside your landing zone, then create the metastore in your primary region.
4. Create catalogs per environment with catalog-level managed storage, and bind the prod catalog to the prod workspace.
5. Stand up the first bronze-to-gold pipeline for one high-value source, with expectations and tags in place.
6. Connect Power BI to a serverless SQL warehouse, publish metric views, and only then add the next source.

Teams that follow that order spend the second quarter adding sources instead of reworking permissions. If you want a second pair of eyes on the foundation, our Databricks consulting work starts with that operating model, and our data governance consulting covers the Unity Catalog design in depth.

Frequently Asked Questions (FAQs)

What is the architecture of Azure Databricks?

Azure Databricks has a control plane managed by Databricks and a compute plane where your workloads run, with data stored in your own ADLS Gen2 accounts. Unity Catalog governs data and AI assets across workspaces. Around that lakehouse core sit ingestion, transformation, serving and orchestration services.

How many workspaces does a mid-market team need?

Three is typical: dev, test and prod in one region, sharing one Unity Catalog metastore. Add more only for a hard isolation requirement, such as a separate business unit or a regulated dataset that needs its own workspace binding.

Can I have more than one Unity Catalog metastore?

You can have only one metastore per region, and every workspace in that region shares it. Isolation between environments or business units happens at the catalog level, not by creating extra metastores.

Do I still need Azure Data Factory with Databricks?

Often, but for less. Use Lakeflow Connect and Auto Loader where they reach your sources, and keep Data Factory for on-premises or unsupported systems. Let one orchestrator own the end-to-end schedule so failures surface in one place.

How does Azure Databricks pricing work?

Databricks charges for compute in DBUs, billed per second of use. Azure bills storage, networking and any classic virtual machines separately. Serverless compute folds the virtual machine cost into the Databricks charge, which simplifies forecasting.

Where does Power BI fit in the architecture?

Power BI connects to Databricks SQL warehouses and reads gold tables or metric views, with Unity Catalog permissions applied. Keep business logic in gold and in metric views rather than in Power BI datasets, so every tool reports the same numbers.