Hive Metastore to Unity Catalog Migration: Plan, UCX and What Breaks

At a Glance

A Unity Catalog migration moves table definitions, permissions and the code that references them from the legacy Hive metastore into Unity Catalog. Existing workspaces are not forced to move by 30 September 2026; that date only changes how new workspaces are provisioned. Plan by table type: SYNC for external tables, deep CLONE or UCX for managed ones, and code fixes last.

• Cost: the tools are free; the cost is engineering time, some duplicated storage for cloned tables, and a period of dual running.
• Effort: driven by the number of managed tables in DBFS root, mounts, and notebooks with hard-coded paths, not by total data volume.
• Risk: jobs that silently keep writing to the old tables, lost table history, and permissions that do not map one to one.
• When it fits: now, if you plan new workspaces, want lineage and audit, or are building AI on governed data.

The date everyone is quoting deserves a careful read. According to Databricks’ own documentation, starting 30 September 2026 new workspaces in all accounts will be provisioned without access to DBFS root and mounts, the Hive metastore, no-isolation shared clusters, and Databricks Runtime versions before 13.3 LTS. The same page states that existing workspaces and their workflows are not impacted.

So this is not an end-of-life notice for your current workspace. It is a signal about direction, and it bites the day you create a new workspace for a new region, business unit or environment. Unity Catalog has also been enabled automatically for every workspace created after 8 November 2023, which is why many estates now run both catalogs side by side.

That split estate is the real problem. Lineage covers half the tables, grants live in two places, and a failed job takes longer to trace. In Fivetran’s 2026 benchmark, large enterprises already averaged about 13 hours to resolve a pipeline failure. Two catalogs over one lakehouse do not make that number smaller.

What the 30 September 2026 Change Means for You

Three situations cover almost every team we talk to.

You only run existing workspaces. Nothing stops working on 30 September. Your Hive metastore tables, mounts and clusters keep running. You still carry the governance gap, and any future workspace will not be able to use the old patterns.

You plan a new workspace. It will be Unity Catalog only. Any job you intend to copy across that reads from a mount, writes to DBFS root, or runs on a no-isolation cluster needs rework before it moves. This is where most unplanned migration work comes from.

You are building AI or a governed BI layer. Genie, retrieval pipelines and shared metric definitions are all built on Unity Catalog. Migrating the tables those use first is the fastest way to value.

The Migration Tools, Compared

Databricks documents several ways to upgrade Hive tables to Unity Catalog. The right one depends on the table type, not on preference.

Two details save a lot of pain. SYNC supports a DRY RUN mode that checks whether tables can be upgraded without creating anything, and it is designed to be rerun, so it can keep UC tables current while old jobs still write to Hive. UCX, per its GitHub repository, is provided for exploration and is not formally supported with SLAs, so test it on a non-production workspace first.

A Migration Plan That Works Table by Table

We run these projects through AIM-IT. The steps below are the Implement stage; the Assess and Innovate work comes first so you migrate a clean process, not a messy one.

1. Inventory and classify. Run a UCX assessment or your own query to list every Hive table as external, managed outside DBFS root, or managed in DBFS root. Record the jobs and notebooks that read or write each one.
2. Retire before you migrate. Every estate we assess has tables nobody reads. Archive them. The cheapest table to migrate is the one you delete.
3. Design the target. Decide catalogs per environment, schemas per domain or medallion layer, storage credentials and external locations, and the groups that will own each catalog.
4. Build storage access first. Create storage credentials and external locations before any table moves, then test them with a read and a write.
5. Move external tables with SYNC. Dry run, then sync, then schedule reruns during the dual-running period.
6. Move managed tables with deep CLONE or UCX. Record table history you need for audit before cloning, since it does not carry over.
7. Map permissions to groups. Translate legacy table ACLs into Unity Catalog grants on groups, not users.
8. Fix the code. Update two-part table names to three-part names, replace mount paths with volumes or external locations, and move jobs to supported cluster access modes.
9. Cut over by dependency group. Switch a source table and every job that depends on it together, compare row counts and key totals, then freeze the Hive copy.

Cutting over by dependency group, not table by table, is the single choice that most reduces rework. It keeps a downstream report from reading a half-migrated chain. We cover the broader sequencing in our data warehouse modernization guide.

What Breaks During the Move

These failures compound, which is why the order matters. Fix the first and the later ones shrink.

1. Managed tables in DBFS root. SYNC cannot touch them, so they need a deep clone or UCX. Teams that plan everything around SYNC discover this late.
2. Mount paths in notebooks. Code that reads /mnt/ paths keeps working in the old workspace and fails in a Unity Catalog only one. Search the code before you estimate.
3. Jobs still writing to Hive. A job nobody updated keeps landing data in the old table, so the new one goes stale quietly. Scheduled SYNC hides this during dual running, which is exactly when you need to find it.
4. Permissions that do not map cleanly. Legacy grants given to individuals, or the same group name meaning different people in different workspaces, turn a mechanical step into a policy discussion.
5. Partition commands and lost history. Hive commands that manipulate partitions directly are not supported on Unity Catalog managed tables, and CLONE does not bring table history across. Any process that relied on either needs a new approach.

Every item on that list is a process gap that surfaced during a technical task. Our data governance consulting work starts by fixing the ownership and access request process, so step 7 above becomes a lookup rather than a debate.

Dual Running and Cutover

Plan for a period where both catalogs hold the same data. Keep it short and deliberate. Scheduled SYNC keeps external tables aligned, and a simple reconciliation job comparing row counts and a few business totals catches drift early.

Set an explicit freeze date for each dependency group. After it, the Hive table becomes read-only and any job that still writes to it fails loudly, which is what you want. Silent success on the old path is the worst outcome of a migration, because the numbers look fine until someone compares them.

Once the last group is cut over, the lakehouse finally has one catalog, one set of grants and complete lineage. That is the point of the exercise: a lakehouse governed from one place. It is also the base our warehouse to lakehouse migration projects build on.

Ask These Before You Sign or Build

Whether you run the migration in house or hire help, get clear answers to these first.

• How many of our Hive tables are managed in DBFS root, and what is the plan for each one?
• Which notebooks and jobs use mount paths or two-part table names, and who will change them?
• Which tables can be retired instead of migrated, and who has the authority to say so?
• What is the dual-running window per dependency group, and how will drift be detected?
• How will legacy grants be translated to groups, and who approves the new access model?
• If we use UCX, what is our fallback when a Labs tool does something unexpected, given there is no vendor SLA?
• What table history do we need for audit, and where will it live after a clone?
• What does done mean: every table migrated, or the Hive metastore frozen and read-only?

If those questions expose gaps, our Databricks consulting team can run the assessment and hand you a sequenced, dependency-grouped plan.

Frequently Asked Questions (FAQs)

Do I have to migrate to Unity Catalog by 30 September 2026?

No. From 30 September 2026, new workspaces are provisioned without Hive metastore, DBFS root and mounts, no-isolation shared clusters or runtimes before 13.3 LTS. Databricks states that existing workspaces and their workflows are not impacted, and no end date for Hive metastore in existing workspaces has been set.

Does UCX copy my data?

It depends on the table. External tables are typically upgraded in place, with Unity Catalog pointing at the same files. Managed tables moved to Unity Catalog managed tables are copied into new storage. Check each table type in the assessment before estimating storage.

Can I keep the Hive metastore alongside Unity Catalog?

Yes. Hive metastore federation creates a Unity Catalog catalog that mirrors your Hive metastore, which is useful as a bridge. It is best treated as a transition step, because the full benefits of lineage and managed tables come from migrating.

How long does a Unity Catalog migration take?

It depends far more on code and table types than on data volume. An estate of mostly external tables with clean code moves quickly. Many managed tables in DBFS root, mount paths throughout notebooks, and individual-user grants all extend the timeline, so size it after an assessment rather than before.