Highlight
This was the one part of the migration where the word “catalog” almost hid the real work. We were not just moving metadata; we were rebuilding a platform with Azure Landing Zone thinking and Unity Catalog governance baked in.
Lesson 1: Migration is 80% governance
The challenge we faced
Our starting point was a very large Azure platform in a single subscription. The old environment had a central Hive metastore and more than 1,000 Databricks workspaces connected to it.
Governance was the hidden migration cost. The technology change itself was only one small piece of the problem: the bigger risk was who owns the account, who can create catalogs, and how the organization is aligned around those decisions.
The solution
We treated the migration like a landing zone project. That meant:
- Building a brand-new Azure environment instead of bending the old one.
- Designing Unity Catalog as part of the platform, not as a sidecar.
- Mapping project ownership to the organization structure, rather than to individual workspaces.
Why this matters: if you build Unity Catalog as an afterthought, you end up with a messy set of adhoc permissions and a platform that is hard to operate.
Lesson 2: Unity is not just managed Hive metastore
When we started, many teams assumed Unity Catalog was simply a managed version of the old metastore. That is not the case.
In Unity we found:
- row-level security (RLS) and column-level security (CLS)
- data lineage and audit metadata
- table, schema, and external location controls
- shareable assets across cloud and cross-tenant boundaries
- a tenant-level account object with powerful switches
So yes, the old H metastore analogy is useful, but it is only the first line of the story.
What we learned
Trust me: if your team thinks Unity is just a catalog, you will miss the biggest migration work. Unity Catalog is a product platform with new governance and security primitives.
Lesson 3: Build the right new environment
We were migrating between two Azure platforms. That meant we had a clean opportunity to build the new environment using more modern landing zone patterns.
Our design goals were:
- separate the Unity account team from subscription teams,
- avoid workload teams owning tenant-wide toggles,
- standardize project onboarding through a schema + external location package,
- make Unity Catalog the default for every new and migrated project.
That last point was important: after the first two quarters, Unity was not optional for us anymore. It became the default.
Catalog is a Hive Metastore, just baked in
This thing on the left is a Hive Metastore, so if you use it before UC, then this applies to you too.
What’s different
Here are high level differences between hive and UC
How to map Azure objects to UC
Here are high level differences between hive and UC
| Azure Component | Databricks (Unity Catalog) Object | Description |
|---|---|---|
| Entra ID | Databricks Account | Identity provider used for authentication and account-level governance |
| Hive Metastore | Metastore | Central metadata repository managing tables, schemas, and permissions |
| Built-in Catalog | Catalog | Top-level container for organizing data assets |
| Hive Metastore (DB level) | Schema | Logical grouping of tables within a catalog (similar to databases) |
| External Metastore (SQL) | Catalog / Schema (externalized) | External metadata source integrated into Unity Catalog |
| Access Connector | Storage Credential | Secure identity used to access cloud storage (via managed identity or service principal) |
| Data Lake Folder | External Location | Registered cloud storage path governed by Unity Catalog |
Result
We moved about 100 projects within a few months and made Unity Catalog the standard for new work. The migration was large, but the decision to treat it as a landing zone project helped us keep control of scope and ownership.
