Highlight
One of the biggest surprises in our Unity Catalog migration was how much of the design depended on product limits. If you do not know which limits are soft and which are fixed, you can build a design that stops before it starts.
Lesson 1: Limits are part of the design
The challenge we faced
Unity Catalog is powerful, but it is also opinionated. We found that some limits are soft and can be raised with support, while others are fixed and may require a product change.
The migration was not just about data and permissions. It was about making those limits part of the architecture conversation.
What we tracked
The important limits we focused on were:
- storage credentials,
- service credentials,
- external locations,
- identities,
- catalogs and schemas.
If you have 300 projects and hundreds of Azure subscriptions, these values matter.
Lesson 2: soft limits can be raised, fixed limits are different
The good news
Many of the Unity Catalog limits are soft. We raised several key quotas by working with Databricks support.
For example:
- default storage credential limit: 200
- our raised limit: 10,000
That is not a small increase. It means the platform can handle hundreds of subscriptions without forcing a shared credential design.
The warning
Some limits are fixed. In our experience, identity-related limits are usually fixed.
That means:
- the design must fit the product capability,
- there may be no quick ticket-based fix,
- you might need to reduce object proliferation or simplify the object graph.
Lesson 3: know your scales early
We were migrating an entire platform, not just one project.
That meant we had to answer these questions before we built the catalog structure:
- how many external locations will each project need?
- how many identities will be active in the account?
- how many schemas and catalogs will the platform require?
If you do not ask those questions early, you risk the design being blocked by a limit later.
Lesson 4: design for quota growth
The platform design should assume the quotas will grow, but not the product limits.
For example:
- position storage credentials as reusable objects,
- do not create unique credentials for every tiny folder,
- prefer shared external locations when it makes sense,
- keep schema counts sane.
How to avoid the worst case
The least risky path is to build with the limit model in mind:
- use a small number of reusable credentials,
- centralize where it makes sense,
- avoid creating one object per micro-service,
- validate your expected counts with Databricks support.
