Highlight
In this SQLDay 2025 session, we explore the realities of migrating enterprise Databricks environments to Unity Catalog.
Intro
Drawing from a large-scale migration project involving hundreds of workspaces and a complete platform redesign, we discuss the technical, governance, and organizational challenges that teams should expect during their migration journey.
Skills Learned
- Understand the key differences between Hive Metastore and Unity Catalog
- Plan and execute large-scale Unity Catalog migrations
- Navigate governance and organisational challenges
- Design a scalable access management model
- Avoid common migration pitfalls and surprises
- Leverage Unity Catalog capabilities beyond metadata management
High level topics
- Why migrate to Unity Catalog
- Hive Metastore vs Unity Catalog
- Governance considerations and organizational impact
- Access control and permissions model changes
- Data lineage and data sharing capabilities
- Migration planning and execution strategies
- Lessons learned from a real enterprise migration
- Common challenges and recommendations
- Building future-ready Databricks platforms
Agenda
- 00:00 Introduction
- 01:05 Migration project background
- 03:30 Understanding Hive Metastore and Unity Catalog
- 08:15 Unity Catalog capabilities beyond metadata management
- 14:20 Governance and organizational considerations
- 21:40 Migration planning and architecture decisions
- 31:10 Access management and permissions model
- 40:25 Common migration challenges
- 49:30 Lessons learned from the field
- 56:45 Recommendations and best practices
- 01:00:00 Summary and key takeaways
