Quick answer: Unity Catalog permissions control who can read, write and manage data in a Databricks lakehouse. Grant every permission to groups synced from Entra ID, never to individual emails. Use a Databricks service principal for jobs, pipelines and apps, and reuse the one each Databricks App already has instead of creating a new one.
Most lakehouse permission failures I see are not breaches. They are orphaned accounts, stalled deployments and automation running with admin rights it never needed. Users are people, groups hold the permissions, and a Databricks service principal is the non-human identity that runs jobs and apps.
I have spent more than 30 years implementing enterprise systems inside IBM, Ernst & Young, Johnson & Johnson and McKesson. On three separate client calls, I watched the same permissions mix-up stall a deployment. An IT team tried to issue a new Databricks service principal for a Databricks App that already had one.
The fix is fewer identities, organized by group. Stop creating identities by hand and run every grant through directory groups. Secure. Solve. Scale.
Where does legacy manual access provisioning break the pipeline?
- Duplicated application identities: IT creates a new Databricks service principal in the account console for a new app. The app already has its own identity, so permissions conflict and deployments fail.
- Direct user grants: An engineer grants catalog access to a person’s email instead of a group. When that person changes roles or leaves, every query that depends on the grant fails.
- Workspace-only users: Admins invite users straight into one workspace instead of syncing them at the account level. Access now lives in several places, and orphaned accounts survive after people leave.
- Audit blind spots: Orphaned workspace accounts and one-off grants leave gaps in SOC 2, HIPAA and GDPR access reviews. When access is not tied to a directory group, you cannot show an auditor who could see which table, or why.
- Hardcoded tokens in CI/CD: Pipelines authenticate with a long-lived personal access token pasted into a config file. The token belongs to a person, rarely gets rotated and outlives the role it was issued for.
- Over-privileged automation: Teams give a Databricks service principal workspace admin rights to get around schema permissions. That breaks least privilege and exposes the whole data foundation.
- Cross-cloud identity drift: Each cloud carries its own Databricks account. A lakehouse that spans Azure and AWS ends up with two sets of groups that drift apart unless both sync from the same identity provider.
- Redundant PII copies: Teams copy whole tables to hide sensitive columns instead of masking them. Storage costs climb, and the lakehouse loses its single source of truth.
Users, groups, service principals: who is who?
Sorting out identities is the first move in the Assess phase of our AIM-IT Framework. Your environment has three kinds. Users are people who sign in through single sign-on. They write queries, read Power BI dashboards and build models.
Groups hold the permissions. They sync from your identity provider, and every grant goes to a group, never to a person. When someone needs a silver-layer table, IT adds them to the right group.
A Databricks service principal is the non-human identity that runs jobs, pipelines and applications. It does not depend on anyone’s personal account. If the engineer who scheduled a production job leaves, the job keeps running because it authenticates as the service principal.
The group-only rule: why never grant to an email?
The rule is simple: never grant Unity Catalog permissions to an individual email address. Databricks data engineering works best when you reason from first principles, and the first principle here is that people move while roles stay. A grant tied to a person breaks the day that person moves.
Run GRANT SELECT directly to john.doe@company.com and you have built a brittle dependency. When John’s account is deactivated, every view and dashboard that relied on his grant stops working. Your team then traces failures one grant at a time.
Create a group such as data_engineers_prod instead and apply the Unity Catalog grants there. New engineers join the directory group and get the right access on day one. This is zero trust applied to data: access follows a role you can audit, never a person you have to remember. Lean Six Sigma teaches you to remove variation, and group-level grants remove the variation of one-off user provisioning.
How does Entra ID sync to the account console and workspaces?
Your identity provider is the single source of truth. For most mid-market and enterprise teams, that is Microsoft Entra ID. Set up automated SCIM (System for Cross-domain Identity Management) provisioning to push users and groups from Entra ID into the Databricks account console. Do not manage users locally.
Once identities live in the account console, identity federation assigns groups to specific workspaces. Control sits in one place. When IT disables a user in Entra ID, the change reaches the account console on the next provisioning cycle and removes access across every connected workspace.
Inviting users straight into a workspace bypasses that sync. It leaves orphaned accounts and makes access audits harder. Route every identity change through the central directory.
How do catalog, schema and table grants work with PII column masks?
The Unity Catalog three-level namespace runs catalog, then schema, then table. Grant at the highest level that makes sense. USE CATALOG and USE SCHEMA let a group reach the container, and SELECT lets it read the rows.
PII needs precision. Legacy systems built separate redacted tables to hide pay data or Social Security numbers. That duplicated data and broke the medallion pipeline.
In the lakehouse, you attach column masks to the table itself. A SQL function checks group membership: members of finance_team see the real value, and everyone else sees XXX-XX-XXXX. You keep one gold-layer table, and Unity Catalog applies the mask at query time.
Group grants are role-based access control (RBAC). They decide who reaches a table. Masks and row filters add the attribute-based layer (ABAC) that decides what each group sees inside it.
Should you create a new service principal for a Databricks App?
No. This is the exact mix-up from those three client calls. When you deploy a Databricks App, the platform automatically creates a dedicated service principal for it. The app runs as that identity.
The trouble starts when IT gets a ticket asking for app access. Out of habit, an admin creates a new service principal in Entra ID, generates a client secret and tries to point the app at it. That does not work.
Find the app’s existing service principal in the account console instead. Add it to the right data access group. The app picks up the table access it needs, and the deployment goes through with no secrets to juggle.
CI/CD pipelines and external tools that call Databricks are different. Give each one its own service principal and authenticate with OAuth machine-to-machine (M2M) instead of a personal access token. The access tokens expire within the hour, so a leaked config file stops being a standing key to your data.
What belongs in a permissions review checklist?
Check your environment against these five rules. Each anti-pattern is a gap you can close this week.
| Asset type | Correct access method | Common anti-pattern | Required action |
|---|---|---|---|
| Human users | Entra ID group sync | Manual workspace invites | Remove local accounts and enforce SSO sync. |
| Automated jobs | Databricks service principal | Running jobs as a human user | Transfer job ownership to the service principal. |
| CI/CD pipelines | Service principal with OAuth M2M | Personal access token in a config file | Replace the token with OAuth M2M credentials. |
| Databricks Apps | The app’s built-in service principal | Creating a new identity and client secret | Add the built-in app identity to a data group. |
| Sensitive data (PII) | Column masks | Duplicating tables for different teams | Apply masks based on group membership. |
Which grant should you revoke first?
Start with the direct email grant on a production gold table. It is the one that fails without warning the day someone changes roles, and it is the one an auditor will find. Query system., filter for grantees that contain an @ sign, and sort by catalog so production comes first. Revoke each one and reassign it to the matching group.
Next, move every job that runs as a person onto a service principal. Here is the contrarian part: do not open a ticket for a new identity until you have checked whether one already exists. On every one of those three client calls, the identity the team needed was already sitting in the account console.
Want a second set of eyes on that first audit? Our AI governance and compliance work sets up group-based access, column masks and service principal hygiene in your lakehouse from day one.
Frequently Asked Questions (FAQs)
What is a Databricks service principal and when should you use one?
It is a non-human identity for automated jobs, pipelines, CI/CD and applications. Use one whenever a workload must keep running after the person who built it leaves or resets a password.
Why should we avoid granting Unity Catalog access to individual users?
Grants tied to an email break when people change roles, and they need constant manual fixes. Directory groups adjust access automatically as membership changes, so your controls hold as you scale.
What is the difference between RBAC and ABAC in Databricks Unity Catalog?
RBAC grants access by group, such as SELECT on a catalog, schema or table. ABAC applies policies based on governed tags on the data, such as a PII tag, so row filters and column masks follow the tag to every table that carries it. Most teams use both: groups decide who reaches a table, and tags decide what they see inside it.
How do Databricks Apps handle non-human identities?
The platform creates a dedicated service principal for each app automatically. Add that identity to the right data access group instead of creating a new one in your directory.
How does Entra ID sync with the Databricks account console?
Admins set up a SCIM connector in Entra ID that pushes users and groups into the account console. When you deactivate a user in Entra ID, the next provisioning cycle removes their access across every connected workspace.