Databricks cluster policies are the policy-definition and API mechanism historically used to constrain how users create classic compute clusters. Current Databricks product documentation generally calls the feature compute policies, while API and CLI surfaces still include cluster-policy names such as /api/2.0/policies/clusters and the cluster-policies command group. The approved title uses the familiar legacy term; the current administrative concept is compute policy.
Within Databricks Data Engineering, the value of a policy is deterministic guardrails around runtime version, node types, autoscaling, autotermination, access mode, pools, tags, libraries, and cost-related attributes.
This page focuses on the JSON rule language and API-style controls; Databricks Compute Policies focuses on current admin workflows, policy families, permissions, and compliance enforcement.
A policy definition is a set of attribute rules
Each JSON key maps to a cluster configuration attribute and a policy rule describing what values are fixed, allowed, forbidden, defaulted, or bounded.
The policy engine validates compute configuration before creation/edit.
Good policies encode organization standards in machine-enforceable form instead of relying on users to remember which runtime, instance type, or termination timeout is approved.
Fixed rules remove choices entirely
A fixed rule forces one specific value, optionally hidden from the UI.
This is appropriate for non-negotiable controls such as an access mode, required tag, security profile, or specific runtime family.
Do not fix values merely to simplify the interface if workloads legitimately need variation; use allowlists or ranges for controlled flexibility.
Allowlist and blocklist rules bound user choice
Policies can restrict an attribute to approved values or prevent known-bad values.
For node types and runtimes, allowlists are often easier to audit because the policy clearly states the supported set.
Blocklists are useful for isolated exceptions but can age badly as new values appear; prefer positive approval for high-impact infrastructure choices.
Range rules express cost and scale boundaries
Numeric attributes such as worker counts or autotermination can be constrained with minimum/maximum values.
This lets teams self-serve within an approved envelope without asking an administrator for every cluster resize.
Set ranges from workload patterns and budget; a minimum that is too high wastes spend, while a maximum that is too low encourages users to bypass the policy.
Unlimited rules can still provide defaults
The unlimited policy type can allow a user-entered value while supplying a default and optionality behavior.
This is useful for lower-risk settings where the organization wants a recommended starting point rather than a hard restriction.
Defaults should be chosen deliberately because many users will accept whatever the UI prepopulates.
Wildcard policy paths govern repeated or nested attributes
Cluster configuration contains arrays and maps such as init scripts, Spark configuration, environment variables, and tags.
Policy definitions can address nested paths and wildcard patterns subject to current policy-language rules.
Test these carefully because array index semantics can matter during compliance enforcement and can replace values positionally rather than reordering them.
Policy definitions should control required cost-allocation tags
Custom tags can be fixed or constrained so every cluster carries application, environment, owner, cost center, or data-classification metadata.
Tag enforcement improves chargeback and incident ownership.
Keep tag values standardized; free-form owner strings undermine reporting and can create inconsistent cloud-resource tags underneath Databricks.
Libraries can be enforced through policy configuration
Current compute policies can enforce cluster-scoped library installation.
This can standardize security/monitoring agents or required internal packages, but it also couples cluster startup to library availability.
Version enforced libraries and test upgrades because one broken package can prevent every cluster using the policy from starting successfully.
Cluster-policy APIs support infrastructure automation
Databricks exposes APIs and CLI commands to create, list, get, edit, and delete policy definitions.
Store the canonical JSON in source control and deploy it through automation so policy drift is visible.
Do not make production policy changes only through the UI if the organization expects repeatable multi-workspace governance.
Deleting a policy does not terminate existing compute
Current documentation notes that compute governed by a deleted policy can keep running, although editing may be restricted unless the user has broader permissions.
Decommissioning therefore requires finding the clusters/jobs that still reference the policy and migrating or terminating them first.
A deleted policy ID should not become an orphaned dependency discovered during the next cluster edit.
Cluster policies succeed when the JSON represents intentional platform standards
The mature platform uses a small set of reusable policies, version-controls definitions, tests nested rules/defaults, standardizes tags/libraries, and pairs policy changes with current compliance enforcement.
The legacy “cluster policy” name may remain in APIs, but the engineering principle is current: compute creation should be constrained by code-level policy rather than tribal knowledge.
Policy paths should be derived from the current Clusters API schema, not from copied examples written for older runtimes. Attributes evolve, and a rule on an obsolete path can create false confidence. Keep policy tests that attempt both allowed and disallowed cluster definitions against the current workspace/API version.
Hidden fixed values are useful for reducing UI complexity, but they can also make troubleshooting harder because users do not realize a setting is being forced. Document hidden rules in the policy description or internal platform catalog so teams can explain why one cluster always receives a tag, runtime, or security mode.
Use separate policies for materially different access modes or hardware classes. A single JSON definition with dozens of allowlists and conditional patterns can become difficult to reason about. Personal interactive compute, shared analytics, GPU ML, and production jobs often deserve distinct policies with a common inherited baseline.
Policy IDs should be treated as configuration dependencies. Jobs, IaC modules, and user workflows can reference them directly. Maintain a mapping from logical policy name/version to workspace policy ID and avoid deleting/recreating policies merely to update JSON when an edit preserves a stable dependency better.
Workspace-by-workspace rollout should be automated but staged. Deploy the updated JSON to a test workspace, create representative clusters/jobs, inspect compliance and UI behavior, then roll through production workspaces. Policy syntax errors or overly restrictive rules have organization-wide blast radius when deployed centrally.
Policy exceptions should expire. If one workload needs a temporarily broader node type, runtime, init script, or Spark configuration, create a narrowly scoped exception policy with owner and review date. Permanent exceptions accumulate until the standard policy no longer reflects how the estate actually runs.
Cost controls should combine cluster-policy attributes with cloud/DBU observability. A rule can prevent huge workers or cap DBU/hour, but it cannot guarantee a monthly team budget when many compliant clusters run continuously. Use policy as the preventive layer and FinOps reporting/budgets as the aggregate layer.
API automation should validate the policy after create/edit. Retrieve the deployed policy, compare canonical JSON, and test a small cluster definition with expected acceptance/rejection. Successful API response means the object was stored; it does not prove the policy behaves as platform engineers intended.
Policy definitions should minimize direct Spark configuration overrides unless they are necessary. Spark configs are powerful and can affect security, performance, and feature compatibility in ways that are harder to reason about than first-class cluster attributes. Prefer supported platform fields and restrict custom Spark config keys to an approved set.
Runtime policies should balance security patching with reproducibility. For interactive compute, an allowlist of current LTS/approved runtimes can keep users off obsolete releases; for tightly validated production jobs, pinning may be needed until testing completes. Keep a retirement schedule so old runtime allowances do not remain forever.
Instance-pool rules can prevent users from bypassing cost or security assumptions through a shared pool. A policy can forbid pools, require a specific pool, or constrain related attributes depending on the workload. Align pool policy with node-type and autoscaling rules so the resulting UI does not offer combinations that can never launch.
Policy change reviews should include the user experience. An overly hidden or fixed policy can make self-service confusing, causing users to request unrestricted permissions. Test the create-compute UI as an ordinary user and ensure descriptions/defaults explain the approved choices.
Policy unit tests can be simple but powerful: one JSON compute definition that must pass and several that must fail for each critical rule. Run them in CI against a staging workspace after policy changes. This catches path mistakes, default behavior, and new API attributes before users discover that the policy either blocks every cluster or allows the setting it was meant to forbid.
Keep legacy terminology in automation intentionally. CLI/API names may continue to say cluster policy while the UI says compute policy. Internal wrappers can expose one current logical name and translate to the underlying API, reducing confusion when engineers search documentation or migrate scripts across Databricks releases.
Cluster-policy repositories should include human-readable examples showing the resulting compute UI and accepted/denied configurations. JSON alone can be hard for reviewers outside the platform team to interpret, while examples make the policy’s practical effect on users, cost, and security much easier to approve.
Keep policy tests current whenever the Clusters API or supported runtime attributes change.
Policies should be reviewed for both restriction and developer ergonomics. If the compliant configuration is too difficult to request or understand, teams will create exception pressure. Clear defaults, meaningful error messages, and documented ownership make secure self-service more durable.
