Compliance Security in Azure Databricks: A Deep Dive into PCI-DSS Enablement

Note: Information in this document is valid as of June 9, 2026. Features and compliance support may change as the platform evolves.

Abstract

With the increasing volume of sensitive data processed in cloud environments, organizations are required to adhere to strict compliance standards to ensure data confidentiality, integrity, and security. Azure Databricks offers a Compliance Security Profile (CSP) feature that hardens workspaces to meet various regulatory frameworks, including Payment Card Industry Data Security Standard (PCI-DSS). This paper provides a detailed, technical, and structured analysis of the Compliance Security Profile when enabled for PCI-DSS workloads on Azure Databricks. We explore what the feature is, why it is used, the prerequisites for enabling it, and the operational limitations that data engineers and architects must consider.

1. Introduction

Azure Databricks is a collaborative data analytics platform that integrates Apache Spark, machine learning, and data engineering capabilities within the Azure ecosystem. When handling regulated datasets, particularly those involving payment cardholder information, organizations are obligated to comply with standards such as PCI-DSS.

To address these regulatory demands, Azure Databricks provides an option called “Enable compliance security profile” during workspace creation. When selected alongside PCI-DSS as the compliance standard, Databricks configures the workspace with enhanced security controls, stricter compliance auditing, and platform-level restrictions designed to satisfy industry-specific requirements.

2. What is the Compliance Security Profile (CSP) in Azure Databricks?

The Compliance Security Profile (CSP) is a platform-hardening configuration applied at the workspace level during Azure Databricks creation. It enforces security enhancements and compliance-specific guardrails required by regulatory frameworks such as PCI-DSS, HIPAA, and FedRAMP.

Key Characteristics

• Hardened Compute Environment
  
  Clusters run on hardened VM images with additional OS-level configurations and automatic security patching.
  
  
• Mandatory Encryption
  
  TLS 1.2+ for all communications and encryption at rest for data stored in Azure storage accounts.
  
  
• Strict Auditing
  
  Enforces enhanced logging and monitoring, ensuring activities can be traced for compliance verification.
  
  
• Preview Feature Restrictions
  
  Only generally available (GA) features are allowed unless explicitly approved for regulated workloads.
  
  
• Cluster Restrictions
  
  Allows only supported VM families that pass compliance certifications.

3. Why Enable CSP with PCI-DSS on Azure Databricks?

Enabling CSP with PCI-DSS ensures that the workspace is capable of securely processing, storing, and analyzing payment card data. PCI-DSS sets requirements around data protection, access control, encryption, monitoring, and logging, which Databricks enforces when this option is selected.

3.1 Business Drivers

• Regulatory Obligation
  
  Organizations processing cardholder data must comply with PCI-DSS to avoid penalties and legal liabilities.
  
  
• Risk Mitigation
  
  CSP ensures proper encryption, access control, and monitoring to reduce data breach
  
  
• Audit Preparedness
  
  Built-in compliance logging enables easier evidence collection during audits.
  
  
• Data Governance
  
  Ensures that workloads involving sensitive data are isolated and controlled according to compliance requirements.
  
  

4. Prerequisites for Enabling CSP with PCI-DSS

When creating an Azure Databricks workspace with the compliance security profile enabled, several technical and administrative prerequisites must be met.

4.1 Network & Infrastructure

• Customer’s existing Vnet
  
  
• Outbound Port Allowance: Must open port 2443 for compliance telemetry services.

4.3 Compute Requirements

• Supported VM Families:
  
  
  • General-purpose series: Dv5, Dsv5, Ev5, Esv5, v6
    
    
  • Compute-optimized series: Fsv6
    
    
• Unsupported Configurations:
  
  
  • ARM-based VMs
  • Older Gen2 SKUs
  • Non-hardened custom images
    
    

4.4 Workspace Creation Considerations

• CSP must be enabled at creation time; it cannot be retrofitted later.
  
  
• Once enabled, it cannot be disabled — the workspace is permanently compliance-.
  
  

5. Limitations After Enabling CSP with PCI-DSS

Enabling CSP has significant operational impacts on data engineering workflows in Azure Databricks.

5.1 Feature Availability

• Preview Features Disabled
  
  Experimental or early-access functionalities are blocked until they are formally GA-certified for PCI-DSS.
  
  
• Genie Code and Genie Spaces
  
  As of May 2026, Genie Code Agent mode is available by default in PCI-DSS CSP workspaces, no admin action required. Genie Spaces, however, are still disabled by default and require a workspace admin to enable Partner-powered AI features first. Note that Account-level Genie does not aggregate data from CSP-enabled workspaces regardless this is a hard platform limitation.
  
  

5.2 Serverless Workload Restrictions

• Serverless compute is supported in PCI-DSS CSP-enabled workspaces, covering serverless SQL Warehouses, serverless notebooks, workflows, and Lakeflow Spark Declarative Pipelines. However, serverless support for PCI-DSS is region-limited. It is available in the following Azure regions only: australiaeast, australiasoutheast, canadacentral, eastus, eastus2, germanywestcentral, northeurope, and uksouth. Workspaces deployed in other regions will not have serverless available under PCI-DSS. Classic compute is supported across all regions. As of June 2026, Genie chat is also becoming available by default for CSP-enabled workspaces.
  
  

5.3 Compute & Resource Constraints

• Only compliance-approved VM families are allowed, ensuring that all compute resources meet regulatory certification requirements.
• Autoscaling behavior may be impacted due to restrictions on instance pools and available SKUs, which can limit flexibility in scaling strategies.
  
  

5.4 Network Dependencies

• If outbound network traffic is restricted, port 2443 must be explicitly allowed to enable communication with Databricks compliance telemetry services
  
  

5.5 Irreversibility

• Permanent Configuration: Once the Compliance Security Profile (CSP) with PCI-DSS is enabled, the workspace becomes permanently hardened and cannot be downgraded or reverted.
• If a non-compliant environment is required, a separate Databricks workspace must be provisioned.

5.6 AI & Advanced Feature Restrictions

• Genie Spaces
  
  Genie Spaces are natural-language data interfaces that allow business users to query data, generate SQL, and explore results without writing code. They are disabled by default in CSP-enabled workspaces and require a workspace admin to enable them by turning on Partner-powered AI features. Note that Account-level Genie does not aggregate data from CSP-enabled workspaces regardless of whether Genie Spaces are enabled this is a hard platform limitation.
• Genie Code (formerly Databricks Assistant)
  
  Genie Code is the AI coding assistant built into Databricks notebooks, the SQL editor, jobs, and dashboards. It helps developers write, optimize, explain, and fix code. As of May 2026, Genie Code Agent mode is available by default in PCI-DSS CSP workspaces with no admin enablement required. When Partner-powered AI features are disabled, it falls back to Databricks-hosted AI models and remains governed by Unity Catalog permissions.
• Model Serving (Foundation Model APIs)
  
  Model Serving is not supported for pay-per-token workloads under PCI-DSS. That deployment mode is only available when the compliance standard is set to HIPAA or None. Provisioned throughput workloads do support PCI-DSS in certain regions but require dedicated capacity planning. For most organizations this makes Model Serving via Foundation Model APIs effectively unavailable in a PCI-DSS workspace.

• Lakebase (Managed Postgres OLTP)
  
  Lakebase is a fully managed, serverless PostgreSQL-compatible OLTP database engine integrated into the Azure Databricks platform, generally available since March 2026. It is designed to unify transactional and analytical workloads on the same platform without the need for external databases or ETL pipelines.
  
   Lakebase is not supported in PCI-DSS CSP workspaces. The Lakebase tab does not appear in the Compute section of the workspace at all it is fully hidden. It is currently available only when the compliance standard is set to HIPAA, C5, TISAX, or None.
  
   Workaround: Organizations requiring both PCI-DSS compliance and Lakebase should provision a separate non-PCI-DSS workspace for Lakebase workloads, with strict network segmentation and Unity Catalog governance in place to ensure no cardholder data crosses into that environment. This boundary must be validated with your compliance team.

6. Best Practices for PCI-DSS-Compliant Databricks Workspaces

To ensure smooth operations after enabling CSP with PCI-DSS:

1. Use VNet Injection to isolate the workspace from public exposure.
   
   
2. Enable Private Endpoints for secure connectivity to Azure Storage and Key Vault.
   
   
3. Leverage Azure Key Vault for managing credentials and secrets instead of hardcoding them.
   
   
4. Implement Role-Based Access Control (RBAC) for strict permission governance.
   
   
5. Route Logs to SIEM (e.g., Azure Monitor or Sentinel) for audit evidence and security monitoring.
   
   
6. Validate SKUs before deploying clusters or jobs to avoid unsupported configurations.
   
   
7. Avoid Relying on Preview Features when designing regulated workloads.
   
   

7. Conclusion

Enabling the Compliance Security Profile with PCI-DSS on Azure Databricks significantly strengthens the security posture of the platform, making it suitable for handling regulated payment data. However, it introduces operational constraints, requiring careful architectural planning, supported VM selection, and proactive governance practices.

For organizations working with PCI-scoped datasets, this configuration offers a robust foundation for data protection, regulatory adherence, and audit readiness. That said, the platform continues to evolve — Genie Code is now available by default in PCI-DSS workspaces, Genie Spaces can be enabled by a workspace admin, and emerging capabilities like Lakebase have a clear compliance roadmap even if PCI-DSS support is not yet available. Data engineers, architects, and compliance officers must collaborate closely to design workloads that meet both security requirements and operational efficiency, while staying current with Databricks’ release notes as the compliance coverage of these features continues to expand.

Author : Mohamed Jamal