Azure AI Lakehouse Module
A private Azure Databricks workspace for data and machine learning work. Clusters run without public IPs, the workspace is reached over Private Link, Unity Catalog governs who can see which rows and columns, and notebooks call Azure OpenAI through a private endpoint.
- Fixed fee
- $35,000
- Delivery
- 1–2 weeks
- Public endpoints
- None
- Before we start
- Setup review or a tier
How the workspace is isolated
Cluster subnets injected into your network, Private Link for the workspace and control plane, and private endpoints for storage and Azure OpenAI.
(cluster VMs)"] ContainerSubnet["Container subnet
(driver and worker runtimes)"] HostSubnet <--> ContainerSubnet end subgraph PrivateEndpoints["Private endpoint subnet"] PE_UI["Private Link: workspace UI and sign-in"] PE_CP["Private Link: control plane"] PE_Storage["Private endpoint: ADLS Gen2
(bronze, silver, gold)"] PE_AI["Private endpoint: Azure OpenAI"] end subgraph SecurityPlane["Keys and governance"] AKV["Azure Key Vault
(customer-managed keys)"] UC["Unity Catalog metastore
(access control and lineage)"] end end VPN --> PE_UI HostSubnet --> PE_Storage HostSubnet --> PE_AI HostSubnet --> PE_CP AKV -.->|"Encryption keys"| PE_Storage UC -.->|"Row and column filters"| HostSubnet
Why data platforms drift into exposure
Data science teams moving fast often create Databricks workspaces or notebook clusters on public networks with shared storage keys. Confidential data and model artifacts can then sit behind public endpoints with no record of who read them.
Without Unity Catalog, table permissions get granted one at a time and are hard to evidence in a SOC 2 or HIPAA security review. This module puts the workspace inside your private network from the start, with access control and audit logging in one place.
What it covers
Private workspace setup, Unity Catalog configuration, and security sign-off with your data lead.
-
VNet injection, no public IPs
The Databricks runtime deployed into subnets in your network. Driver and worker nodes get no public IP addresses.
- Secure cluster connectivity (no public IP) enabled
- Dedicated host and container subnets, sized for your cluster plans
- Network security groups scoped to cluster traffic and approved egress
-
Front-end and back-end Private Link
No public internet path for workspace access or for cluster traffic to the Databricks control plane.
- Front-end Private Link for the web UI and single sign-on
- Back-end Private Link for cluster management
- Private DNS for the regional Databricks endpoints
-
Unity Catalog
The metastore, storage credentials, external locations, and catalog, schema, and table permissions, provisioned and documented.
- Managed identity access connector for metastore storage
- Row filters and column masks for sensitive personal data
- Data access audit logs sent to Log Analytics
-
ADLS Gen2 storage
Bronze, silver, and gold containers behind private endpoints, encrypted with customer-managed keys.
- Public network access disabled on every storage account
- Delta Lake optimize and vacuum schedules
- Lifecycle rules that move cold data to Cool or Archive tiers
-
Private Azure OpenAI access
Notebooks call private Azure OpenAI deployments for inference and fine-tuning without leaving your network.
- Private endpoint to Azure OpenAI
- Managed identity authentication, so no API keys sit in notebooks
- Model serving endpoints behind internal ingress only
-
Cluster policies and cost controls
Policy templates that restrict expensive instance types and stop idle clusters.
- Idle auto-termination enforced by policy
- Spot instance pools for non-production batch work
- Tags that attribute Databricks usage to cost centres
What you provide
Subscription rights
Contributor on the target Azure subscription, to register Databricks and create private endpoints.
Databricks account admin
Account admin in the Databricks account console, for Unity Catalog setup.
A data platform owner
One person to approve Entra ID group mappings and cluster cost limits.
Questions
Does our data leave our Azure subscription?
Your tables live in ADLS Gen2 storage accounts in your own subscription, encrypted with keys in your Key Vault, and clusters reach them over private endpoints. The Databricks control plane manages workspace metadata, not your table data.
Can clusters reach on-premises SQL Server or Oracle data?
Yes. Because the clusters run in your spoke network, they resolve on-premises hostnames and connect over your ExpressRoute or VPN connection without crossing the public internet.
How does Unity Catalog use our Entra ID groups?
We set up SCIM provisioning from Entra ID into the Databricks account, so your existing security groups control catalog and SQL permissions.
How do we stop runaway compute bills?
Cluster policies block GPU instances for general analyst use, enforce idle auto-termination, and cap the number of worker nodes.
How is this different from the private Databricks and Foundry implementation?
Both use the same private network and Unity Catalog pattern. This module is the fixed-fee version with Azure OpenAI connectivity. The Foundry implementation adds a Palantir Foundry integration and is priced after an architecture assessment.
Do we need one of the tiers first?
Not necessarily. We start by reviewing your current Azure setup. If it already has the governance and networking the workspace needs, we build on it. If it does not, we build that foundation with you through one of our tiers first.
Related
- Private Azure Databricks with Palantir Foundry
The same private lakehouse pattern, with a governed integration to Palantir Foundry.
- Case study: private Databricks for a healthcare network
How a healthcare provider network governed protected health information across a private lakehouse and Palantir Foundry.
- Preparing Azure for a hospital security review
Private endpoints, customer-managed keys, and audit logging against the HIPAA Security Rule's technical safeguards.
Give your data team a private, governed workspace.
$35,000 fixed fee, 1 to 2 weeks. Not sure which foundation it should sit on? Answer three questions.