Azure Databricks Administrator

Who You Are

We are seeking an experienced Azure Databricks Administrator to manage, maintain, secure, and optimize Databricks environments in Azure. This role is focused on platform administration, governance, operational support, performance, reliability, and cost efficiency across the organization’s data analytics infrastructure.

The successful candidate will serve as a platform administrator and operational partner to data engineering, analytics, cybersecurity, compliance, and business teams, helping ensure Databricks is reliable, well-governed, secure, and effectively supported.

What You'll Do

• Configure, maintain, and administer Azure Databricks workspaces and clusters across development, staging, and production environments.
• Monitor cluster health, performance metrics, resource utilization, and platform availability.
• Optimize cluster, SQL warehouse, and workspace configurations for performance and cost efficiency.
• Manage workspace governance, including job scheduling, notebook organization, audit logging, and operational standards.
• Administer user access, workspace permissions, SQL warehouse configurations, and group-based security controls.
• Troubleshoot infrastructure issues, performance bottlenecks, connectivity problems, and workload failures.
• Support disaster recovery and high-availability planning, validation, and operational readiness for Databricks workloads.
• Collaborate with data engineering teams to establish Databricks usage standards, support workload reliability, troubleshoot platform issues, optimize Delta Lake performance, and promote governed platform adoption.
• Administer and enforce security policies, RBAC, encryption standards, and access controls.
• Maintain documentation and runbooks for operational procedures.
• Support capacity planning, cost optimization, and platform improvement initiatives.

Security Governance

• Maintain security governance standards for Azure Databricks in a financial services environment, including RBAC, least-privilege access, privileged access controls, and group-based permissions.
• Administer Unity Catalog permissions, metastore access, catalogs, schemas, tables, volumes, and external locations to enforce governed data access.
• Manage workspace permissions, admin roles, service principals, groups, SCIM provisioning, and Microsoft Entra ID integration to support secure identity governance.
• Configure and review compute policies, cluster permissions, instance pool access, SQL warehouse permissions, and job ownership to reduce unnecessary privilege and control platform usage.
• Oversee secrets management, token usage, storage credential access, and Key Vault-backed configurations to protect sensitive financial, customer, and operational data.
• Coordinate recurring access reviews, access certification, and timely remediation of excessive, inappropriate, or outdated permissions to support internal control requirements.
• Review Databricks audit logs, security events, access changes, and control findings to identify risks, policy exceptions, segregation-of-duties concerns, and required remediation actions.
• Support audit and regulatory readiness by maintaining evidence for access controls, configuration standards, policy enforcement, operational procedures, and change management activities.
• Partner with cybersecurity, compliance, risk, infrastructure, and data governance teams to align Databricks controls with financial services security, privacy, and compliance expectations.

What You'll Need

• 5+ years of experience administering cloud-based data platforms or analytics environments, with a focus on operational support, governance, and reliability.
• 3+ years of hands-on experience administering Azure Databricks workspaces, Unity Catalog, clusters, jobs, SQL warehouses, and platform settings.
• Strong experience managing Databricks users, groups, workspace permissions, admin roles, service principals, SCIM provisioning, and Microsoft Entra ID integration.
• Experience administering Unity Catalog, including metastores, catalogs, schemas, tables, volumes, permissions, storage credentials, and external locations.
• Ability to configure and maintain compute policies, cluster permissions, instance pools, SQL warehouses, job ownership, and platform usage controls.
• Experience monitoring Databricks platform health, cluster performance, job execution, workload failures, audit logs, and resource utilization.
• Strong troubleshooting skills for Databricks platform issues, Spark workloads, connectivity problems, permission errors, and performance bottlenecks.
• Working knowledge of Python, SQL, PowerShell, Databricks CLI, REST APIs, and automation practices used to support platform administration.
• Experience supporting operational controls, access reviews, audit evidence, change management, runbooks, and platform documentation.
• Knowledge of Databricks security governance, RBAC, least privilege, encryption, secrets management, audit readiness, and secure data access patterns.
• Understanding of Azure networking and security concepts, including private endpoints, firewalls, network security groups, storage access, Key Vault, and secure connectivity.
• Experience partnering with data engineering teams to support reliable pipelines, workload troubleshooting, performance tuning, platform standards, and secure Databricks operations.