How to store secrets in Azure Databricks

December 19, 2020

Background

In Azure Databricks, we can write code to perform data transformation on data stored in various Azure Services, e.g. Azure Blob Storage, Azure Synapse. However, as other programs, sometimes, you want to protect credentials used in Azure Databricks, Azure Databricks provides a solid secret management approach to help you achieve that.

Steps

Prepare Databricks command-line interface (CLI) in Azure Cloud Shell

Configure your cloud shell environment

Open Cloud Shell & make sure you select “Bash” for the Cloud Shell Environment.

Launch bash cloud shell Launch bash cloud shell

Set up Virtual Environment

Create Virtual Environment with below command.

# Bash
virtualenv -p /usr/bin/python2.7 databrickscli

Create virtual environment Create virtual environment

Activate your virtual environment

Activate your virtual environment with below command.

# Bash
source databrickscli/bin/activate

Activate virtual environment Activate virtual environment

Install Databricks CLI

Install Databricks CLI with below command.

# Bash
pip install databricks-cli

Install Databricks CLI Install Databricks CLI

Create secret in Azure Databricks

Set up authentication

Before you can create a secret, you need to authenticate as a user of the Azure Databricks, which requires your Azure Databrics workspace’s URL and a token

Get your Azure Databricks workspace’s URL

You can navigate to your Azure Databricks workspace and copy its URL.

Get Databricks URL Get Databricks URL

Generate Access Token for your Azure Databricks workspace

You can follow below steps to retrieve access token

  1. Launch Databricks workspace Launch Databricks Workspace
  2. Click 'User Settings' Click 'User Settings'
  3. Click 'Generate New Token' Generate access token
  4. Configure access token & click 'Generate' Configure access token
  5. Copy access token Copy access token

Create Secret Scope

After authentication, you need to first create a secret scope which you may group several secrets.

If your databricks is in Standard plan, you can only create secret scope which will be shared with other users in the same workspace.

Check databricks plan Check databricks plan

# Bash
databricks secrets create-scope --scope <<scope>>

# Example
databricks secrets create-scope --scope storage --initial-manage-principal users # Standard Plan
databricks secrets create-scope --scope storage # Premium plan

Create secret scope Create secret scope

Create Secret

You can use below command to create secret under the specified scope.

# Bash
databricks secrets put --scope <<scope>> --key <<key name>>

databricks secrets put --scope storage --key blob #Example

Type command to launch secret editor Type command to launch secret editor

Type your secret and save Type your secret and save

Use Secret in Notebook

You can use secret by below command in notebook.

# Python
dbutils.secrets.get(scope=<<scope>>,key=<<key>>)

dbutils.secrets.get(scope=storage,key=blob) #Example

Use stored secret in notebook Use stored secret in notebook

Blog: https://joeho.xyz

LinkedIn: https://www.linkedin.com/in/joe-ho-0260758a

© 2021 — created by Joe