BlogTagsProjectsSubscribe

How to store secrets in Azure Databricks

December 19, 2020
🏷 Azure
🏷 Data Engineering

Join our Newsletter

Subscribe to get beginner friendly DevOps Guide.
    We won't send you spam. Unsubscribe at any time.

    Background

    In Azure Databricks, we can write code to perform data transformation on data stored in various Azure Services, e.g. Azure Blob Storage, Azure Synapse. However, as other programs, sometimes, you want to protect credentials used in Azure Databricks, Azure Databricks provides a solid secret management approach to help you achieve that.

    Steps

    Prepare Databricks command-line interface (CLI) in Azure Cloud Shell

    Configure your cloud shell environment

    Open Cloud Shell & make sure you select “Bash” for the Cloud Shell Environment.

    Launch bash cloud shell Launch bash cloud shell

    Set up Virtual Environment

    Create Virtual Environment with below command.

    # Bash
virtualenv -p /usr/bin/python2.7 databrickscli

    Create virtual environment Create virtual environment

    Activate your virtual environment

    Activate your virtual environment with below command.

    # Bash
source databrickscli/bin/activate

    Activate virtual environment Activate virtual environment

    Install Databricks CLI

    Install Databricks CLI with below command.

    # Bash
pip install databricks-cli

    Install Databricks CLI Install Databricks CLI

    Create secret in Azure Databricks

    Set up authentication

    Before you can create a secret, you need to authenticate as a user of the Azure Databricks, which requires your Azure Databrics workspace’s URL and a token

    Get your Azure Databricks workspace’s URL

    You can navigate to your Azure Databricks workspace and copy its URL.

    Get Databricks URL Get Databricks URL

    Generate Access Token for your Azure Databricks workspace

    You can follow below steps to retrieve access token

    1. Launch Databricks workspace Launch Databricks Workspace
    2. Click 'User Settings' Click 'User Settings'
    3. Click 'Generate New Token' Generate access token
    4. Configure access token & click 'Generate' Configure access token
    5. Copy access token Copy access token

    Create Secret Scope

    After authentication, you need to first create a secret scope which you may group several secrets.

    If your databricks is in Standard plan, you can only create secret scope which will be shared with other users in the same workspace.

    Check databricks plan Check databricks plan

    # Bash
databricks secrets create-scope --scope <<scope>>

# Example
databricks secrets create-scope --scope storage --initial-manage-principal users # Standard Plan
databricks secrets create-scope --scope storage # Premium plan

    Create secret scope Create secret scope

    Create Secret

    You can use below command to create secret under the specified scope.

    # Bash
databricks secrets put --scope <<scope>> --key <<key name>>

databricks secrets put --scope storage --key blob #Example

    Type command to launch secret editor Type command to launch secret editor

    Type your secret and save Type your secret and save

    Use Secret in Notebook

    You can use secret by below command in notebook.

    # Python
dbutils.secrets.get(scope=<<scope>>,key=<<key>>)

dbutils.secrets.get(scope=storage,key=blob) #Example

    Use stored secret in notebook Use stored secret in notebook

    Blog: https://joeho.xyz

    LinkedIn: https://www.linkedin.com/in/joe-ho-0260758a

    Subscribe