Cosmos DB Configuration Management


black-board-bright-695266I often found the need to automate Cosmos DB deployments.

I like to have users run an ARM template and be ready to go.  Automation is key component to repeatability.  It enables automated testing and increase agility in many other ways.

ARM Templates don’t go beyond a database account.

Take the article Setup for populating Cosmos DB with random data using Logic Apps.  On top of providing an ARM template, we needed to add bunch of steps in the Portal to setup Cosmos DB.

This is why I built cosmos-db-target-config, available on GitHub.  It mimics what a full blown ARM Template provider could do for Cosmos DB.

ARM Template limitations

Cosmos DB’s ARM template provider only deploys a database account.

Azure Command Line Interface (CLI) goes further.  We can script databases & collections.  We can’t script anything beyond a collection (e.g. stored procedures & functions).

The tool covering the entire Cosmos DB REST API is its SDK.  It is available on several platforms, including .NET, Java, Python & Node.js.

Enter Target Config

Cosmos DB Target Config works by reading a target config and applying it to a Cosmos DB Database Account.

It can create databases from scratch.  It can also evolve existing databases.  It is a target state system.

Cosmos DB Target Config takes the form of a .NET Core Console Application.  It can easily be used as a Container Instance in an ARM Template deployment.  We discussed this automation approach here.

The Linux Container image is available on Docker Hub.  It takes a Cosmos DB Account endpoint, access key and a target configuration file public URL.

We can use SAS token to make a private blob accessible.

Target Configuration

A Target Configuration file is a simple JSON file.  For instance:


{
   "$schema": "https://raw.githubusercontent.com/vplauzon/cosmos-db-target-config/master/documentation/cosmos-target-schema.json",
   "destructiveFlags": [
     "database",
     "collection",
     "storedProcedure"
   ],
   "databases": [
     {
       "name": "simple-db",
       "collections": [
         {
           "name": "partitioned-with-sproc",
           "partitionKey": "/dept",
           "requestUnits": 1200,
           "storedProcedures": [
             {
               "name": "oneSproc",
               "targetUrl": "sproc1.js"
             }
           ]
         }
       ]
     }
   ]
 }

This specific file creates a database named simple-db.  It creates one collection in that database named partitioned-with-sproc.  It finally creates a stored procedure within that collection.  The file defines the throughput of the collection and its partition key.

At the beginning of the file we specify destructiveFlags.  Those define the type of resources Cosmos DB Target Config can delete.  By default, it will only add resources and won’t delete anything.

Trying it

Let’s give it a try.  Let’s deploy a sample by clicking the following button:

image

This should lead to the following Azure Portal pane:

The only template parameter is the Demo Target Config File.  This allows us to choose a demo file.  Those files are all the same folder on GitHub.

Let’s start with the first one:  Single Unpartitioned Collection.

Deployment usually takes less than 5 minutes.  Once it’s completed, we should have the following resources in the resource group:

image

Let’s first look at the Azure Cosmos DB account.  The name of the resource is appended a unique string so it doesn’t clash with other accounts in Azure.

In the overview page of the account we can see the account isn’t empty but has a database with one collection.

image

In Data Explorer, we can see the collection has a stored procedure:

image

With only one ARM template we got everything deployed in Cosmos DB!

The magic:  Container Instance

Let’s look at how that happened.  Going back to the resource group, let’s pick the container group.  We’ll likely need to hit Refresh to see the logs.

image

The container first outputs the environment variables passed by the ARM Template.  The first two are credentials while the third is the target URL of the configuration file.

That target configuration content itself is then output.

If we scroll down, we can see the operations it did on the Cosmos DB account:

image

Update

Let’s now update a Cosmos DB account.

First, we’ll need to delete the container group.

image

We do this so that the ARM template can recreate a container group and run the new target configuration.  This step is necessary given how Container Instance work.

Now, let’s deploy again:

image

Let’s specify the same resource group so that we will update the same Cosmos DB account.

Let’s select Single Partitioned Collection.

image

After the deployment, we can see the previous database and its collection are gone.  They are replaced by another database and collection.

image

If we look at the logs of the container, we can see what occurred:

image

We can see that Cosmos DB Target Config behaves like an ARM Template.  It takes a Cosmos DB account from any state and brings it to the state described in configuration.

Here it

  • Kept the database in place
  • Removed a collection
  • Added another collection
  • Added a stored procedure

Container Group Configuration

We integrate cosmos-db-target-config with ARM template by using Azure Container Instance.  We pass parameters to the container as environment variables.


"environmentVariables": [
  {
    "name": "ACCOUNT_ENDPOINT",
    "value": "[reference(variables('Account Id')).documentEndpoint]"
  },
  {
    "name": "ACCOUNT_KEY",
    "value": "[listKeys(variables('Account Id'), '2015-04-08').primaryMasterKey]"
  },
  {
    "name": "TARGET_URL",
    "value": "[variables('Target URL')]"
  }
]

The first two parameters are credentials to access Cosmos DB Account.  They do not need to be hardcoded.  They can be inferred by as we did here, pointing to the Cosmos DB Account resource.  The last variable is the target URL.

Learning more

cosmos-db-target-config is open sourced on GitHub.  This is where its documentation lives.

Summary

We presented cosmos-db-target-config.

Hopefully this could be useful to people using Cosmos DB.

Feel free to leave feedback about it.

Advertisements

3 thoughts on “Cosmos DB Configuration Management

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s