Linda Sheard lindacmsheard

Notes on Azure ML environments

(Ref: https://docs.microsoft.com/en-us/azure/machine-learning/reference-yaml-environment#yaml-syntax)

The environment item in an AzureML cli v2 job specification yml file can leverage an AzureML curated environment directly, referenced as follows:

environment: azureml:AzureML-pytorch-1.10-ubuntu18.04-py38-cuda11-gpu@latest

A list of available curated environments can be found here, or from wtihin the environments tab in AzureML Studio.

Context: this question

The following commands can be exectuted from within the Data Explorer interface of an Azure Cosmos DB Gremlin API database resource, or from other gremlin consoles such as a local TinkerPop installation.

Create required vertices

This can be in an empty graph, like that created when adding a graph in Azure Cosmos DB Gremlin API or added to an exsisting graph, like for example the 'modern' sample graph from the Tinkerpop Getting Started documentation.

g.addV("person").property("pk","pk").property("id","a").property("name","Alice")
g.addV("person").property("pk","p

Last updated: July 2021

-- disclaimer: This is my personal preference, always review the official documentation for updates.

Where to go for AzureML Docs

Examples of how to get started

This is a very currently active repo with examples and tutorials.

Use this also to raise github issues with any specific questions or issues.

#create a dataframe from filepaths

directory = '/path/on/dbfs'

file_paths = dbutils.fs.ls(directory)

#e.g
print(file_paths[0].path)
print(file_paths[0].name)

Pre-reqs

From the terminal running on the AML compute instance, ensure that conda is accessible in your current shell session.

which conda
#-> should return a path to a conda executable

If conda is not found, then verify that the executable is present where expected

ls /anaconda/bin/conda

Connect Azure ML to Azure DataLake Gen 2 with a Service Principal

Note that service principal role assignments may take a short while to become available, so give it a few minutes before testing the access.

Create a service principal

az ad sp create-for-rbac --name myAMLWorkspaceRep

Note the appId (client ID) and password (client secret) returned:

Databricks CLI Cheatsheet - managing secrets

(optional) install cli into a virtual python env

virtualenv -p /usr/bin/python2.7 databrickscli
source databrickscli/bin/activate
pip install databricks-cli

Alternatively just install natively

Dealing with merge conflicts

Merge conflicts are not scary. Any good text editor like VSCode for example will help with buttons and visualisations of what you need to do. But here's the gist...

When a merge conflict occurs, git will actually insert some lines in the file to mark the spot and show both versions of lines that are conflicting:

<<<<<<< HEAD
my current text in the file
=======

Disclaimer: There are many subtleties to using git - this is a very basic workflow. Discuss in your team how you want to work!

Pre-req

This gist assumes that you have authorisation to clone/pull/push from your repo. Either it is public, or you have set up ssh authentication.

Git basics

git clone <http or ssh link from git portal>
# change into the folder that is created as a result of the cloning.
cd

	# This terraform spec provisions
	# - Azure resource group
	# - Azure Cosmos DB Account (SQL API, Analytics Storage enabled)
	# - Azure Log Analytics Workspace
	# - Diagnostic Setting that sends Cosmos DB logs to the Log Analytics Workspace
	# - A Cosmos database in the account (SQL API document db)
	# - A container in the database, with throughput and indexing configurations

	# Set up terraform (basic example, local backend)
	terraform {