Azure¶

Azure storage setup¶

To set up Azure Blob Storage, one needs to complete the following steps:

Create a valid Azure account and subscription.
Create a new Resource Group.
Set up a new Azure Machine Learning Workspace.
Create a Storage Account to store your images. (Azure automatically creates one with a workspace, but you can use it or create a separate account.)
Configure authentication and authorization to enable programmatic access to the images.

Create a Resource group and a workspace¶

Login into the Azure portal (given you already have a valid account and a subscription)
Create a new resource group, under Home / Resource groups, hit Create, invent a name (e.g. pkliui-rg) and select a region for the group (e.g. Central US)
Next, create a new workspace. In All services / Azure Machine Learning, hit Create and select New workspace Specify the resource group, give the workspace a name and finish with Create. More details about Azure workspaces can be found here.

Create a Storage account and upload data¶

Now we need to make a new storage account to keep our images.

Under Home / Storage accounts, hit Create
Select the resource group (e.g. pkliui-rg) and specify a globally unique storage account name (e.g. skinetstorage), the region (US Central), performance and redundancy (LRS for the lowest costs and non-critical scenarios) and finish with Create
Enter the newly created storage account and under Storage browser / Blob containers select Add container and give a name to it e.g. skinet-container, click Create
A newly created container will appear under Blob containers
Upload the images into the container.

Create a datastore¶

Instead of issuing users with keys for a storage account and interacting with it directly, data are accessed via a datastore. An Azure Machine Learning datastore serves as a reference to an existing Azure storage account and it is a child resource of the respective workspace.
Datasores can be authenticated either using credentials or via idenity-based method that will be described below.
You need to link the storage account that holds your data to a new datastore using an account key in order to give AzureML permissions to manage it. In the storage account go to Security + Networking / Access keys and copy one of the keys.
In the workspace, hit Studio web URL to open Machine Learning Studio
Select the workspace and under Assets / Data select Datastores and Create
Enter a name for this new datastore e.g. skinetdatastore
Select Datastore type e.g. Azure Blob Storage
Select your newly created storage account e.g. skinetstorage and a respective blob container e.g. skinet-container
Pick “Save credentials with the datastore or data access”, under “Authentication type” select “Account key” and paste the storage account key from the clipboard, click Create.

Identification of datasets¶

The DatasetKey Enum is used to uniquely identify datasets throughout the code. The value of each enum member is called dataset_name.

Example Enum:

@unique
class DatasetKey(Enum):
    PH2 = "PH2_DATASET"
    ISIC2017 = "ISIC2017_DATASET"

Create a YAML config file (e.g., azure_settings.yaml), where add a section PATH_ON_DATASTORE mapping dataset_names from DatasetKey to the respective relative paths to data on the datastore.
The dataset_names in DatasetKey Enum must match the YAML keys under PATH_ON_DATASTORE. The corresponding YAML value data_root_on_azure points to relative paths to data on the datastore.

Example YAML config (values as in SkiNet/Azure/azure_settings.yaml)

PATH_ON_DATASTORE:
    PH2_DATASET: "PH2DATA/"
    ISIC2017_DATASET: "ISIC2017DATA_256"

PH2 is the dataset currently stored on the Azure datastore; the Azure examples below use it. ISIC2017 training data is held locally / on Kaggle (see data.md).

Enum Member (dataset_key)	Enum Value or YAML key (dataset_name)	YAML Value (data_root_on_azure)
PH2	“PH2_DATASET”	“PH2DATA/”
ISIC2017	“ISIC2017_DATASET”	“ISIC2017DATA_256”

Access data in Azure Blob Storage¶

Data in Azure Blob Storage can be accessed in various ways. In SkiNet, we use AzureMachineLearningFileSystem on local and Blobfuse2 mounts both on local and on Azure hosts.

Access via AzureMachineLearningFileSystem using Service Principal authentication¶

AzureMachineLearningFileSystem is mainly used to access the storage account programmatically during local experimentation (creation of metadata, inspection of images etc.).
To access AzureMachineLearningFileSystem, we set up a Service Principal. Users are issued with the SP’s credentials, including a secret.
And the SP is granted rights to perform defined actions on particular resources. This approach enables precise control over what each Service Principal can access and manage within Azure.
More about Service Principals and application registrations can be read here. Please note that using Managed Identities eliminates the need to manage credentials, but this type of authentication is available only for Azure machines.

Generate SP credentials¶

Begin setting up a SP as follows:

In Azure, go to Home / App registrations and start a new registration with a name e.g. skinet-sp. Select “Accounts in this organizational directory only” and then “Register”
Enter the newly created service principal application by clicking on its name in the list of registered applications
Note “Application (client) ID”, “Directory (tenant) ID”, you will need them for azure_settings.yaml file to be able to access data programmatically.
Under Certificates and secrets, hit New client secret and create a new token for the service principal. The token will appear only once, so note it well. To keep this token, either create an environment variable, e.g. in python os.environ["AZURE_CLIENT_SECRET"] = TOKEN_VALUE or keep it in a separate text file for reading out later.
In case the same service principal will be used by multiple users, all of them need to have this saved token on their machines.
Add credentials to the YAML config file (e.g., azure_settings.yaml)

AZURE_TENANT_ID: "<tenant-id>"
AZURE_CLIENT_ID: "<client-id>"
SUBSCRIPTION_ID: "<subscription-id>"
RESOURCE_GROUP: "<resource-group>"
WORKSPACE_NAME: "<workspace-name>"
DATASTORE_NAME: "<datastore-name>"
PATH_ON_DATASTORE:
    PH2_DATASET: "PH2DATA/"
    ISIC2017_DATASET: "ISIC2017DATA_256"

Now when we have set up the service principal, we need to grant it with appropriate rights

Grant right to SP via role assignments¶

In order to be able to access blob’s data with the SP credentials and AzureMachineLearningFileSystem, we need to grant righs to the SP, which is done via role assignment.
Go to your Azure Machine Learning Workspace (not studio but the workspace in Azure portal) and under Access control (IAM) / Role assignments choose “New role assignment”
Select AzureML Data Scientist, pick the newly created service principal and finish by clicking Save. You should be able to access the data in your data storage account programmatically using the service principal credentials and AzureMachineLearningFileSystem tools
Note that in this case an access to Azure Blob Storage is granted at the level of the Workspace and the role enables for data access through AzureMachineLearningFileSystem. For any other way of access, e.g. data mounted via blobfuse, a different role and a different level will be required.

Access datasets from code¶

from SkiNet.Azure.azure_setup import AzureSetup, service_principal_authentication
from SkiNet.Utils.experiment_keys import DatasetKey

service_principal_authentication()  # module-level function, not an AzureSetup method

dataset_name = DatasetKey.PH2.value  # "PH2_DATASET"
fs = AzureSetup.get_azureml_filesystem(dataset_name)

In this example, dataset_name (DatasetKey.PH2.value == "PH2_DATASET") matches the YAML key PH2_DATASET, which maps to the path “PH2DATA/” on the Azure datastore. get_azureml_filesystem is a classmethod that resolves the URI via get_azure_uri (which loads azure_settings.yaml); service_principal_authentication and managed_identity_authentication are module-level functions in SkiNet.Azure.azure_setup, not methods of AzureSetup.

Access via Blobfuse2 mounts using managed identity and SP authentication¶

Azure Blob Storage can be mounted on an Azure machine via Blobfuse2 and accessed without credentials by using an Azure Managed Identity. Blobfuse mounts can also be set up on local machines (in this case other kind of authentication such as SP is required).

Create and set up a new Managed Identity (MI)¶

To be able to access data whilst on an Azure machine, one should set up managed identity (MI) authentication for that machine.
In Azure, go to Home / Create a resource and select “Infrastructure services”
Under “Idenity” select “Managed Identity” and in a new windows press “Create”
Specify the resource group, name and region for the MI.
Now we want to use this MI to access data in the Azure Blob Storage from within the Azure machine
Go to the Storage account and under Access Control, select “Add role assignment”
Select “Storage Blob Data Contributor” and assign access to the managed identity.

Set up Blobfuse¶

BlobFuse is an open-source virtual file system driver that integrates Azure Blob Storage with Linux environments. Both local and Azure machines are possible.
By using BlobFuse, you can mount Azure Storage account containers as a file system, making blob data accessible through standard Linux file operations
More information about blobfuse2 mounts is here
You need to st up a BlobFuse config file that contains blob storage account name, container and endpoint. For more info see here
Depending on the mode of access (SPN for service principal and MSI for managed idenity), other fields are required. In example below, the mode of access is set to SP, so the fields for SP credentials are included. In MI mode, the code will ignore these fields and modify the mode accordingly.

Example YAML config (showing only the storage credentials section)

azstorage:
  type: block # type: block|adls
  account-name: skinetstorage #<name of the storage account>
  container: skinet-container #<name of the storage container to be mounted>
  endpoint: skinetstorage.blob.core.windows.net #<example - https://account-name.blob.core.windows.net>
  mode: spn #key|sas|spn|msi|azcli
  clientid: '${AZURE_CLIENT_ID}'  # AZURE_CLIENT_ID
  tenantid: '${AZURE_TENANT_ID}' # AZURE_TENANT_ID
  clientsecret: '${AZURE_CLIENT_SECRET}'

Access datasets from code¶

Example data mount on Azure machine using MI

For training, Azure Blob Storage is expected be mounted on a computational instance with an assigned managed identity
In this case, data are available to the MI within the scope granted to it and can be accesed from the code without specifying any credentials.

from SkiNet.Utils.project_paths import BLOBFUSE2_CONFIG_PATH  # Default blobfuse2 config path

from SkiNet.Azure.azure_blob_mounter import AzureBlobMounter
mounter = AzureBlobMounter(
    mount_path=path_on_azure_machine_where_to_mount_datas,
    config_path=BLOBFUSE2_CONFIG_PATH,  # Uses default config unless overridden
    is_azure_mount=True
)
mounter.mount()

This is wrapped in the “mount_data.py” script which can be callable for Azure mounts as

python mount_data.py --mount-path="SOME_AZURE_MOUNT_PATH" --is-azure-mount --config-path=BLOBFUSE2_CONFIG_PATH
# If you do not specify --config-path, the script will use the default value (BLOBFUSE2_CONFIG_PATH) automatically.

Example data mount on local using SP authentication under the hood

It is possible to mount the Blob Storage on a local machine. In this case, a SP auth is required (and has been set up in code).

python mount_data.py --mount-path="SOME_LOCAL_MOUNT_PATH" --config-path=BLOBFUSE2_CONFIG_PATH
# If you do not specify --config-path, the script will use the default value (BLOBFUSE2_CONFIG_PATH) automatically.

 python mount_data.py --mount-path /mnt/data --config-path SkiNet/Azure/blobfuse2.yaml

Running experiments on Azure¶

Create a new Azure compute and set it up using run_on_azure.sh¶

Create a new Azure compute machine and provide it with a startup script “run_on_azure.sh” and the name of the Docker image as an argument, e.g. “pkliui/skinet:gpu” assuming that one is publicly available.
Each time the machine starts, the script sets environment variables, downloads and installs git, docker and blobuse2; also the startup script will pull the latest changes in the repo, mount the Azure Blob Storage as specified in Blobfuse2 config , pull the latest version of the Docker image and run it
Perhaps it will make sense to split this into 2 scripts: install do not need to be repeated every time machine starts.
Note: Docker should be run with the special options allowing access to the mounted Azure Blob Storage

docker run --rm -it\
  --cap-add=SYS_ADMIN \
  --device=/dev/fuse \
  --security-opt apparmor:unconfined \
  --mount "type=bind,src=$HOST_REPO,dst=$CONTAINER_REPO" \
  --mount "type=bind,src=$AZURE_MOUNT_PATH,dst=$CONTAINER_AZURE_MOUNT_PATH" \
  -w "$CONTAINER_REPO" \
  "$IMAGE"

This command bind-mounts the repository from the host to the container and also bind-mounts the Azure Blob Storage (mounted on the host) into the Docker container as specified.

Azure¶

Azure storage setup¶

Create a Resource group and a workspace¶

Create a Storage account and upload data¶

Create a datastore¶

Identification of datasets¶

Access data in Azure Blob Storage¶

Access via AzureMachineLearningFileSystem using Service Principal authentication¶

Generate SP credentials¶

Grant right to SP via role assignments¶

Access datasets from code¶

Access via Blobfuse2 mounts using managed identity and SP authentication¶

Create and set up a new Managed Identity (MI)¶

Set up Blobfuse¶

Access datasets from code¶

Running experiments on Azure¶

Create a new Azure compute and set it up using run_on_azure.sh¶

Run code inside the docker environment as on local¶