Using Blobfuse2 to mount Azure Blob Storage using managed identity from Azure Container Instance privileged containers

#azure-privileged-containers.md

Using Blobfuse2 to mount Azure Blob Storage using managed identity from Azure Container Instance privileged containers

• You can mount Azure Blob Storage as a filesystem on Linux using Blobfuse2. This includes support for both blob containers and data lake hierarchical namespaces.
• You can also perform this mount on Linux containers as long as the container has mount privilege provided by the CAP_SYS_ADMIN capability.
• You can run containers with privileged capabilities on Azure PaaS using Azure Container Instance confidential containers.
• You can authenticate Blobfuse2 against Azure Storage using an Entra workload identity, including Azure Container Instance managed identity.

The Bicep template azure-privileged-containers.bicep provides an example of how to:

• Mount an Azure Storage data lake container using Blobfuse2
• Within a privileged Azure Container Instance confidential tier instance
• Running an Azure Linux based container
• Authenticated against Azure Storage using the Azure Container Instance system assigned managed identity
• Exposing the mounted filesystem using the Filebrowser web-based file manager
• Serving the Filebrowser UI using a Caddy reverse proxy which provides automatic HTTPS with a public CA issued TLS certificate for the Azure Container Instance default domain name (<name>.<location>.azurecontainer.io)

Note

It might take a few minutes for Azure role assignments to propagate and/or the TLS certificate to be issued.

Azure privileged containers

Azure hosted containers often run in multi-tenant or multi-application environments and as such have restricted capabilities in order to maintain container isolation. One exception is Azure Container Instance confidential containers which allow running containers in privileged mode. This is possible because confidential containers run within a hardware based Hyper-V isolated trusted execution environment.

Azure Container Instance confidential containers are configured with privileged capabilities by setting the privileged flag on the securityContext configuration:

resource containerGroup 'Microsoft.ContainerInstance/containerGroups@2024-10-01-preview' = {
  properties: {
    // securityContext privileged only available with confidential linux containers
    sku: 'Confidential'
    osType: 'Linux'
    containers: [
      {
        properties: {
          securityContext: {
            privileged: true
          }
        }
      }
    ]
  }
}

Azure Kubernetes Service also provides options to configure containers with privileged capabilities. This is a high risk configuration flagged by Azure Policy since such privilege might allow a malicious actor or a compromised application to break out of the container boundary.

Azure container hosting capabilities comparision

Below is an overview of the capabilities provided by different Azure PaaS container hosting options (checked using capsh):

Capability	Container App	App Service	Container Instances Standard	Container Instances Confidential	Container Instances Confidential Privileged
cap_audit_control					x
cap_audit_read	x	x	x	x	x
cap_audit_write					x
cap_block_suspend					x
cap_bpf					x
cap_checkpoint_restore					x
cap_chown	x	x	x	x	x
cap_dac_override	x	x	x	x	x
cap_dac_read_search					x
cap_fowner	x	x	x	x	x
cap_fsetid	x	x	x	x	x
cap_ipc_lock					x
cap_ipc_owner					x
cap_kill	x	x	x	x	x
cap_lease					x
cap_linux_immutable					x
cap_mac_admin					x
cap_mac_override					x
cap_mknod	x	x	x	x	x
cap_net_admin					x
cap_net_bind_service	x	x	x	x	x
cap_net_broadcast					x
cap_net_raw	x	x	x	x	x
cap_perfmon					x
cap_setfcap	x	x	x	x	x
cap_setgid	x	x	x	x	x
cap_setpcap	x	x	x	x	x
cap_setuid	x	x	x	x	x
cap_sys_admin					x
cap_sys_boot					x
cap_sys_chroot	x	x	x	x	x
cap_sys_module					x
cap_sys_nice					x
cap_sys_pacct					x
cap_sys_ptrace					x
cap_sys_rawio					x
cap_sys_resource					x
cap_sys_time					x
cap_sys_tty_config					x
cap_syslog					x
cap_wake_alarm					x

azure-privileged-containers.bicep

param location string = resourceGroup().location
param username string = 'admin'
@secure()
param password string = newGuid()

resource storageAccount 'Microsoft.Storage/storageAccounts@2023-05-01' = {
  name: 'st${uniqueString(resourceGroup().id)}'
  location: location
  kind: 'StorageV2'
  sku: {
    name: 'Standard_LRS'
  }
  properties: {
    allowSharedKeyAccess: false
    isHnsEnabled: true
    defaultToOAuthAuthentication: true
    minimumTlsVersion: 'TLS1_2'
    allowedCopyScope: 'AAD'
  }
  resource blobService 'blobServices' = {
    name: 'default'
    resource container 'containers' = {
      name: 'mount'
    }
  }
}

// https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/storage#storage-blob-data-owner
var roleDefinitionId = 'b7e6dc6d-f1e8-4753-8033-0f276bb0955b'
var roleDefinitionResourceId = subscriptionResourceId('Microsoft.Authorization/roleDefinitions', roleDefinitionId)
resource roleAssignment 'Microsoft.Authorization/roleAssignments@2022-04-01' = {
  scope: storageAccount
  name: guid(storageAccount.id, roleDefinitionResourceId, aci.id)
  properties: {
    principalId: aci.identity.principalId
    roleDefinitionId: roleDefinitionResourceId
    principalType: 'ServicePrincipal'
    description: 'Gives Azure Container Instance "${aci.id}" access to the storage account'
  }
}

var uniqueDnsNameLabel = uniqueString(resourceGroup().id)
resource aci 'Microsoft.ContainerInstance/containerGroups@2024-10-01-preview' = {
  name: 'aci-${uniqueDnsNameLabel}'
  location: location
  identity: {
    type: 'SystemAssigned'
  }
  properties: {
    // SecurityContextOnlyConfidential: Container security context options are only available with SKU 'Confidential'
    sku: 'Confidential'
    osType: 'Linux'
    // restart to retry in case of role assignment propagation delay
    restartPolicy: 'OnFailure'
    ipAddress: {
      type: 'Public'
      // must use unsecure to have deterministic DNS name injected at deployment time into the container
      // since ACI does not inject the DNS hash component into the container runtime
      // generates DNS in the form: <uniqueDnsNameLabel>.<location>.azurecontainer.io
      // https://learn.microsoft.com/en-us/azure/container-instances/how-to-reuse-dns-names#understand-the-dns-name-reuse-policy
      autoGeneratedDomainNameLabelScope: 'Unsecure'
      dnsNameLabel: uniqueDnsNameLabel
      ports: [
        {
          port: 80
        }
        {
          port: 443
        }
      ]
    }
    containers: [
      {
        name: 'azurelinux'
        properties: {
          // https://github.com/microsoft/azurelinux
          // https://mcr.microsoft.com/en-us/artifact/mar/azurelinux/base/core/about
          image: 'mcr.microsoft.com/azurelinux/base/core:3.0'
          resources: {
            requests: {
              cpu: 1
              memoryInGB: 1
            }
          }
          ports: [
            {
              port: 80
            }
            {
              port: 443
            }
          ]
          environmentVariables: [
            // https://github.com/Azure/azure-storage-fuse/blob/main/README.md#environment-variables
            {
              // the storage account to be connected
              name: 'AZURE_STORAGE_ACCOUNT'
              value: storageAccount.name
            }
            {
              // account type 'block' (blob) or 'adls' (data lake)
              name: 'AZURE_STORAGE_ACCOUNT_TYPE'
              value: storageAccount.properties.isHnsEnabled ? 'adls' : 'block'
            }
            {
              // name of the container to be mounted
              name: 'AZURE_STORAGE_ACCOUNT_CONTAINER'
              value: storageAccount::blobService::container.name
            }
            {
              // storage authentication: Key, SAS, MSI, SPN
              name: 'AZURE_STORAGE_AUTH_TYPE'
              value: 'MSI'
            }

            // https://filebrowser.org/cli/filebrowser
            {
              // filebrowser username
              name: 'FB_USERNAME'
              value: username
            }
            {
              // filebrowser password plain text (will be hashed)
              name: 'FB_PASSWORD_TEXT'
              secureValue: password
            }

            // https://caddyserver.com/docs/command-line#caddy-reverse-proxy
            {
              // inject ACI DNS name used by Caddy to issue TLS certificate
              name: 'ACI_DNS_NAME'
              value: '${uniqueDnsNameLabel}.${location}.azurecontainer.io'
            }
          ]
          command: [
            '/bin/bash'
            '-c'
            join([
              // update the package cache and install blobfuse2, CA root certificates, jq and tar
              'tdnf update -y && tdnf -y install blobfuse2 ca-certificates jq tar'
              // download the latest release of filebrowser and extract it to /usr/local/bin
              'curl -fsSL https://github.com/filebrowser/filebrowser/releases/download/v2.31.2/linux-amd64-filebrowser.tar.gz | tar -xzf - -C /usr/local/bin filebrowser'
              // download the latest release of caddy and extract it to /usr/local/bin
              'curl -fsSL https://github.com/caddyserver/caddy/releases/download/v2.8.4/caddy_2.8.4_linux_amd64.tar.gz | tar -xzf - -C /usr/local/bin caddy'
              // create a cache directory and a storage directory
              'mkdir /root/storage && blobfuse2 mount /root/storage --block-cache --allow-other'
              // start filebrowser on port 8080 with no authentication and a database file as a background process
              // https://filebrowser.org/cli/filebrowser
              'filebrowser -p 8080 -d /root/filebrowser.db -r /root/storage --password "$(filebrowser hash "$FB_PASSWORD_TEXT")" &'
              // start caddy as a reverse proxy to filebrowser acquiring a TLS certificate for ACI_DNS_NAME
              // https://caddyserver.com/docs/command-line#caddy-reverse-proxy
              'caddy reverse-proxy --from $ACI_DNS_NAME --to :8080'
            ], '\n')
          ]
          securityContext: {
            // when Blobfuse2 is mounted on a container, SYS_ADMIN privileges are required for it to interact with the fuse driver
            // https://github.com/Azure/azure-storage-fuse/blob/main/README.md#limitations
            privileged: true
          }
        }
      }
    ]
  }
}

@description('The URL to access the filebrowser UI with the username and password through Caddy reverse proxy')
output url string = 'https://${aci.properties.ipAddress.fqdn}/'
@description('The username to access the filebrowser UI')
output username string = username
#disable-next-line outputs-should-not-contain-secrets
@description('The password to access the filebrowser UI')
#disable-next-line outputs-should-not-contain-secrets
output password string = password