Azure Function App PowerShell script to backup Blob storage to an Azure File share

run.ps1

# Input bindings are passed in via param block.
param($Timer)

# Define variables
$SrcStgAccURI = "https://sourceblobstg.blob.core.windows.net/"
$SrcBlobContainer = "myblobs"
$SrcSASToken = SAS_TOKEN_SOURCE
$SrcFullPath = "$($SrcStgAccURI)$($SrcBlobContainer)?$($SrcSASToken)"

$DstStgAccURI = "https://destinationfilestg.file.core.windows.net/"
$DstFileShare = "backupstaging"
$DstSASToken = SAS_TOKEN_DESTINATION
$DstFullPath = "$($DstStgAccURI)$($DstFileShare)?$($DstSASToken)"

# Get the current universal time in the default string format
$currentUTCtime = (Get-Date).ToUniversalTime()

# The 'IsPastDue' property is 'true' when the current function invocation is later than scheduled.
if ($Timer.IsPastDue) {
    Write-Host "PowerShell timer is running late!"
}

# Write an information log with the current time.
Write-Host "PowerShell timer trigger function ran! TIME: $currentUTCtime"

# uncomment to delete azcopy and force download again
# del azcopy.exe

# Test if AzCopy.exe exists in current folder
$WantFile = "azcopy.exe"
$AzCopyExists = Test-Path $WantFile
Write-Host "AzCopy exists:" $AzCopyExists

# Download AzCopy if it doesn't exist
If ($AzCopyExists -eq $False)
{
    Write-Host "AzCopy not found. Downloading..."
    
    #Download AzCopy
    Invoke-WebRequest -Uri "https://aka.ms/downloadazcopy-v10-windows" -OutFile AzCopy.zip -UseBasicParsing
 
    #Expand Archive
    write-host "Expanding archive..."
    Expand-Archive ./AzCopy.zip ./AzCopy -Force

    # Copy AzCopy to current dir
    Get-ChildItem ./AzCopy/*/azcopy.exe | Copy-Item -Destination "./AzCopy.exe"
}
else
{
    Write-Host "AzCopy found, skipping download."
}

# Set env values for AzCopy
$env:AZCOPY_LOG_LOCATION = $env:temp+'\.azcopy'
$env:AZCOPY_JOB_PLAN_LOCATION = $env:temp+'\.azcopy'

# Run AzCopy from source blob to destination file share

Write-Host "Backing up storage account..."
./azcopy.exe copy $SrcFullPath $DstFullPath --recursive --overwrite=ifsourcenewer

Hi Andrew. I'm wondering how you got this working in an Azure Function. When I run your code I get:

2021-06-12T05:14:30.152 [Information] Executing 'Functions.func2' (Reason='This function was programmatically called via the host APIs.', Id=5bee6e4d-c5b3-4760-b0b7-2b5e123e53fd)
2021-06-12T05:14:30.152 [Information] Executing 'Functions.func2' (Reason='This function was programmatically called via the host APIs.', Id=5bee6e4d-c5b3-4760-b0b7-2b5e123e53fd)
2021-06-12T05:14:30.204 [Information] INFORMATION: AzCopy exists: False
2021-06-12T05:14:30.208 [Information] INFORMATION: AzCopy not found. Downloading...
2021-06-12T05:14:30.587 [Error] ERROR: Could not find file 'C:\home\site\wwwroot\AzCopy.zip'.

Exception             : 
    Type       : System.IO.FileNotFoundException
    Message    : Could not find file 'C:\home\site\wwwroot\AzCopy.zip'.
    FileName   : C:\home\site\wwwroot\AzCopy.zip
    TargetSite : 
        Name          : ValidateFileHandle
        DeclaringType : System.IO.FileStream
        MemberType    : Method
        Module        : System.Private.CoreLib.dll
    StackTrace : 
   at System.IO.FileStream.ValidateFileHandle(SafeFileHandle fileHandle)
   at System.IO.FileStream.CreateFileOpenHandle(FileMode mode, FileShare share, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
   at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize)
   at System.IO.File.Create(String path)
   at Microsoft.PowerShell.Commands.StreamHelper.SaveStreamToFile(Stream stream, String filePath, PSCmdlet cmdlet)
   at Microsoft.PowerShell.Commands.InvokeWebRequestCommand.ProcessResponse(HttpResponseMessage response)
   at Microsoft.PowerShell.Commands.WebRequestPSCmdlet.ProcessRecord()
   at System.Management.Automation.Cmdlet.DoProcessRecord()
   at System.Management.Automation.CommandProcessor.ProcessRecord()
    Source     : System.Private.CoreLib
    HResult    : -2147024894
CategoryInfo          : NotSpecified: (:) [Invoke-WebRequest], FileNotFoundException
FullyQualifiedErrorId : System.IO.FileNotFoundException,Microsoft.PowerShell.Commands.InvokeWebRequestCommand
InvocationInfo        : 
    MyCommand        : Invoke-WebRequest
    ScriptLineNumber : 17
    OffsetInLine     : 5
    HistoryId        : 1
    ScriptName       : C:\home\site\wwwroot\func2\run.ps1
    Line             : Invoke-WebRequest -Uri "https://aka.ms/downloadazcopy-v10-windows" -OutFile AzCopy.zip -UseBasicParsing
                       
    PositionMessage  : At C:\home\site\wwwroot\func2\run.ps1:17 char:5
                       +     Invoke-WebRequest -Uri "https://aka.ms/downloadazcopy-v10-windows …
                       +     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    PSScriptRoot     : C:\home\site\wwwroot\func2
    PSCommandPath    : C:\home\site\wwwroot\func2\run.ps1
    InvocationName   : Invoke-WebRequest
    CommandOrigin    : Internal
ScriptStackTrace      : at <ScriptBlock>, C:\home\site\wwwroot\func2\run.ps1: line 17

It seems the file cannot be downloaded, but running the iwr line in a local console works just fine.

Hi Agazoth, that sounds odd. If you add a dir at line 41 does it show the zip file as successfully downloaded?

Yes, the file was downloaded.

Actually I solved my need by using Start-AzStorageBlobCopy: https://docs.microsoft.com/en-us/powershell/module/az.storage/start-azstorageblobcopy?view=azps-6.0.0

Extracting zip-files and running .exe files in an Azure Function seemed a little too exotic for my taste ;-)

@andrew-kelleher I'm trying to do something similar to what you have above. I've already gotten it loaded and unzipped, and then set the log and plan paths based on what you have above, and am now getting a "panic: invalid state, azcopyAppPathFolder should be initialized by root" error when I try to login in AzCopy.

It seems to be an access issue, so I've tried several locations for the executable and log/plan paths ("$($env:temp).azcopy", "D:\home\site\wwwroot\AzCopy", and others), with hopes of getting past the permissions, but still no luck. Any ideas?

I'm on a Function App on a Standard S1 service plan, runtime version 4, PowerShell 7.2.

Thanks!

@andrew-kelleher I'm trying to do something similar to what you have above. I've already gotten it loaded and unzipped, and then set the log and plan paths based on what you have above, and am now getting a "panic: invalid state, azcopyAppPathFolder should be initialized by root" error when I try to login in AzCopy.

It seems to be an access issue, so I've tried several locations for the executable and log/plan paths ("$($env:temp).azcopy", "D:\home\site\wwwroot\AzCopy", and others), with hopes of getting past the permissions, but still no luck. Any ideas?

I'm on a Function App on a Standard S1 service plan, runtime version 4, PowerShell 7.2.

Thanks!

Hi @kevsor2,
Did you ever get to the bottom of this, I'm experiencing the exact same problem?
Any help would be appreciated.
Thanks

Hi @stevenjpbrown,
No, I ended up having to drop attempting this method all together because the security team doesn't want to allow using SAS tokens. AzCopy unfortunately requires using a SAS token on at least one end of the copy, so I can't use it at all.
Sorry, wish I was more help. :(

@stevenjpbrown it's been a while since I last looked at this. I suspect something's changed in one of Function App's updated runtimes or libraries, causing this to fail, unfortunately.

Depending on your requirements, there's now a native Azure operational backup capability which I covered here. HTH.

Hi @stevenjpbrown, No, I ended up having to drop attempting this method all together because the security team doesn't want to allow using SAS tokens. AzCopy unfortunately requires using a SAS token on at least one end of the copy, so I can't use it at all. Sorry, wish I was more help. :(

:-( thanks for the reply, kind of stuck at the same point myself

@stevenjpbrown it's been a while since I last looked at this. I suspect something's changed in one of Function App's updated runtimes or libraries, causing this to fail, unfortunately.

Depending on your requirements, there's now a native Azure operational backup capability which I covered here. HTH.

Hi Andrew, thanks for the update but this isn't really what I'm after. I need to trigger a copy to another storage account once it uploads to one storage account. It is a file share with single deep root folders I need to copy only.

hey @stevenjpbrown, i've integrated azcopy.exe in my package folder along run.ps1 and run & "./name of your function/azcopy.exe". It would have been better to directly downloaded it from internet but unfortunalty i run into the same error as you

thanks for the update @GuetarniTarik I take it you didn’t get any further with it? I raised a ticket with MS but didn’t get any further myself.

at the end it worked for me by adding azcopy.exe directly in the package. I suggest you to do the same

I did do that but couldn’t get it working. It didn’t like the path to the log files. It was changing the slash on a windows function app to / rather than . Did you use Linux or windows?

@GuetarniTarik - whats the way to integrate azcopy.exe with the package folder?

Hello Andrew,

I tried to execute this script, but it is not working for me, can you pass on any documentation if it is handy with you.

Thanks!