Skip to content

Instantly share code, notes, and snippets.

@sheldonhull
Last active January 22, 2021 01:34
Show Gist options
  • Select an option

  • Save sheldonhull/9bdb6ae57121e3c9ddc3fa594119c476 to your computer and use it in GitHub Desktop.

Select an option

Save sheldonhull/9bdb6ae57121e3c9ddc3fa594119c476 to your computer and use it in GitHub Desktop.
[Install S5Cmd With PowerShell 4.0 Support] How to install s5cmd without all the magical tools like PowerShell Get #powershell #s3 #aws

Get S5Cmd

Older versions of PowerShell (4.0) and the older AWSTools don't have the required ability to sync down a folder from a key prefix. This also performs much more quickly for a quick sync of files like backups to a local directory for initiating restores against.

In testing down by the tool creator, they showed this could saturate a network for an EC2 with full download speed and be up to 40x faster than using CLI with the benefit of being a small self-contained Go binary as well.

Once the download of the tooling is complete you can run a copy from s3 down to a local directory by doing something similar to this, assuming you have rights to the bucket. If you don't have rights, you'll want to set environment variables for the access key and secret key such as:

Import-Module aws.tools.common, aws.tools.SecurityToken
Set-AWSCredential -ProfileName 'MyProfileName' -Scope Global
$cred = Get-STSSessionToken -DurationInSeconds ([timespan]::FromHours(8).TotalSeconds)
@"
`$ENV:AWS_ACCESS_KEY_ID = '$($cred.AccessKeyId)'
`$ENV:AWS_SECRET_ACCESS_KEY = '$($cred.SecretAccessKey)'
`$ENV:AWS_SESSION_TOKEN  = '$($cred.SessionToken)'
"@

You can copy that string into your remote session to get the access tokens recognized by the s5cmd tool and allow you to grab files from another AWS account S3 bucket.

NOTE: To sync a full "directory" in s3, you need to leave the asteriks at the end of the key as demonstrated.

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
$ToolsDir = 'C:\tools'
$ProgressPreference = 'SilentlyContinue'
$OutZip = Join-Path $ToolsDir 's5cmd.zip'
Invoke-WebRequest -Uri 'https://github.com/peak/s5cmd/releases/download/v1.2.1/s5cmd_1.2.1_Windows-64bit.zip' -UseBasicParsing -OutFile $OutZip
# Not available on 4.0 Expand-Archive $OutZip -DestinationPath $ToolsDir
Add-Type -assembly 'system.io.compression.filesystem'
[io.compression.zipfile]::ExtractToDirectory($OutZip, $ToolsDir)
$s5cmd = Join-Path $ToolsDir 's5cmd.exe'
&$s5cmd version
$ErrorActionPreference = 'Stop'
$BucketName = ''
$KeyPrefix = 'mykeyprefix/anothersubkey/*'
$Directory = "C:\temp\adhoc-s3-sync-$(Get-Date -Format 'yyyy-MM-dd')\$KeyPrefix"
New-Item $Directory -ItemType Directory -Force -ErrorAction SilentlyContinue
$s5cmd = Join-Path $ToolsDir 's5cmd.exe'
Write-Host "This is what is going to run: `n&$s5cmd cp `"s3://$bucketname/$KeyPrefix`" $Directory"
Read-Host 'Enter to continue if this makes sense, or cancel (ctrl+c)'
&$s5cmd cp "s3://$bucketname/$KeyPrefix" $Directory
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment