Last active
September 21, 2023 13:48
-
-
Save seanbamforth/9388507 to your computer and use it in GitHub Desktop.
Calculate Amazon eTag hash for named file. Powershell
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
#Calculates an eTag for a local file that should match the S3 eTag of the uploaded file. | |
$md5 = new-object -TypeName System.Security.Cryptography.MD5CryptoServiceProvider | |
$blocksize = (1024*1024*5) | |
$startblocks = (1024*1024*16) | |
function AmazonEtagHashForFile($filename) { | |
$lines = 0 | |
[byte[]] $binHash = @() | |
$reader = [System.IO.File]::Open($filename,"OPEN","READ") | |
if ((Get-Item $filename).length -gt $startblocks) { | |
$buf = new-object byte[] $blocksize | |
while (($read_len = $reader.Read($buf,0,$buf.length)) -ne 0){ | |
$lines += 1 | |
$binHash += $md5.ComputeHash($buf,0,$read_len) | |
} | |
$binHash=$md5.ComputeHash( $binHash ) | |
} | |
else { | |
$lines = 1 | |
$binHash += $md5.ComputeHash($reader) | |
} | |
$reader.Close() | |
$hash = [System.BitConverter]::ToString( $binHash ) | |
$hash = $hash.Replace("-","").ToLower() | |
if ($lines -gt 1) { | |
$hash = $hash + "-$lines" | |
} | |
return $hash | |
} |
Thank you. This helped a lot.
I'm using this code to calculate the content-MD5 header.
So I had to convert the md5 to base64 (and remove the $hash = $hash + "-$lines
part)
$bytes = New-Object -TypeName byte[] -ArgumentList ($hash.Length / 2)
for ($j = 0; $j -lt $hash.Length; $j += 2) {
$bytes[$j / 2] = [System.Convert]::ToByte($hash.Substring($j, 2), 16)
}
$EncodedText =[System.Convert]::ToBase64String($bytes)
return $EncodedText
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Thank you for a great script.
The actual value of eTag is different from the calculated value of eTag when I tested this script in case that file size is just 16MB because the multi-part upload into S3 works when file size is greater than "or equl to" 16MB.
I think that in line 14
if ((Get-Item $filename).length -gt $startblocks) {
should beif ((Get-Item $filename).length -ge $startblocks) {
.