Quick shell script to decrypt client-side encrypted file using QNAP HBS *Sync* jobs. Only work with version 2 files - Doesn't work with QNAP HBS Backup jobs!

qnap-clientenc-decrypt.sh

#!/bin/sh

set -ue

[ "$#" -lt 2 ] && { echo "Syntax: $0 <filename> <key>"; exit 1; } 

filename="$1"
key="$2"

# Compose the key by repeating the user input until we have 32 characters (64 hex digits) 
openssl_key=""
while [ "$(printf '%s' "${openssl_key}" | wc -m)" -lt 64 ]; do
  openssl_key=${openssl_key}$(printf '%s' "${key}" | od -An -tx1 | tr -d '\ \n')
done
openssl_key=$(printf '%s' "${openssl_key}" | cut -c1-64)

# Determine if the cleartext has been compressed
compressed="$(dd if="${filename}" bs=1 skip=9 count=1 2>/dev/null | od -An -tx1 | tr -d '\ \n')"
[ "${compressed}" -ne 0 ] && echo "Compressed, use bzip2 to decompress" >&2

# Decipher the header
header="$(dd if="${filename}" bs=1 skip=16 count=64 2>/dev/null | \
  openssl aes-256-ecb -d -K "${openssl_key}" -nopad | od -An -tx1 | tr -d '\ \n')"

# Extract key and iv from the header
ckey="$(printf '%s' "${header}" | cut -c17-80)"
salt="$(printf '%s' "${header}" | cut -c81-112)"

# Decipher the file
dd if="${filename}" bs=1 skip=80  2>/dev/null | openssl aes-256-cbc -d -K "${ckey}" -iv "${salt}"

I mean 11.5, yes Big Sur.
Since you mentioned passphrase, I am thinking that there could be a problem with the creation of openssl_key here.
Is your passphrase longer than 32 characters, by any chance?

Okay, so there is one minor version difference in the OS but as you said that shouldn't make the difference. I checked the settings and I deactivated QDedup. Maybe this is what you meant with compression because I couldn't find any other compression option? Password is for testing just 123456 so it is not exceeding 32 char limit.

I copied your script in a decrypt.sh and did the following with a hello_world.txt with the Content "Hello World!" which is encrypted with 123456 passphrase.

cat hello_world.txt | base64
SGVsbG8gV29ybGQh

# Then Qnap does the Backup with encryption Job

./decrypt.sh hello_world.txt 123456
bad decrypt
4651892396:error:06FFF064:digital envelope routines:CRYPTO_internal:bad decrypt:/System/Volumes/Data/SWE/macOS/BuildRoots/e90674e518/Library/Caches/com.apple.xbs/Sources/libressl/libressl-56.60.2/libressl-2.8/crypto/evp/evp_enc.c:521:

# the encrypted file
cat hello_world.txt 
Kʔr^?1?W?7?b?s??z[)H)Cޠ??????{????}?m?~=??????}{u?A?1V???t??0Sż(?????>g?????

# the encrypted file content base64 encoded
cat hello_world.txt | base64
S8qUcl6DHDEBAAAAAAAAANJX8jemYtdz9MF6WylIKUPeoBOCngSQHT/C2nsF5avdyfN9wG3hfj2AhoG3jsl9e3WuQe8xVpm36nTv9DBTxbwo//YEv6EDnj5npIqrlsMK

I added the base64 stream of the binary encrypted files and also of the source file so to make sure it is reproducable on a binary accurate level on other machines.

Thanks for any hints.

It's very hard to tell what's going on...
I need to do some testing. One thing that looks different is that I am doing a sync job, with encryption, while you are doing a backup.
The "external" header seems to match. But there is an additional header, from which I extract key and iv, which may be different...

Will look into it later :)

@ceelian do you mind trying https://github.com/alexkazik/qnap-decrypt ?

I did some testing.
I think that the encryption key used by QNAP HSB Backup is derived from the user password with a different algorithm.
From Sync jobs, the password used to decrypt the header is a 64 bytes long concatenation of the user password.

Maybe the backup job concatenates something to the user password... but honestly is hard to tell!

@dguerry thank you, I was 2 weeks AFK. Will try soon and let you know. Thank you a lot for the investigations.