AWS SDK client-side encryption testing. Also see https://gist.github.com/stefansundin/428c1f3dfcbddf58202477c48d4c8fef.

1.README.md

Documentation:

• https://docs.aws.amazon.com/general/latest/gr/aws_sdk_cryptography.html
• https://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html

Ruby:

• https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/s3-example-client-encryption.html
• https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3.html
• https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/S3/Encryption.html

Go:

• https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/s3-examples-encryption.html
• https://docs.aws.amazon.com/sdk-for-go/api/service/s3/
• https://docs.aws.amazon.com/sdk-for-go/api/service/s3/s3crypto/

How S3 client-side encryption works

Objects are encrypted in the S3 client and then uploaded to S3. The objects can additionally be encrypted on the server side as well, if desired. If not encrypted on the server side, the S3 console will say "Encryption: None". The S3 console doesn't care whether or not the object used client-side encryption, you can still download the ciphertext.

The aws-cli and S3 Management Console does not support client-side encryption. They will just display the ciphertext.

Client-side encryption uses envelope encryption, meaning that the client generates an data key that is used to encrypt the object contents. The data key is then encrypted using the encryption key configured (RSA key or KMS key). Not all SDKs support all ciphers (see below). The encrypted data key is stored alongside the object as metadata. The result of this is that all objects are encrypted with a unique data key.

AWS also has an "Encryption SDK", which is separate and not interoperable with S3 client-side encryption. It is also not available for all languages (Ruby and Go are NOT supported at the moment). See https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/introduction.html

Cloud compatibility

Google Cloud has an implementation of the S3 API. The "Interoperability API" has to be enabled.

When using RSA, there is no dependency on AWS KMS. I doubt that Google Cloud KMS has any compability with AWS KMS. It is perfectly possible to store objects in Google Cloud and use AWS KMS to encrypt the data key. The example code here supports uploading the objects to both AWS S3 and Google Cloud Storage, when using AWS KMS to encrypt the data key.

Azure does not seem to support the S3 API. I found several examples where people use proxies and are able to use the AWS SDK that way. I did not try this.

Google Cloud

Create a Storage Bucket in Google Cloud. Remember to enable the "Interoperability API". Generate an access key.

Set up a s3test_gcloud profile in ~/.aws/credentials. To use KMS, also set up an s3test_aws profile with a user that has access to the KMS key (and optionally to an S3 bucket).

[s3test_gcloud]
region=does-not-matter
aws_access_key_id=ABC
aws_secret_access_key=XYZ

[s3test_aws]
region=us-west-2
aws_access_key_id=ABC
aws_secret_access_key=XYZ

Supported ciphers

https://docs.aws.amazon.com/general/latest/gr/aws_sdk_cryptography.html

The Ruby SDK can use:

• AES/ECB key wrap (not recommended)
• RSA key wrap
• AES/CBC content encryption (Encryption Only mode)
• AES/GCM content encryption (Strict Authentication mode)
• AES/CTR content encryption (Authenticated mode only used for decrypting in range GETs)

The Go SDK can use:

• AES/CBC content encryption (Encryption Only mode)
• AES/GCM content encryption (Strict Authentication mode)

The Ruby SDK currently is not configurable to encrypt with AES/GCM, but it can decrypt it. When using KMS as a cipher provider, it will always use AES/CBC to encrypt.

Newer ciphers, such as AES/CBC and AES/GCM, use KMS to encrypt the data key.

There seem to be some support to extend the Go SDK to support additional ciphers and additional KMS implementations, but I did not experiment with this.

RSA key

openssl genrsa -out private_key.pem 2048
openssl rsa -in private_key.pem -outform PEM -pubout -out public_key.pem

Conclusion

This investigation was just a short spike to understand how the AWS SDKs implement client-side encryption.

If there is a desire to store encrypted data in places other than S3, then the AWS SDK will be difficult to use.

In my opinion, this can be considered on a case-by-case basis. For data that will be stored in other places, I think Vault should be considered instead. Vault has SDKs for a lot of languages, and I suspect most of them have good feature parity.

download-kms.go

package main

import (
	"bytes"
	"fmt"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/endpoints"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/kms"
	"github.com/aws/aws-sdk-go/service/s3"
	"github.com/aws/aws-sdk-go/service/s3/s3crypto"
)

func check(e error) {
	if e != nil {
		panic(e)
	}
}

func main() {
	s3Bucket := "s3test-bucket"
	s3Key := "secret-aes-gcm.txt"

	kmsSession := session.Must(session.NewSessionWithOptions(session.Options{
		Profile:           "s3test_aws",
		SharedConfigState: session.SharedConfigEnable,
	}))
	kmsClient := kms.New(kmsSession)
	handler := s3crypto.NewKMSWrapEntry(kmsClient)

	// To use AWS:
	// s3Session := session.Must(session.NewSessionWithOptions(session.Options{
	// 	Profile:           "s3test_aws",
	// 	SharedConfigState: session.SharedConfigEnable,
	// }))

	// To use GCloud:
	customResolver := func(service, region string, optFns ...func(*endpoints.Options)) (endpoints.ResolvedEndpoint, error) {
		if service == endpoints.S3ServiceID {
			return endpoints.ResolvedEndpoint{
				URL: "https://storage.googleapis.com",
			}, nil
		}
		return endpoints.DefaultResolver().EndpointFor(service, region, optFns...)
	}
	s3Session := session.Must(session.NewSessionWithOptions(session.Options{
		Profile:           "s3test_gcloud",
		SharedConfigState: session.SharedConfigEnable,
		Config: aws.Config{
			EndpointResolver: endpoints.ResolverFunc(customResolver),
		},
	}))

	s3DecryptionClient := s3crypto.NewDecryptionClient(s3Session, func(client *s3crypto.DecryptionClient) {
		client.WrapRegistry[s3crypto.KMSWrap] = handler
	})
	respGet, err := s3DecryptionClient.GetObject(&s3.GetObjectInput{
		Bucket: aws.String(s3Bucket),
		Key:    aws.String(s3Key),
	})
	check(err)
	buf := new(bytes.Buffer)
	buf.ReadFrom(respGet.Body)
	fmt.Println(buf.String())
}

download-kms.rb

#!/usr/bin/env ruby
require 'aws-sdk-s3'


s3_bucket = 's3test-bucket'
s3_key = 'secret-aes-cbc.txt'
# s3_key = 'secret-aes-gcm.txt'
kms_key_id = '01234567-0123-0123-0123-01234567890a'

kms_client = Aws::KMS::Client.new(
  profile: 's3test_aws',
  region: 'us-west-2',
)
s3_client = Aws::S3::Encryption::Client.new(
  # To use AWS:
  profile: 's3test_aws',

  # To use GCloud:
  # profile: 's3test_gcloud',
  # endpoint: 'https://storage.googleapis.com',
  # region: 'does-not-matter',

  # Common options:
  kms_client: kms_client,
  kms_key_id: kms_key_id,
)

resp = s3_client.get_object(
  bucket: s3_bucket,
  key: s3_key,
)
puts resp.body.read

download-rsa.rb

#!/usr/bin/env ruby
require 'aws-sdk-s3'
require 'openssl'

s3_bucket = 's3test-bucket'
s3_key = 'secret-rsa.txt'
private_key = OpenSSL::PKey::RSA.new(File.read('private_key.pem'))


s3_client = Aws::S3::Encryption::Client.new(
  # To use AWS:
  profile: 's3test_aws',

  # To use GCloud:
  # profile: 's3test_gcloud',
  # endpoint: 'https://storage.googleapis.com',
  # region: 'does-not-matter',

  # Common options:
  encryption_key: private_key,
)

resp = s3_client.get_object(
  bucket: s3_bucket,
  key: s3_key,
)
puts resp.body.read

Gemfile

source "https://rubygems.org"

gem "aws-sdk-s3"

Gemfile.lock

GEM
  remote: https://rubygems.org/
  specs:
    aws-eventstream (1.0.3)
    aws-partitions (1.295.0)
    aws-sdk-core (3.93.0)
      aws-eventstream (~> 1.0, >= 1.0.2)
      aws-partitions (~> 1, >= 1.239.0)
      aws-sigv4 (~> 1.1)
      jmespath (~> 1.0)
    aws-sdk-kms (1.30.0)
      aws-sdk-core (~> 3, >= 3.71.0)
      aws-sigv4 (~> 1.1)
    aws-sdk-s3 (1.61.2)
      aws-sdk-core (~> 3, >= 3.83.0)
      aws-sdk-kms (~> 1)
      aws-sigv4 (~> 1.1)
    aws-sigv4 (1.1.1)
      aws-eventstream (~> 1.0, >= 1.0.2)
    jmespath (1.4.0)

PLATFORMS
  ruby

DEPENDENCIES
  aws-sdk-s3

BUNDLED WITH
   2.1.4

go.mod

module s3test

go 1.14

require github.com/aws/aws-sdk-go v1.30.6

go.sum

github.com/aws/aws-sdk-go v1.30.6 h1:GuWgFWWR9CF8mO9SM6N9oZt0vM0yzgPCMDDZOEQb8l4=
github.com/aws/aws-sdk-go v1.30.6/go.mod h1:5zCpMtNQVjRREroY7sYe8lOMRSxkhG6MZveU8YkpAk0=
github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
github.com/go-sql-driver/mysql v1.5.0/go.mod h1:DCzpHaOWr8IXmIStZouvnhqoel9Qv2LBy8hT2VhHyBg=
github.com/jmespath/go-jmespath v0.3.0 h1:OS12ieG61fsCg5+qLJ+SsW9NicxNkg3b25OyT2yCeUc=
github.com/jmespath/go-jmespath v0.3.0/go.mod h1:9QtRXoHjLGCJ5IBSaohpXITPlowMeeYCZ7fLUTSywik=
github.com/pkg/errors v0.9.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
github.com/stretchr/testify v1.5.1/go.mod h1:5W2xD1RspED5o8YsWQXVCued0rvSQ+mT+I5cxcmMvtA=
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
golang.org/x/net v0.0.0-20200202094626-16171245cfb2/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=

upload-aes-cbc.rb

#!/usr/bin/env ruby
require 'aws-sdk-s3'


s3_bucket = 's3test-bucket'
s3_key = 'secret-aes-cbc.txt'
body = 'this is a very secret string'
kms_key_id = '01234567-0123-0123-0123-01234567890a'

kms_client = Aws::KMS::Client.new(
  profile: 's3test_aws',
  region: 'us-west-2',
)
s3_client = Aws::S3::Encryption::Client.new(
  # To use AWS:
  profile: 's3test_aws',

  # To use GCloud:
  # profile: 's3test_gcloud',
  # endpoint: 'https://storage.googleapis.com',
  # region: 'does-not-matter',

  # Common options:
  kms_client: kms_client,
  kms_key_id: kms_key_id,
)

resp = s3_client.put_object(
  bucket: s3_bucket,
  key: s3_key,
  body: body,
)
puts resp.inspect

sleep(1)
puts
puts "object metadata:"
resp = s3_client.head_object(bucket: s3_bucket, key: s3_key)
puts resp.metadata.inspect

upload-aes-gcm.go

package main

import (
	"bytes"
	"fmt"
	"time"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/endpoints"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/kms"
	"github.com/aws/aws-sdk-go/service/s3"
	"github.com/aws/aws-sdk-go/service/s3/s3crypto"
)

func check(e error) {
	if e != nil {
		panic(e)
	}
}

func main() {
	s3Bucket := "s3test-bucket"
	s3Key := "secret-aes-gcm.txt"
	body := "this is a very secret string"
	kmsKeyID := "01234567-0123-0123-0123-01234567890a"

	kmsSession := session.Must(session.NewSessionWithOptions(session.Options{
		Profile:           "s3test_aws",
		SharedConfigState: session.SharedConfigEnable,
	}))
	kmsClient := kms.New(kmsSession)
	handler := s3crypto.NewKMSKeyGenerator(kmsClient, kmsKeyID)

	// To use AWS:
	// s3Session := session.Must(session.NewSessionWithOptions(session.Options{
	// 	Profile:           "s3test_aws",
	// 	SharedConfigState: session.SharedConfigEnable,
	// }))

	// To use GCloud:
	customResolver := func(service, region string, optFns ...func(*endpoints.Options)) (endpoints.ResolvedEndpoint, error) {
		if service == endpoints.S3ServiceID {
			return endpoints.ResolvedEndpoint{
				URL: "https://storage.googleapis.com",
			}, nil
		}
		return endpoints.DefaultResolver().EndpointFor(service, region, optFns...)
	}
	s3Session := session.Must(session.NewSessionWithOptions(session.Options{
		Profile:           "s3test_gcloud",
		SharedConfigState: session.SharedConfigEnable,
		Config: aws.Config{
			EndpointResolver: endpoints.ResolverFunc(customResolver),
		},
	}))

	s3EncryptionClient := s3crypto.NewEncryptionClient(s3Session, s3crypto.AESGCMContentCipherBuilder(handler))
	respPut, err := s3EncryptionClient.PutObject(&s3.PutObjectInput{
		Bucket: aws.String(s3Bucket),
		Key:    aws.String(s3Key),
		Body:   bytes.NewReader([]byte(body)),
	})
	check(err)
	fmt.Println(respPut)

	time.Sleep(1 * time.Second)
	s3Client := s3.New(s3Session)
	respHead, err := s3Client.HeadObject(&s3.HeadObjectInput{
		Bucket: aws.String(s3Bucket),
		Key:    aws.String(s3Key),
	})
	fmt.Println()
	fmt.Println("object metadata:")
	for k, v := range respHead.Metadata {
		fmt.Printf("%v: %v\n", k, *v)
	}
}

upload-rsa.rb

#!/usr/bin/env ruby
require 'aws-sdk-s3'
require 'openssl'

s3_bucket = 's3test-bucket'
s3_key = 'secret-rsa.txt'
body = 'this is a very secret string'
public_key = OpenSSL::PKey::RSA.new(File.read('public_key.pem'))

s3_client = Aws::S3::Encryption::Client.new(
  # To use AWS:
  profile: 's3test_aws',

  # To use GCloud:
  # profile: 's3test_gcloud',
  # endpoint: 'https://storage.googleapis.com',
  # region: 'does-not-matter',

  # Common options:
  encryption_key: public_key,
)

resp = s3_client.put_object(
  bucket: s3_bucket,
  key: s3_key,
  body: body,
)
puts resp.inspect

sleep(1)
puts
puts "object metadata:"
resp = s3_client.head_object(bucket: s3_bucket, key: s3_key)
puts resp.metadata.inspect