Corecrypt, a tool for secure read-only storage of large files. It is designed for flexibility and speed.

corecrypt.go

// corecrypt
//
// A command line tool for secure read-only storage of large files. It supports
// the following operations (specified by -mode):
//
// encrypt: encrypts the plaintext on STDIN and writes the ciphertext to STDOUT.
// The algorithm is AES128 in Galois counter mode, which takes a 16-byte key, a
// nonce, associated data, and a plaintext, and outputs the ciphertext. The key
// and associated data are specified by the command line options -key (required)
// and -adata (optional) respectively; the 16-byte nonce is generated randomly.
// The output is the nonce followed by the ciphertext.
//
// decrypt: decrypts the ciphertext on STDIN and writes the plaintext to STDOUT.
// If the ciphertext is inauthentic, then corecrypt outputs an error to STDERR.
// This application uses a 16-byte nonce; as such, the first 16 bytes of STDIN
// are interpreted as the nonce, and the remaining as the ciphertext. As with
// encrypt, the key and associated data are provided on the command line.
//
// verify: checks that the ciphertext on STDIN is authentic and writes nothing
// to STDOUT. If the cipehrtext is inauthentic, then it outputs an error to
// STDERR.
//
// readfile (-fn=FILE -start=I -count=N): deciphers N bytes of the ciphertext
// stored in file FILE beginning at the I-th byte. (Note that indexing starts at
// the beginning of the ciphertext, so the I-th byte of the ciphertext is the
// (I+16)-th byte of FILE.) It outputs the N bytes of plaintext to STDOUT.
// Note that this operation does not depend on the associated data used to
// encrypt the file.
//
// WARNING: the decrypt and readfile operations are not designed to prevent
// release of unverified plaintext. To do so, you must first run verify.
// However, the decrypt operation will output an error if the ciphertext just
// decrypted was inauthentic.
//
// WARNING: if the plaintext exceeds (2^32 - 2) * 16 bytes (roughly 63.99
// Gigabytes), then the program will panic. This is the maximum size of a GCM
// plaintext.
//
// This program uses an implementation of GCM optimized to support the above
// features. The code is an extension of Go's GCM implementation and can be
// found at github.com/cjpatton/sgcm. You'll need to run:
//
//  go get github.com/cjpatton/sgcm
//  go install github.com/cjpatton/sgcm
//  go build

// Design rationale
//
// I grant that AES-GCM is a controversial choice for this application, given
// how brittle it is to nonce misuse. Here were my main criteria for choosing
// the AEAD scheme:
//
//  (1) Online authenticated encryption. Plaintexts are possibly huge, and the
//  overhead is critical for many applications. Encryption and decryption must
//  only require one pass, including for computing the tag.
//
//  (2) Random access to the ciphertext. Authorized users should be able to
//  inspect the file without having to decrypt the whole thing. This is crucial
//  for the intended application of handling large core dumps or databases.
//
//  (3) A short, "simple" key. Since the key is provided by the user, it should
//  easy to work with. In particular, it should fit into any standard KEM, it
//  should play nice with key management systems, etc. This is an important
//  usability property.
//
//  (4) Local non-malleability. Short of verifying the entire ciphertext, we
//  cannot provide non-malleability because of criterion (1). However, we could
//  ensure that changes to the ciphertext affect the plaintext in a way that
//  might look fishy: for example, a block of random-looking bytes.
//
// I considered a number of encryption schemes, each having their own set of
// issues. XTS mode is designed for disk encryption, and so would seem a natural
// candidate. But it doesn't provide authentication on its own, and its key is
// already long (32 bytes). A similar construction with a short key is OCB.
// However, it doesn't satisfy criterion (2), at least not efficiently.
// (X)ChaCha20-Poly1305 would be better, but access to the ciphertext is still
// not constant time. The best schemes w.r.t (2) are CTR-mode (plus HMAC) and
// GCM; I went with GCM because it has a short key. Neither CTR nor GCM provides
// local non-malleability (4), but at the end of the day, efficient random
// access is much more important.
//
// With respect to nonce misuse: this program makes the handling of the nonce
// opaque to the user so as to minimize the risk of repeating the nonce.
package main

import (
	"crypto/aes"
	"crypto/rand"
	"flag"
	"io"
	"log"
	"os"
	"strings"

	"github.com/cjpatton/sgcm"
)

const (
	// The nonce size. The standard nonce size for AES-GCM is 12 bytes; we need
	// an extended nonce because we generate it randomly. For nonceSize=12, the
	// collision probability is below the birthday bound, but not for
	// nonceSize=16.
	nonceSize = 16

	// The size of the read buffer. This can any positive integer; it needn't
	// even be a multiple of the block size.
	chunkSize = 1024 * 1024
)

// Encrypt encrypts the plaitnext on STDIN in Galois counter mode for AES128.
// The key and associated data are provided on the command line and the nonce is
// randomly generated.
func Encrypt(out io.Writer, in io.Reader, enc sgcm.AEADEncryptor) {

	inBuf := make([]byte, chunkSize)
	outBuf := make([]byte, chunkSize)
	for {
		bytes, err := in.Read(inBuf)
		inBuf = inBuf[:bytes]
		if bytes == 0 {
			if err == nil {
				continue
			} else if err == io.EOF {
				break
			} else {
				log.Fatalf("encrypt: %s", err)
			}
		}

		// NOTE(cjpatton) Next() appends the next ciphertext fragment to its
		// first argument and returns the updated slice. I suspect this is
		// making things slow.
		//
		// This code would go much faster if Next() wrote to a buffer allocated
		// here and didn't bother with appending to dst. This would require
		// changing the AEADEncryptor interface. I wrote this interface this way
		// in order to align with the cipher.AEAD interface as much as possible.
		// This change would make the code less "pretty", but it would
		// definitely be faster.
		outBuf = enc.Next(outBuf[:0], inBuf)
		if _, err = out.Write(outBuf); err != nil {
			log.Fatalf("encrypt: %s", err)
		}

		if err != nil && err != io.EOF {
			log.Fatalf("encrypt: %s", err)
		}
	}

	outBuf = enc.Finalize(outBuf[:0])
	if _, err := out.Write(outBuf); err != nil {
		log.Fatalf("encrypt: %s", err)
	}
}

// DecryptOrVerify either verifies or decrypts and verifies the ciphertext on
// STDIN. If used to decrypt, the plaintext is written to STDOUT. The key and
// associated data are provided on the command line; the nonce is the first
// nonceBytes bytes of STDIN.
func DecryptOrVerify(out io.Writer, in io.Reader, dec sgcm.AEADDecryptor) {

	inBuf := make([]byte, chunkSize)
	outBuf := make([]byte, chunkSize)
	for {
		bytes, err := in.Read(inBuf)
		inBuf = inBuf[:bytes]
		if bytes == 0 {
			if err == nil {
				continue
			} else if err == io.EOF {
				break
			} else {
				log.Fatalf("decrypt: %s", err)
			}
		}

		// NOTE(cjpatton) See note in Encrypt().
		outBuf = dec.Next(outBuf[:0], inBuf)
		if _, err = out.Write(outBuf); err != nil {
			log.Fatalf("decrypt: %s", err)
		}

		if err != nil && err != io.EOF {
			log.Fatalf("decrypt: %s", err)
		}
	}

	outBuf, err := dec.Finalize(outBuf[:0])
	if err != nil {
		log.Fatalf("decrypt: %s", err) // inauthentic
	}
	if _, err := out.Write(outBuf); err != nil {
		log.Fatalf("decrypt: %s", err)
	}
}

func main() {

	// Process command line arguments.
	mode := flag.String("mode", "encrypt", "encrypt, decrypt, verify, readfile")
	ad := flag.String("adata", "", "associated data")
	key := flag.String("key", "", "AES128 key")
	fn := flag.String("fn", "", "name of the file to read (-mode=readfile)")
	start := flag.Int("start", 0, "index of first byte of ciphertext (-mode=readfile)")
	count := flag.Int("count", 0, "number of bytes (-mode=readfile)")
	flag.Parse()

	if len(*key) == 0 {
		log.Fatalln("missing key")
	} else if len(*key) != 16 {
		log.Fatalf("key length is %d, expected %d", len(*key), 16)
	}

	aes, err := aes.NewCipher([]byte(*key))
	if err != nil {
		log.Fatal(err)
	}

	nonce := make([]byte, nonceSize)

	if strings.Compare(*mode, "encrypt") == 0 {
		// Choose a random, 16 byte nonce. This is done to ensure that user of
		// the application does not accidently reuse a nonce. (This would be
		// catestrophic for AES-GCM!)
		if _, err := rand.Read(nonce); err != nil {
			log.Fatal(err)
		}

		// Set up the streaming GCM encryption context.
		enc, _, err := sgcm.NewStreamingGCMWithNonceSize(aes, nonceSize)
		if err != nil {
			log.Fatal(err)
		}
		enc.Initialize(nonce, []byte(*ad))

		// The first nonceSize bytes of the ciphertext are the nonce.
		if _, err = os.Stdout.Write(nonce); err != nil {
			log.Fatal(err)
		}

		Encrypt(os.Stdout, os.Stdin, enc)

	} else if strings.Compare(*mode, "decrypt") == 0 || strings.Compare(*mode, "verify") == 0 {
		// Read the nonce from the beginning of STDIN.
		if bytes, err := os.Stdin.Read(nonce); err == io.EOF || bytes != nonceSize {
			log.Fatalf("decrypt/verify: nonce: read %d bytes: expected at least %d", bytes, nonceSize)
		} else if err != nil {
			log.Fatal(err)
			log.Fatalf("decrypt/verify: %s", err)
		}

		// Set up the streaming GCM decryption context.
		_, dec, err := sgcm.NewStreamingGCMWithNonceSize(aes, nonceSize)
		if err != nil {
			log.Fatal(err)
		}

		if strings.Compare(*mode, "decrypt") == 0 {
			dec.Initialize(nonce, []byte(*ad))
		} else {
			dec.InitializeVerifyOnly(nonce, []byte(*ad))
		}

		DecryptOrVerify(os.Stdout, os.Stdin, dec)

	} else if strings.Compare(*mode, "readfile") == 0 {
		if len(*fn) == 0 {
			log.Fatal("readfile: missing file name")
		}

		// Set up the context for accessing the ciphertext.
		st, err := sgcm.NewGCMWithNonceSize(aes, nonceSize)
		if err != nil {
			log.Fatal(err)
		}

		fd, err := os.Open(*fn)
		if err != nil {
			log.Fatalf("readfile: %s", err)
		}
		defer fd.Close()

		// Check that the query does not exceed the length of the ciphertext
		// (excluding the tag).
		if fi, err := fd.Stat(); err != nil {
			fd.Close()
			log.Fatalf("readfile: %s", err)
		} else if int64(*start+*count+nonceSize) > fi.Size()-int64(st.Overhead()) {
			fd.Close()
			log.Fatalf("readfile: query out of range")
		}

		// Read the nonce from the beginning of the file.
		if bytes, err := fd.Read(nonce); err == io.EOF || bytes != nonceSize {
			fd.Close()
			log.Fatalf("read nonce: read %d bytes: expected at least %d", bytes, nonceSize)
		} else if err != nil {
			fd.Close()
			log.Fatalf("readfile: %s", err)
		}

		// Seek to the specified starting byte.
		if _, err = fd.Seek(int64(*start+nonceSize), io.SeekStart); err != nil {
			fd.Close()
			log.Fatalf("readfile: %s", err)
		}

		// Read the specified number of bytes.
		buf := make([]byte, *count)
		if bytes, err := fd.Read(buf); err == io.EOF || bytes != *count {
			fd.Close()
			log.Fatalf("readfile: read %d bytes: expected at least %d", bytes, *count)
		}

		// Decipher the bytes in the buffer.
		st.(sgcm.RandomAccessStream).XORKeyStream(buf, buf, nonce, *start)

		// Write the plaintext to Stdout.
		if _, err = os.Stdout.Write(buf); err != nil {
			fd.Close()
			log.Fatalf("readfile: %s", err)
		}
	}
}

racecar.sh

#!/bin/bash
#
# Creates a 22MB file for testing corecrypt. Try the following:
#  ./racecar.sh | ./corecrypt -key "$KEY" > fella
#  ./corecrypt -key "$KEY" -mode=readfile -fn=fella -start=13424232 -count=1000
#
# Or try some benchmarking:
#  time ./racecar.sh | cat | > /dev/null
#  time ./racecar.sh | ./corecrypt -key "$KEY" > /dev/null
#  time ./racecar.sh | ./corecrypt -key "$KEY" | ./corecrypt -key "$KEY" -mode=verify
STRING="Jerry was a racecar driver."
N=800000
for i in `seq 0 $N` ; do echo -n "$STRING "; done

test.sh

#!/bin/bash
#
# Run tests for corcrypt:
#  go build && ./test.sh
#
# Expected output:
#  Hello, and welcome to corecrypt!
#  Hello, corecrypt!
#  2018/01/29 12:28:43 decrypt: cipher: message authentication failed
TESTFN=test_file
KEY="1234123412341234"
AD="This is some associated data"
GREETING="Hello, and welcome to corecrypt!"

# Test encrypt/decrypt.
echo "$GREETING" | ./corecrypt -key "$KEY" -adata "$AD" | \
                   ./corecrypt -key "$KEY" -adata "$AD" -mode=decrypt

# Test file reading.
echo "$GREETING" | ./corecrypt -key "$KEY" -adata "$AD" > $TESTFN
./corecrypt -key "$KEY" -mode=readfile -fn=$TESTFN -start=0 -count=7
./corecrypt -key "$KEY" -mode=readfile -fn=$TESTFN -start=22 -count=10
echo

# Test verification.
echo "$GREETING" | ./corecrypt -key "$KEY" -adata "$AD" | \
                   ./corecrypt -key "$KEY" -adata "Attack!" -mode=verify
rm -f $TESTFN