Skip to content

Instantly share code, notes, and snippets.

@IAmJSD

IAmJSD/cemtp-1-spec.md

Last active July 27, 2023 20:40
Show Gist options
  • Save IAmJSD/eeedd125e9194a2a5bcbc10ef2f8ad7a to your computer and use it in GitHub Desktop.
Save IAmJSD/eeedd125e9194a2a5bcbc10ef2f8ad7a to your computer and use it in GitHub Desktop.
Download ZIP
The specification for Client-side Encrypted Mail Transfer Protocol (CEMTP)
Raw
cemtp-1-spec.md

Client-side Encrypted Mail Transfer Protocol (CEMTP) 1.0 Specification

Created By: Astrid Gealer, Web Scale Software Ltd [email protected]

Created On: 23rd July 2023

Terminology

  • A "client" refers to a mail server or user wishing to connect via the protocol to the host server.
  • A "user" refers to a user of the host mail server.

1.0. Rationale

There are many mail providers that exist today that provide end to end encryption within their own platforms. These all have similar problems when it comes to their implementation:

  • Building a email client that supports all of these different platforms encryption standards is complex. This protocol aims to fix this by making a simple specification on how to implement encrypted mail.
  • Building a encrypted mail client even for a specific service over HTTPS involves non-standardised API's. This aims to fix this by making the mail transfer protocol a simple HTTPS API.
  • To send a email that will be encrypted in a host agnostic way, you have to hunt for the users PGP key. This creates problems because many people will not have this information or the tool will be split from the mailing system, making it non-trivial enough that a lot of people will just send the email in a non end to end encrypted fashion.
  • Since there is no standardised way to do this, there becomes the problem of mail is often sent without end to end encryption when from a different platform and then encrypted. You have to trust that your mail provider is not sniffing SMTP packets during this process.

The solution to all of these problems is to build a simple HTTPS based API based on existing well implemented technologies such as PGP. This solves these problems because the mail server can deliver the PGP keys, the other mail servers delivering to this server can watch and warn for differences, and there is a standardised way of performing end to end encryption.

2.0. Upgrading a SMTP Connection

To prevent man in the middle or downgrade attacks, a client wanting to make a CEMTP connection should establish a encrypted SMTP connection first. From here, we can establish if this server supports our CEMTP version. As per RFC 821, we should run the command HELO cemtp1.spec.webscalesoftware.ltd. The response should be 250 CEMTP_SUPPORTED <url> or 250 CEMTP_SUPPORTED_WITH_PLUGGABLE <url> (meaning it supports 2.1. Pluggable Transports), where <url> is a RFC 1738 compliant URL. If we get another response, we should assume the server is not CEMTP compliant.

If the response was valid and <url> was a valid URL, we should parse it. We need to check that the protocol is https, and when we split the cemtp_versions query param by comma after making it lower case, cemtp1.0 should be present within it. If it is not, consider this a connection failure and handle this how you find appropriate.

2.1. Pluggable Transports

In countries where censorship is common, it is possible HTTPS connections to a CEMTP endpoint may be blocked but the SMTP endpoint might not be due to that meaning that citizens could not use e-mail with that provider. In that instance, If your language and mail server (it sends the with pluggable response above) supports it, you can send the command TREAT_AS_CEMTP_HTTPS and the SMTP socket will become a HTTPS one. Send all content to that socket in the same way as if you established a HTTPS connection with that URL with that URL's hostname as the host.

The URL specified should still be a valid CEMTP endpoint due to the fact that not all languages easily support this.

2.2. Forcing a upgrade via DNS

Whilst it is not possible to specify a alternate hostname to connect to over DNS, if your mail server is on another IP address, this may be a faster way for you to accept connections. If a mail server specified cemtp_spec=cemtp1.0 in a TXT record on the same domain the client is querying, your client SHOULD resolve the MX record to a IP and do a HTTPS connection with the hostname you were initially querying with that IP address as the resolved host.

This may not be desirable if you run other HTTPS content for the same host on the same IP address.

3.0. Errors

After we have established that we are running over the CEMTP 1.0 specification, any errors SHOULD have the following headers:

X-Cemtp-Version: 1.0
Content-Type: application/json; charset=utf-8
Content-Length: <length of json body>
Access-Control-Allow-Origin: *

<length of json body> should be replaced with the length of the JSON body.

Unless explicity specified by a error code, a error MUST be sent with the status code 400. However, even if out of spec, any non 2xx status code MUST be treated as a failure by clients. The JSON body should be the following:

  • error_code: The code should be one of the following:
    • ERR_NOT_SPEC_COMPLIANT: The request is not specification compliant in some way.
    • ERR_AUTHENTICATION_FAILURE: Failed to authenticate. Should use status code 403.
    • ERR_ADDITIONAL_AUTH: This is used for user Basic authentication when a token authentication flow is required. This adds the following additional fields to the JSON body:
      • token_flow_url: The user should have this URL open up for them. You can inject redirect_uri if you wish into this URL to make tracking progress easier.
      • token_fetch_url: The URL that we should navigate to after the flow is complete. Will either get a error or a 200 OK JSON body with a token inside of it. This should be used for future requests.
    • ERR_NOT_FOUND: Used when the user is not found. Should use status code 404.
    • ERR_MAIL_REJECTED: Mail rejected for a other reason.
  • description: A human readable description of the error.

4.0. Authentication

Now we are connected, all servers and users connected are required to validate their identities in some form. Note optional forms during authentication are not required, but not having them present may result in errors due to the domain being unverified.

4.1. Server > Server Authentication

To connect to another server, the server MUST inject the domain query parameter into the URL. This MUST be a RFC 1035 compliant domain. The following query parameters SHOULD also be present in the query:

  • smtp_email: The email that will be used in the VRFY command ran by the host server. This SHOULD be unique.
  • smtp_key: A 24-byte key that MUST be present in the response to the SMTP query. This SHOULD be unique.

To verify the domain actually belongs to the requesting server, if present, the host server SHOULD verify the email and key. To do so, it SHOULD make a SMTP connection to the requesting servers apparent domain and run VRFY <smtp_email> where <smtp_email> is the email listed above. The host server SHOULD check that the key is contained within the returned bytes then quit the connection. If there is a connection failure, a error of type ERR_AUTHENTICATION_FAILURE MUST be returned.

If the key is either not contained in the mail servers response or the query parameters to verify are not present, the host server SHOULD treat this suspciously.

4.2. User Authentication

A authenticating user should use the Authorization header. They can authenticate in one of 2 ways:

  • Basic: Basic username and password authentication. Should follow RFC 7617. The password MUST be SHA-512 hashed and stored as a base64 encoded hash (most hashing implementations do this by default). On your server, DO NOT store the hash in plain text, instead hash and salt your users password as the hash. Note that a error of type ERR_ADDITIONAL_AUTH COULD be returned by the server when you use this.
  • Bearer: A user token, useful if you wish to do token authentication in some form or Basic authentication has been upgraded. Note if this method of authentication is used, the client SHOULD listen for the X-New-Token header in responses to listen for cycled tokens.

The server MUST validate this. If it is unable to, a error of type ERR_AUTHENTICATION_FAILURE MUST be returned.

5.0. Responses

To be valid, all responses MUST have X-Cemtp-Version: 1.0 in their headers and SHOULD have the following other headers:

Content-Type: application/json; charset=utf-8
Content-Length: <length of json body>
Access-Control-Allow-Origin: *

<length of json body> should be replaced with the length of the JSON body. The response will depend on the request type specified in Requests

6.0. Requests

Requests MUST have the following headers and be sent in utf-8:

X-Cemtp-Supported-Specifications: cemtp1.0
Content-Type: application/json; charset=utf-8
Accept: application/json
Content-Length: <length of json body>

<length of json body> should be replaced with the length of the JSON body.

The JSON body of the request should be the following:

  • t: The type of the request. Anything 6.x. is a valid request type.
  • d: Any required request data. Specified in the specification type if required. Should be null if not specified.

All requests should be POST requests to the URL specified above.

6.1. GET_PGP_KEY

Tries to get the public PGP key of a user. The request data should be a string with the e-mail address of the user you wish to get the PGP key of. Returns either a 200 OK with a JSON string containing the shielded PGP public key, or a ERR_NOT_FOUND status code.

6.2. SEND_EMAIL

Tries to send a e-mail. The request data should be a object. The contents of this object depend on if the encrypted boolean is set to true.

If it is, the client MUST encrypt a RFC 2822 encoded email in the following way:

  • The From header must be removed from the email headers and PGP encrypted with the person who you are intending to send the email to's key seperately.
  • The Subject header must be removed from the email headers and PGP encrypted with the person who you are intending to send the email to's key seperately.
  • The remainder of the email must then be PGP encrypted.

The following MUST then be sent within the object:

  • to: The unencrypted email which this is to. Note this should JUST be the email and not include any other information.
  • public_key_used: A string of the shielded PGP public key which was used. This SHOULD be from a previous GET_PGP_KEY call.
  • encrypted_from: A shielded PGP encrypted version of the From header which was encrypted above.
  • encrypted_subject: A shielded PGP encrypted version of the Subject header which was encrypted above.
  • encrypted_remainder: A shielded PGP encrypted version of the remainder of the encoded e-mail which was encrypted above.

Note that for the encrypted body, when sent via a authenticated user, encrypted_from should be changed to from and sent un-encrypted to the server. This is to prevent forgery incidents. It MUST be re-encrypted before being sent to external servers.

If the encrypted boolean is set to false, the following MUST be set instead:

  • body: A RFC 2822 encoded email including a From and To.

The mail server SHOULD also store the domain that sent the email encrypted with the users PGP key and whether it was successfully verified for comparison later. The mail server MUST return either null with a 200 OK or a error for this.

6.3. ME

Gives information about the requestor. If this is a server requesting, returns the following:

  • domain: The domain the client is claiming to be.
  • verified: A boolean on if this could be verified.

If this is a user requesting, returns the following:

  • encrypted_name: The PGP encrypted name of the user.
  • emails: A string array of emails owned by the user. Each string is un-encrypted.
  • pgp_public_key: A string containing the shielded PGP public key.
  • encrypted_pgp_private_key: A object containing the encrypted private key. If null, the user SHOULD be prompted for the key. Contains the following data:
    • encrypted_key: A AES-256-GCM encrypted version of a shielded version of the PGP private key. The encrypted data should have a 12-byte IV at the start and then be base64 encoded.
    • strategy: The strategy used for encryption. A object containing one of the following:
      • Password Strategy: The encrypted key is encrypted with a SHA-256 of the password. The object should just be {"type": "password"}.
      • Prompt Strategy: Ask for the key. The object should be the following:
        • type: Should always be prompt.
        • sha256: Whether the password should be SHA-256 hashed. If not, it needs to be 32 bytes exactly.
  • folders: A array of the following objects:
    • folder_id: The ID of the folder. Can be either a integer or string.
    • encrypted_folder_name: A shielded PGP encrypted version of the folder name.
    • count: The number of emails in a folder.

6.4. GET_EMAILS

This route is for users only. You MUST error if a server requests this.

Gets the users emails. The request data MUST be a object. It COULD contain the following:

  • page: Defines the page of emails that the user wants. MUST be a integer if specified.
  • limit: Defines the limit of emails that the user wants for each page. MUST be a integer if specified.
  • folder_id: If you want to request emails for a specific folder, you can specify the ID here. Returns the ERR_NOT_FOUND error if not specified.
  • since: Only get emails since a specific unix millis. Can be a string or integer. If this is not present, does not time check emails.

Returns a JSON object with the following:

  • pagination: A object with information about pagination:
    • limit: The servers limit of content.
    • current_page: The current page you are on.
    • next_page: A boolean suggesting if there is a next page.
  • emails: A JSON array of objects containing emails:
    • email_id: A string or integer that is the ID of the email.
    • folder_id: A string or integer that is the ID of the folder it is in.
    • public_key_used_hash: The SHA-256 hash of the shielded PGP public key which was used.
    • encrypted_domain: A shielded PGP encrypted version of the domain which the server was told the requestor was.
    • domain_verified: Whether the domain was verified to be owned by the requestor.
    • encrypted_from: A shielded PGP encrypted version of the From header. The from header SHOULD be verified to be from the domain above.
    • encrypted_subject: A shielded PGP encrypted version of the Subject header which was encrypted with the key specified.
    • encrypted_remainder: A shielded PGP encrypted version of the remainder of the encoded e-mail which was encrypted with the key specifiec.
    • timestamp: Either a integer or a string containing the time of the email in unix millis.

6.5. DELETE_EMAIL

This route is for users only. You MUST error if a server requests this. Deletes a email. The request data should be the string or integer ID for this email. Returns the ERR_NOT_FOUND error if the email does not exist or null if successful.

6.6. MOVE_EMAILS

This route is for users only. You MUST error if a server requests this.

Moves a users emails. The request data MUST be a object. It MUST contain the following:

  • email_ids: The email ID's to move.
  • folder_id: The folder ID to move them to.

Returns the ERR_NOT_FOUND error if any of the emails or the folder do not exist, or null if successful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment