Outline of a modern CLI-based email system

cli-email-system.md

Overview

Getting user-friendly email to work on the terminal is difficult. There's a lot of information out there, but a lot of it is ill-suited or overkill for the average user, who simply wants to check Gmail or some other web mail account in their terminal. This post is meant to be an opinionated overview of what you need to get such a system in place.

Modern CLI email is about synchronization

In the old days, it was expected that if you were setting up anything in the terminal involving email, that you were also setting up your computer to send and receive mail directly (e.g., setting up an SMTP server). In a modern setup, we assume that your email provider (e.g., Gmail) does the sending and receiving for you.

The task of setting up email in the terminal is therefore about synchronization. You will maintain copies of the emails that are hosted on your email provider's servers. When you read an email, you'll actually be reading the local copy. When someone sends you an email, it will be received by your email provider and your computer will download the message to your local copies so that you can read it. When you delete an email, you'll actually be deleting your local copy, and your computer will synchronize the deletion to your email provider. Any changes to emails are done to your local copies first and then synchronized to the remote server, while the remote server remains the source of truth.

mbsync is a mail client that synchronizes remote mailboxes using the standard IMAP protocol. lieer is another client that we'll use specifically for Gmail or Google Mail-based domains (e.g., work email; more on Lieer later).

Modern CLI email is about saving each local copy as a file

Our setup uses the Maildir format of saving emails locally.

The two historical standards are Maildir and mbox. The former saves each email as its own file whereas the latter concatenates all your emails to a single file.

As you can imagine, the Maildir format is much more amenable to indexing and saving thousands or even millions of messages, which is not uncommon for many users today. All the tools we use in our setup assume Maildir format. You don't need to know all the specifics, but knowing that each message is saved as its own file will help you understand how certain email operations are setup later.

Modern CLI email is about indexing

Most people use email as an informal database, a place where you can save and search for appointments, invoices, or anything else that you might need to remember. Indexing therefore becomes crucial so that full-text searches can be as fast as possible.

How do local changes propagate to a remote server?

IMAP	Maildir	notmuch
\Seen	Seen	unread (for messages without S flag)
\Answered	Replied	replied
\Flagged	Flagged	flagged
\Deleted	Trashed
\Draft	Draft	draft
\Recent	move to tmp
	Passed	passed

Declarative, stateful transactions

• Operations like reading or replying to an email add/remove flags from a given message.
• Each part of the email system synchronizes a message's flags with whichever parts that it communicates with. For example, if mbsync detects a new message in the Maildir that has a D flag, it may issue an IMAP command to flag a message as \Draft. notmuch will add the draft tag to the same message within its internal database.
• Flags don't necessarily align perfectly. For example, there's no IMAP flag to represent that a message has been passed (i.e., forwarded). Similarly, notmuch doesn't sync the T Maildir flag to indicate that a message needs to be deleted.

Most email providers → IMAP

Most email providers offer a straightforward IMAP interface.

Mailboxes

Folders are generally exposed as IMAP mailboxes on a 1:1 basis, including the special INBOX mailbox.

Folder	IMAP mailbox
Inbox	INBOX
*	*

If a message appears in a certain mailbox, it's assumed that it only exists in the mailbox's corresponding folder on the server (i.e., there are no duplicates on the server).

Flags

Each message's state is mapped to the standard IMAP flags. How each server saves the message state is an implementation detail, but the mapping below gives a general idea of how it works for most servers:

State	IMAP flag
Seen	\Seen
Replied to	\Answered
Urgent	\Flagged
Trash	\Deleted
Draft	\Draft

The IMAP standard also defines the flag \Recent, which is supposed to correspond to messages that have "'recently' arrived" in a mailbox. In practice, the definition is vague and the flag is often ignored by IMAP servers and clients alike.

Note that the semantic information conveyed by a message's flags may seem to overlap with the meaning of the mailbox it's placed within, yet the two factors are independent of each other. For example, a message may exist in the account's "Trash" mailbox and also have the \Deleted flag, which would mean that the server has moved the message into the "Trash" folder and has slated it for permanent deletion. A message in the "Trash" folder that doesn't have the \Deleted flag might mean that the message is archived: no longer in the inbox and also not slated for deletion.

Gmail/Google mail → IMAP

Gmail/Google's implementation of IMAP handles mail differently from standard IMAP.

Mailboxes

Internally, Gmail has no conception of folders. All messages have labels attached to them, similar to notmuch's tags, and the combination of labels determines which "folders" a message appears in, which in turn are exposed as IMAP mailboxes.

Gmail labels	Gmail folder	IMAP mailbox
Not SPAM or TRASH	All Mail	[Gmail]/All Mail
DRAFT	Drafts	[Gmail]/Drafts
IMPORTANT	Important	[Gmail]/Important
INBOX	Inbox	INBOX
SENT	Sent Mail	[Gmail]/Sent Mail
SPAM	Spam	[Gmail]/Spam
STARRED	Starred	[Gmail]/Starred
TRASH	Trash	[Gmail]/Trash
User-defined label	N/A	*

Note that because the same message can have multiple labels, it can appear in multiple mailboxes. A draft message will appear in [Gmail]/Draft as well as [Gmail]/All Mail. Similarly, a starred sent message will appear in [Gmail]/Starred and [Gmail]/Sent Mail. This behavior contrasts with how most email providers use IMAP, where it's generally assumed that a message only exists within one mailbox and one folder at the same time.

There are also standard Gmail labels that aren't exposed as IMAP mailboxes, such as CATEGORY_PERSONAL, CATEGORY_SOCIAL, and CATEGORY_PROMOTIONS, which correspond to the "Personal," "Social," and "Promotions" tabs in the Gmail interface.

Flags

Gmail uses standard IMAP flags except for \Recent and appears to follow the expected behavior of most email providers (see above). It additionally defines the following non-standard IMAP flags, which may or may not be used by your client:

• $Forwarded
• $Junk
• $NotJunk
• $NotPhishing
• $Phishing
• JunkRecorded
• NotJunk

IMAP → Maildir

A client like mbsync or offlineimap consumes an IMAP interface, downloading a remote server's messages and saving them locally in Maildir format (or another format, such as mbox, but we'll assume that you're using Maildir).

Directory structure

Each user has their own Maildir folder, typically ~/Maildir, ~/.mail, or something similar. The structure of a Maildir folder generally looks like this:

Maildir/
  [Account]/
    [Mailbox]/
      tmp/
      new/
      cur/

[Account] and [Mailbox] are optional but common and correspond to user-defined accounts and the mailboxes that correspond to them, respectively. For example, you might have a personal account using a generic IMAP provider and a work account using Google mail, in which case your directory structure might look like:

Maildir/
  personal/
    Inbox/
      tmp/
      new/
      cur/
    Sent
      ...
  work/
    Inbox/
      ...
    [Gmail]/
      All Mail/
        ...
      Spam/
        ...
    ...

The exact mapping of accounts and IMAP mailboxes to Maildir subdirectories depends on how you configure your IMAP client. For example, mbsync maps IMAP mailbox names into hierarchical directories by default, using / as the delimeter in the mailbox name to indicate levels of hierarchy. mbsync can also be configured to use a different character as the delimeter or to flatten all hierarchies using a delimeter (e.g., [Gmail]/All Mail → [Gmail].All Mail).

tmp, new, and cur are mandatory parts of the Maildir format:

• tmp contians mail that's waiting to be delivered.
• new contians mail that's been delievered but not seen yet by a mail application.
• cur contains mail that's already been seen by a mail application.

IMAP concept	Maildir concept
Account	(Optional) Account subdirectory
Mailbox	(Optional) Mailbox subdirectory
Message	File