My email setup with Emacs, Rmail, msmtp/mpop, mairix, and then some more

[cadadr]

Introduction

A lot of "my email setup" posts start with telling how the author deals with thousands or millions of messages with probably tens or hundreds of interactions each day.

This blog post instead is about how I deal with a humane level of email messaging, with a local-first approach, using a smallish variety of software, which you can just set up and forget about.

Let me start with a list of software, then go on with describe the setup.

• GNU Emacs
• Rmail
• msmtp and mpop
• procmail
• mairix
• pass(1)
• gnupg
• Python 3
• K-9 Mail app for Android

Of this list, the following can easily be factored out if you want: Python 3, mairix, pass, procmail, gnupg, and K-9 Mail. Furthermore, it should actually be possible to replicate this whole setup with only ever using GNU Emacs and Rmail (which comes with Emacs itself), if your system has movemail handy, or if your Emacs includes it (which was common up until recently, but is a workflow deprecated by Emacs developers). Then, you might ask, why do you bother? The answer is simple: if I use external programs to fetch and send mail, then I can use multiple email clients with the same setup. E.g., I sometimes use the mutt email client with this setup with no modifications or fragile repetitions. I'll in fact talk a bit about how you can use mutt and another tool to just totally factor Emacs out of this setup while retaining a similar workflow.

With that unrefined intro done, let's get to talking about what, why and how.

The general workflow here is this: view mail on mobile using IMAP and K-9 Mail, and archive/respond on the computer using Emacs et al. That's "what". The "why" is two fold: (i) I can't pull out a laptop out of my pocket just to check out a notification with a flask in my other hand or when holding onto a bar on public transport, and (ii) typing and searching on mobile is the abject rock bottom of human-computer interaction: slow, imprecise, and unforgiving (when's the last time you used a mobile app with great undo? Yes, me too, never). In that light, this is my e-mail algorithm:

1. I check email manually on mobile, or a notification arrives
2. There's new interesting mail
3. I read the mail, and decide to reply...
1. Email is synced to computer, deleted from the server
2. I do whatever I meant to do with email
4. ... or I don't do anything and mail piles up for a while on the remote server
1. Every few days (no fixed schedule or anything) I sync it to computer anyways

I guess that is somewhat evident of the whole "why" question, so now it's time to go on to "how" question. I'll start from receiving email, and explain every program and config on the way.

The Computer Setup

One thing worth mentioning here is the directory structure on the computer side. Everything goes under ~/posta, which looks like this:

[In: ~/Notes; Cts Ara 12 11:31; on branch master#; ^1]
[8] g@ulgen (0)$ tree -d ~/posta
/home/g/posta
├── drafts
├── mairix
├── mpop_uidls
├── queue
├── RCS
└── tmp

We'll talk about what each directory is for as they come up. In addition, we'll see a simple Makefile and a couple Python scripts for working with this setup in a more convenient way.

The messages are stored in mbox files under ~/posta.

Avoid typing passwords all the time with gpg-agent

pass stores your passwords as encrypted text files using gnupg, and everytime you interact with your passwords they are decrypted. If you set up gpg-agent, this becomes totally automatic, so you only need to type your password in once every time you log on to your computer.

But an even better way to do this is to rely on your desktop environments credential store, and have your GNUPG sessions managed automatically. If your desktop asks you to save the password, go ahead and let it do that.

If you're using a custom desktop, you probably do know how to run a process in the background, so it's probably a nice idea to start a gpg-agent process inside your .xinitrc or equivalent.

Receiving mail with mpop

mpop is a client for the POP3 protocol. POP3 is the older brother of the more widely used IMAP. Using POP3 over email has some advantages: it's a simpler protocol, leading to simpler clients which are simpler to configure (as we'll see in a bit with mpop), which leads to a more robust setup. No need to bother with syncing between multiple devices or mismatching metadata. POP3 also doesn't support folders or live notifications. All this means POP3 is great for just grabbing mail off of the server and transferring it to your computer for once and all. And this is why I use it on the computer setup, where I do want to do exactly that. We'll see that in the mobile setup, I use IMAP because that setup is only for reading; the mail stays on the server til it's fetched into local storage via the computer.

mpop's config is as simple as it gets. My actual configuration has some added complexity because it uses GNU m4 text preprocessing macro language to factor in some secrets from an external file, and pass to get passwords for the email accounts, but both of these are optional. I already use pass for all my passwords, and a couple simple m4 macros allow me to publish my config without leaking passwords or other secrets.

Below is a simple $HOME/.mpoprc that works fine on Debian and its derivatives (tls_trust_file may require fixing for other operating systems):

# -*- mode: conf-unix -*-
# mpoprc --- mpop config.

# Set default values for all following accounts.
defaults
tls on
tls_starttls off
tls_trust_file /etc/ssl/certs/ca-certificates.crt
keep off
password

# My default account
account default
host pop.example.com
port 995
user cadadr@example.com
delivery mda "/usr/bin/procmail -f '%F' -a default"
uidls_file ~/posta/mpop_uidls/1
passwordeval "pass show cadadr@example.com"

The section that starts with the defaults line is the default setting for all accounts. The other block sets up an account called default. First of all, the keep off setting is the most important: if it's on, mpop will mark messages on the server that it has fetched as read, but will not delete them from the server. If it's off, no copies will be left on the server. Generally speaking, if you want to use keep on, you might want to use an IMAP setup instead, which is better at syncing messages and metadata between multiple clients using one central server. Going back to the mpoprc, the next important line is the delivery ... line, which passes messages to procmail. This is not strictly necessary, but this intermediary step means that if you want to integrate local spam checking or to split up mail using procmail's great capabilities, the machinery to do so is there. This line has two arguments: with -f '%F' we're passing a From ... line that mpop thinks is nice enough (notice, it's not From: ..., tho the difference is not relevant here) to procmail, and with -a default we're passing the account's name to it, which is not immediately useful, but can be handy if you ever want to filter based on mpop account in procmail. Finally, the last important line here is the passwordeval ... line, which asks pass to give us the password we saved as cadadr@example.com.

It's trivial to add as many accounts as you want, just copy/paste the account block and modify it to your liking.

(Not necessarily) filtering mail with procmail

The protocol client, in our case mpop fetches email from the server and relays it to the Mail Delivery Agent, whose role is to decide where to put incoming mail, and actually put the mail there. The MDA we'll use is the tried and time tested procmail. This is the step where, if you want, do any filtering you want. You can use software like SpamAssassin and/or ClamAV check for spam and viruses locally, or filter and distribute incoming mail into as many inboxes as you want. Personally, I like to have one big inbox that globs up everything, so my $HOME/.procmailrc does just that. Refer to the procmailrc(5) man page if you want to add sophisticated roles.

# procmailrc --- Deliver mail. -*- comment-start: "# "; mode: conf-unix -*-

### Environment:
SHELL=/bin/sh
MAILDIR=$HOME/posta
SENDMAIL=/usr/bin/msmtp
LOGFILE=$HOME/log/procmail.log
VERBOSE=yes
LOGABSTRACT=yes
DROPPRIVS=yes

### Rules:

# Catch-all
:0:
inbox

You can pretty much use this as is.

Little recap time. So, what's happening here is this: mpop takes mail from my servers, using multiple accounts. All of those mail is fed into $HOME/posta/inbox using procmail. So, inbox is the staging ground for email. Soon, we'll see that how I kick this off from within Emacs with a single command, and how Rmail collects email from inbox and a couple other places. But before, we'll briefly look at mairix, which helps with searching email in multiple mboxes fast.

Setting up mairix to search mail

There's no point to storing email if you can't find messages you need after fetching and reading them, and this is where mairix comes into play. mairix can quickly index and search a multitude of mboxes, and integrates well with Emacs and Rmail. We'll talk about how to use it later, but here, I'll show you a simple $HOME/.mairixrc:

# mairixrc --- Mailbox search.

# Set this to the directory where your maildir folders live
base=/home/g/posta
mbox=current:archive*:outbox*:lists*
mfolder=/home/g/posta/mairix/search
mformat=mbox
database=/home/g/posta/mairix/index

Again, you should be able to use this as is if you're following along, tho the most important setting is mbox which sets the patterns and file names to index. These are pretty much like shell wildcards, and we'll soon see why I set it like above.

msmtp for sending email

The last external tool we'll configure is msmtp, which we use to send email. Another option here is sendmail, but it's old, clunky, and sometimes even unsafe, which makes msmtp, with it's simple config and terrible name, a great alternative. Below is an example ~/.msmtprc:

# msmtprc --- Send mail.

# Set default values for all following accounts.
defaults
auth on
tls on
tls_trust_file /etc/ssl/certs/ca-certificates.crt
tls_starttls off
logfile ~/log/msmtp.log
protocol smtp
# Leave ↓ blank for check netrc
password

# self@
account self
host smtp.example.com
port 465
from cadadr@example.com
user cadadr@example.com
passwordeval "pass show cadadr@example.com"

I bet you feel like this is pretty similar to ~/.mpoprc, which it is, because the same guy wrote both. So whatever I said about ~/.mpoprc applies to this file too. See msmtp(1) man page for details.

One important detail is that, do not forget to set from lines for all your accounts, because later on we'll make use of those lines to implement sending from multiple accounts in Emacs. Just set it to the accounts full email address.

Inside Emacs

Now we're done explaining the process for fetching email, it's time we see how Emacs kicks things off and then how Rmail takes that mail in how we use it to read and (re)file the messages.

Utility variables and functions inside Emacs

Apart from the variables and functions that directly pertain to the configuration of Rmail and Mairix inside Emacs, there are some other variables and functions in my setup that the former use.

First of all, we load Rmail and Mairix packages (mairix.el is included in Emacs, so you don't need to install it separately), and other modules we use:

(require 'cl-lib)
(require 'mail-source)
(require 'mairix)
(require 'message)
(require 'rmail)
(require 'rmailsum)
(require 'rx)
(require 'sendmail)

Then, here are the functions from my .emacs that my Rmail setup uses:

(defun gk-executable-ensure (command &optional silent)
  "Err-out if COMMAND is not found."
  (if-let* ((ex (executable-find command)))
      ex
    (when (not silent)
      (warn "Program is absent: %s" command))))

(defun gk-gui-p ()
  (or window-system (daemonp)))

(defun gk-send-desktop-notification (summary message &optional icon)
  "Show a notification on the desktop."
  (unless (gk-gui-p)
    (error "Cannot send desktop notification in non-GUI session"))
  (make-process
   :name "gk-desktop-notification"
   :buffer (get-buffer-create " *Desktop Notifications*")
   :command
   (cond
    ((executable-find "notify-send")
     (list "notify-send" (concat "[Emacs] " summary) "-i" (or icon "") message))
    ((executable-find "kdialog")
     (list "kdialog" "--passivepopup" message "10"
           "--title" (concat "[Emacs] " summary))))))

(defun gk-rmail-view-html-part-in-browser ()
  "View the HTML part of the message in this buffer in the

browser."
  (interactive)
  (save-excursion
    (goto-char (point-min))
    (re-search-forward
     (rx bol "["
         (optional digit (zero-or-more (and "/" digit)) ":")
         "text/html "
         (or "Hide" "Show")
         " Save:"))
    (point)
    (forward-char 1)
    (let ((button (button-at (point)))
          (filename
           (concat (make-temp-name
                    (expand-file-name
                     "gkrmailout" gk-mail-temporary-file-directory))
                   ".html")))
      (browse-url
       (concat "file://" (gk-rmail-mime-save-to-tmp button filename))))))

(defun gk-rmail-mime-save-to-tmp (button output-file-name)
  "Save the attachment in BUTTON in OUTPUT-FILE-NAME.

Return the file name, expanded."
  ;; Adapted from ‘rmail-mime-save’ in order to automatically export
  ;; to HTML and open in external browser.
  (let* ((rmail-mime-mbox-buffer rmail-view-buffer)
	 (data (button-get button 'data)))
    (prog1 (expand-file-name output-file-name)
      (if (and (not (stringp data))
	       (rmail-mime-entity-truncated data))
	  (unless (y-or-n-p "This entity is truncated; save anyway? ")
	    (error "Aborted")))
      (with-temp-buffer
        (set-buffer-file-coding-system 'no-conversion)
        ;; Needed e.g. by jka-compr, so if the attachment is a compressed
        ;; file, the magic signature compares equal with the unibyte
        ;; signature string recorded in jka-compr-compression-info-list.
        (set-buffer-multibyte nil)
        (setq buffer-undo-list t)
        (if (stringp data)
	    (insert data)
	  ;; DATA is a MIME-entity object.
	  (let ((transfer-encoding (rmail-mime-entity-transfer-encoding data))
	        (body (rmail-mime-entity-body data)))
	    (insert-buffer-substring rmail-mime-mbox-buffer
				     (aref body 0) (aref body 1))
	    (cond ((string= transfer-encoding "base64")
		   (ignore-errors (base64-decode-region (point-min) (point-max))))
		  ((string= transfer-encoding "quoted-printable")
		   (quoted-printable-decode-region (point-min) (point-max))))))
        (write-region nil nil output-file-name nil nil nil t)))))

(defun gk-rmail-forward-link-or-button (p)
  "Navigate both links and buttons in Rmail in a ring.

This replaces the use of ‘forward-button’ which only traverses
buttons and skips over links."
  (interactive (list (point)))
  (let (positions)
    (dolist (overlay (overlays-in (point-min) (point-max)))
      (when (memq (car (overlay-properties overlay))
                  '(goto-address button))
        (pushnew (overlay-start overlay) positions)))
    (setq positions (sort positions #'<))
    (if (>= p (car (last positions)))
        (goto-char (first positions))
      (goto-char (first (cl-remove-if ($ (<= $1 p)) positions))))))

(defun gk-rmail-backward-link-or-button (p)
  "Navigate both links and buttons in Rmail in a ring.

This replaces the use of ‘forward-button’ which only traverses
buttons and skips over links.

This is the reverse counterpart of
‘gk-rmail-forward-link-or-button’."
  (interactive (list (point)))
  (let (positions)
    (dolist (overlay (overlays-in (point-min) (point-max)))
      (when (memq (car (overlay-properties overlay))
                  '(goto-address button))
        (pushnew (overlay-start overlay) positions)))
    (setq positions (sort positions #'<))
    (if (<= p (first positions))
        (goto-char (car (last positions)))
      (goto-char (car (last (cl-remove-if ($ (>= $1 p)) positions)))))))

These are some global variables we use in the setup:

(defvar gk-mail-home (expand-file-name "~/posta")
  "Where all mailboxes etc. are.")

(defvar gk-mail-temporary-file-directory
  (expand-file-name "tmp" gk-mail-home))

(defvar gk-mail-inboxes
  (list (expand-file-name "inbox" gk-mail-home))
  "Where to look for mail.")

(when-let* ((spool (getenv "MAIL")))
  (pushnew spool gk-mail-inboxes))

(setf message-mail-user-agent t
      read-mail-command 'rmail)

(setf
 ;; Gmail does not like parens.
 message-from-style 'angles)

;; Ensure that a safe movemail is used.  I configure Emacs to use system
;; movemail at build time, but if somehow it doesn't, try to ensure it
;; does here.

(unless (string-match "with-mailutils" system-configuration-options)
 (setf mail-source-movemail-program (gk-executable-ensure "movemail")))

These are peripheral to the workflow, so I won't detail them any further. You can check their docstrings via C-h v for more information.

Fetching mail in Emacs

I have a command in Emacs that runs mpop -Q -a inside a subprocess, which is bound to C-c <, so fetching mail inside Emacs is a single keybinding operation.

(defun gk-fetch-mail ()
  "Run mail retrieval scripts."
  (interactive)
  (make-process
   :name "gk-fetch-mail" :buffer (get-buffer-create "*Fetch Mail*")
   :command (list "mpop" "-Q" "-a")
   :sentinel
   (lambda (process event)
     (let ((msg ""))
       (unless (process-live-p process)
         (when (zerop (process-exit-status process))
           (dolist (f gk-mail-inboxes)
             (when-let* ((f (file-attribute-size (file-attributes f))))
               (when (> f 0)
                 (setf msg "You have unread mail! "))))
           (when (and (gk-gui-p) (not (string-empty-p msg)))
               (gk-send-desktop-notification "New mail" msg "mail-message-new")))
         (message "%sFetch mail process %s" msg (string-trim event)))))))

(global-set-key (kbd "C-c <") #'gk-fetch-mail)
(define-key rmail-mode-map "<" #'gk-fetch-mail)

When this command is run, an asynchronous subprocess executes mpop, which fetches mail and executes procmail to deliver all mail to ~/posta/inbox. Neat, innit? I mean, if you read up until this point and didn't yet barf onto your keyboard, you must find this somewhat neat, no? ;-P

If everything went well, a few seconds after hitting C-c <, and there's new mail in inbox, a popup notification should notify you about that. That means we successfully fetched mail and are ready to finally read it. With luck, no Nigerian princes or generals have their money stranded in overseas tax havens today!

Reading mail inside Emacs

Our new email now resides in inbox. Think of inbox as the little mailbox outside your home: new mail is stored in there, but you don't read it all while standing on the pavement, right? You take it inside your home, make yourself comfortable on your soft armchair, grab your silver paper knife, gently cut the spine of the envelopes one by one, and go through your messages in the shelter of your cozy study, as dry logs of wood burn away inside the red bricken fireplace, as the pile of mail to be read rests on your soft leather desk set.

The current mbox is that desk set. Rmail grabs new mail from inbox (and $MAIL, if present), and puts it all in current. Why the distinction? Well, inbox provides a safe buffer for incoming mail, and current is an isolated workspace. I can continually fetch mail, even using e.g. a cronjob, and it won't mess my workspace so long as I don't ask Rmail to fetch from inbox. It also allows me to collect mail from the system spool, which is rarely used, but sometimes very useful (e.g. a cronjob wants to report something).

Below, we first see how I set Rmail up to behave like this, and go through the process of actually reading mail.

Rmail setup

Let's cut to the chase here first:

(setf
 rmail-primary-inbox-list gk-mail-inboxes
 rmail-secondary-file-directory gk-mail-home
 rmail-secondary-file-regexp "spam\\|outbox\\|archive$"
 rmail-file-name (expand-file-name "current" gk-mail-home)
 rmail-default-file (expand-file-name "archive" gk-mail-home)
 gk-rmail-archive-file (expand-file-name "archive" gk-mail-home)
 rmail-displayed-headers
 (rx (and bol (or "to" "date" "from" "cc" "subject" "message-id" "list-id"
                  "delivered-to")))
 rmail-mime-prefer-html nil)

This builds on the utility variables we defined earlier, and sets Rmail up to behave like we just described. Again, I won't go into details here, but you can look into the docstrings of the variables in order to learn more about them.

Now, we're ready to run M-x rmail and read our messages.

Reading in Rmail

So we did run C-c <, and got the notification saying we do have new messages. Now, we type M-x rmail RET, et voilà, the Rmail buffer appears, showing the first unread message in current. Here, we navigate messages using n and p, view a list of messages using h, etc. For other possible commands, I suggest you enable menu-bar-mode, and look through Rmail's menu items a bit. There's not many of them, and they are worth looking through in order to quickly learn what you can do with Rmail.

Archiving mail

With this approach to doing email, I suggest that you delete useless mail with d, like marketing spam, newsletters not worth saving, reports for your cargo and orders which have already been delivered and you won't be returning, etc. Useless mail is junk mail, and there's no point in keeping it. Because the rest, we'll be keeping it: when we're done with messages in current, we move them into archive mbox. I have a couple functions for doing that nicely.

First, gk-rmail-advance, bound to N:

(defun gk-rmail-advance ()
  "Advance to the next message in default mbox.

This command will not run unless in an RMAIL buffer visiting
‘rmail-file-name’.  It will output the current message to
‘gk-rmail-archive-file’ and delete it, advancing to the next
message in the RMAIL file.  This is a utility for the email
workflow where a temporary inbox is used for working with current
email and archiving read mail in another file."
  (interactive)
  (unless (string= (buffer-file-name) rmail-file-name)
    (user-error
     "This is not your default RMAIL file, won't run ‘gk-rmail-advance’ here"))
  (rmail-output gk-rmail-archive-file)
  (rmail-delete-forward))

(define-key rmail-mode-map "N" #'gk-rmail-advance)

This command is like n, but just before advancing to the next unread message, it copies the current message to archive, and deletes it from current.

Second, a slight variation on gk-rmail-advance, with a lambda bound to b, works exactly like the former, but also opens the HTML part of the message in the browser:

(define-key rmail-mode-map "b"
  (gk-interactively (gk-rmail-view-html-part-in-browser)
                    (gk-rmail-advance)))

Rmail (and other Emacs MUAs) can render HTML messages using shr.el which is bundled in Emacs, but the evil artifact of Satan that is HTML email is inevitably best viewed in a browser.

A typical reading session of mine is thus as follows: if I want to keep the message in current for a while, hit n, if I don't want to keep it any further, hit d, if I want to move it to archive, hit N, and if I want to view the HTML part in the browser and archive the message, I hit b. I don't delete the HTML files Rmail writes (because sometimes I like to bookmark them or don't have time to look at them for a long time), so there's no point in keeping the messages in current.

Splitting archive

With mboxes, mail is stored in a single mbox per "folder". This means, over time these files can grow huge, as mail accumulates, and they'll become impossibly slow to operate, if not treated. One option is to split the mboxes, archive in particular, in order to not put a strain on Rmail and Emacs. I do this using a makefile and a Python script.

Here is the Python script, which I have under ~/bin/splitmbox.py:

#!/usr/bin/env python3
# splitmbox.py --- split up an mbox

import mailbox
import sys
import os

if len(sys.argv) < 4:
    print("usage: splitmbox.py MEGABYTES MBOX NEWNAME")
    exit(1)

_, split_size, source, newname = sys.argv

split_size = int(split_size) * 1024 * 1024

mbox = mailbox.mbox(source)
idx = 1

# Find the next slice number.
files = list(filter(lambda fil: fil.startswith(newname + "-"), os.listdir('.')))
if files:
    files.sort()
    idx = int(files[-1].split("-")[1])


def newmbox():
    return "%s-%04d" % (newname, idx)


target = mailbox.mbox(newmbox())

for _, message in mbox.items():
    target.add(message)
    if os.stat(newmbox()).st_size >= split_size:
        target.flush()
        target.close()
        idx += 1
        target = mailbox.mbox(newmbox())

os.rename(newmbox(), newname)

This only uses Python 3 and its standard library.

And this is the makefile, at ~/posta/Makefile:

# $Id: Makefile,v 1.1 2019/07/04 09:57:52 g Exp $
# Makefile

help:
	@echo "	split			split all mailboxes"
	@echo "	split-archive		split archive"
	@echo "	split-outbox		split outbox"

split: split-archive split-outbox

split-archive:
	splitmbox.py 32 archive archive
	mairix

split-outbox:
	splitmbox.py 32 outbox outbox
	mairix

.PHONY: help split split-archive split-outbox

When my archive goes above 32 megabytes, which Emacs is kind enough to nag me about, I go to ~/posta and run make split to split into little bits. So the directory has all these little ~32MB mboxes:

[In: ~/posta; Cts Ara 12 14:18; ^1]
[12] g@ulgen (0)$ ls archive* outbox*
archive       archive-0009  archive-0018  archive-0027	archive-0036  outbox-0001
archive-0001  archive-0010  archive-0019  archive-0028	archive-0037  outbox-0002
archive-0002  archive-0011  archive-0020  archive-0029	archive-0038  outbox-0003
archive-0003  archive-0012  archive-0021  archive-0030	archive-0039  outbox-0004
archive-0004  archive-0013  archive-0022  archive-0031	archive-0040  outbox-0005
archive-0005  archive-0014  archive-0023  archive-0032	archive-0041  outbox-0006
archive-0006  archive-0015  archive-0024  archive-0033	archive-0042
archive-0007  archive-0016  archive-0025  archive-0034	archive-0043
archive-0008  archive-0017  archive-0026  archive-0035	outbox

I kinda see your popping eyes as you wonder how on earth would you ever be able to find any messages ever in this mess, but fear not! mairix comes to our rescue, and we never touch these files ever manually.

Searching mail with mairix

There are many local mail search programs out there. mu and it's Elisp companion mu4e, and notmuch with its own Emacs interface can help you with maildirs. But if you're not in team D. J. Bernstein, you're not alone, you have mairix. It helps us index and search as many mboxes as we need or want.

Above, I provided a ~/.mairixrc that works nice with the setup that I'm describing here. Below is the corresponding Emacs setup:

(setf
 mairix-file-path (expand-file-name "mairix/" gk-mail-home)
 mairix-search-file "search")

It's as simple as it gets, especially in comparison with the rest of this huge ass setup. You can run a mairix search with M-x mairix-search RET, which will read a search string from the minibuffer, or using M-x mairix-widget-search RET which brings up a search form with informative labels for all the fields.

Running a mairix search will copy matching messages into an mbox called search, and display it to you using none other than the good old Rmail itself.

Improving Rmail a bit

Rmail is a bit spartan, so the following keybindings help make it a bit more doric:

(define-key rmail-mode-map (kbd "<tab>") #'gk-rmail-forward-link-or-button)
(define-key rmail-mode-map (kbd "<backtab>") #'gk-rmail-backward-link-or-button)

;; ‘q’ is normally bound to #'rmail-summary-quit, which is simply
;; useless.
(define-key rmail-summary-mode-map "q" #'bury-buffer)

Here is what these do: TAB and Shift+TAB in Rmail generally only navigate multipart buttons. We replace them with commands which can navigate to links too. And, in summary buffers, i.e. those which you access by hitting h in Rmail buffers, hitting q normally closes both the summary buffer and the Rmail buffer, which is pretty annoying. So we have it to just bury the summary buffer instead.

And that's pretty much it for reading mail in Emacs. Fetch, read, archive. Next up, we look at the setup for sending mail with message-mode and msmtp.

Sending mail inside Emacs

Sending mail is in general simpler than receiving it, but I add a dash of complexity in order to conveniently send from multiple msmtp accounts.

Basics of message-mode setup

Let's set some variables first:

(setf
 message-send-mail-function 'message-send-mail-with-sendmail
 message-sendmail-f-is-evil t
 message-sendmail-envelope-from 'header
 sendmail-program (gk-executable-ensure "msmtp"))


;; Spammers are everywhere.
(setf user-mail-address (concat "cadadr" "@" "example" "." "com")
      user-full-name "Göktuğ Kayaalp")

(setf
 message-citation-line-function 'message-insert-formatted-citation-line
 message-citation-line-format "On %Y-%m-%d %R %Z, %f wrote:")

(setf
 message-default-headers (format "Fcc: %s/outbox" gk-mail-home)

 ;; Drafts directory.
 message-auto-save-directory (expand-file-name "drafts" gk-mail-home)
 ;; Ask for confirmation before sending a message.
 message-confirm-send t)

and some QoL improvements:

(add-hook 'message-sent-hook #'bury-buffer)
(define-key message-mode-map (kbd "C-c C-c") 'message-send)

These settings are some sane defaults, and I won't bother explaining them in detail. But I'll touch upon two key settings: first, we set sendmail-program to "msmtp", in order for Emacs to use that program to send email (Emacs has an SMTP client implementation bundled with it), and then we add an FCC header to message-default-headers so that messages we sent are saved to ~/posta/outbox, which if we didn't, they'd be sent with no trace anywhere, offline or on your mail server.

Sending from multiple msmtp accounts

This is something hard to get working in Emacs. If you have multiple email accounts you might have to send messages from, it'd be pretty inconvenient if each time you'd have to go into your ~/.msmtprc and change the default account or change some variable in Emacs and don't forget to edit your From: line in the message-mode buffer. With the little function and hook below, you'll just need to change your From: line in that buffer, and the function will find the matching account from your ~/.msmtprc and set things the right way just before sendmail-program is run.

(defun gk-mail-set-msmtp-account ()
  "Find account name for email address in From: line."
  (let ((from (save-excursion
                (goto-char (point-min))
                (or (re-search-forward "^From: .*? <" nil t)
                    (user-error "No From: line or an empty one"))
                (buffer-substring (point) (1- (line-end-position))))))
    (with-current-buffer (find-file-noselect "~/.msmtprc")
      (goto-char (point-min))
      (or (re-search-forward (concat "^from " from) nil t)
          (user-error "No msmtp account for ‘%s’" from))
      (re-search-backward "^account ")
      (end-of-line)
      (setf
       message-sendmail-extra-arguments
       (list "-a" (substring-no-properties (thing-at-point 'symbol)))))))

(add-hook 'message-send-mail-hook #'gk-mail-set-msmtp-account)

And this should conclude the core of it. Next up, we'll integrate this setup with the OS so that mailto: links end up in Emacs.

Routing mailto: links to Emacs

It's called FreeDesktop because they are free to have you deal with a billion *.desktop files.

— Me, just now.

What if I took this mailcap file and .xinitrc and rigged them with explosives?

— A FreeDesktop dev, just about the Dusk of Computing.

Most operating systems support the FreeDesktop specification for associating mimetypes and link types with applications. It's a complex machinery nobody needs to understand. The magic trick is, if you put the below text into ~/.local/share/applications/emacsclient-mailto.desktop, Emacs will be able to handle mailto: links, granted you have the Emacs server running (if not, see (emacs) Emacs Server):

[Desktop Entry]
Categories=Office;Network;Email;
Comment=Emacsclient for mailto links
Exec=emacsclient -nc -eval '(browse-url-mail "%u")'
Icon=emacs
Name=emacsclient-mailto
MimeType=x-scheme-handler/mailto;
NoDisplay=false
Terminal=false
Type=Application

Mobile setup

After the elegant monstrosity of the Emacs setup above, the mobile setup is a breeze of fresh air: just obtain K-9 mail via F-Droid or Google Play, and set your accounts up. Nothing fancy or out of the ordinary there. Even which MUA you use is not important, given we only read email here. Personally, I only type stuff on a touchscreen if I have no other options.

But K-9 is a great open source Android mail client, and I really suggest you try out the new beta, which is a UI overhaul done right IMHO.

Miscellanea

I'll briefly mention some ideas to improve this setup with some sidekicks.

Gnus for mailing lists through NNTP

One weakness of this whole setup is that it's really hard to keep up with mailing lists with it. It's not impossible, some people like Richard Stallman do pull it off, but you and I are not RMS, so we may fancy our kind selves a bit more pampering of a solution to deal with the huge volume of messages a mailing list can throw at us.

One option is to use Gnus, another bundled email (and more) client in Emacs, to access the mailing lists using NNTP. It's fairly easy to set up, tho I haven't set this up yet. You'll want to configure Gmane as your NNTP server, which is a mailing-list to NNTP gateway by the great Lars Ingebrightsen, to whom we're also indebted for Gnus too.

An alternative setup is to just subscribe to the mailing lists and use procmail to sort messages into local mboxes, but it gets old too quickly, as gigabytes and gigabytes of useless data pile up each month, and your mail fetch time grows by orders of magnitude.

This setup, without Emacs

I like your setup, but I like my pinkies more.

— You.

Well, first of all, do yourself a great favour and run setxkbmap -option "ctrl:nocaps". If you still don't want to use Emacs tho, it's pretty easy to replicate the essence of this workflow using mutt. The mutt config below should be a nice starting point:

# muttrc                                                 -*- mode: conf -*-

### Personal details:

set realname = "İ. Göktuğ Kayaalp"
set from = "self@gkayaalp.com"

### Folders:

# This attempts to replicate the mail flow I have in Rmail: all mail is
# under ~/posta, it comes into `inbox', gets read in `current', then either
# gets appended to `archive' or `spam' (generally); outgoing mail is recorder
# in `outbox'.

set folder = ~/posta
set mbox = +archive
set spoolfile = +current
set record = +outbox
# In Emacs I save drafts one per file, so use a separate mbox for Mutt's
# drafts.
set postponed = +drafts/postponed
mailboxes = +inbox

### Sending mail:

set edit_headers = yes
set envelope_from = yes
set sendmail = /usr/bin/msmtp

I keep this as a backup for an emergency situation (e.g. my Emacs config is broken and I don't have the time to fix), thus it's not much fancy, but it should be possible to drastically improve this, as mutt is very capable. I also know that NeoMutt exists, but I can't compare it with mutt, as I haven't used it before.

There's a replacement for Gnus, too, for your large mailing lists: slrn. There probably are some alternatives, but I haven't tested them myself. Wikipedia is your friend.

BTW, I know some vim users like to remap Caps Lock to ESC. It should be achievable with setxkbmap, but I don't know the particular incantation, because I fancy a better editor.

Working offline

It's neat that you can read mail offline, but sometimes you need to compose offline and send in bulk. msmtp has a solution for that: two shell scripts that manage a queue of messages in a local directory and send them in bulk using msmtp. They are included in msmtp's distribution but I've customised these scripts to work with my setup, which you can find below. In order to use these scripts with Emacs, first set the environment variable MAILQUEUE to ~/posta/queue, then make these scripts executable and put them somewhere on the $PATH, and change the SMTP client in Emacs to the shell script in the configuration above:

sendmail-program (gk-executable-ensure "msmtp-enqueue.sh")

I have these couple convenience commands in Emacs to list the queue and run the queue:

(defun gk-runq ()
  "Run outgoing email queue."
  (interactive)
  (async-shell-command "msmtp-runqueue.sh" "*runq*"))

(defun gk-listq ()
  "Show email queue."
  (interactive)
  (async-shell-command "msmtp-listqueue.sh" "*listq*"))

Now the scripts themselves. msmtp-enqueue.sh:

#!/usr/bin/env bash

QUEUEDIR=$MAILQUEUE

# Set secure permissions on created directories and files
umask 077

# Change to queue directory (create it if necessary)
if [ ! -d "$QUEUEDIR" ]; then
	mkdir -p "$QUEUEDIR" || exit 1
fi
cd "$QUEUEDIR" || exit 1

# Create new unique filenames of the form
# MAILFILE:  ccyy-mm-dd-hh.mm.ss[-x].mail
# MSMTPFILE: ccyy-mm-dd-hh.mm.ss[-x].msmtp
# where x is a consecutive number only appended if you send more than one 
# mail per second.
BASE="`date +%Y-%m-%d-%H.%M.%S`"
if [ -f "$BASE.mail" -o -f "$BASE.msmtp" ]; then
	TMP="$BASE"
	i=1
	while [ -f "$TMP-$i.mail" -o -f "$TMP-$i.msmtp" ]; do
		i=`expr $i + 1`
	done
	BASE="$BASE-$i"
fi
MAILFILE="$BASE.mail"
MSMTPFILE="$BASE.msmtp"

# Write command line to $MSMTPFILE
echo "$@" > "$MSMTPFILE" || exit 1

# Write the mail to $MAILFILE
cat > "$MAILFILE" || exit 1

# If we are online, run the queue immediately.
# Replace the test with something suitable for your site.
#ping -c 1 -w 2 SOME-IP-ADDRESS > /dev/null 
#if [ $? -eq 0 ]; then
#	msmtp-runqueue.sh > /dev/null &
#fi

exit 0

msmtp-runqueue.sh:

#!/usr/bin/env bash

QUEUEDIR=$MAILQUEUE
LOCKFILE="$QUEUEDIR/.lock"
MAXWAIT=120

OPTIONS=$@

# wait for a lock that another instance has set
WAIT=0
while [ -e "$LOCKFILE" -a "$WAIT" -lt "$MAXWAIT" ]; do
	sleep 1
	WAIT="`expr "$WAIT" + 1`"
done
if [ -e "$LOCKFILE" ]; then
	echo "Cannot use $QUEUEDIR: waited $MAXWAIT seconds for"
	echo "lockfile $LOCKFILE to vanish, giving up."
	echo "If you are sure that no other instance of this script is"
	echo "running, then delete the lock file."
	exit 1
fi

# change into $QUEUEDIR 
cd "$QUEUEDIR" || exit 1

# check for empty queuedir
if [ "`echo *.mail`" = '*.mail' ]; then
	echo "No mails in $QUEUEDIR"
	exit 0
fi

# lock the $QUEUEDIR
touch "$LOCKFILE" || exit 1

# process all mails
for MAILFILE in *.mail; do
	MSMTPFILE="`echo $MAILFILE | sed -e 's/mail/msmtp/'`"
	echo "*** Sending $MAILFILE to `sed -e 's/^.*-- \(.*$\)/\1/' $MSMTPFILE` ..."
	if [ ! -f "$MSMTPFILE" ]; then
		echo "No corresponding file $MSMTPFILE found"
		echo "FAILURE"
		continue
	fi
	msmtp $OPTIONS `cat "$MSMTPFILE"` < "$MAILFILE"
	if [ $? -eq 0 ]; then
		rm "$MAILFILE" "$MSMTPFILE"
		echo "$MAILFILE sent successfully"
	else
		echo "FAILURE"
	fi
done

# remove the lock
rm -f "$LOCKFILE"

exit 0

and finally msmtp-listqueue.sh:

#!/usr/bin/env bash

QUEUEDIR=$MAILQUEUE

for i in $QUEUEDIR/*.mail; do
	egrep -s --colour -h '(^From:|^To:|^Subject:)' "$i" || echo "No mail in queue";
	echo " "
done

Managing a database of contacs

You can use Emacs to manage your address book via BBDB. Below is my setup for it:

(require 'bbdb)
(require 'bbdb-vcard)

(setf bbdb-file (expand-file-name "~/Documents/bbdb")

(add-hook 'message-setup-hook 'bbdb-mail-aliases)

With this setup BBDB integrates with message-mode so that you can tab complete To:, Cc:, etc. lines.

With bbdb-vcard, you can import/export .vcf (vCard) files (RFC 2425 and RFC 2426). This way you can sync with your phone's contacts, for example. IIRC there are automatic solutions for this too, but I hadn't used them before. ^notGoogle is your friend.

Converting to mbox from NNML and Maildir

Finally, I want to share two Python 3 scripts which you can use to convert your message archives from Gnus' NNML format or the widespread Maildir format to mbox files you can use with Rmail. After using these scripts, you should do as follows: first, open the generated mbox file in Rmail via C-u M-x rmail RET, then run M-x rmail-sort-by-date RET to sort the messages, and hit s to save the mbox. Beware that these steps can take a long time with large mboxes. Then, use splitmbox.py from above to split the big mbox into smaller ones.

When converting from a Maildir, I'd first collect all the cur/ subfolders and use them to create a big archive mbox. Then I'd generate current from all the tmp/ and new subfolders.

Here are the scripts. They only depend on Python 3 and its standard library. maildir2mbox.py

#!/usr/bin/env python3
# maildir2mbox.py --- convert maildir to mbox files.

import sys
import os
import mailbox
import email

if len(sys.argv) < 2:
    print("usage: maildir2mbox.py FOLDER... TARGET")
    exit(1)

target = sys.argv.pop()
folders = sys.argv[1:]

messages = []

for folder in folders:
    msgs = os.listdir(folder)
    idx = 0
    try: msgs.remove(".overview")
    except ValueError: pass
    for msg in msgs:
        idx += 1
        messages.append((os.path.join(folder, msg), idx))

messages.sort(key=lambda i: i[1])

mbox = mailbox.mbox(target)

for path, _ in messages:
    with open(path, mode="rb") as f:
        try:
            text = f.read()
            msg = email.message_from_bytes(text)
            mbox.add(msg)
        except Exception as e:
            print("Exception while processing %s: %s" % (path, e))

and nnml2mbox.py:

#!/usr/bin/env python3
# nnml2mbox.py --- convert Gnus NNML groups to mbox files.

import sys
import os
import mailbox
import email

if len(sys.argv) < 2:
    print("usage: nnml2mbox.py FOLDER... TARGET")
    exit(1)

target = sys.argv.pop()
folders = sys.argv[1:]

messages = []

for folder in folders:
    msgs = os.listdir(folder)
    try: msgs.remove(".overview")
    except ValueError: pass
    for msg in msgs:
        messages.append((os.path.join(folder, msg), int(msg)))

messages.sort(key=lambda i: i[1])

mbox = mailbox.mbox(target)

for path, _ in messages:
    with open(path, mode="rb") as f:
        try:
            text = f.read()
            msg = email.message_from_bytes(text)
            mbox.add(msg)
        except Exception as e:
            print("Exception while processing %s: %s" % (path, e))

Conclusion

Congratulations! If you read all that and are curious why would I ever bother, well, the answer is, I never did. I started with a pretty minimal setup around Gnus many years ago, and it evolved and evolved. With software like Emacs and Unix-likes, one can take existing tools and modify them to their needs, and end up creating a highly personal and complex solution like this. FWIW, I could've kept this much more terse, but I wanted to try and provide not only a comprehensive explanation, but a fully working setup. Honestly tho, 6 hours ago, if I knew it'd be this long, maybe I wouldn't have started :)

If you found this useful, I'm glad that it helped. If you enjoyed the read, I'm happy. If you hate me for writing this, well, here you're at the very final line of a long ass post, so maybe you didn't hate that much after all :)