A reproducible IMAP sandbox with Docker

Introduction

Trying an IMAP client requires a mail server and a populated mailbox — which usually means real credentials, a provider’s rate limits, and results that cannot be reproduced anywhere else. This vignette shows how to run mRpostman against a disposable local IMAP server (Dovecot, in a Docker container) whose mailbox is populated deterministically by the package itself: populate_sandbox() uploads a fixed synthetic corpus with the package’s own APPEND implementation, so any user, on any machine, obtains exactly the same messages — and therefore exactly the same search, fetch, and decoding results shown here.

This is useful for:

• trying every feature of the package without a real mail account (no OAuth2 setup, no “less secure apps” hurdles);
• teaching and demonstrations, with reproducible outputs;
• testing code that uses mRpostman against a fast, offline, rate-limit-free server.

None of the code chunks in this vignette are evaluated when the package is built (there is no Docker server on CRAN’s machines); copy and paste them into your session.

NOTE: the sandbox server is for local experimentation only — it uses plaintext authentication, no TLS, and a static password.

Starting the server

You will need Docker. The Dockerfile and the Dovecot configuration ship with the package; find them with:

system.file("docker", package = "mRpostman")

From that folder, build the image and start a container, exposing the IMAP port as 1430 on the host:

docker build -t mrpostman-sandbox .
docker run -d --name mrpostman-sandbox -p 1430:143 mrpostman-sandbox

The server accepts any username with the password sandbox, and each username gets its own empty mailbox.

Connecting

library(mRpostman)

con <- configure_imap(url = "imap://localhost:1430",
                      username = "testuser",
                      password = "sandbox",
                      use_ssl = FALSE)

con$list_server_capabilities()

Note the use_ssl = FALSE and the imap:// (not imaps://) scheme: the sandbox has no TLS. Dovecot advertises, among others, the ESEARCH, SORT, THREAD, MOVE, UNSELECT, and SPECIAL-USE capabilities, so the package’s extension-based methods can be exercised too.

Populating the mailbox

populate_sandbox() generates the synthetic corpus of sandbox_corpus() — 200 RFC 822 messages by default, driven by a fixed RNG seed — and stores it using the package’s own IMAP operations (APPEND, CREATE, STORE):

info <- populate_sandbox(con, n = 200)
#> Appending 200 synthetic messages to INBOX at imap://localhost:1430 ...
#>   50/200
#>   100/200
#>   150/200
#>   200/200
#>
#> Done. INBOX now reports:
#> $EXISTS
#> [1] 200
#> ...

The returned (invisible) info data.frame describes what was embedded in each message, so every result below can be checked against it:

head(info)
#>   id                     from                  subject ...
#> 1  1 [email protected]     Meeting minutes     ...

The corpus is designed so that each feature of the package has matching messages to act upon:

• Date: headers spread over the year 2020, for the SENT* date searches;
• a subset of large bodies, for size searches;
• accented subjects in MIME encoded-words and quoted-printable bodies, for the decoding helpers;
• CSV, PNG, and one-page PDF attachments (some with repeated filenames), for the attachment methods, binary base64 decoding, and filename deduplication — see info$attachment_type;
• reply chains (In-Reply-To/References), for SORT and THREAD;
• \Seen, \Flagged, and \Answered flags, adjusted after the upload, for flag searches. (libcurl’s APPEND stores every message with the \Seen flag, so populate_sandbox() removes it from the messages planned as unread — exercising STORE in both directions.)

For use from the shell, the wrapper script inst/docker/populate_mailbox.R does the same: Rscript populate_mailbox.R 200.

Ingesting real data: maildirs and the Enron corpus

The synthetic corpus is not the only way to fill the sandbox. ingest_maildir() uploads any local maildir-style directory (one RFC 5322 message per file) to a server folder via APPEND — mail server backups, exported archives, or public corpora:

manifest <- ingest_maildir(con, path = "~/backup/maildir/archive2020",
                           folder = "archive2020")
table(manifest$appended)

Built on top of it, enron_sandbox() turns the sandbox into a real-data laboratory: it downloads the public Enron e-mail corpus once (~423 MB, with your consent, cached under tools::R_user_dir("mRpostman", "cache")), selects a subset by custodian, folder-name pattern, and Date: header window, and ingests it — one server folder per custodian:

con2 <- configure_imap(url = "imap://localhost:1430",
                       username = "enron", password = "sandbox",
                       use_ssl = FALSE)
manifest <- enron_sandbox(con2,
                          custodians  = c("kaminski-v", "lay-k", "skilling-j"),
                          sent_since  = "2000-01-01",
                          sent_before = "2002-07-01")
table(manifest$custodian, manifest$appended)

After the ingestion, everything shown in the guided tour below works on real data too — e.g. con2$search_string(expr = "SEC", where = "TEXT") on a custodian’s folder. The same internal-date caveat applies: query ingested corpora by date through the SENT* searches, which read the Date: header.

A guided tour

con$list_mail_folders()
con$examine_folder("INBOX")
con$select_folder("INBOX")

# messages "sent" in the first quarter of 2020 (Date: header)
ids_q1 <- con$search_sent_period(since_date_char = "01-Jan-2020",
                                 before_date_char = "01-Apr-2020")

# large messages
ids_large <- con$search_larger_than(size = 8000)

# flagged messages
ids_flagged <- con$search_flag("FLAGGED")

ids <- con$search(request = OR(string(expr = "sales", where = "SUBJECT"),
                               string(expr = "[email protected]",
                                      where = "FROM")))

con$esearch_count(flag = "SEEN")
con$esearch_min_id(flag = "FLAGGED")

meta <- ids_q1[1:5] %>%
  con$fetch_metadata(attribute = c("INTERNALDATE", "RFC822.SIZE"))

texts <- ids_q1[1:5] %>%
  con$fetch_text()

utf8_ids <- info$id[info$is_utf8_body]
raw_text <- con$fetch_text(msg_id = utf8_ids[1])
clean_msg_text(raw_text)
#> [[1]]
#> [1] "Segue em anexo o relatório com os números consolidados. ..."

att_ids <- info$id[info$has_attachment]

# list attachment filenames without downloading
con$fetch_attachments_list(msg_id = att_ids[1:5])

# download them (note the automatic deduplication of repeated filenames)
con$fetch_attachments(msg_id = att_ids[1:5])

con$sort(by = "SUBJECT", use_uid = TRUE)
con$thread(algorithm = "REFERENCES")

con$copy_msg(msg_id = ids_flagged[1], to_folder = "ProjectAtlas")
con$add_flags(msg_id = ids_flagged[1], flags_to_set = "\\Deleted")
con$expunge()

Resetting and tearing down

Since the server accepts any username — and each one gets its own independent mailbox — the quickest way to start over is to simply connect with a new username, with no need to touch the container:

con2 <- configure_imap(url = "imap://localhost:1430",
                       username = "testuser2", # any new name = pristine mailbox
                       password = "sandbox",
                       use_ssl = FALSE)
populate_sandbox(con2, n = 200)

This is also handy for keeping several corpora side by side (e.g. one mailbox per experiment or per seed).

Alternatively, the mailboxes live inside the container, so recreating it resets everything at once:

docker rm -f mrpostman-sandbox
docker run -d --name mrpostman-sandbox -p 1430:143 mrpostman-sandbox

Either way, a new populate_sandbox() call recreates the exact same corpus.