Back to Blog

Introducing IMAP Migration Tools: Moving 100k+ Emails Without Losing My Mind

Javier Callico
#Python#Email#Open Source#Engineering

The 100,000 Email Problem

Migrating email accounts is one of those tasks that sounds simple in theory but is a nightmare in practice. A few weeks ago, I needed to migrate a Google account with over 100,000 emails to a new provider.

I tried the usual suspects. I tried desktop clients like Thunderbird (drag-and-drop crashed after a few hundred messages). I tried proprietary migration services (expensive and black-box). I tried open-source scripts that hadn't been updated in five years.

They all failed. They timed out, they crashed on weird character encodings, or they simply couldn't handle the sheer volume of data without hitting API rate limits.

So, as engineers do, I decided to build my own.

Enter imap-migration-tools

My goal was simple: Reliability over everything. I needed a tool that could run for days if necessary, handle network interruptions gracefully, and ensure that exactly zero emails were lost.

The result is imap-migration-tools, a suite of Python scripts designed for heavy-duty email operations.

Key Features

1. Smart De-duplication

The biggest risk in migration is running a script twice and ending up with duplicates. My tools index messages by their unique Message-ID header. You can stop the script, restart it, or run it effectively in "sync" mode, and it will skip everything it has already processed.

2. Multi-threaded Performance

IMAP is a chatty protocol. Waiting for a server to acknowledge every single APPEND command sequentially takes forever. I implemented a thread pool to allow parallel copying, which maximizes throughput while keeping within server connection limits (configurable via MAX_WORKERS).

3. The "Gmail Mode"

Gmail is weird. It doesn't strictly follow standard IMAP folders; it uses labels. An email with three labels appears in three different IMAP folders, but it's physically just one message. I built a specific --gmail-mode that:

  • Migrates from [Gmail]/All Mail to ensure we get every message exactly once.
  • Reconstructs the label structure on the destination by "copying" (labeling) the message into the appropriate folders.

4. Verification

Trust, but verify. I didn't want to just hope it worked. I built compare_imap_folders.py, a tool that connects to both source and destination and produces a side-by-side report of message counts per folder. Seeing those "MATCH" statuses in green is incredibly satisfying.

The Toolset

The repository (available on GitHub) includes four main tools:

  • migrate_imap_emails.py: The main workhorse. Direct server-to-server migration.
  • backup_imap_emails.py: Downloads your entire account to local .eml files (perfect for archiving).
  • restore_imap_emails.py: Pushes a local backup back up to an IMAP server.
  • count_imap_emails.py: A quick diagnostic tool to size up the job before you start.

Lessons Learned

Building this taught me a lot about the fragility of email protocols. RFC standards are often treated as "suggestions" by different providers. Handling edge cases—like subjects with mixed encodings, or servers that abruptly close connections—was 80% of the work.

But now, I have a toolkit I can rely on. If you're facing a massive email migration and existing tools are letting you down, give imap-migration-tools a try. It’s open source, MIT licensed, and built to get the job done.

Check it out here: https://github.com/JCallico/imap-migration-tools