Parsing emails into clean JSON, without tears

Every developer eventually has the same idea: "the data I need arrives by email, so I'll just parse the email." Order confirmations, sign-up alerts, invoices, monitoring notifications: it all lands in an inbox, and an inbox is just text, right? Then you open a raw message for the first time and discover that email is a 40-year-old archaeological dig of extensions layered on extensions.

A single message can contain multiple alternative bodies, nested attachments, three different text encodings and headers folded across lines with their own escaping rules. None of it was designed for machines to consume. It was designed for 1980s mail clients to render. This guide walks through the real pain points first, then the practical approaches for getting structured JSON out the other side.

A modern email is a MIME tree, not a flat document. The top level is usually multipart/mixed (body plus attachments), which contains a multipart/alternative (the same content as both text/plain and text/html), which may itself contain multipart/related (the HTML body plus the inline images it references). Yes, that's a tree three or four levels deep for a routine newsletter.

Your parser has to walk this tree and make decisions: which alternative do you treat as "the body"? The text/plain part is easier to process but senders often neglect it. It can be empty, truncated or just say "view this email in a browser". The HTML part is authoritative but has to be parsed as HTML. Most robust pipelines extract both, prefer HTML for structured extraction, and keep plain text as a fallback.

Each MIME part declares a Content-Transfer-Encoding, commonly quoted-printable (those =E2=82=AC sequences) or base64, plus a charset that might be UTF-8, ISO-8859-1, Windows-1252 or something rarer. Decode with the wrong one and you don't get an error; you get mojibake in your database three weeks later.

Headers are worse. Subject lines and sender names use RFC 2047 "encoded words" (=?UTF-8?B?...?=), which can be split across multiple chunks mid-word, in different charsets. Never parse these by hand: every mainstream language has a battle-tested library (Python's email package, mailparser in Node) that handles the decoding. Use it, then treat everything downstream as UTF-8.

HTML bodies reference inline images via cid: URLs pointing at other MIME parts, so a naive "grab the img tags" approach yields links that resolve to nothing outside the message. Tracking pixels add noise: single-pixel images you almost never want in your extracted data. And attachments can be misdeclared: a PDF labelled application/octet-stream, a filename only recoverable from a Content-Disposition parameter with its own encoding scheme.

The practical stance: extract attachments by walking the tree and sniffing content types when the declared one is generic, strip or ignore cid: references unless you specifically need the images, and treat everything in the body as untrusted input, because it is.

Before touching bodies, mine the headers. From, To, Subject, Date, Message-ID and In-Reply-To are semi-structured and reliable. Machine-generated mail often includes bonus headers too: List-Id identifies newsletters, Auto-Submitted flags automated senders, and many platforms add custom X- headers identifying the triggering event.

For many automation use cases (routing, deduplication, threading, sender classification), headers alone answer the question, and you never need to parse a body at all. Cheap wins first.

One caveat: headers lie. From addresses are trivially forged, and Date reflects the sender's clock, not reality. If a decision has security consequences, check the authentication results your receiving server records (SPF, DKIM, DMARC) rather than trusting what the sender wrote about itself, and prefer the received timestamp your own infrastructure stamped on arrival.

For HTML bodies from a consistent sender, CSS selectors are the sharpest tool. Order confirmations, receipts and alerts are rendered from templates, so the total is always in the same table cell. Parse the HTML properly (cheerio, BeautifulSoup) and select. Don't regex raw HTML.

For plain-text bodies and simple patterns, regex still earns its keep: verification codes, order numbers, monetary amounts and URLs all have recognisable shapes. Anchor patterns to nearby labels ("Order #", "Your code is") rather than matching bare digits, and expect templates to change under you eventually.

AI fills the gap the other two can't: classification and extraction across senders you don't control. "Is this a shipping notification, and what's the tracking number?" works across a hundred different courier templates without writing a hundred selectors. The pragmatic pipeline is layered: headers first, deterministic selectors and regex for known senders, AI for classification and the long tail. Each layer only handles what the cheaper layer beneath it couldn't.

Everything above is genuinely necessary if you run your own ingestion. But it's undifferentiated plumbing: no user ever thanked a product for its excellent quoted-printable handling. This is exactly the layer HideMy.world exists to absorb: you create an address like orders@yournick.hidemy.world, point a sender at it, and inbound mail arrives at your webhook as clean JSON: decoded, de-MIMEd, with headers, text and HTML bodies neatly separated. Rules and AI rules filter before delivery, and transformers reshape the payload to exactly the fields you want.

Whether you build it or buy it, the shape of a good email pipeline is the same: decode once at the boundary, extract with the cheapest tool that works, and never let raw MIME leak past your first function.