Email Triage with NLP: Classify, Route, Escalate

Opening answer (BLUF)

Email triage with NLP is a production operations problem, not a spam filter with extra labels. The job is to classify inbound mail, route it to the right queue, draft a reply when the case is routine, and escalate when it is not. Binary spam detectors on public corpora now report accuracy near 99 percent in research settings.[7] Multi-class operations mail is a different task. In a 2025 case study, Travelers Insurance and AWS reported 91 percent classification accuracy across 13 service-request categories after prompt engineering, up from 68 percent without it.[4] That residual is the operating system: a human review queue for low-confidence and high-harm messages, and an audit trail of who classified what and whether a draft was sent. NIST's January 2023 AI Risk Management Framework treats accuracy as more than a headline score. It includes false positives, false negatives, human-AI teaming, and representative test sets, and it calls for human intervention when a system cannot detect or correct its own errors.[2] Build email triage with NLP around those exceptions, or the automation will quietly misroute the mail that matters.

Classification accuracy is the operating constraint

Operations inboxes mix billing questions, coverage changes, vendor notices, complaints, legal holds, and social-engineering attempts. A classifier that is excellent at spam versus ham can still fail at that mix. An XLNet spam detector published in 2025 reported 98.69 percent accuracy on SpamAssassin and 98.92 percent on Enron, with matching F1 scores.[7] Those figures describe a two-class problem on well-studied corpora. They do not describe a shared mailbox that must distinguish an address change from a coverage endorsement, or a payroll update from an exposure change.

Travelers formulated inbound service email as a 13-class text classification task. The labeled set held more than 4,000 emails, and about 25 percent carried PDF attachments (mostly ACORD forms) that the pipeline parsed into the prompt. Initial foundation-model testing without prompt engineering produced 68 percent accuracy. After class definitions, few-shot examples, category condensation, and document processing changes, accuracy reached 91 percent (90 percent on a smaller, faster model).[4] That still leaves nearly one message in ten for a person.

Academic work on intent, rather than spam, shows why a single accuracy figure misleads. A 2024 Engineering Applications of Artificial Intelligence study used pre-trained transformers in zero-shot and few-shot settings to map email text to intents and then to process activities. Task-mapping F1 reached 0.79. Intent-classification F1 was 0.55, because several intents collapse into the same next action.[5] Few-shot labels (between three and seven examples) improved both tasks relative to zero-shot.[5] For operations, that gap is the design brief. Route on the action when the action is stable. Hold the intent when the intent is ambiguous.

NIST, citing ISO/IEC TS 5723:2022, defines accuracy as closeness of observations to true or accepted values, and it requires false-positive and false-negative rates, human-AI teaming, and test sets that match expected use.[2] A blended 91 percent on 13 classes can hide a class that is unusable. Measure precision and recall per class, and report them by action: auto-route, review, or escalate.

Route with confidence, not with a single label

Email triage with NLP should emit a label, a confidence, and a recommended action, not a silent folder move. High-confidence, low-harm classes (order status, password reset receipts, routine vendor acknowledgments) can auto-route. Medium-confidence mail should land in a review queue with the suggested label, the top alternatives, and the phrases the model used. Low-confidence mail, executive senders, regulated products, and anything tagged legal, safety, or suspected phishing should escalate. NIST MAP 2.2 requires documented knowledge limits and a record of how humans will use and oversee outputs. MAP 3.5 requires defined, assessed, and documented human-oversight processes.[2]

The review queue is not a failure of the model. It is the product. Operations mail is full of mixed-purpose messages: a coverage question that is also a complaint, a form that does not match the body, a forwarded thread with three requests. Travelers' pipeline improved when categories were condensed and class definitions were written with business experts.[4] Start with a taxonomy operations already uses, then split a class only when reviewers keep relabeling it.

Shared mailboxes in Charlotte, NC and satellite operations groups in Raleigh, NC, Asheville, NC, and Philadelphia, PA often share a taxonomy and disagree on edge cases. Capture those disagreements as labeled examples. Three to seven well-chosen shots per class are not a toy setup. They are the amount of supervision the 2024 transformer study found useful for intent and task mapping when labeled mail is scarce.[5] Keep the examples current. A taxonomy frozen in January will drift by September.

Draft replies. Hold the send.

Drafting is the step that turns a classifier into an operations system, and it is the step that most often needs a hold. NIST's July 2024 Generative AI Profile defines confabulation as confidently stated but erroneous or false content (the colloquial "hallucination"), including outputs that diverge from the prompt or contradict earlier statements in the same context. The profile flags the risk that users will act on that content because it sounds sure, and it names automation bias and over-reliance under human-AI configuration.[3] A drafted reply that invents a policy number, a ship date, or an opt-out status is not a productivity win. It is an incident.

Constrain drafts to facts the system already holds: ticket fields, CRM records, last invoice, published policy language. Leave a placeholder rather than guess. Require a reviewer on any draft that would change money, coverage, employment, or legal position. NIST's 2023 framework is explicit that risk management may need human intervention when the system cannot detect or correct errors, and that explainable systems are easier to document, audit, and govern.[2] Show the reviewer the source sentence, the retrieved fact, and the proposed sentence side by side.

Do not auto-send a draft that would turn a transactional reply into a commercial pitch. The FTC's CAN-SPAM guidance covers all commercial messages, including business-to-business mail. Each violating email is subject to penalties of up to $53,088. Subject lines must reflect content. Headers must identify the sender. Commercial messages need a working opt-out, honored within 10 business days, and the mechanism must remain available for at least 30 days after send.[1] 15 U.S.C. § 7704 also requires a valid physical postal address and a clear identification that a commercial message is an advertisement or solicitation (unless the recipient gave prior affirmative consent).[9] A triage model that rewrites a service confirmation into a promotion can create a compliance event at inbox scale. Classify outbound drafts the same way you classify inbound mail: transactional, commercial, or mixed. Hold mixed.

Escalate phishing as a first-class class

Phishing is not a leftover "other" bucket. It is a class with a different action: do not reply, do not click, escalate to security. A June 2025 University of Oslo study evaluated large language models on intent-based phishing detection using only subject and body text (the view a person has in the inbox). Larger models exceeded 95 percent detection accuracy. When asked to sort phishing into MITRE-aligned categories (link, attachment, service), category accuracy on the three strongest models ranged from 76 percent to 95 percent, depending on the prompt.[6] Few-shot examples helped some models and hurt others, which is a warning against treating prompt recipes as portable.

MITRE ATT&CK T1566 frames phishing as attachment, link, or service (a lure that moves the victim to phone, SMS, or another channel).[10] Put those three plus "suspected social engineering" in the taxonomy. Never auto-draft a reply to a phishing class. Never route it to a customer-service queue because the subject looks like a password reset. Pair the NLP label with authentication results (SPF, DKIM, DMARC) already on the message. Text-only intent detection is a complement to metadata filters, not a replacement, which is the point of the Oslo experiments.[6]

Keep an audit trail that can answer "what happened"

If a regulator, a client, or a director asks why a message was auto-replied, the system must answer without a scavenger hunt. Microsoft 365 message trace follows a message as it is received, rejected, deferred, or delivered, and records actions taken before final status. Summary traces cover recent mail. Extended reports can reach 90 days of historical data, with events such as receive, deliver, fail, DLP rule, and transport-rule matches.[8] That is the mail-flow layer. Email triage with NLP needs a second layer: model or prompt version, taxonomy version, predicted class, confidence, retrieved facts, draft text, reviewer identity, and send or no-send.

NIST treats accountability and transparency as trustworthiness characteristics. Transparency is information about the system and its outputs for people who interact with it. Accountability presupposes that transparency. Documentation, in the GOVERN function, is meant to improve human review and bolster accountability.[2] MEASURE requires testing before deployment and regularly in operation, with uncertainty estimates, benchmark comparisons, and formal reporting.[2] For an inbox system, that is a weekly scorecard: volume by class, auto-route rate, review-queue age, per-class precision and recall, override rate, phishing catch and miss, and time from receipt to first human action on escalations.

Do not store raw mail in a logging bucket that outlives your retention policy. Store the decision record. Redact or hash message bodies if policy requires it. Keep the Network Message ID (or equivalent) so the NLP record can join to mail-flow trace.[8]

Practical takeaways

1. Treat email triage with NLP as four actions (classify, route, draft, escalate), each with a different risk of being wrong.
2. Do not quote spam-corpus accuracy as operations accuracy. Measure per class, and split metrics by auto-route, review, and escalate.[2][4][7]
3. Write the taxonomy with the people who already work the mailbox. Condense classes when reviewers cannot tell them apart.[4]
4. Use confidence thresholds. Low-confidence and high-harm mail always go to a person. Document knowledge limits (NIST MAP 2.2).[2]
5. Few-shot examples (even three to seven per class) are worth collecting. Keep them current.[5]
6. Draft against retrieved facts. Hold send on money, coverage, employment, legal, and any commercial rewrite. Confabulation is a named generative-AI risk, not an edge case.[3]
7. Honor CAN-SPAM on any draft that is commercial: honest headers and subjects, physical address, working opt-out, 10-business-day processing.[1][9]
8. Make phishing a first-class class. Do not auto-reply. Pair text intent with authentication results.[6][10]
9. Join NLP decision logs to mail-flow traces so a review can reconstruct receipt, classification, draft, and send.[8]
10. Retest in production. Accuracy on last quarter's labels is not a guarantee on this week's vendors, forms, and lures.[2]

How we can help

We design email triage with NLP as operations infrastructure: a taxonomy your team already recognizes, a classifier measured on your classes, routing rules that leave exceptions in a review queue, draft generation that cites source records, and an audit trail that joins to mail flow. That work sits with our AI business tools practice. We are based in Charlotte, NC, and we support operations and technology leads whose shared inboxes span Raleigh, Asheville, and Philadelphia. We do not treat a 91 percent score as a finished system. We treat the other 9 percent as the part of the design you can defend.

Have more questions or want to get in touch?

https://ideaforgestudios.com/contact-us-idea-forge-studios/ · (980) 322-4500 · [email protected]

Citations

1. Federal Trade Commission, "CAN-SPAM Act: A Compliance Guide for Business" (2024)
2. National Institute of Standards and Technology, "Artificial Intelligence Risk Management Framework (AI RMF 1.0)" (2023)
3. National Institute of Standards and Technology, "Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile" (2024)
4. Amazon Web Services / Travelers Insurance, "How Travelers Insurance classified emails with Amazon Bedrock and prompt engineering" (2025)
5. Khandaker, Senderovich, Zhao, Cohen, Yu, Carbajales, and Chan, "Transformer models for mining intents and predicting activities from emails in knowledge-intensive processes" (2024)
6. Eilertsen, Mavroeidis, and Grov, "LLM-Powered Intent-Based Categorization of Phishing Emails" (2025)
7. Shrestha, Oluoch, Sun, et al., "An accurate spam email detection mechanism using XLNet" (2025)
8. Microsoft Learn, "Message trace in the new EAC in Exchange Online" (2025)
9. Legal Information Institute, "15 U.S. Code § 7704 - Other protections for users of commercial electronic mail" (2003)
10. MITRE ATT&CK, "Phishing (T1566)" (2025)