Agent Mail Gateway

Agent Mail Gateway

Give every AI agent its own email address — without giving it the keys to the mailbox.

Agent Mail Gateway is a small self-hosted service (one Docker container) that sits between your AI agents and ordinary mailboxes on your existing mail server. Each agent gets one API key for exactly one mailbox. Through the gateway it can read and answer mail, send attachments and manage calendar invitations — but only with the people you allow, and every outgoing message is checked before it leaves.

Agents talk to it over MCP (Model Context Protocol) or a plain REST API. Mail arrives as Markdown instead of HTML, which saves a lot of tokens.

---

Why

Giving an agent the IMAP/SMTP password of a mailbox means it can read everything, write to anyone, and a single prompt injection in an incoming email can make it leak data or spam people. The gateway keeps the password to itself and enforces your rules on every request:

Features

Mailbox access

• List, search, read, mark, delete; attachments in and out — as text, base64 or straight from a received message, so agents never have to juggle base64 for a CSV.
• PDFs and images as text for the agent (get_attachment with format: "text"): PDF text layer built in, images via a vision model or local OCR (Tesseract).
• Reply (incl. reply-all) and forward — the gateway fills in recipients, Re:/Fwd: and threading.
• HTML → Markdown for reading, Markdown → HTML for sending.

Calendar

• Send, update and cancel invitations; they update cleanly in Outlook, Gmail and Apple Calendar.
• See who accepted or declined your invitations.
• Read invitations you receive and accept, decline or tentatively accept them.

Control and safety

• Allow lists per mailbox for who the agent may receive mail from and write to (name@domain or *@domain). Everything else is invisible to the agent and moved to Trash (or kept).
• Forged senders are rejected (SPF/DKIM/DMARC as checked by your mail server).
• Attachment and content checks on incoming mail, each set to hide, flag or off and skippable per sender: dangerous files (programs, macros, disguised file types — on by default), virus scan with ClamAV, and a prompt injection check of text, PDFs and images (details below).
• Send limit per hour, audit log of every send, rejection and deletion.
• No fast retries on wrong passwords, so your server's fail2ban never blocks the gateway.

Review before sending (details below)

• Built-in rules catch empty mails, attachment-only mails, "see attached" without attachment, leftover placeholders and accidental duplicates — on by default.
• Optional LLM review (e.g. a local model in Ollama) for "is this mail complete and sensible?".
• Optional policies: your own rules in plain language, for everyone or for specific recipients — e.g. *"never share financial information"*, *"never mention gifts to Alex"*.

Integration

• MCP over Streamable HTTP, a stdio bridge for stdio-only clients, and a REST API with OpenAPI docs at /docs.
• Signed webhooks when new mail arrives (compatible with Hermes Agent webhook routes).
• Hermes Agent plugin with a skill that teaches the agent how to use its mailbox.
• Works with any IMAP/SMTP server: Plesk, IONOS, Outlook, Postfix/Dovecot, …

How it works

 Agent ──MCP / REST + API key──▶ ┌────────────────────────────────────────┐

│ Auth key → exactly one mailbox │ Agent ◀──signed webhook──────── │ Policy who may write / be written │ │ Review rules · LLM · your policies │ │ Inbound file types · virus · inject.│ │ Converter HTML ⇄ Markdown │ │ Calendar invitations and replies │ │ Mailbox IMAP + instant new-mail push│──IMAP──▶ your mail server │ Sender SMTP + copy to "Sent" │──SMTP──▶ └────────────────────────────────────────┘ config.yaml + .env (read-only)

Mail stays on your mail server; the gateway only keeps a small SQLite file with its own state.

Quick start

Use a dedicated mailbox for each agent (for example assistant@yourmailserver.eu), not your

personal one: by default, mail from senders that are not on the allow list is moved to Trash.

1. Get the files:

   mkdir agent-mail-gateway && cd agent-mail-gateway

curl -LO https://raw.githubusercontent.com/dominikamann/agent-mail-gateway/main/docker-compose.yml curl -L -o config.yaml https://raw.githubusercontent.com/dominikamann/agent-mail-gateway/main/config.example.yaml curl -L -o .env https://raw.githubusercontent.com/dominikamann/agent-mail-gateway/main/.env.example

2. Edit `config.yaml`: one entry per agent with its mail server, login and allow lists.

1. Put the secrets into .env (openssl rand -hex 32 makes a good API key).
2. Start it and check:

   docker compose up -d

curl http://localhost:8080/health # {"status":"ok"}; with a key also that mailbox's state curl -H "Authorization: Bearer " http://localhost:8080/v1/mailbox

A minimal mailbox entry:

mailboxes:

• name: assistant address: youragent@yourmailserver.eu api_key: ${AGENT_API_KEY} imap: { host: mail.yourmailserver.eu, port: 993, security: tls } smtp: { host: mail.yourmailserver.eu, port: 465, security: tls } username: youragent@yourmailserver.eu password: ${AGENT_MAIL_PASSWORD} allow_receive_from: [you@yourmailserver.eu, "*@yourcompany.eu"] allow_send_to: [you@yourmailserver.eu]

Every option is explained in docs/configuration.md.

Review before sending

Agents are sometimes sloppy — an email with only an attachment, "please find attached" without a file, Hello {name}. The gateway checks every outgoing email and invitation before it is sent. If something is wrong, nothing is sent and the agent gets a clear error with the reasons (review_rejected), so it can fix the message and try again. Rejected attempts don't count towards the send limit.

There are three layers; you choose per mailbox:

Rules and the LLM review can block the message, only warn (send anyway and report it), or be switched off; each policy rule either blocks or warns. The check covers new mails, replies, forwards, invitations and their changes, and comments in invitation answers.

    review:

rules: block # block | warn | off llm: # any OpenAI-compatible endpoint, e.g. a local Ollama url: http://ollama:11434/v1 model: llama3.1:8b mode: warn # quality check: warn | block | off policies: mode: block # what a violation does: block | warn rules:

• rule: Never share financial information such as revenue, prices, invoices, bank details or salaries.
• rule: Never mention gifts, presents or surprise plans. recipients: [alex@yourmailserver.eu]
• rule: Never send calendar invitations. recipients: [sam@yourmailserver.eu]

Policies also read attachments: text files, PDFs, and images when attachments.read_images is set. A rule with recipients applies only when one of those people receives the message (To, Cc, Bcc or invitation attendee). With Ollama the review runs entirely on your own machine; no mail content leaves your server. Long messages are checked in parts — nothing is cut off; part size and limits are configurable to fit your model's context window. The built-in review prompt can be replaced or extended — see docs/configuration.md.

Incoming mail checks

Allow lists decide *who* can reach your agent; these checks look at *what* arrives. For each check you choose what a finding does: hide the message (like a sender that is not allowed), flag it (the agent sees it with a warning and cannot download dangerous files), or off. With skip_senders a check is skipped for trusted senders — so you can still send your agent instructions by mail.

    attachments:

clamav: { host: clamav, port: 3310 } # optional, see docker-compose.yml read_images: { engine: ocr, languages: [eng, deu] } # or a vision model inbound_checks: prompt_injection: action: flag skip_senders: [you@yourmailserver.eu] # your own instructions are not flagged dangero