Secure SMTP Relay Checklist: Test Authorization and TLS
Summarize with AI
A secure SMTP relay should reach production only after it proves which systems may submit mail, which sender addresses they may use, where they may relay, and whether the first hop requires encryption. Approve the relay when every allowed path works, every paired unauthorized path fails, and the logs identify the rule that made each decision. A successful delivery by itself proves too little.
Define the SMTP Security Boundary
This checklist covers the application-to-relay submission boundary and the relay's decision to accept or refuse a message. It does not prove inbox placement, configure SPF, DKIM, or DMARC, or show that every later server-to-server hop is encrypted.
SMTP submission and public mail exchange follow different rules. RFC 6409 says a message submission agent must reject a MAIL command by default when the session lacks authentication or another form of authorization, such as a protected network. RFC 3207 says a publicly referenced mail server cannot require STARTTLS for every connection to its own domain, while a private submission service can require TLS before accepting a message.
Write down the exact hop under review: the sending application, its source network, the relay endpoint, and the relay rule or connector expected to match. A test against a public MX endpoint answers a different question from a test against a restricted submission endpoint.
Use a Relay Security Acceptance Matrix
The matrix below turns policy into an approval record. Replace each general condition with the identities and domains approved for your system.
| Control | Approval condition | Denied-path test | Evidence to retain |
|---|---|---|---|
| Connection identity | Relay recognizes an authenticated account, certificate, or explicitly approved source | Connect from an unapproved IP or present an invalid credential or certificate | Source IP, authenticated identity, certificate name, matched rule, SMTP response |
| Envelope sender | Approved identity may use only its permitted `MAIL FROM` domains or addresses | Submit with an unauthorized `MAIL FROM` | Envelope sender, authenticated identity, rejection response, policy log |
| Recipient scope | Approved path may reach only its intended local or external recipients | Attempt an external `RCPT TO` from a path limited to local delivery | Recipient, relay decision, matched rule, rejection log |
| TLS state | The application-to-relay connection meets the documented TLS requirement and validates the intended server | Attempt submission without TLS; test an invalid or mismatched certificate in a controlled environment | TLS version, certificate result, endpoint name, SMTP response |
Authentication should identify the connection. Sender authorization should decide whether that identity may use the requested envelope sender. RFC 6409 allows a submission service to reject a MAIL FROM identity that lacks submission rights.
Do not authorize relaying from HELO or MAIL FROM alone. RFC 2505 warns that both values are easily forged and requires an MTA to control or refuse unauthorized relay use.
Run the Denied-Path Tests
Use a controlled test account, domain, and recipient. Do not test unauthorized relay by sending unsolicited mail or by probing infrastructure you do not own or have permission to assess.
- Unapproved source: repeat a known-good submission from an IP or network outside the approved range. The relay should refuse the message before external relay succeeds.
- Invalid identity: present a wrong credential, expired certificate, or certificate name that does not match the approved identity. Record the stage at which the session fails.
- Unauthorized envelope sender: keep the approved connection identity but change
MAIL FROMto an unapproved address or domain. The relay should reject the sender rather than trusting the visible From header. - External recipient from a restricted path: use an accepted sender but issue
RCPT TOfor a domain outside the local scope. The relay should refuse external delivery unless that connection has explicit relay permission. - No TLS: attempt the submission without negotiating the required TLS mode. The relay should refuse credentials and message acceptance on a TLS-required endpoint.
RFC 5321 recommends filtering relay use to known or identifiable sources and an appropriate 550 response for a policy rejection. Exact response text varies, so approval should depend on the denied outcome and its matching log, not one expected phrase.
Verify TLS and Credential Handling
If the relay uses SASL PLAIN authentication over TLS, the client must validate the server certificate and match it to the intended hostname before sending credentials. RFC 4954 says the client must not attempt PLAIN authentication when the certificate is missing, invalid, or mismatched.
Run the no-TLS and bad-certificate cases from the actual application or a faithful test client. Capture whether the client aborts before credentials leave the system and whether the relay records the refusal.
Keep the TLS claim narrow. STARTTLS protects traffic after negotiation. RFC 3207 describes how an active attacker can alter pre-handshake traffic and says implementations must be configurable to require successful TLS for selected peers.
Build an Approval Record
The reviewer should be able to trace each result without opening the relay's configuration screens. Store:
- endpoint and test date;
- application owner and relay owner;
- source IP and authenticated account or certificate identity;
- envelope sender and recipient scope;
- TLS mode, certificate validation result, and negotiated version;
- SMTP response, message ID when one exists, and relay-log reference;
- matched connector, rule, or policy;
- expected review trigger, such as an endpoint, certificate, source IP, sender-domain, or relay-policy change.
Vendor controls differ, but the acceptance questions stay the same. Use our Google Workspace SMTP relay guide or Office 365 SMTP relay guide for product-specific implementation details, then return to this matrix for approval testing.
Ready to Review Your Outbound Infrastructure?
We can map relay ownership, sending identities, denied paths, and evidence into an outbound system your team controls. Explore our services or book your free discovery call →.
Frequently Asked Questions
A strong positive reply rate for B2B cold email is 1.5–3%. Top-performing campaigns with tight targeting and personalized copy can hit 4–5%. If you're below 1%, it usually signals a deliverability or messaging problem, not a volume problem.
The safe range is 30–50 emails per inbox per day for warmed inboxes. That's why outbound systems use multiple inboxes (we use 80) to reach 40,000+ monthly sends while keeping each inbox well within safe limits. Sending more than 50/day from a single inbox risks spam folder placement.
Yes. The CAN-SPAM Act permits unsolicited commercial email as long as you include a physical address, an unsubscribe mechanism, accurate headers, and non-deceptive subject lines. Unlike GDPR in Europe, the US does not require prior opt-in consent for B2B cold outreach.
Domain warm-up typically takes 2–3 weeks. During this period, sending volume gradually increases while the email warm-up tool generates positive engagement signals (opens, replies) to build sender reputation. Skipping or rushing warm-up is the most common cause of deliverability problems.
Cold email is targeted, relevant outreach to a specific person based on their role, industry, or company, with a clear business reason. Spam is untargeted mass messaging with no personalization or relevance. The distinction matters legally (CAN-SPAM compliance) and practically (deliverability depends on relevance signals).
