Say you have to choose only one among the following authentication types for your own SMTP server:


LOGIN, PLAIN
CRAM-MD5
DIGEST-MD5
NTLM/SPA/MSN


Which one would you recommend for optimal security?

PS: The list is the authentification types given in man swaks

With SSL/TLS it's okay to use LOGIN / PLAIN.

You should provide SMTP on top of an SSL-encrypted connection. While some schemes from your list (e.g. DIGEST-MD5) can keep a password secure even over an untrusted channel, they won't protect users from a man-in-the-middle attacker tampering with their session. (Commonly, email servers wrap SMTP via direct TLS or a connection upgrade with STARTTLS at the ports 465/587.)

Any SMTP auth type, regardless if you usePLAIN or an advanced method, just provides application level authentication. But what you want is transport level security. After a user is authenticated over SMTP, there will be no automatically encrypted connection. Per the SMTP protocol, commands and emails are exchanged with the server in plain text, allowing a man-in-the-middle attacker to read and modify the communication and inject new commands. That's why you should provide it on top of SSL encryption, just like HTTPS provides HTTP on top of SSL.

The HTTP analogy: If you secure your website with HTTPS, then it doesn't matter that the a login form actually transmits your password as a plain string in the POST body of the HTTP request, because the data transport is SSL-encrypted. Enabling CRAM-MD5 for SMTP is analogous to implementing a challenge-response scheme in Javascript before transmitting login credentials to a website. (You can occasionally see that technique in router interfaces which don't provide HTTPS but it's not very common.)

As for a real-life example, GMail is fine with offering LOGIN / PLAIN authentication (where credentials are sent in plan text) after having established a secure SSL connection:


$ openssl s_client -starttls smtp -connect smtp.gmail.com:587 
 ...
250 SMTPUTF8
EHLO foo
250-smtp.gmail.com at your service, [127.0.0.1]
250-SIZE 35882577
250-8BITMIME
250-AUTH LOGIN PLAIN XOAUTH2 PLAIN-CLIENTTOKEN OAUTHBEARER XOAUTH
250-ENHANCEDSTATUSCODES
250-PIPELINING
...


(As you can see, they also provide some methods you didn't list, e.g. XOAUTH2 for OAuth2 tokens which might be interesting if you're after passwordless authentication.)
