Is trying to fix (E)SMTP really worth it? [part 1]

[digg=http://digg.com/security/Is_trying_to_fix_E_SMTP_really_worth_it_part_1]
This one question has been in my mind for quite some time already. I mean, everyone uses SMTP (knowingly or not) when sending out emails and everyone sending emails also knows what SPAM is and receives SPAM messages.

However, few know how old SMTP actually is, and that, even though it serves everyone well, it has been designed in a time when everyone was thinking of Spam as canned meat. Back in 1982 SMTP was a great achievement and a lot of kudos should go to its creators, but now, in 2008, SMTP has become more of a liability than a great tool.

Originally, I wanted to write a single article covering all shortcomings of SMTP and possible solutions to these problems, but while writing the article a lot of text came up, so this is the first of two articles I am going to write on this topic. The first part is about the problems with SMTP and how fix-ups for SMTP are, even though they do work to some extent, a proper solutions to today's issues.

Due to the way SMTP was designed and the way the Internet was back then it is prone to various things, like SPAM messages, sender spoofing, data manipulation and so forth. A few attempts have been made at fixing some of the shortcomings of SMTP, like ESMTPA (SMTP-AUTH) or SPF, Callback Verification, and DKIM, but none of them has really fixed all problems that exist and all of these modifications are in my opinion mere workarounds. Let us have a look at why both SPF and DKIM fail to fix the all problems SMTP has right now.

SPF

First let's have a quick look at how SPF works: When an email is received a special TXT DNS record at the sender's domain is used to verify that the sending computer (using its IP address) of an email is actually allowed to send email for a domain. A great mechanism that in theory would work perfectly well. Reality is a bit different though.

There are some domains which are actually using SPF and do have valid SPF records on their DNS servers. However, those are only some of the millions of domains on the Internet. How should one treat emails coming from a domain without SPF record? The messages could be real, non-SPAM messages, that should be delivered, but on the other hand these could be SPAM messages. Also, the more people start using SPF, the more likely it becomes that spammers are simply going to use sender-domains which do not have SPF records.
Also, there are some organizations that have domains with improperly configured SPF records and there are even well-known ones, such as Technorati (I covered this in one of my articles). So one cannot even trust SPF records and valuable messages could be lost if a mail server is configured to drop all messages for which SPF authentication fails.

And there is a third problem: Sending emails from other places than your default one (office, home, etc.) and ISPs not allowing external users to use their SMTP servers (even not with authentication). A good example of this would be Austrian ISP UPC (their SMTP server tells me that the AUTH extension was not advertised, even though it was; long story short, I cannot log in from outside) and I am quite sure there are a lot of others.

And I can come up with yet another problem: What about email relaying? Think about downloading all messages from all your email accounts into a single one, using fetchmail for example. This makes SPF useless, as no checks can be done anymore, due to the sending system's IP address not being the original sender. If one assumes that every mail server uses SPF this is not a problem, but I like doing my checks on my server rather than relying on some other server.

Maybe there are even more problems with SPF, such as what to do when an email is received from a nonexistant domain or when there is a temporary DNS failure on either side, but the ones listed above are those I am confronted with most often.

Callback Verification

On to the next topic. Callback verification is a simple method used by mail servers to try verifying that the sender actually exists. Whilst this works for some SPAM messages which use non-existent senders, it does not help much as soon as the sender address does exist, and it does not even matter if the message was actually sent by the user owning the address. I guess there is nothing to add, even though it is a nice method to get rid of some spam it does not help with a lot of such messages.

DKIM

DKIM, or DomainKeys Identified Mail (originally named DomainKeys) is a method that is not meant to prevent abuse (such as SPAM), but rather to make tracking abuse easier. It works by the sender (usually the MSA of the sender on behalf of him) adding another header, "DKIM-Signature", which contains a cryptographic signature of the message body. The signature is generated using public-key cryptography, where the public key is stored in a DNS TXT record and can thus be used by the receiving end to verify that the message contents have not been tampered with during transport and that the mail actually originated from that domain.

This method, even though being one of the most advanced ones today, is prone to replay attacks and does not protect from tampering with message headers. In short this means that even though the message body cannot be modified without the receiving end detecting the modification, the headers can, and thus the message can be redirected. Also, it is possible to intercept the transmission of a message, generate a thousand messages with the same content but a different recipient and this way flood a mailbox with a message that would stand DKIM verification.

The email relaying problem that's present in SPF is not a problem here anymore, but the mobile-mail, the nonexistent domain and the DKIM-not-in-use problem still exist.
Also, DKIM seems not to be used by a lot of email servers on the Internet. Thinking about it for a second I can just come up with two names of well-known organizations using DKIM: Google and Yahoo.

The message format

The next part of this article is about the SMTP message format. This part is not directly related to the SPAM problem, but should provide you with some more information that verifies that SMTP is outdated nowadays.

RFC822 messages (or emails) usually consist of two parts: a message header and a message body. Originally these messages were designed to contain 7bit-encoded ASCII data, which is plain text. This means that there were only 128 different characters which could be transferred via email, without support of special characters, like German "Umlaut" characters. A solution has then be developed, not only to support special characters in emails, but also to support transferring of binary data (such as images).

MIME is the name of this solution, and it enables every one of us to send binary attachments and special characters via email today. MIME allows the email client to include more than just 7-bit plain text messages, including attachments. This is achieved by special header, "MIME-Version", which indicates that the contents of a message are MIME encoded. This header is then followed by a "Content-Type" header, identifying the type of content. For simple messages just consisting of a message body this would be "text/plain", telling the client that there is just text in the mail.

However, how can emails then consist of both text and attachments? Well, there is a special value for the "Content-Type" header: "multipart/mixed". This one indicates that there are several parts of a message, and every part comes with a separate "Content-Type" header. This way contents of a message can be organized in a tree, for example, containing the message body and a forwarded message.

An attachment is added by specifying an additional part of the message, usually with a "Content-Transfer-Encoding: base64" header, that says that the data has been base64 encoded. This way binary data can be represented using 7-bit ASCII.

But what does that mean? First of all, even though a message is split into several parts, there is only one body. Now if you are downloading a message via POP3 for example, there is no way of only downloading the actual text. You always need to download the whole message. Everyone knows this situation: You are downloading a message and have to wait for all attachments to be downloaded, even though you might not be interested in those attachments at all.

Also, encoding binary data using base64 creates a lot of overhead, as every byte (which can have 256 different values and corresponds to a single letter of text in 7-bit ASCII) is represented using only 64 possible values. Talking numbers here this means that messages encoded using base64 are usually 137% the size of the data they contain.

Conclusion

I hope that I have shown you what the problems with SMTP are right now. SMTP initially was designed to transport only text and had no way of verifying either the sender of a message or the integrity of data. Some workarounds have been created to get rid of these issues, but even though some helped a lot, none has really fixed any of those problems.

Also, one should never forget how much time and money has been used to try fixing SMTP, whilst a lot less money might have been sufficient for creating something new, something better, something that is built for the needs of the Internet today, and not for the needs of the Internet back in 1982.

Personally I believe that the days of SMTP are long over and that there is need of a proper replacement. I do understand that SMTP and the current email infrastructure are still in use because an infrastructure exists, but SMTP really deserves being retired, after serving us pretty well for more than 25 years.

The next part in this series of articles will be about what my idea of a successor of SMTP and the whole email infrastructure is, what it could look like and also how it could work. So stay tuned.