Wes Cutshall

One of the biggest problems I’ve run into as a commercial website owner is consistently delivering emails generated by my websites. The issues in doing this have been various and at each step of the way I’ve learn something that helps us to to deliver messages there on out.  But each lesson comes from a shortcoming. Hopefully I can spell out some of the things I’ve learned that have helped to increase email deliverability for our websites.

The Number One Problem

The number one problem that affects email delivery is the server from which the emails originate. Or that server’s “reputation” at the time that you send your emails. If you run your website on a shared hosting environment, you will have problems getting emails delivered consistently to your users. When you have no control over what emails are coming from your email server’s IP address you put your own deliverability results in jeopardy. One of the misunderstandings we had along the way was that having a dedicated web server meant that we had a dedicated email server. Like many programmers we simply used PHP’s mail feature and never really knew where the emails went from there. Although our website was delivered from a dedicated server, all outgoing emails were routed through a small collection of email servers. It’s not a problem that your email server has a different IP address than your website, it’s just an issue that you can’t keep Joe Schmoe from sending spam, or just sending too much email – which looks like spam, through “your” (shared) email server. The webhost in question did not allow us to run our own email service on our web server, so we had problems using their email server until we later moved to Rackspace. I’m a programming, and not a network admin guy, so I don’t know even know how to set up a Linux/Unix web server to send mail, I was just told it was not allowed with our webhost, even though we had a dedicated server. With this host it was much more of a managed server, and we could do a lot less than we can do now at Rackspace. Not that this type of hosting bad, it’s just not ideal for what we needed. So during the time that we had a dedicated web server but not email server we ended up using a 3rd party email service provider (ESP) to help get emails delivered more consistently. This definitely helped, complaints were down, but it wasn’t the ideal solution, and costs more than sending emails strictly through your own server. The very best scenario is to either have your own separate email server (overkill) or have your web server box configured to be your own email server. This way you have control over your server’s reputation with the ISP’s and email providers of the world.

Reply To?

This tip is one that seems a little ridiculous, but I tested it and it’s completely true. When generating emails programmatically I never thought to populate the “ReplyTo” field. I just thought that email clients reply to the From address unless the ReplyTo happens to be filled out. So why would I ever put something in the ReplyTo property? It turned out that with one major email service provider if I sent an email without the ReplyTo specifically set to the same address as the From field, it would send the email to the spam folder. Weirdest thing to me, but when I discovered this I just changed all of my code to add the From address as the ReplyTo address. This kept emails from going to the spam folder for this one major email provider.

DNS Entries

A few things at the DNS level can be done to help ensure email delivery. I don’t know these settings, or entries, well enough to deeply discuss setting them up, since I set them up once with Rackspace based on articles I read, and haven’t touched them since, so I’m just going to touch on the concepts to guide you in the right direction.

1. One of those settings is reverse DNS lookup entry. It’s one thing for Example.com to resolve to 98.129.58.195 when you attempt to visit the website, but if an email server gets emails from 98.129.58.195 that claim they are from example@example.com, how can it test to see if the sending server really hosts Example.com? Reverse DNS lookup. I won’t attempt to discuss the technical details, because I don’t know them, but be sure that your DNS records have reverse DNS set up so that email servers can verify that your emails are coming from the correct server.
2. Another DNS record that can help email delivery is an SPF (Sender Policy Framework) record. The idea behind an SPF record is for your domain server to be able to be queried and asked for a list of email servers that are allowed to send email for your domain. Even if you’re sending emails from the same server IP address, it helps to project to the email servers out there that your emails are allowed to be created from this IP address for your domain name.
3. The third DNS record that can help with delivery is DomainKeys. This was a little more complicated to set up, but luckily Plesk took care of all the hard work. DomainKeys is a more sophisticated way of identifying that a specific message was sent from a specific server. DomainKeys works in a public-key scenario, so a key is sent in the email that can be verified by querying the DNS server. Google it, or contact your webhost to find out how to set this up. It will be a combination of server settings and DNS entries.

Abuse@ and Postmaster@

Another thing that email servers can detect is the existence of certain email addresses. It’s a common tip that having a valid abuse@example.com and postmaster@example.com helps to deliver emails. This one I can’t vouch for, but I do have them set up because I’ve read it multiple times.

Handle Bounced Emails

One thing that can make you look like a spammer is sending emails to bad email addresses. Whether they never existed or are expired, it just looks bad. You can minimize this by handling bounced emails correctly to keep for resending emails to bad addresses. The way we do this is to:

1. Set the Return-Path value in your email headers that contains the email address being sent to for each email. In doing this your giving yourself a way to know where an email was sent when it bounces back to your saver. An example of what you can put in the Return-Path header field is user=yahoo.com@example.com. It’s the user’s email address in front of the @example.com with the “to” address’s @ sign replaced by “=” to keep it valid. So an email to user@yahoo.com has a return path of user=yahoo.com@example.com.
2. Have a default email address handle all mail to invalid email addresses for your domain (user=yahoo.com@fidofinder does not exist). So we have all emails sent to an invalid address funnel to bounce@example.com.
3. Pipe the Return-Path “address” to a script that can process them, most likely adding them to a database table of emails to not send to. I have a PHP script that can do this for qmail. Contact me if you need it. It just reads the email left of the “@” and replaces the “=” and puts it in a table. The code is a little weird as it has to read the stream of data being piped to it. It took me a while to find the correct code to do this.
4. Obviously, you now consult the database table when sending emails to stop sending emails to addresses that have bounced in the past. This again just prevents you from looking like a spammer.

Timing and Volume

Two things that affect if your valid email looks like spam is the timing of the email and the volume. Large email providers know if you are sending the same email to thousands of addresses. And even if this is to registered users of your website, it can look spammy. Avoid this by sending emails in spurts. If you need to email 100K users write a script to do this a little at a time throughout a day. And avoid sending emails between midnight and 5 AM, even though it’s the choice of most programmers to do maintenance tasks at low-volume times. Spam too is mostly sent during these hours. Another tip is that if you want your emails to not appear to be spammy, send more emails on a regular basis. I’ve always felt that sending fewer emails would be better, but I’ve seen suggestions of the opposite.

Unsubscribe and Postal Address

They say that the major email providers can tell if you included an unsubscribe link at the bottom of your email or not. Whether they actually do this I do not know, but since I’ve read that they do, I include it. Don’t assume that the URL has to go to a fancy automatic unsubscribe system. It’s fine if it goes to a page where a user can sign in to change their settings or delete their account. And since in some states it’s required to have a postal address included in email marketing, throw your postal address at the bottom as well. I’ve read that this too is detected by the major providers.

That’s It!

Well there you have it. There are probably other tips, but those are the ones I’ve learned over the past 6 years. It’s been a huge learning process, and a thorn in my side at times, but currently emails are delivered at a fairly good rate which I can only measure by a low volume of user complaints.