Why are people not receiving my newsletters?

We provide  this explanation to provide some education on what is unfortunately a complex issue.

While we've been working hard to resolve issues with Newsletter delivery, there are industry wide problems related to reliably sending bulk mail from small business web-sites. The issues are explained below.

About SPAM

SPAM is a major and growing issue for all internet services providers (ISPs).

Because SPAM is so prevalent, mechanisms to combat SPAM exist at many levels.

Mail originating from an e-mail address not in the recipients address book with suspicious information in the e-mail headers or with suspicious content can erroneously be categorized as SPAM.

Understanding Reputation Based Lists (RBLs)

For the past few years, most mail providers have relied on "RBLs". There are hundreds of RBL providers who routinely list internet servers suspected to be involved in appropriate activity.

For NeatWorx, maintaining our reputation with these providers is understandably important. We use a service called MxToolBox to regularly monitor our status with RBL providers. We also have an automated notification service that advises us if we become listed on an RBL for any perceived violation of their policies.

Searching the IP address for one of our servers shows (that at the time of this writing) we are "clean" with no outstanding issues. Occasionally we will show up on blacklists for reasons that we cannot control. For example if one of our customers enables an auto-responder to their e-mail we can become vulnerable to forged headers and become guilty of "backscatter" (and interesting read here). Not the fault of our customer, but evidence of the complexity of this topic.   Sometimes we've found ourselves to be blacklisted through no fault of our own simply because a third party has been up illicit activity and their IP address is in close proximity to ours. Sometimes blacklist providers (a lazy improper thing to do in our view) blacklist entire networks because they become suspicious there may be other "spammy" hosts in our network neighborhood.

(This is like saying that because you're neighbor evaded his taxes last year you should go to jail!) - I'll spare you my "rant" on this topic.

MXToolBox has a nice interface to check the most popular black list providers with a single click as shown above. The results below show the the NeatClubs server is not listed anywhere and that six of the 147 RBL providers are down or not responding.

Mail Providers Outgrow Blacklists

Just as we experience issues with Blacklists, so too does everyone else. The business practises of a few blacklists providers are becoming very dubious (in my opinion). We have had experience with providers admitting that they know we are not spamming from our servers, but requesting a fee or requesting a donation in order to be de-listed from their blacklist.

In other words - " we know you're 100%  innocent, but your bail stands at $1,000.00"

I'll avoid naming the company here because I don't have time for a frivolous law suit and don't feel the need to make enemies but it should come as no surprise that there are some very unethical companies on the web.

These kinds of issues are well documented and are increasing in frequency - some of the at the link below.

http://en.wikipedia.org/wiki/DNSBL#Criticisms

How are E-Mail Providers responding?

Well, the first thing most Canadian providers are doing is kind of "throwing up their hands" and realizing they just cannot possibly manage all this complexity. As a result, most Canadian service providers have outsourced their mail to companies with a deep competency in e-mail management. For our good friends in the U.S. this is certainly true of smaller companies stateside as well.

As a few examples and evidence of this assertion:

What does all this mean? These companies are all clearly excellent and world-class so this is not entirely a bad thing. The issue however becomes that when dealing with a particular provider, they may not able to directly help you troubleshoot e-mail issues because they may not actually understand the issues.  The out-sourced partner certainly does, but they may not.

A specific problem we are seeing with big e-mail providers

Mail providers are sophisticated and are constantly improving how they filter e-mail. Most are not so worried about a few improper e-mails getting through, but they do become very worried if their customers are flooded with tens of thousands of e-mails. Thus, many of the sophisticated mail filters we've encountered may let ~50 or so e-mails through within a particular time period, but if they decide the sender is suspicious they may stop accepting e-mail after this arbitrary limit.  Other providers will allow mail through as long as it is "paced". This makes it obvious that we are not engaged in nefarious activities like trying to overwhelm the recipients server. NeatWorx intentionally "paces" outgoing e-mail such that we send no more than 100 messages per 5 minute period and we pause in between to avoid being viewed as spammers.

Sometimes a NeatWorx customer sends a large volume of e-mail messages, and after a set number are received by a particular major e-mail provider the provider becomes nervous about the volume of messages and blocks us, affecting all customers on a physical server.

Why "Big is Good" when sending e-mail?

Major e-mail providers have good reason to be concerned about non fortune 500 e-mail senders. They don't know us, and they don't know our customers.  If a mail provider is delivering mail to their customers and trusting it is not SPAM, they like to know where it is coming from. A first line of defence is SPF records. (explained here). Some mail providers use SPF records and others don't. SPF records allow recipients of e-mail to query to domain's DNS record to see who the owners of the domain are sanctioned to send e-mail on their behalf. If you're sending address doesn't match the SPF record, you don't get by.

Another way to detect if someone is legitimate is to discern their IP address, translate that back to an internet domain name using a reverse DNS lookup and see if the name matches the sender. If they match, the ISP has increased confidence that the e-mail is "legit", but if they don't match, they may choose to block  the e-mail for a bunch of good reasons.

The problem with reverse DNS lookups

So after all this a pre-amble to where we get to the heart of the real issue. Increasingly we're seeing policies where mail providers after a threshold number of received e-mails will perform a reverse DNS lookup on the sender. This works well if you are sending e-mail from a large organization with their own dedicated servers like Microsoft.COM. If you are a NeatWorx customer running on our dedicated servers, a reverse lookup will result in a name like NeatWorx.COM, NeatWrx2.COM or NeatWrx3.com being returned rather than your own domain name. Because of this mismatch, the e-mail provider gets suspicious and starts blocking the e-mail. They understand full well that they may well be blocking legitimate e-mail, but they also know that they are not blocking anyone "big" as large organizations tend to have their own dedicated infrastructure such that the reverse looks will match the sending e-mail.

What's the solution?

We .. err... this is kind of the point! - there really isn't a single good solution because the issue has nothing to do with us - it is about ISPs being overly aggressive in SPAM filtering.

Option #1: Invest in a dedicated server

One way to solve it is to give every customer their own dedicated server so that the reverse DNS lookups will match when a lookup is requested. Problem is that the baseline cost of a well supported dedicated server is around $300.00 per month, and we incur significant additional work to manage each additional physical server. (Charging $200 per month and incurring costs of more like $1,000 per month is not a strategy to stay in business long! ) For those that need their own fully managed dedicated server we can certainly do this - our costs however will be in the range of $1,000.00 per managed server per month inclusive of all costs - Very reasonable for a very large company, but almost certainly too expensive for most clubs and assoiations.

Option #2: Get Comfortable with a Bulk Mailing Service

Another solution (we think this is more promising actually) is to make use of a Newsletter mailing service like Constant Contact or MailChimp.COM. Both of these services are excellent. MailChimp has the very attractive property that it is completely free for those with a small mailing list up to 500 recipients. We are currently working on an integration with MailChimp.COM so that we can dump mailing lists from the NeatClubs Members and Participant directories directly into a MailChimp.COM profile to simplify the sending of targeted newsletters.