Annalivia Ford

-44 of you think I'm insane for suggesting that bacon doesn't make everything better. Of course it makes everything better. What are you, mad?
-8 of you don't like bacon (Really? Is that possible?)
-11 of you think it should only be eaten for breakfast. With what else, I wonder?
-44 of you demand MOAR BACON!

(Get your bacon right here: Nueske's applewood smoked bacon is a wonderful thing and I have some here, with which I will celebrate the closing of this poll. I figure their marketing scheme has got to be bulletproof - anything that has bacon like this in it cannot fail.* )


...what should the next poll be?


*This is not an official endorsement. I have never run Nueske's numbers so I don't truthfully know. I simply love this stuff. I'm just sayin'. It's BACON.

One really, really good way to shoot your IP reputation in the foot is to send marketing mail over a transactional IP - cue the age-old argument over what constitutes transactional mail!

Bottom line though, is when it comes to IP reputation, (recipient end-user) perception is reality.

Have an excellent weekend, folks. Bring on the snow!

Some folks have been reporting issues with extra text breaking ARF processing. Official update here.

I wrote a post shortly before Christmas that discussed my irritation with a business that was sending me emails that were telling me to hurry, Hurry, HURRY because I had a very limited time to save, Save, SAVE! Bad enough that I got these shrieking subject lines delivered to me once a day for weeks. When the frequency rose to three emails in one morning, I got so annoyed I unsubscribed (and haven't resubscribed since). I had regularly bought items from them, and had been on their mailing lists for several years. I even recommended them to other people. That company lost a client, and also lost my word of mouth advertising. I now use them as a poster child for how not to do a Christmas campaign.

Mark Brownlow recently wrote an article about the topic of urgency. My take-away from his post is that the strategy of "getting the immediate sales and we'll worry about the rest later" is not sustainable. I agree. I also believe that the current methods some senders use to get around the system instead of changing with it will not work much longer and will in fact have very negative repercussions. This will be a very interesting and challenging year in the email marketing scene. We are, I think, at a tipping point.

Which side will your email operation come down on?

Ah, the joys of reputation systems!

Businesses that have relied for years on staying under the radar are quite visible these days. Those masses of marginal email that flow in, hiding behind the worse streams that draw off human attention? We can see you now. Hi!

I believe that the general receiver focus is shifting to eliminating the above class of mail. It costs a lot of money to transport, filter and store it, and most of that cost is borne by the receivers. My speculation is that across the board, at all major ISPs, the historical tolerance for marginal mail will rapidly evaporate. ISPs are tightening things down. Anti-spam vendors are stepping up and offering new products that I hope will prove very effective. I expect this trend to accelerate dramatically.

Engagement and relevance will be increasingly critical in getting mail delivered - not just to the inbox, but at all. That translates not just to well thought-out marketing and excellent targeting, but also to list hygiene and client vetting. The particulars of reputation systems do vary from ISP to ISP but fundamentally, the successful mailers in the long term will be the ones who do the most thorough and best due diligence before letting (desired, relevant, and targeted to an engaged audience) email leave their network. This isn't news, really. It has always been true, but it now more crucial than ever. I was telling people to Get Ready, that Reputation Was Coming and would change everything...at least 2 years before it started to roll out. It's gathering momentum rapidly now. The paradigm is changing very quickly, and those who don't change with it will be left behind.

It's evolution in action. Darwin would have been thrilled.

Parrots repeat things over and over again, and so do deliverability consultants. They repeat themselves in the hope that someone will listen. Parrots want crackers, but consultants want people to hear what they're saying. Laura Atkins' post from yesterday is very relevant. Folks, listen to her. She knows what she's talking about. Al Iverson, too. There are many good articles being published these days that address this problem:

The days of "what do I tell the ISP?" are long gone. The ISPs care about what bulk mailers do, not what they say. Reputation systems are a different paradigm from the old days of whitelisting and blocking.

It used to be that you could call up someone at an ISP, and if you were either honest enough or clever enough, it was possible to convince them to whitelist your IP, and then the IP in question had a free ride until someone noticed a Bad Thing and revoked the whitelisting. Hard blocking was done mostly by hand, which meant there was (relatively) not that much of it going on. A mailer had to do something pretty awful to catch the attention of a person and get blocked. The bad guys, predictably, ruined it; staying under the radar let mailers get away with very poor practices, and a great many of them did just that. The paradigm stopped working. The ISPs had to come up with a better way before their networks died and all their customers went elsewhere.

Reputation was born.

The wonderful thing about reputation systems from an ISP point of view is that once they are functional and right, they catch an awful lot more bad mail than mere people could. They're catching goodly portions of the enormous quantities of marginal mail that no-one really seems to want: pummeling the ISPs and their subscriber bases with huge quantities of "blah" mail won't work any more - they don't want to transport billions of emails that their members don't care about: it's expensive, both in overhead costs and in losing members to fatigue.

And here's where I'm repeating myself: reputation systems don't care about business models, protestations of opt-in, legitimacy, or urgency. They care about the response generated by a given stream of bulk mail. Whitelisting no longer provides bullet-proof protection from blocks; whitelisting is mostly dynamic and dependent on reputation. Spam-foldering is also now largely driven by reputation. It's a much more fluid environment, wherein how mail is treated can change by the moment, and in which is it much, much easier to drive reputation down than it is to bring it back up.

Don't worry about what to tell the ISPs. If your mail isn't being treated the way you want it to be, look at the mail! Figure out what is causing the issue, and fix it. If you can't figure it out, hire someone that can; there are ESPs and consultants who do it for a living and are very good at it.

(continued...)

Oh, what was happening?! I even used my other laptop to download the latest wireless card driver for this laptop, stuck it on a thumb drive, and updated the driver. Nothing. It obviously was my laptop at fault and not my network...but guess what I did next?

Right. I tore my LAN apart and rebuilt it. By then it was about 2AM, and I was completely freaked out: my precious laptop - my connection to everything outside my house - my work, my friends, my parents, the news, television, access to web-search...wasn't connecting. Oh, my God. I was really upset! I was swearing like a drill sergeant. I had yelled at my dogs and nearly kicked my cat, and I'm pretty sure I flung a shoe at the wall in a fit of enraged frustration.

4 hours had passed, and I had done everything I could think of that made sense, and a number of things that clearly did not, and I was no further along than when I opened the lid the first time. While I was sitting there pondering the problem - and panicking - I told myself to slow down and think. Think. THINK. What was the last thing I had done with the laptop prior to the problem? Why, I'd set it down, of course....but! It had caught on my sleeve! Then my eyes registered what I should have seen immediately: the little green WiFi light on the top part of my keyboard was not lit. ...O, RLY? I looked at the side of the laptop, and what do you know? Yeah. There's a nearly invisible slider bar on it that turns the wireless on and off. I flicked it back on, and everything worked just as it should.

"..."

Panic over, I sat there feeling incredibly stupid. All that, over a tiny little slider bar I didn't even know existed *and* when I had another, perfectly useful laptop? Really? Yes. Really.

Moral of the story: Techpanic is utterly irrational and can happen to anyone.

If you're still with me, here's the point of my post: Be nice to people who are caught in it. They're going to feel plenty stupid and embarrassed enough when it's over, without any help from anyone else. In fact, it is what makes a lot of those angry people in support queues...angry. They *know* they're doing something stupid or ignorant, and asking for help in that situation puts people - especially people who are specialists in computers in some way, or in some other field - on the angry defensive right out of the gate. I know perfectly well that if I had called support during that little episode I would have been aggressive and very unpleasant, primarily because I was really embarrassed.

Being familiar with the pathology of this particular emotional cycle can be really valuable when trying to defuse someone in that mood: in order to actually fix a problem, you need a cooperative user, and angry people just plain don't cooperate very well. Getting them past the fear, rage and humiliation into a functional mode is the key to getting them off your phone, out of your inbox, and out of your queue.

Final thought: The above is impossible if you, too, are perceptibly angry.

Truly final thought: Never trust a telephone's mute button. ;)


--
P.S. For those that did not get the references, the titles of this post and the previous one are riffs on the immortal Hitchhicker's Guide To The Universe. Douglas Adams, you are greatly missed.

The links to panic case studies that I included in my previous post are fascinating, and I encourage you to read them, even if they are long. The two from "asktog.com" were written by Bruce "Tog" Tognazzini, a guy who specializes in human-computer interaction. The cell-phone-sunrise one illustrates the very affecting level of panic that is often induced in the users of electronic contraptions. If this sort of thing can happen to people who are extraordinarily tech-savvy, imagine how your average end-user feels?

I'm not an average end user. I did top-tier tech support back in the Windows dial-up days when "work-flows" didn't exist - I sat under my call-center desk with my eyes closed and my head firmly jammed into a corner to keep it from exploding, and I could tell what was wrong with a modem by the sound of it. I've worked in a NOC, troubleshooting big circuits and token-ring networks. I physically built part of the Internet (granted, a really small part). I work on the Internet, I socialize on the Internet, I watch TV on the Net. My laptop is never out of reach - I used to sleep with the wretched thing when I was on 24/7 call for a very unstable network. I've provided hands-on and remote/phone Windows tech support to my family and friends for an unthinkable array of problems. I've built my own computers and set up my own LAN and VoIP phone. I think one can safely say I'm generally quite comfortable with computers and Internet technology. I am relating these details to set up what follows...

Check out what happened few months ago one workday evening when I opened my laptop - the one I'm typing on right now, that I've had for a long time - and the Internets failed to work. My first reaction was "Eh, my cable barfed, it happens..." so I went to reboot the cable modem. It came back up fine. I rebooted the WAP and the laptop just in case, and tried again. Nothing. "Hm," says I, "I wonder if something upstream from me is broken?" So I dug out the other laptop and fired it up. It connected just fine. "Uh-oh, " said my inner voice "This Is Bad! BAD!!"

...and that is when the panic set in. I distinctly remember the feeling of an icy wash sweeping over my body, followed by a hot flash. I was sweating, my breathing was short, and my hands had a fine tremor. Now, this was patently absurd. I had another, functional laptop! But that wasn't relevant to me right then: THIS one didn't work!! I opened up a command window, used ipconfig to flush everything. Nothing - and no new IP assigned, either. Oh, what was happening?! I even used my other laptop to download the latest wireless card driver for this laptop, stuck it on a thumb drive, and updated the driver. Nothing. It obviously was my laptop at fault and not my network...but guess what I did next?

(to be continued...)

As I go through my ticket queue I am often confronted with some pretty wacky stuff. When people lose their grips, the results can range from funny to disastrous. One of the tickets I worked today reminded me of this remarkable story from a few years back:

The City Manager of Tuttle, Oklahoma loses the plot. (you only need to read the first few to get the idea, I promise)

This is a perfect case-study of how bluster and threats can backfire in the most unexpected ways. Mr. Taylor, I am certain, never expected to become the laughingstock of the Internet overnight. The site got 516,147 hits total, and I remember that it got most of them within 48 hours of the original post. It spread like wildfire, and the howls of derision rang from one end of the world to the other. Emails were sent to his superiors, to his City Hall, to anywhere people could think of to spread the word to the town's inhabitants and the world that Mr. Taylor had made a serious tactical blunder. He himself was sent so much email that their mail server fell over, and his email address was removed from the city website, as was his photo shortly thereafter. The Tuttle City webserver made a small sad sound, and died under the load. The tale made it to several large online newspapers, including The Register, in the UK. In a town with less than 5000 inhabitants, it must have been a very uncomfortable time for the guy.

I feel a certain sympathy for him. His initial reaction was a classic panic response, which happens to everyone sooner or later. Where he made his mistake was getting into a land war in Asia...^H^H oh, wait - was in rapidly escalating the situation to threats of FBI involvement, instead of actually reading the emails being sent to him by the CentOS developer that detailed how to correct his issue. My sympathy largely evaporates right about there.

Panic is a natural human response to stress. Getting out of a panic-inducing situation generally involves taking a step back and using rational thought processes, even if you only have a split second to do it - say, your parachute fails to deploy at 3000 feet. Uh-oh! Now that's cause for panic! Mr. Taylor wasn't falling out of the sky at 120MPH, though. He was safely on the ground, sitting in front of a computer and freaking out about his town's website. He had the time to stop and think, but he didn't use it, and the end result was world-wide mockery that is archived for the public, in all its painful glory, to this day (and probably until the end of time).

I get this sort of thing a lot. People don't pause to read and think, or do a quick online search, or follow links provided in the bounce they got - they make assumptions, draw faulty conclusions, lose their tempers and come in swinging with blood in their eyes - or worse, hysterically weeping. One of my particular skills is talking this kind of person down out of the trees. I am almost always successful, but it's hard work, and I do wish people wouldn't do it to themselves - or me. Often when the panic is over, they wind up thanking me for trying to help them out, and if applicable, I explain how to avoid $FOO next time, then hang up hoping that they will indeed stop and think the next time.

Here's a handy (but non-exhaustive) guide to avoiding email-related freak-outs*:

• * Panic first. Think, second. Act, third or even fourth.
• * Follow links in error messages, and read what they have to tell you. If you don't understand what they say...
• * Web-search is your friend.
• * Back up important emails. Better yet, print them. Your inbox is not a bulletproof storage place.
  
• * For really crucial contacts, have a phone number, or at least a secondary email address.
• * Always double-check the "to" before you hit "send" on an email.
• * Don't put stuff in an email you would cringe to see published in the Washington Post.
• * ISPs do not have magic powers or time machines. If you deleted an email in 2006 and it's now critically important to a legal case, you are out of luck. That email is gone, never to return.
• * Don't ever believe an email that's offering you money, or telling you "confirm your account details or bad things will happen". If it sounds too good to be true, it is. No modern institution with a lick of sense will send you such a confirmation email either. If you do business with one that is stupid enough to send an account-details-or-else email of the sort that phishers love to use, fire them.
• * Always check the file extension of an attachment. If it's a .exe, proceed with extreme caution. Like, don't open it, pick up the phone and call that number I told you to have for your contacts, and ask them if they meant to send you an executable file.
• * If you do fall for an email with an .exe in it, DONT REBOOT. Back away from the machine and get help.
  
• * Abusing the person who is trying to help you is rarely going to have a good outcome for you.
• * Did I already mention backing up important emails and contacts? Yeah? Well, it's worth repeating. Back-ups will save your hide.

Include all pertinent information:

*IPs
*Error messages or detailed description of symptoms
*Headers and bounce if you have it..and if you don't:
*Sender & Recipient/Date/time/time zone for an example of problem/missing mail
*When the problem began.
*What you have done to solve the problem.
*What kind of mail the IP is sending - or should be sending!
*Do your troubleshooting. Don't make me ask you to do the basics, or have to do it for you.
*If you can't get mail *from* AOL, send a test from an AOL account and include the results, and give it enough time to time out and send you a bounce.
*If you're an Exchange admin, don't expect me to make sense of your insanely configured MTA by sending me screen shots of the admin console.


This is not pointed at ESPs in particular. In fact, what provoked this was a corporate IP. It is a huge waste of everyone's time when I'm asked to lift a block on an IP, I look at the complaints and see that clearly it was compromised...and have to send an email asking if it was fixed. 90% of the time, it was fixed before the ticket was opened. Say so the first time! The less guesswork I have to do, the faster I can solve a problem and the faster I can move on to the next one.

From the AOL Postmaster blog:

The Postmaster team has suffered significant staff reductions, and any tickets opened will see slower processing times. Thank you for your understanding and patience.

----

And from me, thank you all so much for your kind words, job leads, and condolences. They were all passed along to my team.

Most of the US Postmaster team just got laid off, including Christine. I am still employed, but of the non-programmers, I'm all that's left in the US. *

This is a devastating blow. If people reading this have job leads for any of my folks...ex-folks...please drop me a line, or post in the comments and I'll pass it on.

Jobs needed: Sysadmin, spam fighting/abuse, postmaster-y stuff, deliverability, programming, mail gateways/MTA, and of course management. We lost a lot of people outside the immediate Postmaster team also: anti-spam programming, mail/MTA, database, etc.

These people are the best of the best...
...and I am going to miss them more than I can even begin to express. Eight years is a long time.



*yes, Madkins is safe.

"I'll worry about that on Thursday."

My previous post was slightly incorrect. I posted this:

ERROR [state=rcpt_to, code=550, text= 5.1.1 : Recipient address rejected: aol.com

The true error returned by AOL is slightly shorter. The formatting above was specific to the reporting MTA. This one is the actual one:

550 5.1.1 : Recipient address rejected: aol.com

My apologies for any confusion. The official blog post is here.

I think the RSS issue has been solved. Many thanks to Matt Vernhout for stepping in and taking this in hand. I was getting really frustrated.

My apologies to everyone whose feed got blown up by my blog.

The more I consider the common uses of this obfuscation service, the more it confounds me. I cannot see any reason for a legal business entity to use it. If Joe User sets up a domain, keeps a personal webpage on it and wants to keep his registration information private, more power to him. I'd certainly use it in that situation since I don't particularly enjoy stalkers. But, a business? Really? Why?

Here's a perfect example of the lack of awesome: non-technical family member A is attempting to exchange email with definitely-not-technical family member B who is in something of a sticky situation. The server that relays these crucial emails is timing out connections, inbound and outbound. I wanted to help, so I figured I'd look up the technical contact for the domain and give them a nudge. I tried it.

Yep, Dear Reader, you guessed it. Their contact information is hidden. The domain's website provides no avenue for support either, although it is a government site. So this family - and how many more? - remains fractured, frustrated and angry, their email continues to be deferred, and no-one can do anything about it because the admins who could fix it remain in blissful ignorance of the problem.

Genius!

Tell me again why a non-criminal business needs to hide who they are and how to reach them?

Our new MTA is returning a different error code for invalid users now:

ERROR [state=rcpt_to, code=550, text= 5.1.1 : Recipient address rejected: aol.com

Senders using scripts to remove unknown users automatically should be looking for both the new error code and the old one.

The Postmaster site will be updated ASAP. More information on the migration here.

In a previous post, I mentioned a nifty service that will scan any file you upload to it with multiple AV programs. It is both useful and socially conscious, since any problems found are reported to all the participating anti-virus makers so that they can continue to improve their products to the benefit of everyone.

Naturally, if there is light there must be darkness: a couple weeks later I found several articles including one by Brian Krebs, that refer to a new breed of such scanners which are based on the premise that they will NOT share their findings with the AV makers, and make no pretense of being created to serve anyone but malware authors. For $1 per file, they can test their nasty little products against the big guys and no-one will be the wiser. And of course, the currency used is virtual, so there's no trail to follow there, either.

When I was a kid and I was reading cyberpunk books, I thought it was all amazingly cool. Now that I'm living in the future, I'm finding it deeply alarming. The more I learn about the shadow economy, the more it scares me.

I fondly imagine that the people who are reading this blog are savvy enough to have their computers locked down tight, know how to spot phish, and are generally security conscious. Please, share the knowledge. Go to your parents and sibs and friends and help them learn. The bad guys are ahead of the game, so why not make it a little harder for them?

I get a number of emails from this blog asking me for consulting services. At the moment, I am employed by AOL and cannot consult as it would be rather a conflict of interest, though if I should get laid off, that may change.

However! There are a number of consultants out there, and the latest addition to the pack is:

Mickey Chandler, author of Spamtacular.com & SpamSuite.com, and owner of Whizardries, Inc has recently hung out his consultant's shingle. He's sharp, he knows a great deal about email, deliverability, and internet law, he's fair, delivers on his promises and on top of all that he's a thoroughly nice person. I know these things not only because he is my friend, but because I have worked with him for years in my position at AOL, and have never been anything but impressed.

I should probably make clearer that my post about "red flags" is aimed entirely at marketing mail. It is in a class of its own. Listserv/groups, private mail servers, corporations, government IPs, peer ISP MTAs, newspapers, social networks, gaming sites, etc are all treated differently from each other by our reputation systems and also differently than bulk mail is.

If you're not a marketing mailer, and you're having issues sending mail to AOL, we really want to hear from you so we can fix it! If you are experiencing delivery issues -- and your IP does not send any bulk, marketing, or advertising mail -- please fill out a support request and let us know what kind of mail you send so that we can treat it appropriately.

A commenter asked me:

What do you expect me to start a complaint with, then? really ...what is wrong with "We are a legitimate business, we do not send spam!"?

There's nothing inherently wrong with "We are a legitimate business, we do not send spam!" - but every single business that sends email says they're legitimate, whether their mail stream generates massive complaints and a poor IP reputation, or not. The immediate "We're legitimate!" is a red flag in the same way "Officer, I wasn't doing anything wrong!" is. Starting off the conversation by protesting your innocence when you don't know what precipitated the stop (or worse, when you do know) can only lead to heartache, and a big pain in your wallet.

As an ISP abuse desk rider, I can tell you that sender claims of legitimacy, explanations of business models, assertions of opt-in, of CAN-SPAM compliance, and of observance of best practices are not actually relevant to the situation - what I care about is the response that the email streams get from my users. The reputation systems don't pay any attention to business models. They pay attention to results.

What does "a legitimate business" even mean in this context? That it has a brick and mortar address somewhere? That it sells genuine products and isn't a scam? That they don't buy email lists? It could mean anything, and often means nothing.

What "legitimate" means to me is "Does this IP have a good reputation? Do the recipients of this email really want it? Is it relevant and desired? Does it have an engaged audience? If this IP is new, has the sender followed best practices in terms of warm up and feedback loop set-up?"

If someone is opening a support request because their marketing mail is not getting through, the answers to those questions are probably "no", for whatever reason. There will be occasions where the failure of email to get delivered is a technical problem on the ISP end, but on the whole those aren't very common.

So, to more directly answer the question that was asked of me, when opening a support request I would skip any explanation of your business model, legitimacy, opt-in, etc, and get right to the heart of the matter - "My email is not being delivered, here's the IP, here's the error message, this is what we have done to clean up the problem. Thanks for your time."

...and if it's a reputation driven error/tempfail you're getting, don't open a ticket at all. Fix the underlying problem and the delivery issue will go away.

I've been doing this job for a long time now, and over the years Ive seen certain types of phrases in support requests repeated over and over again. These correlate to bad IP stats so often that I've built a mental filtering system that flags these phrases. Some examples:

"All of the email we send is not spam, but legitimate communication that the recipient has requested from our clients."

"The contents of the email follow all of the CAN-SPAM guidelines!"

"Please remove the spam blocks so that our mutual customers can get these urgent and valuable offers that they specifically asked for."

"We are a legitimate business, we do not send spam!"

"Why are you targeting my 100% opt-in emails?"

"Let me explain our business model..."

There are many more such gems. Every single one of them sounds a loud "AOOGAH!! DIVE! DIVE!" alarm in my head, and inevitably leads me to IPs that are sending junk mail that people clearly don't want to get. The result of that finding should not require further elaboration here.*

The bottom line is that if a marketer is sending mail that is truly valuable and desired, he will not experience blocks...and if he does, he will have sufficient knowledge of what his customer did wrong that he will not use such verbiage in the trouble ticket.



*There is one exception to this rule. If I see "We are a legitimate business, we do not send spam!" and the rDNS of the IP indicates a corporate or government machine, 99.999% of the time they have a compromised host on their network, which generally horrifies them when they're alerted to the fact... and they clean it up as fast as humanly possible.