MBoffin.com

Something witty this way comes.

Using Gmail as a Spam Filter

I recently got my Gmail account up and running, but I wasn’t finding a whole lot of use for it since I have a primary account on MBoffin.com that I use for everything. Just for kicks, I decided to have all my MBoffin.com e-mail (spam and all) forwarded to my Gmail account to see how well it would handle my e-mail habits and workflows, and to see how well it would handle the spam.

Gmail did surprisingly well on all fronts. Much has been written in other circles about the nice and not so nice parts of Gmail’s user interface, so I won’t get into that discussion here.

After a couple of days using Gmail as my primary e-mail account, I started missing my MBoffin.com account. I like my MBoffin.com e-mail address and I’m not quite ready to leave it behind. But what to do with this nice new Gmail account? Then it hit me....

Could Gmail be used as a spam filter for my MBoffin.com e-mail account? The answer is yes, it can. And here’s how I figured it out.

Basic Procedure

Gmail allows you to forward incoming messages to any other e-mail address. Go to the Settings page and then to the Forwarding and POP tab. In the Forwarding option, set Gmail to forward all incoming mail to your regular e-mail account, and keep a copy in Gmail’s inbox.

(In this explanation, I will assume your regular e-mail address is user@domain.com and your Gmail address is user@gmail.com, and I apologize to Mr. User over at Domain.com and Mr. User over at Gmail if they get any extra mail from people following the steps in this article too literally.)

Once that forwarding rule is set on Gmail, all incoming mail to user@gmail.com will get spam filtered and anything left over will be forwarded to user@domain.com, with a copy left at Gmail. But that doesn’t help you much yet, because people are still sending spam directly to your user@domain.com account.

Now, over at your user@domain.com’s mail server, create a server-side filter to check the headers of any incoming e-mail. Have it forward to your Gmail account if it does not find the following in the header:

X-Forwarded-For: user@gmail.com user@domain.com

In English, the filter would be written: “Any mail that does not contain ‘X-Forwarded-For: user@gmail.com user@domain.com’ in the mail header should be forwarded to user@gmail.com”.

Once this server-side filter is in place, only mail on its way back from user@gmail.com (already filtered for spam) will be passed to your user@domain.com account's inbox. Everything else will be forwarded on to user@gmail.com to be filtered and forwarded back.

Another Advantage: Backup

Since Gmail is keeping a copy of all the mail it’s forwarding on, you now have an online backup of all your e-mails. If you were to somehow lose all the e-mails stored on your hard drive, you wouldn’t have to worry, as they would all be safely backed up over at Gmail, already spam filtered and everything. This means you can keep your local e-mail client clean and tidy, deleting e-mails as you see fit without worry that they are being deleted forever. They’re all backed up on Gmail the instant you receive them.

Will Spammers Now Add Headers to Prevent Being Forwarded to Gmail?

Sure, they could easily add the required header and trick your user@domain.com account into accepting spam, thinking it had already been spam filtered by Gmail. But here’s the kicker: They need to know your Gmail account address too.

Final Thoughts

Keep in mind that I only just figured this out tonight. I ran some tests and confirmed that it works, but that’s as far as it has gone. This hasn’t been tested over weeks and weeks, so if it continues to work well in the long run, great. Otherwise, I’m sure someone else will come up with a way to improve this.

While doing the testing, actual spam was being sent to my account (I get hundreds a day), and none of it made it through. It was all trapped at Gmail.

Update: Joe is testing this out with his Yahoo! account to see if a similar thing can be done. Report back, Joe! Let us know how it goes. Alex pointed out that you could set up parallel Yahoo! and Gmail accounts and sign up for the same spam lists to see which has better spam filtering.

Replies

I've been doing this since the 17th of February when I first got my gmail account, although I didn't think of setting up a forwarder at my primary account to route all mail through gmail. I don't need to do this however as I receive no spam directed towards my primary account since I've been using disposable email addresses (sneakemail.com and before that spammotel.com) to cloak it since July 2002 or possibly longing if using a secondary hotmail address would count for the same thing.

My primary motivation however was to use gmail as a permanent online mail archive and it was only as an uninteded side-effect that I discovered the spam filtering benefits of using gmail to replicate and then forward any mail directed to me.

I discovered it since I am also a subscriber to a few YahooGroups mailing lists, which have been suffering from a barrage of spam lately, and Yahoo has chosen rather lamely to only implement filters on their Yahoo Mail accounts but have neglected entirely to introduce automatic spam filtering of the mail distributed using their Groups feature.

I look forward to seeing the results of your further enquiries and comparisons with Yahoo Mail although to my mind there is no way that Yahoo Mail could be a serious alternative to GMail for use as a permanent online mail archive since Gmail started dynamically increasing the available storage space on their account.
knilaus - Apr 9, 2005 @ 6:12 AM - Permanent Link
Wow, that's amazing that they would ignore the Yahoo! Groups in the spam filtering. It seems like that's where it would be even more annoying, since it's not just one person who has to see the spam, but a whole group of people.
Dylan - Apr 9, 2005 @ 6:29 PM - Permanent Link
I belong to about eight Yahoo groups. Five of them I am set to receive direct email from the group. Two of them I am the moderator for the list. Four of them are publicly listed in the Yahoo directory. I don't get any spam on any of them.
rnewhouse - Apr 9, 2005 @ 7:06 PM - Permanent Link
Is there a way to use gmail's spam filtering technology to be applied to Microsoft Outlook.? I am currently looking thru the tabs and menus but not seeing too much help...

Thanks

Walcy
snap326 - Apr 9, 2005 @ 10:17 PM - Permanent Link
Unfortunately, it's a server-side thing. The rule that gets set is on your mail server. In other words, when the e-mail gets to your e-mail server, your e-mail server is the one that sends it on to Gmail to be filtered, not your e-mail client (Outlook, Thunderbird, etc.).

Then again, I'd love to be proven wrong on this one. Any takers?
Dylan - Apr 9, 2005 @ 11:34 PM - Permanent Link
Hahah, you could implement something similar soley from within a your mail client now that I think about it but it would be terribly inefficient. A mail filter setup to forward incoming messages to a gmail account setup to forward either back to the same account (with the filter deleting the original received message as well as forwarding) or to another account. I suppose if you leave your mail client open with automatic receiving then the second method might actually be worth it but I can't imagine going so far as deleting the originals would ever make much sense.

On this whole subject, does anyone know of a good all-in-one sort of mail server package for Mandrake or just a workable setup. I've had a half working mess of a mail server setup for vampirical.com for ages and I'm sick of it.

vampirical - Apr 11, 2005 @ 2:09 AM - Permanent Link
After a couple of days using Gmail as my primary e-mail account, I started missing my MBoffin.com account. I like my MBoffin.com e-mail address and I’m not quite ready to leave it behind.

Why not simply go to gmail's "Settings" -> "General" -> "Reply-to Address" and set it to your mboffin email? That way any email you send from gmail looks like it came from mboffin.com. This seems like a much simpler and more stable way to get the same effect.
The Goose - Apr 11, 2005 @ 2:22 PM - Permanent Link
I had pondered doing something similar to this a few weeks back, but hadn't implemented it yet, as I hadn't been able to confirm whether or not GMail applied rules and filters before or after it applied spam filters.

My implementation was to be a little different, however...

Since I have control over the MX for my domain, I was going to set all incoming mail for my various usernames (in the virtual user and alias tables) to kick over to my GMail account, and then have GMail kick it back to a new account on my mail server whose sole purpose was to receive filtered mail from GMail. By then using client-side or server-side filtering on that account, I can then reject any mail that has not come from GMail, under the assumption that it's coming from a spammer broadcasting to all accounts on my box in hopes of hitting a live account somewhere, because the only legitimate e-mail coming to this new account should have the GMail forward tag on it.

Thanks for the tip! I'm glad someone else was thinking along the same lines I was. I didn't know about the headers. Now, if only GMail could add an X-gmail-labels header to it, I'd be in geek heaven.
Manuka - Apr 11, 2005 @ 3:28 PM - Permanent Link
Goose, That works fine if you're one of those weirdos that has only one e-mail address...

Sadly, GMail doesn't let me do profiles like Thunderbird does, where I can select which e-mail address I'm sending "from".

Another GMail tip: If you need an SMTP server when you're on the road, set your client to drop mail into smtp.google.com, and use your gmail login. It even supports SSL connections.

Manuka - Apr 11, 2005 @ 3:30 PM - Permanent Link
Goose, that's a good suggestion. And eventually, it may be something I end up doing. Manuka has a good point, though, about the multiple e-mail addresses. I have some old mailing lists I'm on that will only accept e-mail from a specific e-mail address and the listserv administration options are long since dead, probably on a server now inaccessible by the web. It would be rather a pain to have to switch reply-to addresses each time I have to send mail to those lists.
Dylan - Apr 11, 2005 @ 3:45 PM Last Edited: Apr 11, 2005 @ 3:50 PM - Permanent Link
This is a great idea. However, after going into cpanel and email filtering. I can create filters where the headers contain something but I can not create a filter where the headers do not contain something (such as...)
X-Forwarded-For: user@gmail.com user@domain.com

Any ideas?

Thanks in advance.

SCuM
SCuM - Apr 11, 2005 @ 3:59 PM - Permanent Link
I can create filters where the headers contain something but I can not create a filter where the headers do not contain something


It's up to your hosting service and what software they run.

For example, with Textdrive (using procmail) you need to have a .procmailrc file containing rules and a .forward file containing

|/usr/local/bin/procmail


More tips for setting this up with Textdrive here.
jpwain - Apr 11, 2005 @ 7:09 PM Last Edited: Apr 12, 2005 @ 8:56 AM - Permanent Link
I'm with Goose on this one. This whole post seems to me an added meaningless step. If you want to use Gmail as a spam filter... then use Gmail.

1) Set up an alias for your email address(es) that forwards all incoming mail to your gmail account.
2) Set up POP on Gmail, and keep a copy in Gmail's inbox.
3) Check gmail with your mail client through POP (Thunderbird, whatever).
4) Send email through your mail client (allowing you multiple profiles, etc).

The key here is to add aliases so that all your email addresses forward to your gmail address. Then you're good to go, the only difference is you're mail is in gmail - not your own server.
Trent - Apr 12, 2005 @ 5:26 PM - Permanent Link
In response to SCuM: While this is not the same thing at all, I currently use Cpanel's email filtering to grab all my SpamAssassin tagged emails and forward them to a GMail account. I don't have to download hundreds of spam emails and yet I can always log onto GMail to see if an email was inadvertently flagged as spam. So far, so good, but again, it's not the same thing.
lilithvf1998 - Apr 12, 2005 @ 10:53 PM - Permanent Link
Trent, you have a good point about simply using Gmail as the stop point for all the e-mail, and not having it then send back to your regular e-mail account.

However, and this is just simple niggling of details, instead of just adding a server filter and a setting in Gmail, you have to go change your e-mail client's accounts to now get POP mail through Gmail, instead of just leaving your client's settings alone and doing things as you've always done.

On top of that, what if Gmail's spam filters just aren't cutting it for you, or maybe you want to switch to Yahoo! Mail, or maybe you want to chain Gmail and Yahoo! Mail together to get double filtering, or, or, or? There are a number of situations where leaving your e-mail client to do what it's always done is a more desirable solution.

I think it really comes down to more than one way to skin a cat. In some situations your procedure would be perfectly acceptable, but in others, the procedure I describe in the original post makes sense too.
Dylan - Apr 12, 2005 @ 11:32 PM - Permanent Link
It's a great idea and Gmail does an amazingly good job of spam removal.
However, the bit where I filter the headers and send some back to Gmail is where my system goes wrong because I can't set that up at my email server - they just don't have the facility.
So, is there a way of making Thunderbird do it (I don't think so)?
Or does anyone know how to set up "X-ray Mail Assistant" to do it?
Failing that.... any other ideas?

Cheers,
Pippington
pippington - Apr 17, 2005 @ 3:57 AM - Permanent Link
Gmail handles viruses by not allowing certain file types/attachments through. As a result, forwarding email through Gmail will bounce any emails that have these attachments. This includes Access databases. Yahoo doesn't have this problem and I've had our work email forward through in the way you describe for about a year with no trouble.
jaythere - Apr 18, 2005 @ 9:01 AM - Permanent Link
Jaythere, that's great info. I haven't even looked for what file types are allowed or not allowed. Is there a list somewhere you've found of what file types they don't allow?

Since you've had this setup for a year now, how is the spam filtering on Yahoo using this kind of setup?

And Pippington, I don't know exactly what it would be on X-Ray Mail Assistant, having only looked at the help on their site, but I would think something along the lines of anything that doesn't match this rule:

$X-Forwarded-For != user@gmail.com user@domain.com

should have the "action" of forwarding to your Gmail account.
Dylan - Apr 18, 2005 @ 9:42 AM - Permanent Link
I don't know of a list, only:

http://gmail.google.com/support/bin/answer.py?answer=8493

Gmail blocks viruses in the most direct possible way: by not allowing users to receive executable files (such as files ending in .exe) that could contain damaging executable code. This protects your computer and halts the spread of such viruses. Gmail doesn't accept these types of files even if they are sent in a zipped (.zip, .tar, .tgz, .taz, .z, .gz) format. If executable files are sent to your Gmail account, the message is bounced back to the sender.


Yahoo spam filtering is good as a first filter, then our desktop software gets whatever makes it through.
jaythere - Apr 18, 2005 @ 11:12 AM - Permanent Link
mewhouse> assuming that you are aiming to contribute to the debate on spam filtering I am at a loss to ascertain the meaningful point of your testimony in this thread?
knilaus - Apr 21, 2005 @ 2:18 PM - Permanent Link
Wow, that's a great idea...
Joe - Apr 21, 2005 @ 2:59 PM - Permanent Link
That was my response to your comment about receiving lots of spam from your Yahoo lists: merely that it is not my experience, and it seems like I have enough Yahoo lists going on, and in enough configurations, to get a good sampling.

I'm not particularly motivated to enter this debate, if that's what it is, about spam filtering. For me, the main drawback to using gmail as a spam filter is the prohibition on attachments, which I require heavily in my work.

I am informed that Outlook 2003 has some really excellent filtering in place, along with the ability to add senders to a safe list (by user or domain) with a simple right-click. I haven't yet had the opportunity to check it out.

Also note that CNN was carrying some articles about the newest antispam trend being to filter it at its source, which IMHO is where the burden belongs.
rnewhouse - Apr 21, 2005 @ 7:36 PM - Permanent Link
I use disposable email addresses such as the ones offered by sneakemail.com as my method of spam prevention and I have to say it is fool-proof. I have had the same email address for 5 years and do not receive any spam mail addressed to it directly.

One of the yahoogroups to which I subscribe is however a heavy source of spam. It was set up in 1998 and the address has been advertised on several home pages, which of course makes it more susceptible to spam.

We had to introduce moderation of posts from new members simply to stem the influx of spam: At its peak we received about 30 spam mails per day to this address all of which I as a moderator had to flush out of the system manually because the idiotic YahooGroups system does not offer an automatic spam detection scheme.

We then changed the name of the group (and by extension its email address) in order to fool the spammers, who had added us to their lists and at the same time lifted the moderation requirement on new posts.

This helped for a few months but unfortunately we have begun to receive spam on the new address also (despite not having advertised it anywhere on the web) at a rate of about ½ - 1 per week.

In general I am opposed to the principle of spam filtering since I see it as the first step unto a slippery slope whereby increasingly heavy restrictions are being imposed on the content that you may submit as a legitimate email. The ultimate consequence may be a form of censorship. And furthermore I think that spam filtering is simply a way of alleviating symptoms rather than fixing the problem at its roots: Namely that the irresponsible behaviour of users exhibiting their email address in all sorts of places is being exploited by unscrupulous spammers, who harvest these and sell them on for profit.

I consider disposable email addresses as the primary technology to address the root causes of spam.

Take a look at http://www.sneakemail.com for a free and working demonstration of the concept.
knilaus - Apr 22, 2005 @ 3:33 AM - Permanent Link
Just an update. So far this has been working great. The only problem that has come up was when Gmail had two false positives. Considering this is out of a few thousand spam e-mails over two and a half weeks, that's not bad. But, flagging those two as "Not Spam" in Gmail does not seem to forward them back on to my mboffin.com e-mail address. A small price to pay for great spam filtering and automatic backup, though.
Dylan - Apr 27, 2005 @ 9:12 AM - Permanent Link
I just noticed a problem that I had with my setup? It works fine for the first round of spam filtering. But say someone sends me email 1 with topic "hello", and I reply to them with email 2 "re:hello" and they reply to my reply with email 3? that third email doesn't get forwarded back to my original email address for retrieval - rather it gets grouped with the conversation in the archive on the gmail servers!!! Big problem. Has anyone else run into this?? It happened once - and as much as I'm digging the filtering, I'll wade through the spam to avoid this MAJOR issue. I have it set up with my primary account to forward to gmail any email that doesn't have the gmail address (username@gmail.com) in the header.
rycc - Apr 28, 2005 @ 8:21 AM - Permanent Link
Um, this is all cool and everything, but wouldn't it be easier to just setup your personal mail server to forward EVERYTHING to Gmail, set your "replyto" address to your personal mailserver's (so it's still so-and-so@myserver.com instead of your Gmail address) and just let Gmail's filters work? Why make it so complex, when the Gmail interface is so GOOD?

This would also let you use the gmail features like Google searches of your mail on mail addressed to your personal server, as well as any other features Gmail may add.
Wirehead - Apr 28, 2005 @ 9:52 AM - Permanent Link
My SPAM prevention technique is to simply enter Dylan's email address on all forms that I encounter.
DataBind() - Apr 28, 2005 @ 10:18 AM - Permanent Link
If you don't exclude mail with username@gmail.com in the header, you end up with your mail looping back and forth endlessly to gmail without ever arriving in your in box.

If you wanted to just go to gmail to get your stuff, and ignore Outlook, that would work.

I personally rely on some of Outlook's features like collaborative calendering, voting, free-busy info, etc. to interact with other people, so I would not want to give that up.
rnewhouse - Apr 28, 2005 @ 10:20 AM - Permanent Link
Rycc: Wow, that's an interesting one. I ran a test on mine and I didn't have any problems. I even set Gmail to auto-archive all incoming e-mails after it forwards them on to my mail account. It did take a few minutes before the e-mails were passed on, but even subsequent replies were passed on. I would do some explicit testing on your end to really make sure that's happening, and it wasn't just a fluke.

Just in case, maybe try making sure Gmail isn't set to auto-archive incoming e-mails.
Dylan - Apr 28, 2005 @ 10:21 AM Last Edited: Apr 28, 2005 @ 10:21 AM - Permanent Link
Wirehead: Yes, you very well could use Gmail's interface. However, a web interface, no matter how good, is not for everyone. And while you could use Gmail's POP access to just point your local mail client to Gmail and use that, it's not always the right solution. For example, I am on an old mailing list that won't accept e-mails from any other e-mail address than the one I'm signed up with. In other words, simply having the reply-to address is not enough.
Dylan - Apr 28, 2005 @ 10:27 AM - Permanent Link
This looks like a great way I can handle hundreds of spam messages sent to my domain's e-mail. I looked at gmail recently and looks like the Beta is closed to new members. I am currently thinking I will sign up for a Yahoo Plus account ($20 p/y) just to take advantage of your idea here.

I run my own server with procmail but I am not very savvy when it comes to any kind of programming/scripting... Can someone tell me how to construct the .procmailrc file so it will forward anything that doesn't have the correct header info. I know that you posted this line above:

$X-Forwarded-For != user@gmail.com user@domain.com

I am hoping someone could give me the other lines that need to be included in the .procmailrc showing the :0: and lines that would direct the mail to my mailbox, etc.... Please excuse my newbie skills.

Thanks in advance for your help....

ddeo
ddeo - May 2, 2005 @ 7:20 AM - Permanent Link
I have about 60 gmail invites if you want one. No need to spend money on Yahoo. If you'd like an invite, post your email here and I'll send you one (probably best to use a disposable address or one you already get tons of spam to anyway).

I can delete your email address from the post once the invite is sent and accepted to minimize spammage.
Wirehead - May 2, 2005 @ 8:46 AM - Permanent Link
There is now a post you can reply to if you want an invite. It's members-only and moderated. Read the post for details.
Dylan - May 2, 2005 @ 9:19 AM - Permanent Link
ddeo, I seem to obtain quite good results with:
# forward to gmail account for spam filtering - cf http://mboffin.com/post.aspx?id=1636
:0
* !^X-Forwarded-For: user@gmail.com user@domain.com
! user@gmail.com

in procmail. But YMMV...
serkel - May 11, 2005 @ 7:16 AM - Permanent Link
Can someone please tell, or point me somewhere, how to achieve the X-Forwarded check using QMail (and the .qmail file) instead of .procmailrec?
stefpet - May 14, 2005 @ 11:42 AM - Permanent Link
Wish I would have found this post yesterday, I would have saved myself a lot of time. Anyway,

I'd like to emphasise the usefulness of using Google's SMTP server for all your email. Apart from relieving roaming anxiety, it automatically saves a copy of the message on the Google account. So if you redirect your pop mail to Gmail AND use their SMTP server, you'll have ALL your messages archived, not just the received mail. I left some instructions for newby outlook friends here.
mariushendrik - May 16, 2005 @ 8:19 PM - Permanent Link
Great post! Signed up just to reply to this one.

I've been using Gmail for about 3 weeks now to filter my spam. I regularly use the mail aliases X@mydomain.net (X varies), and they now all get forwarded to my Gmail account, which kills the spam and forwards the rest to a private email address.

One problem: the accuracy of Gmail seems to lag behind. In other words, it only eats about 80% of all the spam I receive. It must be said that the people spamming me are pretty creative: they know how to spell 'VIAGRA' in 666 different ways. Still, Gmail doesn't know them all. My client-side spam filter (PopFile) does a far better job.

Waiting for the remaining 20% of the spam to download to my PC is annoying, as is having the spam in my Gmail inbox (I'm keeping a copy of my mail in there).

Two questions here:
1 Anyone got some tips to improve this system?
2 Is Gmail really learning from all the spam I report manually?
BigFurryMonster - May 26, 2005 @ 9:08 AM - Permanent Link
Glad you got use from the article! :)

While you will still have to download the spam, Mozilla Thunderbird has excellent client-side spam filtering if you want to use that as a client.

Gmail is learning from your manual spam reporting. It does improve over time (sometimes longer than others). I have witnessed this happen with the tremendous volumes of spam to my mboffin.com accounts.
Dylan - May 26, 2005 @ 2:22 PM - Permanent Link
I thought this was a great idea and went over to my hosting provider to figure out how to implement it. However, I ran across this post. Basically, forwarding a large amount of spam to an outside account could cause someone like GMail, Yahoo, etc. to blacklist the provider. My provider has had this happen and had to get in touch with MSN and Yahoo to straighten it out. It's now a TOS violation to set this up.

By the way, when I signed up on this site a few minutes ago, the confirmation email was tagged as spam by Gmail.
niwdoog - May 26, 2005 @ 9:04 PM - Permanent Link
Wow, that's an interesting twist. Gut instinct says that Gmail doesn't blacklist you for having large volumes of e-mail forwarded to yoru Gmail account. For one, there are thousands of people who have all their mail now forwarded to Gmail, many of whom had things this way before I wrote this article. In addition, I have had tens of thousands of e-mails forwarded to my Gmail account, yet it still is handling my spam filtering with no problems.

Though, your note that the confirmation e-mail was tagged as spam makes me pause. Then again, it's not a very personal e-mail and with the randomized activatation code in the link, I could understand it looking like spam.

Thanks for the heads-up on this possibly coming up as a problem, niwdoog. Good catch. I'll be interested to see how Gmail handles this over the long-term, as more and more people do this.
Dylan - May 26, 2005 @ 10:47 PM - Permanent Link
What if i can not use procmail, and I have only pine installed on my ISP unix account. any ideas? I cant run procmail, cuz everytime it freezez my emails, so nothing happens
simorgh - Jun 2, 2005 @ 7:02 PM - Permanent Link
Instead of using server-side filter, I've created another account to recieve filtered mails back from gmail. For instance, if my primary email is me@mydomain.com then I create me1@mydomain.com in order to recieve filtered mail and instruct gmail to forward mail to me1@mydomain.com. Then change setting in your mail client to check mail from me1@mydomain.com instead. So no need to bother with server-side setting especially if you're not an administrator.
saran - Jul 14, 2005 @ 8:52 PM - Permanent Link
Anyone know of a good server-side (online) free spam filter to complement Gmail's ?
BigFurryMonster - Oct 6, 2005 @ 4:03 AM - Permanent Link
Can someone comment whether this will foward an entire domain or not? I have my own domain, and my whole family uses it. I want to filter only my mail. According to the post, it seems like this will capture all mail for everyone at my domain, and foward it back to me. The logic seems to rely on you being the only one in a domain. Is there a way to filter this down to one user? I've been trying to do so via a regex, but have been unsuccessful thus far. I think you would typically need a negative lookbehind assertion, but they are fixed width, so I'm not sure that will work. I've also tried conditional regex, and I can't get that to work either.

Looking at the headers of a mail to make a fowarding decision, I'm basically looking to say: If "user@domain.com" exists and not "X-Forwarded-For: user@gmail.com user@domain.com" then foward to user@gmail.com. Sounds simple enough, but I'm at my wits end trying to make this work via regex.

Anyone have any ideas?

Thanks,
Joe
jhs2 - Nov 14, 2005 @ 9:26 AM - Permanent Link
I did not set this up for myself as a domain-wide filter. I use it with my own mboffin.com e-mail address, but there are many other active e-mail addresses on this domain that don't use this filtering process. I'm not sure what mail server you are using, but I was able to create a server-side filter for my account only. In other words, the filter is not being applied on a server level. It is a server-side filter applied only on my account. The mail server this site runs on is IMail by Ipswitch, Inc. It has a web mail interface that allows you do do things like set up filters that are attached to your account.

Any more info you can give me on what system you are using and what your options are for settings filters for an account? I'll help as much as I can.
Dylan - Nov 14, 2005 @ 11:43 AM - Permanent Link
Thanks for the reply Dylan. I have CPanel 10 interface to my domain. I don't have a specific user control panel, it's only for the domain. I honestly don't know what type of mail server I'm running, as I don't have direct access to it. When I click on the Mail icon on the main page for CPanel, I get a subpage with several options. One of those subpage options is "E-mail Filtering". Under this option, I can add a filter. Here, I basically need to select what I want to filter on (From, Subject, To, Body, or Any Header). Then I have to choose a comparrision operator (equals, matches regex, contains, or begins with). Then I specify the matching text. Finally, I choose what to do with it if a match is made. I can either discard the message or foward to any email address I like.

Based upon this, and the fact that I'm making these changes for my domain, I don't know how to limit this down for just my user account. I thought that by using regex, I'd be able to write something quite complex to achieve this goal, but I can't seem to make it work (and I'm fairly good at regex).

I hope this helps to fill you in a bit more. If this is even close to what you have, then perhaps I'm missing something obvious.

Thanks again,
Joe.
jhs2 - Nov 15, 2005 @ 10:14 PM - Permanent Link
Hi there,

Has anyone come across a way to set up the mail filter using Plesk?

Any help would be greatly appreciated - looking forward to trying this out!

Jim
sixmedia - Feb 2, 2006 @ 12:37 PM - Permanent Link
So I have a bit of an issue that I was hoping to get some assistance with. The following is my .procmailrc file
DEFAULT=$HOME/Maildir/
MAILDIR=$HOME/Maildir
PMDIR=$HOME/.procmail
LOGFILE=$PMDIR/log
SHELL=/bin/sh

# forward to gmail account for spam filtering - cf http://mboffin.com/post.aspx?id=1636
:0
* !X-Forwarded-For: spamaccount@gmail.com user@domain.com
{
    :0fw
    | formail -IDelivered-To

    :0
    ! spamaccount@gmail.com
}


user@domain.com is my primary email address on the account and this works fine for me, messages are sent to gmail and then sent back to user@domain.com.

But if I try to change the file for the address user2@domain.com (additional email account - wife's account) then all messages (even for user@domain.com) are sent to user2@domain.com directly. They do not get forwarded to gmail first.

Any ideas?

Also, I would like to see if this can be implemented for multiple users with multiple gmail accounts.

Thank you.
sfarmer - Aug 11, 2006 @ 9:53 AM - Permanent Link
Hi Dylan...

Dunno if you remember me... You and I used to play on the same Q3F servers back in the day.

I was doing a google search for spam filtering and came across this post on your site. Question: What's the mod on the forwarding for sendmail? I'm using a Raq 550 [bluequartz] on CentOS.

Thanks!
Know Your Role - Aug 14, 2006 @ 6:37 PM - Permanent Link
sfarmer, sorry for the delay. Can you post what you are putting when you try adding the second user account? I'll do some searching around and see what I can hunt up. I'll need to figure it out for myself anyway in the near future.

Know Your Role... whoa. :D Just seeing that nick brings back good memories of my Q3F days. Good to see you, man. Talk about a small world. Off the top of my head, I don't know the config to have sendmail do this, but I recall reading a blog where someone did do that, so I'll see what I can find.
Dylan - Aug 15, 2006 @ 9:26 AM - Permanent Link
I just wanted to say THANK YOU for posting about using gmail as a spam filter. Like several posters, I cannot use the reverse filter (going into cpanel and email filtering, I can create filters where the headers contain something but I can not create a filter where the headers do not contain something such as X-Forwarded-For: user@gmail.com user@domain.com). BUT I was getting thousands (yes thousands) of bounced-back spam where the spammers were sending e-mail using my domain name. I can now use g-mail to do the filtering on the forwarded mail which is much more efficient and less cumbersome for me than the SBC Yahoo, and I can get my domain mail easily as previously set up on Yahoo! It has worked like a dream! Thank you thank you, thank you. I was tearing my hair out, calling every computer geek I knew, and no one could help me...till I read this posting.
sheilawin - Aug 21, 2006 @ 1:13 PM - Permanent Link
buyer beware. i have a theory about a flaw in this:

when you forward other mail to gmail, gmail can't filter the spam out as well. this would be because it can't use collaborative filtering for all the spam that is sent via dictionary attacks (or any other mass mailing sent to many gmail addresses). perhaps it even gets spam that spammers never intended to go to gmail addresses as they knew it would get filtered out given a large enough volume.

likewise to (and more importantly than) being big on false negatives, it may also simultaneoulsly cause a high false positive rate since the system just can't lock down the pattern well enough without the collab.

also: since there is no record of you sending out emails, replies to your emails (e.g. "re: movie tonight") may be more likely considered spam.

my evidence points to the symptoms (not necessarily supporting these theorys): many false positives in my spam box. a lot of which are from friends directly to me!

still using this in spite of the issue. just carefully monitoring my (huge) spam folder.
watson - Jan 13, 2007 @ 3:30 AM - Permanent Link
Interesting that you've had that experience, watson. I've had quite different results. I even posted last month how my spam count was reaching over 10,000 a month. Yet GMail was only letting through a fraction of a percent of those e-mails into my inbox. I did that math and it was 0.05% of the spams were making it to my inbox. Also, after a while I was getting so few legitimate e-mails flagged as spam that I stopped even checking. In the past six months, I've only had one that made it in there.

And that's with my catch-all being forwarded to my GMail account as well, so I'm getting all the spam for this entire domain (whether it's to a valid user or not) routed to my GMail account.

But I appreciate you posting your experiences. It does make me wonder what the difference could be that would produce such drastically different results.
Dylan - Jan 13, 2007 @ 7:32 PM - Permanent Link
Hi,
I may have cracked the cPanel problem.....

When you create a filter in cPanel the details are stored in a file called '.filter' in your root directory. Create any old filter (to create the file), then then modify the file contents to do the necessary.
My file contains:

$message_headers does not contain "X-Forwarded-For: user@gmail.com user@domain.com"+++++++user@gmail.com

This appears to be working well - Please someone else try it and see if it works for them.

Thanks
Matt
mattl - Feb 23, 2007 @ 12:16 PM - Permanent Link
As one or more people have already pointed out, be careful with where you forward your emails to for processing. Gmail gladly accepts any forwarded email (because Google is cool IMO and because they have more resources than they know what to do with). But Yahoo won't accept all forwarded mail. It may reject it, it may defer it, or they may choose to blacklist the forwarding server if too much mail is sent. Verizon, Comcast, and AOL will often either blacklist forwarded mail that it feels is spam, reject it, or defer it.

So forwarding from one email account to another is only a good idea if you know that the server that is receiving the forwarded mail is not going to blacklist/reject it but will gladly accept it and process it.

I can definitely see how it is against an email host's TOS to just forward emails anywhere. It's too easy for somebody with an email address that is a large spam target to forward that email (unfiltered) to AOL.COM and then have AOL.COm blacklist their webhost's mail server from sending any mail to AOL. It can very quickly become a problem for many or all of a web hosts customers.

Here is an example that is even worse. isignupforallmaillinglists@lamer.com decides to forward his email to his Yahoo, AOL account, etc. because he doesn't like the job his email host is doing with spam detection. Then this character starts receiving all of his spam at his AOL/Yahoo/etc account and begins to designate it as SPAM at Yahoo/AOL/etc. So now Yahoo/AOL/etc immediately determine that the last IP address that i was relayed through (his webhosts mail server) is a spam generating macine and they blacklist it - or they tag ALL mail coming from that IP as spam. Again this causes the webhost's mail server to become blacklisted or have decreased sending abilities to Yahoo/AOL/etc. In addition, this guy probably will never see even a _valid_ email to isignupforallmailinglists@lamer.com in his AOL/Yahoo account because he just guaranteed that all mail from that server is tagged as spam.

Things that should never be done anymore - 1. autoresponders. 2. email forwards without prior effective spam processing before forwarding (unless you know that the mail server it is being forwarded to will accept it and not blacklist the forwarding server).

Of course, there are some web hosts (shh, I wont mention any names) who do have _some_ customers on mail servers that do not effectively handle spam. That makes it difficult. In those cases, the customer should think about using something like googlemail for domains (whatever the TMd name is called, i forget) and have their webhost/dns provider change the MX to point to Google (or some effective spam filtering email company) to handle their email.

Mike
mtindor - Mar 27, 2007 @ 8:17 AM - Permanent Link
Hi guys,

Awesome article! Just a question about how this works. If I have multiple email addresses that I want to 'wash' through gmail, does that mean I need multiple gmail addresses, or is there a way of telling gmail where to forward particular messages?
Remy Lebeau - Oct 28, 2007 @ 4:06 PM - Permanent Link
Actually, after posting that I noticed that there are filtering options in gmail also! Don't mind me :-)
Remy Lebeau - Oct 28, 2007 @ 4:09 PM - Permanent Link
Filtering is always a time saver for the user's end. I have found that using filters is actually great for organizing simple applications and great for easily sorting out my important information.
AlbertF - Sep 23, 2009 @ 8:41 PM Last Edited: Sep 23, 2009 @ 11:13 PM - Permanent Link
sure one could dump all of one's email into gmail for eternal storage, cross referencing, and data mining

or one could use one's own web host (vps) to run through greylisting, MTA hoops, SpamAssassin, RBL, and other community driven services akin to DCC -- all while retaining your privacy and security

seth - Oct 27, 2009 @ 3:50 PM - Permanent Link
sfarmer, sorry for the delay. Can you post what you are putting when you try adding the second user account? I'll do some searching around and see what I can hunt up. I'll need to figure it out for myself anyway in the near future.

cheap clothes
robbe - May 6, 2010 @ 10:53 AM - Permanent Link
This seems a great idea but one or two things bug me about it.

Firstly; the privacy issue. Big G is becoming very powerful, we don't
want it morphing into Big B(rother).

Also, as noted by niwdoog, the blacklisting issue. When attempting
to set up forwarders with my web host, the user gets the following
message:

"
Forwarding Maintenance
There are known issues with Yahoo forwarders. If you choose to forward your email to a Yahoo address, you run the risk of blacklisting the server.
If this happens, your forwarders will be disabled permanently.
"

They didn't say Google but there must be a risk.
Donal - May 11, 2010 @ 8:32 AM - Permanent Link
Nice article
Andyee - Apr 11, 2011 @ 1:23 AM - Permanent Link
Interesting that you've had that experience, watson. I've had quite different results. I even posted last month how my spam count was reaching over 10,000 a month. Yet GMail was only letting through a fraction of a percent of those e-mails into my inbox. I did that math and it was 0.05% of the spams were making it to my inbox. Also, after a while I was getting so few legitimate e-mails flagged as spam that I stopped even checking. In the past six months, I've only had one that made it in there.

And that's with my catch-all being forwarded to my GMail account as well, so I'm getting all the spam for this entire domain (whether it's to a valid user or not) routed to my GMail account.

But I appreciate you posting your experiences. It does make me wonder what the difference could be that would produce such drastically different results.
Andyee - Apr 12, 2011 @ 6:16 AM - Permanent Link

Post a Reply

Before you may post, you need to either log in or sign up.