What's with bogofilter and spam

[Jonathan Kamens]

On 06/04/2009 07:53 AM, Rodd Clarkson wrote:
> Sadly, I'm not feeling like being manual about this, and I guess that I
> just expect my mail client to work well with the spam software and do it
> for me.  After all, my mail client has a great collection of ham and
> spam so if I can do something like it manually, then surely it can't be
> hard for the spam software to do it without me having to thing about it.
>
> bogofilter used to work well, and I'm hoping that it can once again be
> the great spam filter it was, fast and accurate.
>    
I can't speak specifically to the integration of bogofilter into 
evolution, since I use a completely home-grown bogofilter integration, 
which, as shown here <http://stuff.mit.edu/%7Ejik/#spam>, successfully 
blocks thousands of spam messages and viruses per day.

However, I do want to reiterate what Anne said.  She's right that the 
spammers are getting smarter.  They're /always/ getting smarter -- it's 
a constant battle for the anti-spammers to keep up with the new ideas 
that the spammers come up with.  Therefore, what worked well enough is 
no longer good enough.

For bogofilter to be most effective, here's what needs to happen:

   1. Incoming email needs to be divided into three categories -- ham,
      spam, and unsure -- not just into ham and spam.
   2. Ham and spam messages needed to be added to the bogofilter
      database automatically after they are categorized.
   3. Unsure messages need to be categorized by the user and then added
      to the bogofilter database as either ham or spam, depending on the
      user's categorization.
   4. Incorrectly classified messages need to be reclassified when they
      are detected, e.g., a spam message incorrectly classified as ham
      needs to first be removed from the database as ham and then added
      to the database as spam.
   5. Bogofilter needs to be tuned periodically using a large collection
      of known-ham and known-spam messages.
   6. The bogofilter database needs to be pruned periodically, i.e.,
      words that haven't been seen in any incoming email in a while (I
      personally use 180 days as my threshold) need to be removed,
      preferably before tuning.

All of these are important, but the first four are by far the most 
important.  If the evolution integration doesn't use tristate 
classification, or if it doesn't make it easy for you to identify and 
classify unsure messages and reclassify incorrectly classified ones, 
then it is inevitable that over time, bogofilter's ability to detect 
spam will degrade.

   jik

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://listman.redhat.com/archives/fedora-test-list/attachments/20090604/20a8c392/attachment.htm>