[Xitami-Dev] How to build a spam filter?

[Bob Paddock]

Long ago I asked these questions and received these answers here on the list:

Fri, 11 Jan 2002 10:23:36 +0100, Pieter Hintjens wrote:

>> Before I go reinventing the wheel has any one here done any thing 
>> with POP3 & SFL/SMT?

>Yes, we have a POP3 client written with SMT/SFL.  Let me just check 
>that it's actually in the public CVS... okay, it's not, so we'll put 
>it there and you can access it later from: 

 >   http://cvs.imatix.com./cgi-bin/viewcvs/

Fri, 11 Jan 2002 16:38:55 +0100, Pieter Hintjens wrote:

>Just FYI, there are now two new agents in the SMT library: smtpop3 
>and smtsmtp, plus two matching test programs.  These agents provide 
>multithread POP3 and SMTP access.  All comments welcome!

I know that CVS has been turned off.

I would like to procede with trying to write a spam filter with SFL/SMT.

The one that diverted me for all of these months was http://mailfilter.sf.net/ 
.  Which I still see as a very nice package.  However it has become so 
intangled in the GNU way of ./configure needed things like Flex, Bison, 
ad-infinitum to do a simple C++ compile, that it has literally become 
impossible for it to be built on some systems (like mine).

Getting 100+ spams per day is good motivation.  :-)

I know there are other spam filters out there, but none that I've found that 
will work on both Windows and Linux, with the same configuration file.   I 
know I can run code under Cygwin to solve the problem (like Mailfilter) but 
it is much slower than native code.

My question is can I get the newer POP3 / SMTP code now?

On Saturday 19 January 2002 10:27 pm, Ewen McNeill wrote:

> Finally, yes, making this all easier to build is on our list of things
> to do -- pretty high on our list these days :-)

After 10 months has it made the top of the list yet?  The circular build 
problems are the reason I gave up on SFL/SMT for a while.  Now I wish I had 
stuck with my original plan for my spam filter.

Unlike a web server (Xitami), or biz. apps. (STEP), every one needs a spam 
filter these days.  Look at this project as good marketing for you and yours.  
:-)

This is the concept, taken largely from Mailfilter:

There is a single configuration file.

It contains the following sections:

1) Server info (Where do I get the mail from).
Encrypted passwords (not plan text), a frequent request on the Mailfilter 
list.

2) My address list.  If the message is not sent To/Cc: one of these addresses 
it is deleted as spam.  Mailfilter has such a feature but it is hard to set 
up.

3) List of addresses to allow mail from unconditionally (never delete it).

4) List of addresses/subjects/headers to deny mail from unconditionally 
(delete it).  #3 over rides #4.  To save time processing stops at the first 
matching rule.

5) Some day (in other words I don't know how right now), allow AND/OR/IF type 
logic to the rules.  Anyone here have any suggestions?

#6) As much as practical keep the configuration file compatible with 
mailfilter.

Assume all of the above are using Regular Expressions for matching, unless 
some one here has a better idea.

Only the headers are read on the server (POP3: TOP x 0), never the text of the 
message, for speed reasons.  You can not delete a message based on its 
message content.

Unlike the current version of Mailfilter this would be a two pass system, to 
gain speed.  Normal pass, typically run in a Cron job, identifies the spams, 
but does not delete them.  This keeps the message pointers from being 
renumbers, so that you don't have to do a complete rescan each time.  When 
you actually pick up the mail the messages are really deleted before 
importing them to your mail client (different command line option)

Subtitlies learned from experience:  
 Subjects needed to be normalized before trying to match.
   Subject: Viagra should be the same as these:
   Subject V I A G R A
   Subject V#$#I.A#G@R!A

 Very complex regular expressions break the standard GNU RE library randomly, 
and unpredictably (internal stack over flow?), so don't use it.
  
  RE matching seems like it would be right up the ally of SMT.  Is there such 
a SMT RE library? 

 If not use this RE library:  http://www.ddj.com/articles/2001/0110/

If there is a need for I18N issues, stay away from GetText (Mailfilters 
downfall because of all of the build issues), and use 
http://www-124.ibm.com/icu/ .

 Need to have option of case/or no case sensitivity in rule matching.

Examples:

  From: johndoe@x.com
  To:  MyAddress@here.com
  Subject: Buy your Viagra here.

 Message is deleted because subject Viagra is considered spam per #4 above.

  From: MyWife@home.com
  To:  MyAddress@here.com
  Subject: Bring home some Viagra tonight.

  Message is kept because it is from my wife, and I won't be spending time 
working on this spam filter code tonight.  :-) #3

  From: johndoe@x.com
  To:  IhaveNoClue@WhoThisIs.com
  Subject: Bring home some Viagra tonight.

 Message is deleted because it is not To/Cc: one of my addresses (#2), and not 
#3.

How do we get there?

-- 
      http://www.unusualresearch.com/ http://www.bpaddock.com/