[Xitami-Dev] How to build a spam filter?

[Pieter Hintjens]

Bob Paddock <bpaddock@csonline.net> wrote:

> Long ago I asked these questions and received these answers here on
> the list...

We turned off the CVS because it was almost impossible to construct 
SFL/SMT from it.

In the meantime, we've continued to work on our tools and things are 
looking up.

I think the idea of a portable spam filter is a really good idea.  
But then I love the idea of portable software in general.  I have to 
say that we've been using SpamAssassin at iMatix, and it works 
exceedingly well.

We have packaged all our open source tools into RealiBase, which is 
our base package for realtime software (hence the name).  Xitami is 
separately packaged.

I've put the current build of RealiBase up on imatix.com, at:
   http://imatix.com/pub/rbase_1_0_src.zip for Windows
and 
   http://imatix.com/pub/rbase_1_0_src.tar.gz for Unix

We will start automating this update, and probably use Sourceforge 
for downloads.

To build from the source package you need Linux and gcc, or Windows 
2000 and MSVC.  If you want to use other compilers on Windows, you 
will need to add support into the c.bat file before building.

To build, unzip the source package and run the build.bat or build.sh 
script.  This builds and installs into the default installation path, 
which is c:\imatix on Windows and /usr/local/imatix on Unix.  In both 
cases make sure you have write access to these directories.  You can 
install into another location by setting the IBASE variable first.

e.g. to build and install into a user directory on Linux:

   export IBASE=~HOME/base
   sh ./build.sh

Next, what is in RealiBase?

 - SFL, SMT, Libero, Htmlpp
 - Gaggle, a group of miscellaneous tools
 - Zlib, PCRE, two packages we use in GSL
 - Gurudoc, our replacement for htmlpp, still in progress
 - GSL/3.0

Of which, the last is probably the most interesting.  This is the 
first release of GSL/3.0 to the world.

Now to your question about POP3/SMTP.  The SFL functions did not work 
very well in previous releases.  We've improved them considerably, 
but they remain single-threaded and thus not ideal for volume use.  
We have built SMTP and POP3 agents in SMT, which are more interesting.

But, the most interesting possibility of all is to use GSL.  We have 
integrated PCRE (the Perl Compatible Regular Expression Library) and 
GSL/3 now makes it easy to extend the language by adding extensions 
like this (we also added zipfile functions from zlib).

GSL can script SMT agents.  This means that you could probably use 
GSL to build the spam agent, at least to prototype it.  You can 
define your spam filter lists as XML files, of course.  You can even 
multithread GSL scripts, to - e.g. - clean-up multiple POP3 boxes in 
parallel.

-
Pieter Hintjens
iMatix Corporation