[OS:N:] Open Source and Filtering

["Ken Willett" <kwillett ignisys com>]

As Evan points out, blacklists (and the manual effort to create them) are
the bread and butter of commercial filterware companies. If these lists were
readable, they could be easily converted for use by a competitor's (or a
free) blocking tool. The good filtering databases involve a lot of research,
as well as feedback from customers. The "Artificial Intelligence" approaches
don't work (see the previous comment on trying to research an anatomy
paper).

SquidGuard, which we use, uses primarily an automated analysis (I don't know
the details), together with incorporating feedback from users about sites
which should or shouldn't be on the list. So their list is pretty rough, but
the good news is that you have full access to it. I have found that it tends
to err in blocking too many sites (or at least that's what I am likely to
notice; I don't go looking for obscure porn sites to make sure they are
blocked).

It would be great to have some sort of cooperative open source filtering
database, where contributions from multiple sources could be aggregated.
This would eliminate a lot of duplicated effort. Maybe it could include a
scoring module, so that if there was disagreement among users about how
objectionable a site was, it could be arbitrated...