Sunday, January 2, 2011

DNSBL Safety Report 1/2/2011

UPDATE: See the latest DNSBL Safety Report for current recommendations.

This blog will occasionally look at the weekly DNSBL masscheck statistics.  Our measures indicate that the performance and safety of the smaller DNSBL's can vary wildly from month to month.  If you depend on DNSBL's, you should pay attention to these safety reports in order to protect your users from the likelihood of false positives and losing mail to the spam folder.  This should help you as a SpamAssassin sysadmin to decide which add-on DNSBL's to use, and what score to assign with the goal of maximizing spam filter safety.

Here is a quick look at the safety and efficacy of a few add-on and existing DNSBL's for SpamAssassin.  Today's report looks into Hostkarma, Spam Eating Monkey, Tiopan, MailSpike, NiX Spam and PSBL.

HOSTKARMA_BL has been measured for more than a year now.  It is generated by a sophisticated reputation tracking system that looks at both ham and spam.  The statistics that matter most are labeled "set 0, broken down by message age in weeks" where you see the performance during the past few weeks.  This week and last week's numbers show around 73% of spam caught, with the cost of ~1.5% ham mistakenly flagged and previous weeks not faring much safer.
Recommendation: Avoid for now until safety improves, or use a very low score like 0.3.  Also their other lists BROWN, YELLOW or WHITE lists are useful for other purposes but not spamassassin scoring.

SEMBLACK has been an inconsistent performer for the past year.  This week's statistics indicate a safer False Positive rate than Hostkarma at 0.7% ham hits during the past week.  This level of safety is still not good compared to other DNSBL's, but good enough for a low score.
Recommendation: Maybe, but with a low score like 0.7 points.

Measured for more than the past year, MSPIKE (previously ANBREP) has had consistently good performance and safety.  This week's statistics indicate 67% spam in the past week, and almost zero ham hits in the past 3 weeks, which seems to be very good.  I was told that MSPIKE's database is 50% smaller than usual due to data loss sometime before last week's masscheck.  We are currently not seeing its full potential as they collect data and rebuild their reputation database over the next few weeks.  This blog will keep an eye on their performance in the coming weeks to be sure more listings does not translate into a safety problem.
Recommendation: I recommend following their instructions to setup their rules.  You have the choice of the simple RCVD_IN_MSPIKE_BL which works just fine, or if you prefer MSPIKE_L3, L4, L5 and Z are the components that combine to BL where you can assign more fine-grained scores.  I personally recommend staying below 2.5 points for any DNSBL rule, and use a maximum of maybe 2.1 points for MSPIKE_BL for now.

This is a new rule to the weekly masschecks.  Statistics are not good at 41% spam and 3.4% ham during the past week.  Fortunately this being only the first week of measurement, I have identified a few systematic changes this operator can make to improve his safety rating.  If you operate a DNSBL, ask me if you wish to be measured once or ongoing on a weekly basis.  We might be able to help you to identify systematic problems in your listing methodology in order to improve your safety ratings.
Recommendation: Avoid for now.

NiX Spam is a DNSBL operated by German media outlet  It is a bit unusual compared to other DNSBL's in that it only lists IP addresses for 12 hours.  For this reason we are having a difficult time measuring its performance using the traditional masscheck mechanisms.  Reportedly it works best for European mail.  Anecdotally I can report that it seems to catch merely ~20% of my spam, but for my American users its only consistent problem is disagreement with whitelist provider ISIPP.  Either NiX is wrong here, or this is pointing out real problems in IADB which has been controversial in essentially being a "commercial  pay for DNS whitelisting" service.  In any case the ham hits are such a low score, that NiX alone will not bring it anywhere near the 5 point spam threshold.
Recommendation: Maybe.  If you choose to use NiX, I recommend a low score like 0.9 until we have figured out a better way to measure its performance with actual statistics.

Passive Spam Block List: RCVD_IN_PSBL
spamassassin-3.3.0 added RCVD_IN_PSBL to the default ruleset.  Statistics are excellent for PSBL as one of the safest DNSBL's in existence.
Recommendation: If you are using an older version of spamassassin like 3.2.5 then this is an excellent add-on rule for your server.

header   RCVD_IN_PSBL eval:check_rbl('psbl-lastexternal', '')
describe RCVD_IN_PSBL Received via a relay in PSBL
tflags   RCVD_IN_PSBL net
score    RCVD_IN_PSBL 2.3
How Can You Help SpamAssassin Improve?
SpamAssassin is only as good as the variety of mail represented by the participants in masscheck.  Currently very few people contribute mail samples to the nightly masscheck.  SpamAssassin is nearly PERFECT to my own mail because the variety of mail that I receive is analyzed on a nightly basis.  More mail variety is sorely needed, especially non-English and in non-technology fields.  SpamAssassin can improve substantially in spam detection and safety for those who participate in masschecks.

Do you operate a mail server with spamassassin?  SpamAssassin upstream is in dire need of more participants in the nightly masschecks.  Basically, you put together a folder of ham (and optionally spam).  Every night in cron a script is run to analyze your folders, and logs of statistics are uploaded to SpamAssassin.  Your mail never leaves your server, so there is no privacy problem.  Everyone's logs combine to become the nightly masscheck results you see at RuleQA.  More participants in the nightly masschecks means new rules and DNSBL's can be checked for safety.  It is impossible for spamassassin to improve or to release major versions without masscheck participation.

Read my auto-mass-check page to learn how to get started with nightly masschecks.  It is fairly confusing due to upstream's disorganized and sometimes self-contradictory documentation.  Please e-mail me if you are trying my instructions and you need assistance.

No comments:

Post a Comment