Sunday, May 15, 2011

DNSBL Safety Report 5/14/2011

SpamTips.org occasionally looks at the results of Spamassassin's nightly masscheck at RuleQA in order to analyze the performance and safety of add-on DNSBL's.  It is vitally important to know how a DNSBL is performing before adding it to your Spamassassin custom rules.  Our analysis demonstrates that raw detection numbers alone can be misleading, as ham safety ratings and overlaps with other rules must be taken into consideration before you decide to use a particular add-on rule.

Today's report shows some big changes since our previous report from January 2011 where some previously good rules have turned bad.   Examined below are Hostkarma, SpamEatingMonkey, Tiopan, UCEProtect, Mailspike, Nix Spam and Lashback UBL.  Recommended scores below are what I personally use in production.

Spamassassin-3.3.2 has been released!  [READ MORE]

MailSpike: RCVD_IN_MSPIKE_BL
One month ago I noticed a big improvement in RCVD_IN_MSPIKE_BL.  Last month and again today, MSPIKE_BL demonstrates stellar performance in both spam detection and safety rating, second only to RCVD_IN_XBL.  Nearest high scoring overlap is RCVD_IN_PSBL at ~65% (66% back in January).  This demonstrates continued consistency in both safety and independence of data sources, which further confirms that MSPIKE_BL is the most valuable addition to your production Spamassassin rules.

Recommendation: I highly recommend following their instructions to setup their rules.  You have the choice of the simple RCVD_IN_MSPIKE_BL which works just fine, or if you prefer MSPIKE_L3, L4, L5 and Z are the components that combine to BL where you can assign more fine-grained scores.  I personally recommend staying below 2.5 points for any DNSBL rule, and use a maximum of 2.1 points for MSPIKE_BL for now.

Hostkarma: RCVD_IN_HOSTKARMA_BL (Caution)
Our previous reports from January 2011 and earlier indicated improved safety levels and reasonable independence.  But last month I noticed a big change in HOSTKARMA_BL's overlap ratios, where it overlapped with MSPIKE_BL a startlingly high 88% of the time.  This week it is a 85% overlap. 

Recommendation: HOSTKARMA_BL detects fewer than half as much spam as MSPIKE_BL while being slightly less safe.  Since it is most often redundant to MSPIKE_BL, it can be dangerous to stack both scores together.   For this reason I currently recommend that it probably isn't helpful to use this DNSBL, or if you do, a harmless and informational score like 0.3 would be advisable.

SpamEatingMonkey RCVD_IN_SEMBLACK (AVOID)
Back in January 2011 our analysis found SEMBLACK to have a very good safety rating.  But I cautioned at the time that we need to see consistent performance over the course of many months or years before recommending its use.  This caution turned out to be well warranted as analysis of one month ago and today both indicate a significant decline in safety with false positives on as much as 5-6% of ham.  To make matters worse, continued high overlap of 83% today with RCVD_IN_PBL makes this an extra dangerous rule.

Recommendation: Outright Avoid.

UCEProtect Level 1 (AVOID)
Our latest results indicate relatively consistent but poor performance since our January 2011 analysis: Moderate spam detection, but consistently poor safety at ~1.7% ham false positives.  Overlaps have also been relatively consistent: 88% HOSTKARMA, 87% MSPIKE_BL, 71% PSBL.

Recommendation: Too much overlap with quality rules PSBL and MSPIKE_BL and poor safety on its own means there is no good reason to use this rule.

Nix Spam: RCVD IN_NIX_SPAM (Caution)
NiX Spam is operated by German media outlet Heise.de.  It is a bit unusual compared to other DNSBL's in that it only lists IP addresses for 12 hours.  For this reason we are having a difficult time measuring its performance using the usual masscheck mechanisms.  Reportedly it works best for European mail.

While RuleQA is unable to give any statistics, I can report results from my own servers.  During the past month until May 15th, NIX_SPAM had ~1.7% false positives and 52.5% spam detection on my mail server, consistent with 1.8% FP and ~50% spam measured in January 2011.  Its failures remain consistent: some legitimate Japanese companies, prominent American subscriptions and various senders in the IADB whitelist.  Overlaps today were measured at 78.6% MSPIKE, 63.1% PSBL, 97.9% HOSTKARMA, 62.6% PBL, and 54.3% SEMBLACK.  These unusual overlap measurements is from NIX_SPAM's unusual 12 hour listing methodology, suggesting to what extent these other DNSBL's use a similar real-time data sources for a subset of their listings.

Recommendation: Due to its poor safety rating I cannot recommend that anyone use this DNSBL.  However it has at least shown consistent performance, so it might be worthwhile to use with a low score, especially if your users are mostly in Europe.

RCVD_IN_TIOPAN_BL (Improved, but Still Avoid)
Our last analysis in January 2011 indicated very poor safety (1.6% - 17% FP) and a high overlap of 78% with PBL.  Today's numbers indicate a significant improvement in safety (0.5% - 3% FP) and some improvement in overlap at 70% PBL.  Intriguingly, the next nearest DNSBL overlap is in the low 50%'s suggesting that Tiopan has a very unique source of data.

Recommendation: Tiopan's safety rating is still far too poor for anyone to use.  If it continues to improve in safety while maintaining this unique level of low overlap it could become a promising DNSBL in the future.


Lashback UBL (OUTRIGHT AVOID)
The last time we tested Lashback's UBL in masscheck was during late 2009.  It performed very poorly in that test with 7.9% spam and 2.3% ham detected.  I attempted to test it again locally in January 2011 but it was a complete failure, with their DNS servers responding anywhere between zero and 12 seconds.  Most DNS queries used by spamassassin respond in less than 0.3 seconds.  This can delay your spamassassin filter by several seconds per message before it hits the forced DNS timeout.  The weekly statistics at Intra2Net in past months have shown a marked decline in spam detection rates from earlier levels.

Recommendation: We remain unable to test it because it is so unreliable.  Meanwhile, we are not the only independent tester to suggest that this DNSBL is problematic.  You should outright avoid this DNSBL.

Subscribe to the Spamassassin for Sysadmins Newsletter for occasional news important to your Spamassassin deployments.  Also see our Ultimate Setup Guide for the latest tweaks to maximize your spam filtering effectiveness and safety.

3 comments:

  1. I'm writing you on behalf of the NiX Spam DNSBL. How can we get in contact to improve the performance? First I would like to know which IPs and/or ranges mustn't be on the blacklist.
    Could you please contact me at support (a) ix-lab.de?

    ReplyDelete
  2. Replies
    1. I previously spent hundreds of hours improving spamassassin and working data gathering and statistics to improve the software. Since then other priorities have rendered me unable to work on spamassassin. At this point unless substantial donations came in, I am afraid I need to prioritize moving on with a real career. I am sorry.

      I intend on building new RPMS for RHEL5 and RHEL6 of the upcoming spamassassin-3.4. I did not participate at all in the rule or score generation for this release, so I have no idea if it is an improvement over past releases, or if it remains safe for a few non-English languages that I previously helped to fix.

      If you wish to donate to encourage me to spend more time on Spamassassin, please contact me privately to discuss private consulting.

      Delete