mu.nu - It's mackerel stewed in chocolate!
November 24, 2005
Snark!
Version 1.2 of Snark!™ - our new trackback filtering system - is up and running.
You can see real-time statistics here Hooray for Snark!™ Update: Version 1.3 of Snark!™ is now up and running with a high score list in addition to the running totals. Update: Version 1.4 of Snark!™ offers an automatically generated blacklist. Update: Version 1.5 of Snark!™ tracks and lists IP addresses as well, though that's only been running for a few minutes so there aren't any addresses listed yet. Update: Version 1.51 has the added feature of not crashing when there's no spam, a situation that never arose during testing. Update: Version 1.6 of Snark!™ automatically and permanently blacklists anyone whose spam ranking goes over the magic number. Update: Version 1.7 keeps track of the effectiveness of the various filters. Those stats are a bit screwy right now because of... Well, because. But they will improve over time. Update: Version 1.71 fixes some bug or something. I've forgotten. Update: Version 1.72 makes the decay algorithm fully adaptive. Basically, that means that high-volume spammers are considered particularly annoying. This makes no difference for our internal filtering, but I think makes the stats more reflective of the actual evilness level of particular spammers. Update: Currently running version 1.84, which addresses all known bugs and inefficiencies. It's able to process 20,000 trackbacks per second, removing roughly 99.9% of spam and so far producing no false positives. It is a thing of beauty.
Posted by: Pixy Misa at 07:53 AM | Comments (25) | Add Comment | Trackbacks (Suck)
1
That is sweet!
Posted by: Jim @ at November 24, 2005 02:53 PM
2
I love it!
Posted by: David Boxenhorn @ at November 25, 2005 10:47 AM
3
This is the bit I like:
Blacklist Entries: 18 (plus 55 manual entries)
Session Uptime: 0 hours 18 minutes
Pings Received: 1013
Processing Time: 0.23 seconds
Snark filtered out 1013 trackbacks in 0.23 secooioooooooooooooooooooooooooooooooooo sorry, I just broke one of the keys on my laptop. Hang on. Nnnnngggh. Right.
Snark filtered out 1013 trackbacks in 0.23 seconds. Movable Type takes twice that long to filter out one trackback.
Blacklist Entries: 18 (plus 55 manual entries)
Session Uptime: 0 hours 18 minutes
Pings Received: 1013
Processing Time: 0.23 seconds
Snark filtered out 1013 trackbacks in 0.23 secooioooooooooooooooooooooooooooooooooo sorry, I just broke one of the keys on my laptop. Hang on. Nnnnngggh. Right.
Snark filtered out 1013 trackbacks in 0.23 seconds. Movable Type takes twice that long to filter out one trackback.
Posted by: Pixy Misa @ at November 25, 2005 11:36 AM
4
Another funny this is what I wrote a couple of weeks back: We got 1.3 million trackbacks over the past 30 days, and 550,000 comments. About 99.8% of trackbacks and over 90% of comments were spam. That was just an off-the-cuff estimate, not the result of careful calculation.
But Snark says:
Since Startup: Total 39842, Good 78, Bad 39764, % Spam 99.80
Not bad for a guess!
But Snark says:
Since Startup: Total 39842, Good 78, Bad 39764, % Spam 99.80
Not bad for a guess!
Posted by: Pixy Misa @ at November 25, 2005 01:37 PM
5
I've just been deleting trackback spam from my blog for the third time today.
Dammit, now I want to come back to MT 2.6!
Dammit, now I want to come back to MT 2.6!
Posted by: Pixy Misa @ at November 25, 2005 02:00 PM
6
Tweak tweak:
Blacklist Entries: 23 (plus 60 manual entries)
Session Uptime: 5 hours 44 minutes
Pings Received: 1530
Processing Time: 0.24 seconds
Blacklist Entries: 23 (plus 60 manual entries)
Session Uptime: 5 hours 44 minutes
Pings Received: 1530
Processing Time: 0.24 seconds
Posted by: Pixy Misa @ at November 26, 2005 10:51 AM
7
Latest version (1.84):
Blacklist Entries: 23 (plus 60 manual entries)
Session Uptime: 8 hours 28 minutes
Pings Received: 1024
Processing Time: 0.14 seconds
Blacklist Entries: 23 (plus 60 manual entries)
Session Uptime: 8 hours 28 minutes
Pings Received: 1024
Processing Time: 0.14 seconds
Posted by: Pixy Misa @ at November 26, 2005 11:42 PM
8
Aha! I wondered why my inbox wasn't filled with millions of spam pings here lately. I had attributed it to a higher power. It wasn't, however, the Great Pixy Misa who came to mind.
Posted by: Tig @ at November 27, 2005 02:40 PM
9
Yay!
It does seem to have done the trick. There's still a bit of spam leaking through, but it is enormously reduced.
As I noted, my blog, which is running on MT 3.1 and not protected by Snark, is getting more trackback spam than all the MT 2.6 blogs put together. Since I installed version 1.2, you've got an average of one-third of a spam each. I'm getting thirty a day.
It does seem to have done the trick. There's still a bit of spam leaking through, but it is enormously reduced.
As I noted, my blog, which is running on MT 3.1 and not protected by Snark, is getting more trackback spam than all the MT 2.6 blogs put together. Since I installed version 1.2, you've got an average of one-third of a spam each. I'm getting thirty a day.
Posted by: Pixy Misa @ at November 28, 2005 04:09 AM
10
So now you just shrink wrap and sell for $10 a pop - every blogger in the universe is at your mercy! Yet again, Nobel Prize for Blogging Services goes to Pixy.
Posted by: Simon @ at November 28, 2005 06:50 AM
11
Problem with that is it only works well if you have a large number of blogs. I suppose you could also do it with a one hour delay instead of one minute.
However, by keeping it to ourselves we make munu an even more enticing place. Let's face it, the trackback spam was getting to be a real drag, but now it's just gone. (Well, not for me; I just got another 20 in the past two hours. I'll have to do something about that.)
However, by keeping it to ourselves we make munu an even more enticing place. Let's face it, the trackback spam was getting to be a real drag, but now it's just gone. (Well, not for me; I just got another 20 in the past two hours. I'll have to do something about that.)
Processing 0.01, elapsed 0.0164 seconds.
22 queries taking 0.0128 seconds, 33 records returned.
Page size 12 kb.
Powered by Minx 0.8 beta.