Monday, November 27, 2017

Hit Count and Last Hit

After having left MAPS pretty much alone for a long time, though I did implement "nuke" which helps nuke whole domains, I finally have implemented new functionality for MAPS. You see over this last year or so I've been getting a lot of spam mostly oriented in which the domain name was wildly varying. So lots of mail loops were happening. Whacking whole domains became a favorite past time of mine and hence the "nuke" script (not an official part of MAPS mind you). Still it seemed like a losing battle.

Then I got to thinking - something I've been thinking for a while - how effective are all of my null list entries, now numbering in the 17K area! I thought that I could easily add some fields to the list table, namely a hit_count and a last_hit field, and then during scrubbing time see if these null list entries were at all effective, and if not then to automatically remove them.

So I added the fields and changed MAPS such that it records the hits and when the last hit happened. Then in mapsscrub check to see if last_hit is older than the number of days the user is keeping history. If it's 0 then simply remove that entry. Note this happens only for the null list. White list and black list are to be managed by the user. You purposely add somebody to the white or black list for a reason. Do we want to auto age these off? Probably not.

If the hit count is not 0 then how do we age off these apparently no longer effective null list entries? Well the though was to age them by subtracting 1 from the hit count. Thus if we had a null list entry that was fairly active for a little while and say accumulated a hit_count of 10 but then was largely un-hit, the hit_count would go 9, 8, 7, 6, 5... down to 0 and then the entry would be removed. I might want to change this as some null list entries are getting hit_counts in the thousands now! Assuming the user has a 30 day history then perhaps after 30 days of inactivity I should age any null list entry off the list in 7 days? Note that any activity in those 30 days would immediately reset the clock for at least the next 30 days (assuming a history of 30 days).

With this in place I decided to change the mail loop procedure to auto add the sender to the null list. Previously I was whacking off the whole domain - here we are just auto adding the individual sending address. But it's automatically added, automatically tracked for effectiveness and automatically aged off. Beside I still have the option of perusing the top returning domains and can whack whole domains still. Now any hits on any users in the domain will continue to insure that that domain null list entry will remain and not automatically age away...

No more

Due to an influx of spam getting through MAPS with a format of <something> I finally decided to fix MAPS to handle that better. Now if the from is from (or the current domain) then the user name is looked up in /etc/passwd using getpwent and if there is no user of that name it is simply nulllisted. This is an obvious attempt by spammers to forge from email addresses.

Also removed logging of when the message is obviously garbled or not conforming to standards. Plus I no longer log if the user uses but fails to have a name such as "Andrew DeFaria <>". The former turns out to be mostly Nigerian spam anyway.

Finally, replaced /usr/local/maps with a direct checkout form CVS!

MAPS Scrubber

Updated mapsscrub to also report total number of user emails in the database. Also fixed header.

Added page drop down

Added a page drop down to list.php. This is useful when one wants to say go to the last page.

Move list entry bug

Found a bug today. While adding to the blacklist an entry was on the white list, MAPS correctly adds the entry to the black list and deletes the entry on the white list. However it fails to resequence the white list causing problems down the line.

Login bug

Squashed a nasty bug, well actually a lack of implementation. After converting some of the web pages to PHP I neglected to implement Login in PHP! Now implemented.


Added new "Space Usage" report. Not sure if I like this because 1) it's too slow and 2) the resulting report is not very impressive - it's just a line saying "User is using X bytes in the databsase". Might wish to implement a space counter in the user table. This would require that all procedures updating the database update the space counter. The intent for this space counter is to judge how much space a user is using in the database for the purposes of quotas or perhaps to charge people if they want more space.

Also, currently this space report is only counting space used in the email table. Might wish to add in the log table, which can add up, and the user and useropts tables. The latter two are minimal but including them would make the report more accurate.