Posts filed under 'Technology'
There’s only one thing that drives me nuts about daylight time; people (nearly everyone) who insist on calling it “Daylight Savings Time”. There are no savings folks, you can’t deposit sunlight into your bank account. At least you can’t at my bank, where you’re having a good day if you get to deal with someone who can successfully deposit real money into your account. Other than that, I love daylight time… I get to sleep longer without being bothered by the pesky sun.
It’s Daylight Saving Time. If you need practice, repeat the following sentence a few times before bothering me.
“I don’t want to pay Microsoft $4000 for the hotfix to make the Daylight Saving Time change on my Windows 2000 machines, can you help?”
March 12th, 2007
Apache SpamAssassin 3.1.8 was released on Wednesday. It would be a good idea for most people to update — especially those processing a lot of mail for a lot of users/domains. People using sa-update with channels from sources that they don’t particularly trust to adequately secure their channel to prevent it being compromised should also update. With versions prior to 3.1.8 a channel, that you use, compromised by a malicious party could turn your SpamAssassin install into a spambot (or anything else that could be done with privs sa-update or other SA software runs as on your machine). Cool. The spambot possibility is the reason I pushed for the new –allowcode option (disabled by default).
Speaking of sa-update channels. People using the “Openprotect” channel are shit-out-of-luck for now if they’re using 3.1.8. Apparently they publish a separate txt record for each version of SpamAssassin. Actually, it’s not even each version of SA that supports sa-update — only versions 3.1.3-3.1.7 are currently supported by their channel. I suspect that their free DNS provider (speaking of not totally secure channels — make sure you’re not disabling the gpg key check for this channel since they outsource their DNS) doesn’t allow wildcards. I don’t know why they wouldn’t anticipate new releases in advance, though, and add records for, say, versions up to 3.1.15, or at least one more than the version number in current release.
Oh yeah… people using the “Openprotect” channel may or may not be affected by the new “–allowcode” thing in sa-update. The “Openprotect” folks decided to enable 12 plugins for you in their updates. Some of these plugins aren’t enabled by default in the SA distro and sa-update now won’t load these plugins (using the loadplugin lines in the updates, anyway) without you including the “–allowcode” option in your call to sa-update. Depending on whether or not their rule files include the proper “ifplugin” lines the channel might not pass a lint (and thus not be installed) if you haven’t loaded the required plugins yourself in your system config (which sa-update now uses to load the plugins you’ve enabled in your setup when linting an update).
Now that I’ve probably pissed off everyone involved with the “Openprotect” channel and all those who use it (really that’s not my intention — I’m trying to give people a heads up so that they don’t waste a bunch of time trying to figure out what’s going on) I’ll point out that the SARE sa-update channels that I provide aren’t at all affected by any of this. My channels are also a lot more flexible… you get to pick your own rulesets. See this page for usage details or this post for why I set these channels up.
February 15th, 2007
Twice this month I was reminded why when you’re relying on someone else to backup data to any sort of removal media (or any media at all really, but removal media seems to be the worst) you should always if at all possible keep as many duplicate copies of the data being backed up on the same network (or if necessary, same machine) as the source data.
Why? Without fail, the most technologically clueless person in the orginization will blindly destroy the backups — all of the backups — that they know about whenever there’s a problem accessing the source data on the system. Always. The big red letters on the backup media itself saying not to do a backup if they’re having problems and to call whoever manages the system will never stop them. Nor will all the messages, saying the same thing, that the backup process spits out.
I’ve managed a number of point of sale stations and networks for a number of small (sales under 5 million or so a year) independent retailers for about 15 years. A lot of the systems are quite out dated — lots of Racal InterLan token ring gear, 10base5, Novell 2, NetWare Lite, LanMan, 386s, some Samsung 286s, and a whole lot of dBase. Along with the dBase comes a number of programs that can’t resist corrupting any dBase database they touch, the worst being early 90s era VendorWare. With dBase and VendorWare you’d better have rolling backups coming out of your ass.
Anyway… both times this month it was a case of corrupted dBase databases and VendorWare. As long as you’ve got a recent backup of the database it’s easy to fix after a VendorWare program has trashed the database since 99% of the time it trashes a database that rarely changes or doesn’t have any data you can’t easily get back in sync. Well, what do you know, before contacting me both of the organizations decided (like usual) that they’d try to make it work themselves. The first step in their troubleshooting process… try doing a backup to each set of backup media (overwriting the backups of the working database) and checking to see if things magically started to work.
Here’s the great thing about point of sale systems (POS) these days; they’ve almost always got a tonne of free disk space. This applies to both current POS software and ancient POS software. The ancient stuff was designed to run on systems with 40 or 80MB of space… most of those systems now have at least 512 MB partitioned (usually on a 10 or 40+ GB disk with an overlay on it). With all of that free space you’ve got lots of room to keep a week, or a month, or even a couple months of daily (or more often) copies of their databases right on the same disk. It’s useless if the disk goes up in flames/whatever — that’s why you’ve got the removal media — but it’s priceless for when there’s just data corruption caused by the POS software; you’ve got all the copies you’ll need to get the database back in a functional state.
Of course, if the users didn’t blindly sabatoge any chance of you using the removal media, the on disk copies wouldn’t be necessary (although the on disk copies are so much faster to use anyway), but if there’s one constant I’ve found with these type of installations (or any installation really) is that no amount of training, warnings, documentation, or whatever, is going to stop them from trashing your only backups — if they know about them.
Bottom line, keep backup copies on the same system as the source data and don’t tell them about it. If you tell them about it they’ll surely delete it, somehow. There shouldn’t be any sort of liability issue with it… if they’re purposely trying to destroy data (say to avoid pesky tax auditors, or whatever reason you can imagine someone would want to destroy their data for), anyone competent enough to figure out where the POS software is storing its data should notice the backups on the disk. You can usually get away with making it quite obvious to them too… store the backups in directories called backup.1, backup.2…, in the main volume… the clueless staff won’t notice them (the malicious staff is a different story — but that requires a whole different backup strategy).
February 1st, 2007
Looking at my httpd logs for the sa-update channel for SARE’s 70_sc_top200.cf that I host at 70_sc_top200.cf.sare.sa-update.dostech.net (howto), I’m wondering why about 25% of the IPs (and about 17% of the /24s) use it. If you’re running network tests SpamAssassin already queries SpamCop by default, so if you’re also using this channel (or ruleset via RDJ directly from SARE) you’re just adding a, usually outdated, copy of the same data you’re getting via DNS lookups.
My guess is that there’s a lot of people just not paying too much attention. I can’t imagine that there’s that many systems running without network tests, at least not on purpose (I know there’s a lot of people that think they’re using network test, or all the tests available to them, but don’t realize that their distro disabled network tests by default).
I mentioned that the SpamCop data in this channel/ruleset is usually outdated… here’s a list of the updates since the beginning of December (I don’t know why Fred hasn’t automated the updates, last I heard he was manually uploading the ruleset — which itself is generated automatically):
Dec 1 13:04 200612011100.tar.gz
Dec 1 19:04 200612011700.tar.gz
Dec 2 15:04 200612021300.tar.gz
Dec 2 16:04 200612021400.tar.gz
Dec 5 18:04 200612051600.tar.gz
Dec 6 14:04 200612061200.tar.gz
Dec 6 18:04 200612061600.tar.gz
Dec 9 13:04 200612091100.tar.gz
Dec 13 18:04 200612131600.tar.gz
Dec 23 16:04 200612231400.tar.gz
Jan 2 11:24 200701020900.tar.gz
Jan 8 10:24 200701080800.tar.gz
Jan 9 12:24 200701091000.tar.gz
The update frequency is just weird… in the past I’ve noticed that it is sometimes updated three times in the span of two hours and then not updated again for weeks.
January 11th, 2007
I’ve known of Rogers doing this for years, but of course, I forgot all about it until it happened to me.
About a month ago I was having problems with a Linksys WRT54G locking up sporadically. Knowing some of the Linksys $10 routers have had issues under load, probably running out of memory or something, I configured a server running a SpamAssassin daemon (spamd) to forward all of it’s DNS queries to the DNS server assigned by Rogers via DHCP in an attempt to decrease the amount of state data the router had to keep track of for all the UDP queries.
That seemed to help a little, but not much, a firmware update released in December seems to have fixed it, but that’s besides the point.
The problem with using Rogers’ DNS server is that they run a script on their query log that looks for clients who send a lot of queries that result in NXDOMAIN, indicating a machine that could be a spam zombie or otherwise infected by malware. The problem is that the script doesn’t care what the queries were for, just that they returned NXDOMAIN. So if you’re using SpamAssassin, or any other anti-spam method, and are using any sort of DNSBL, you’re going to end up getting a lot of NXDOMAIN results. Specifically, you’re going to get lots of them for every message you check.
So anyway, without any notice (of course), Rogers disabled the highspeed internet service to the cable modem this was all sitting behind. After spending two hours on hold waiting to talk to someone, I managed to (a) get them to re-activate the service, (b) tell the poor guy who insisted I try and get a position in their networking department why what they were doing was a pretty good idea, but could be implemented better, and (c) find out that the threshold for NXDOMAIN query results in a single day is really low, as in “like way less than 300″. For someone filtering their own mail to one or two addresses, it’ll probably only take them a few minutes (and certainly no more than an hour) to hit 300 NXDOMAIN results. I know that mail to just my personal domain will trigger than in only a few seconds.
Once the service was re-activated, I configured Bind to forward all queries to my own DNS servers which have a huge cache or “spam query” results (so it’s probably faster than just doing the queries recursively on this low volume machine).
Anyway… Rogers could do better by paying attention to the number domains that are causing the NXDOMAIN results. In my case, all of the NXDOMAIN results were in response to queries to only a half dozen domains, like multi.surbl.org and multi.uribl.com. Certainly not a pattern consistent with a spam zombie — at least not an effective one (it could be one who’s master host/domain has been kicked off the net). I’d think that the trade off in not detecting infected, but ineffective, hosts over the false positives in cases like mine would be acceptable, especially considering that Rogers blocks port 25 in and out — which is great.
Anywho… if you’re a Rogers Highspeed Cable Internet customer, and you’re running SpamAssassin, or whatever, or do a lot of DNS queries for some other reason, I’d avoid using their DNS servers if you want to avoid having your connection disabled.
December 31st, 2006
7 weeks ago I called Bell Mobility Data Support to report that my handset was having major issues with receiving text messages. This wasn’t the first time I called them about it. I had called them about it four years ago. I never heard back from them and I never followed up because I didn’t really need it to work.
This time around, the first person I talked to was unusually helpful and knowledgeable. A simple, “I’m not an idiot, I’ve tested what’s available to me to test, it doesn’t work, the (Java) handset crashes periodically, it’s almost certainly the handset” was enough to open an escalation to have it looked at by someone who can actually verify that’s the case.
A week goes by… no real news. Another week goes by, they re-confirm that pending text messages usually fail. Two more weeks and they blame the handset manufacturer (Kyocera) and claim that they’re working with them for a resolution. Another week and they’re still waiting for a resolution from Kyocera. I tell them to send me a new handset - after all, I’ve had the same one for 6 years. That’s six years without them subsidizing a handset (which they usually do for most their users every 18-26 months) for me. I figure that the least they could do is send me a new handset (after I could have got three subsidized handsets from them if I had wanted) to replace the faulty handset (text messages only worked right for the first 4 months I had it) they sold me in the first place.
A week later, now 6 full weeks after I contacted them, they tell me that they’ve confirmed that the handset isn’t functioning correctly… which sounded familiar given that I told them the exact same thing (and the guy I had talked to agreed) when I opened the escalation with them. I repeated the “the least you can do is completely or substantially subsidize a good new handset for me” theory again and was told that “It’s against Bell Mobility’s policy, we’re not going to do that. At best we’ll offer you the crappy subsidy if you sign another contract.”
WTH, would I want to sign a new contract with a company that is currently providing me with crappy service? Heck, even though I’ve been a customer of theirs for a long time, I’ve never had a contract with them. Why would I want one now? Better yet, even though I know that I won’t be switching providers any time soon (I like CDMA and I’m not giving up my number — #&^%@* number portability!) if I were to sign a new contract to get their subsidy, they don’t offer any phones I like. Nobody around here had a Bell Mobility handset in the candy bar form factor available with a subsidy. Bell World didn’t even have candy bar phones — only big and bulky flip phones that don’t fit in your pocket (which isn’t a good place for a phone given the radiation and all, but I don’t care, I’m sure I’ll do something stupid enough to get myself killed long before I die of cancer).
Radio Shack, or whatever the heck they’re called now, had a Nokia 2125i that suited my requirements (candy bar phone, some buttons, receives text messages in theory). Of course, no Bell Mobility subsidy (even though their phones on display all had subsidies available. So I ended up shelling out cash for a new handset. Thanks Bell, you suck. I guess the few hundred dollars I personally send you every month for services isn’t enough. Not much choice for me there, but there’s a few thousand dollars a month in services at work that I could easily move to other providers. The only thing stopping that from happening is there’d be little in cost savings, and I know how bad Bell is. I’m not too sure how bad the alternatives are.
So, Bell Mobility Data Support is sort of good… they eventually identified the cause of a simple problem. Bell (Mobility) on the other hand, is as always, not so good. The worst part of it all is that I really only need text messaging to function so that I can be notified when our various Bell provisioned network links go down.
November 16th, 2006
We released SpamAssassin 3.1.7 today. It includes all the good bug fixes from 3.1.6 that we released on schedule (for the fourth time in a row) last Thursday, except the lint change to sa-update that I broke sa-update, for anyone who defines their own scores, with.
I’ve also added my Nagios plugin (”check_spamd”) for monitoring spamd to the contrib/ directory of the tarball in versions 3.1.6+. It’ll work with versions 3.1.1+ of SpamAssassin, but it’ll work best with 3.1.6+ as those versions deal with some quirks in IO::Socket::INET that can cause weird ping results.
October 10th, 2006
OK, maybe not for spammers, but I’m no more likely to believe that they’re taking efforts to prevent email abuse than that they are not.
Canada.com: Toronto Hydro rolls out wireless in continental first
Toronto Hydro Telecom launched its downtown WiFi project yesterday, offering full wireless Internet access throughout the financial district, with plans to have all of Toronto covered in three years.
Yesterday Toronto Hydro launched their wireless service, which they are providing for free until March 6, 2007. Hopefully they’re taking measures to prevent email (and other) abuse by doing things like port 25 blocking or throttling/limiting, otherwise this makes about as much sense as Google providing free WiFi to Nigerians. Heck in years of dealing with the public I’ve learnt that free anything is always going to be abused.
At least this is currently only in the financial district, so it shouldn’t be open to too much abuse from spammers sitting at home. Then again, some spammers make so much money that they can probably easily afford to live in the financial district, so watch out.
If anyone knows what address space this network is using or knows what is or isn’t being blocked, please let me know.
September 8th, 2006
Tonight NASA announced that despite not resolving the fuel cell coolant pump issue that they scrubbed the Wednesday launch over, they’re going to try and launch Atlantis at 11:41 eastern Friday. Fuel cell motor problem? Can anyone say “Apollo 13″?
Canada.com: Shuttle launch to go ahead despite wonky motor
NASA spent 48 hours trying to diagnose a little electric motor similar to a car starter after a glitch was found in one of the three fuel cells that provides the shuttle with electricity. The motor runs the cell’s coolant pump, which was giving erratic readings.
The problem has grounded the $2-billion shuttle and stalled construction of the $100-billion International Space Station. The little motor was manufactured in 1976 by a company that has changed hands five times since then. And lives depend on it.
In four hours of “intense” closed-door discussions, NASA decided the wonky motor isn’t a safety hazard. It does still work; it just doesn’t work the way it’s supposed to.
“We had another fun and very interesting meeting,” said Wayne Hale, boss of the shuttle program. “If you ever want to see the difference between the old NASA and the new NASA, you should have gone over there today. There was a chance for everyone to participate. All the data was laid out on the table for everyone to examine, from the top of the agency to the most junior engineer.
“It was very nearly unanimous, with just a few folks voicing concern,” he said. A further meeting was set for 1 a.m. today.
The shuttle is scheduled to launch a 11:41 ET today. Saturday is a backup launch date.
If Friday’s launch doesn’t pan out, they’ll try again on Saturday, which is a day later than their original deadline — which is silly too. If they previously decided that it’d be too dark to launch after Friday, why, other than desperation, are they now pushing back their deadline?
At least their decision not to launch outside of optimal lighting conditions is just so that they can film the launch and review the film for debris that may have caused damage doesn’t increase the safety of the launch, it just makes it more likely that they’ll be aware of potential problems upon return.
I’m all for space exploration. I just hope that they don’t write off all the shuttles before I get a chance to go see a launch.
September 8th, 2006
The SpamAssassin Rules Emporium (SARE) has a bunch of rules for SpamAssassin. Many of them are good rules. Some of them get updated regularly. To update these rules Chris Thielen wrote, and continues to update, a bash script called rules_du_jour that automatically downloads and when necessary updates a server’s SARE rules.
The rules_du_jour script works fine and has been in use by thousands of people for quite some time. One problem, though, is that it only runs on *nix systems. It also needs to be updated when new rulesets are released. Not a significant task by any stretch, but it does add to a mail system’s administrative load.
Apache SpamAssassin 3.1.1 and beyond includes sa-update, a cross-platform Perl program that manages updating rulesets available via “channels” in a light weight fashion using the DNS to track ruleset versions and HTTP to download updated rulesets when necessary. SHA1 sums and GPG are also used to verify downloaded rulesets.
So far the only sa-update channel available is the default channel published by the Apache SpamAssassin Project (updates.spamassassin.org). In early July I created my own channels containing up to date SARE rulesets to gain experience with sa-update and to automate rule updates on my own systems. For some reason I have never used rules_du_jour myself. I think it’s size and need for updating when new rulesets released turned me off of it.
After much discussion about “rules_du_jour vs. sa-update” in early August on the SpamAssassin users’ list in this and this thread, I decided to make my SARE sa-update channels public. The discussion was largely a debate between long time users of rules_du_jour who were discounting the utility of sa-update and the SpamAssassin development team, along with a number of Windows users who can’t use rules_du_jour. I figured making the channels public would attract enough users to either quash fears about sa-update or identify any problems with sa-update. I’ll be happy with either result.
So far there hasn’t been any problems reported. There’s only been a couple people that have reported their satisfaction though. Numbers often tell a better story. In the last week there have been a few hundred diverse IPs that have download files from one or more channels (each SARE ruleset is in its own channel). Browsing the server logs it appears that a number of noteworthy engineering and computer networking organizations have been using the channels on their mail clusters too. Since no one has complained yet, I’m assuming all is well. I know, myself, I haven’t had any problems in the last month or so.
For those interested in using these sa-update channels, a brief how-to is available here. I was planning on writing some more detailed documentation, but it really is as simple (on *nix systems at least) as stated in the how-to. If anyone wants to contribute documentation for use on Windows or other systems that the how-to doesn’t seem to cover it would be welcomed.
August 19th, 2006
Next Posts
Previous Posts
