Michael On Security

The Problem and the Perceived Fix

About eight years ago, I worked as a sysadmin for a major telecommunications company in the Midwest. It was good for me. I was a senior admin on a very small team, and we had to be quick in order to keep up with the day-to-day troubles. I had my hands in everything, and it seemed like I was learning a new platform, app, or networking technology every week. Not only learning about it, but having to take ownership of it! Needless to say, I felt like quite the hotshot.

Then the virus came.

I don’t recall what it was that hit us, but it was one of the more prominent bugs that was making a lot of noise in the papers and on the 24-hour news stations. It turns out that there were a few workstations on the network that didn’t have anti-virus installed, creating an excellent point of entry. Shit. We pushed our preferred av client and updated to the latest definitions. After further investigation, I found out that our base workstation image didn’t include the antiviris client, and that it was up to the desktop tech to remember to install it. I worked with our desktop guys to build a new image that included the antivirus client so no one had to worry about remembering.

Approximately four months later, it happened again.

It wasn’t the same piece of malware, and it didn’t do anything other than replicate itself, but still - we were hit again. The cause: one of the junior guys installed a series of servers in the data center, and he forgot to install anti-virus. Damn it. Rinse and repeat.

The Real Fix

The pattern should be obvious now: in each case, we chose the appropriate technological response and nothing ever got better. It doesn’t do any good to purchase anti-virus software if you aren’t going to ensure its proper deployment. Then you have to ensure that it stays deployed on existing systems, and is installed on new builds. If you’re wondering what piece of system management software you can use to make this happen, you’re falling into the same hole as I did. Probably a more expensive one, too

Had I just written a procedure for each of these actions that included the step to install and verify the antivirus software, all would have been well. It took me a long time to learn that simple lesson.

It’s easy to ‘fix’ your security problems with technology. It is also quite reactionary, and usually doesn’t cover things the way you initially expect.

The real solution is one of policy and procedure:

• You must write down your intentions
• You must document the procedures you use to manifest those intentions
• You must regularly audit your environment to make sure that you are actually doing as you say
• You must improve

This solution is a process that you repeat over and over. You must be consistent. If you aren’t thinking like this on a regular basis, your environment will suffer. When you install a piece of software, you expect it to follow a series of predetermined steps. You don’t expect your SMTP server to do whatever it pleases, do you? No. You must act in the same fashion.

When I began following this process several years ago, I saw results. Sure, problems came about, but I’d simply change my policies to adapt so I didn’t get bit next time. I made it a part of my regular routine to question every step I made. Before I’d install a new piece of software, I’d check for a document explaining the installation procedure. If it didn’t exist, I would create it.

I truly believe that this is the core of a solid security methodology. Start following it before you spend a dime on anything else. Get your staff on board.

I’ll be posting more about the individual steps of this process in the near future.