Anyone who has ever worked in the world of IT operations knows how hard it is to stay afloat. We are fighting entropy. We have a seemingly never ending todo list and people never stop interrupting. Ever. It’s the nature of this business, and we obviously like to hurt… err… like a good challange.

So how do you get things done? Everyone has their own system, and I’d like to share mine.

First, Come Up With an Organization System

I use a text file with vim. Seriously. It is located on a remote server that I can access from anywhere, and it has a very obvious format:

Apply patches to front-end IIS servers:
x verify that a policy exists
x schedule change
- apply patch on 2/20/2008 @ 00:01

It’s a no brainer, really. The first line denotes the goal, and the following lines are the tasks. After each task is complete, I prepend an ‘x’ and immediately write down the next action. If the goal is realized, then I cut and paste this to the todo.archive file. Simple as that. In the morning I scan the list and pick my target. Every Monday I go through and make sure that everything still fits with reality.

Eventually, I want to add some vim fanciness so that I can navigate the file easier or at least get some nice syntax highlighting. After that, maybe I’ll roll a simple Python cgi that can serve up the text file as an alternate interface. Hell, maybe it’ll even have a RESTful API of some sorts so that you can have all sorts of interface. Ahhh… to dream… to dream…

I use the unix utility remind to keep up with calendar items. I find the format clear and it is something I can bend to meet my needs without too much hassle. Google Calendar may be a better choice, though. I haven’t decided.

All of this data is backed up on a daily basis to Amazon S3 so I can rest easy.

Second, Keep Your Inbox Empty

Nothing raises anxiety like clutter. When I open my email inbox and see a bajillion read items, my mind races. Rather than do that, make your inbox serve your organization system. Every email is treated in one of three ways:

1. immediately answered
2. the goal is extracted and added to my todo list, and the email is moved to the ‘archive’ folder
3. the email is deleted

By doing this, I keep the stress level down and don’t have to dig through piles or email to figure out what I still have to do. That alone has made a big difference.

Wrapping Up

So, that’s how I do it. I’m no guru, and I’m sure it can be improved.

If you’d like to share your own system and you have a blog, I encourage you to write about it and trackback to this post.

If your IT operations team does not have a document library, I suggest you create one right now.  As mentioned earlier, it defines everything you do and takes very little investment to get started.  Install a wiki, make sure it is being backed up, and you are ready to start.

So how do you organize everything?

I’m one who hates complication, and try to stamp it out whenever I confront (or create) it.  Your front page of your library should contain links to all of your documents.  Don’t spread them out and hide them on separate pages - it just makes it harder to find the right one.  If you make it difficult, people stop using it.  We all know this from experience.

I recommend creating a heading for each service that you provide (Active Directory, DHCP, DNS, FogBugz On Demand, etc, etc), and then bullet point your policies and procedures below.  The first item under each heading can be the general policy for that service, and then order the other documents alphabetically.

Do not worry about getting it right the first time.  You probably won’t.  Since you’re storing this all in a wiki, you can easily refactor as things change.

All of this may sound simple and trite, but again, just make sure you have it in place.  If you aren’t writing it down, then you’re not doing a good job.

Microsoft has provided a preview for their patch release next week.  I counted a total of 9 remote exploitation vulnerabilities, so start making your plans.  Remember, if you don’t have one already, it is time to write up a patch deployment policy.  Early on, the goal is to be consistent so that you can start learning about the right way and wrong way to do things in your particular environment.

…are emails like this one:

Hi,

I look after about a dozen Unix boxes (mainly Fedora Linux, but also
Solaris, HP-UX and Digital Unix). I came to the conclusion after reading
your article that I need to formalize updates and document everything on
the wiki a bit more comprehensively. I also came to the conclusion that
anytime I update any system configuration files on any of the servers I
should check them in to the corporate CVS.

You can’t have too much documentation

Thanks again for a useful resource.

Cheers,
Jan.

You can’t beat that! It is fantastic encouragement, and really makes me feel like I may be able to do some good here.

Wanting to do things right around here, I have decided to summarize why I started this weblog and what I really want it to become. I have run a number of sites in the past, but have had a very hard time keeping them on track. This one is different, as the domain name itself doesn’t leave a whole lot of room for variance.

Every day reminds me that no matter how many years I can claim to have worked in the IT industry, I’m still a beginner. Twelve years ago I was a beginner to Microsoft Windows NT 4.0 and to the still young GNU Linux. Four years later, I was a learning how to keeping my head in a large environment. Five years from then, I had to figure out how to lead a large team of systems folk in an even larger organization and keep my soul. I’m still not sure I made it all the way out of that one.

Today, I’m the sole systems guy in a thriving software development company. I have my hands full with keeping up with day-to-day issues, managing risk, planning for the future, being continuously on call, and trying to build a capable operations team from the ground up. I am regularly amazed that a company so young and so small can challenge me like this - but it is doing a damned good job.

One of the most frustrating realizations is that there isn’t a guidebook to lend a hand. There are shelves full of operations management books out there, but none of them give you any idea as to what the hell you’re going to do come Monday morning. The same can be said about a lot of the technical books out there: it is easy to find a book on your favorite Linux distro or a guide to implement Active Directory, but that doesn’t solve the core problem of how you keep ahead.

Sometimes I think that the technical books on the wall behind me are just pleasant distractions from the mess I live in. It’s far easier for me to tinker with something new or arcane than it is to face up to the more immanent and difficult problems.

So this is what is going on: I am going to build a guidebook of sorts on how one can truly take charge of their environment. It will be concise, and it will be living. I’m going to use my posts to cover the topics, and I will write revisions as necessary. I expect to make mistakes. In the end, I’ll have a process that I trust and can use on a daily basis to not only keep things running in a safe and sane manner, but to keep making improvements.

Now, I will drift from time to time to keep things fresh and interesting. I’ll ramble about code and and related news here or there. As an example, I have a small series of articles detailing typical stack smashing techniques that will be fun to polish and publish. This is a blog, after all - it is allowed to wear different hats.

There you have it, folks. Now you know who I am and what I am up to here. Time to get back to writing.

I’m a proponent of centralized authentication mechanisms, as the complexity of managing the password database for each system can be overwhelming. A lot of organizations are already using Microsoft’s Active Directory product, and it has proven to be rather easy to perform simple authentication against. I’m posting this here because it is certainly security related, and I haven’t seen a quick howto out there covering this specific use case.

Prerequisites

I am testing this on a Debian Etch server, and only had to install the ‘python-ldap’ package.

Code

#!/usr/bin/env python

import ldap, sys

# fully qualified path to your ldap server
# if you'd like to use an encrypted channel, just use 'ldaps' instead of 'ldap'
LDAP_SERVER='ldap://mydomain.local'

# fully qualified AD user name
LDAP_USERNAME='myuser@mydomain.local'

# your password
LDAP_PASSWORD='mypass'

try:
        # build a client
        ldap_client = ldap.initialize(LDAP_SERVER)

        # perform a synchronous bind
        ldap_client.simple_bind_s(LDAP_USERNAME, LDAP_PASSWORD)

except ldap.INVALID_CREDENTIALS, e:
        print "Invalid credentials: ",e
        sys.exit()
except ldap.SERVER_DOWN, e:
        print "Your server appears to be down: ", e
        sys.exit()

# all is well
print 'connected!'
ldap_client.unbind()

If you are seeing this, then things are working correctly. I have moved this site over to WordPress.com. More details can be found here.

You can expect a new article in a couple of days!

[editorial note: this article is part of a series that I am drafting to ultimately be included in a small guide to effective information security. I do not consider my views to be the final word, and am therefore soliciting feedback from anyone who has an opinion on my approach. You may either use the comment form below, or contact me directly here. Thanks!]

In a previous post, I argued that information security is much more effective when approached as a management methodology rather than as a technological fix. It has to be incorporated in your operational framework to be effective. I outlined the four steps of a basic Systems Management Cycle:

• You must write down your intentions (policy)
• You must document the procedures you use to manifest those intentions
• You must regularly audit your environment to make sure that you are actually doing as you say
• You must improve

The next step is one of genesis. Where do you begin? Assuming that you are not building a company from scratch, you’re probably in a pretty active environment. Technologies are already in place, projects are being executed, and your support queue is well worn.

In that situation, I suggest that one shouldn’t go running around trying to figure out what they need to document. That’s a brute force method that guarantees an anxiety attack. Instead, you can coax most of these undocumented goblins out of hiding by instituting one of the most powerful controls that I’ve ever witnessed: change management.

Quickly: What is Change Management?

Change management forces people to state their intentions before acting. Rather than walking up to a production server and installing new software, the sysadmin must fill out a quick form explaining the who, what, when, where and how of his actions.

A policy such as this makes day to day action in your environment transparent. As a manger, you see what your team is doing and you can therefore make better decisions. You can see where improvement needs to be made, and it can act as a set of breaks to put a stop to things before serious mistakes are made.

This is your core policy. You’ll write it down and spread the word. Rather than talk in vague terms about a theoretical document, I’m going to take you through the steps of making one. By the end of the article, you should be on track.

Real Steps to Making it Happen

Before anything else, you’re going to need a place to store your documents. I prefer to use a wiki, as they stay out of your way. If you don’t already have one in play, I recommend FogBugz On Demand [see disclaimer]. It’s just a few clicks to get going, you get a 45 day free trial, and it includes a really nice wiki along with a full blown project management suite. As always, though, choose the solution that meets your needs.

Once you have procured a document library for your team, create a section called ‘Policies and Procedures’. Within that, write your first policy called ‘Change Management Policy’. Follow my example here and tailor to your needs.

I prefer to keep my first versions very simple. Notice that I only ask for one business day notice for a standard change and consider all requests approved by default. In more mature organizations, this will be at least 5 business days and include a formal change management meeting prior to any approval - but you have to start somewhere. Remember, you want to lead your organization out of the wilderness without declaring marshal law. Let everyone acclimate to recording their changes first, as you will gain immensely from that.

As soon as you have it written, send it out there! Let your team know that you’re working to improve things, and that this is where you will start. There might be some resistance from some members, particularly those that have never been introduced to this methodology before. It’s ok. Reinforce that you are only asking people to state their plan of action before acting. If yours is like any organization I’ve ever worked in, you can probably find a nice list of past mistakes that could have been avoided if this simple rule had been followed.

Reaping the Benefits

Your team is now making and recording changes about various systems, some of which you may not have even known to have existed before! Since your policy is requiring that sysadmins create documentation for each task, your library of procedures is actually growing, and everyone is really thinking through their work. When you see planned changes fail, you are able to analyze them better and improve.

As time goes on, your change management policy will mature. You may create a change control board consisting of the key stakeholders in your organization and hold weekly review meetings to approve or deny changes. Full approval from your QA department may be a requirement before any change can be implemented. You’ll have to evolve this policy to best fit your organization.

From the standpoint of a security-minded individual, I don’t see what could be better than starting here. Systems availability is a key term on the CIA Triad, and this paves the path. All change requests can be scrutinized to ensure that best practices are being followed and audits can be scheduled to ensure that new holes were not created by these changes.

Best of luck!

The Problem and the Perceived Fix

About eight years ago, I worked as a sysadmin for a major telecommunications company in the Midwest. It was good for me. I was a senior admin on a very small team, and we had to be quick in order to keep up with the day-to-day troubles. I had my hands in everything, and it seemed like I was learning a new platform, app, or networking technology every week. Not only learning about it, but having to take ownership of it! Needless to say, I felt like quite the hotshot.

Then the virus came.

I don’t recall what it was that hit us, but it was one of the more prominent bugs that was making a lot of noise in the papers and on the 24-hour news stations. It turns out that there were a few workstations on the network that didn’t have anti-virus installed, creating an excellent point of entry. Shit. We pushed our preferred av client and updated to the latest definitions. After further investigation, I found out that our base workstation image didn’t include the antiviris client, and that it was up to the desktop tech to remember to install it. I worked with our desktop guys to build a new image that included the antivirus client so no one had to worry about remembering.

Approximately four months later, it happened again.

It wasn’t the same piece of malware, and it didn’t do anything other than replicate itself, but still - we were hit again. The cause: one of the junior guys installed a series of servers in the data center, and he forgot to install anti-virus. Damn it. Rinse and repeat.

The Real Fix

The pattern should be obvious now: in each case, we chose the appropriate technological response and nothing ever got better. It doesn’t do any good to purchase anti-virus software if you aren’t going to ensure its proper deployment. Then you have to ensure that it stays deployed on existing systems, and is installed on new builds. If you’re wondering what piece of system management software you can use to make this happen, you’re falling into the same hole as I did. Probably a more expensive one, too

Had I just written a procedure for each of these actions that included the step to install and verify the antivirus software, all would have been well. It took me a long time to learn that simple lesson.

It’s easy to ‘fix’ your security problems with technology. It is also quite reactionary, and usually doesn’t cover things the way you initially expect.

The real solution is one of policy and procedure:

• You must write down your intentions
• You must document the procedures you use to manifest those intentions
• You must regularly audit your environment to make sure that you are actually doing as you say
• You must improve

This solution is a process that you repeat over and over. You must be consistent. If you aren’t thinking like this on a regular basis, your environment will suffer. When you install a piece of software, you expect it to follow a series of predetermined steps. You don’t expect your SMTP server to do whatever it pleases, do you? No. You must act in the same fashion.

When I began following this process several years ago, I saw results. Sure, problems came about, but I’d simply change my policies to adapt so I didn’t get bit next time. I made it a part of my regular routine to question every step I made. Before I’d install a new piece of software, I’d check for a document explaining the installation procedure. If it didn’t exist, I would create it.

I truly believe that this is the core of a solid security methodology. Start following it before you spend a dime on anything else. Get your staff on board.

I’ll be posting more about the individual steps of this process in the near future.

Information Security is a topic that I’ve always been interested in, and have been studying to various degrees over the past twelve years. I’ve played many roles, ranging from the over curious teenager, to the sysadmin, to the professional penetration tester. I’ve made progress and done some good, and have also made plenty of mistakes. Everyday experience shows me that I’ll always have to keep learning.

Michael on Software is a forum for the curious. While I’ll be setting the front page topics on a regular basis, you are more than welcome to start your own. All I ask is that you treat each other with courtesy and be open to learning.

What’s in a Name?

The story behind the name of this site is quite simple. First, I wanted to name this site in such a manner that would force me to keep focus. I have authored a number of blogs over the years, all without a dedicated topic. I want to do something different here. I can’t image how I can waver with a name like this.

Second, I’m a long time reader of Joel Spolsky’s Joel on Software. His writing style appeals to me, and I see nothing wrong with the slight imitation that my title implies. It gets right to the point.

It is also fair to note that I am an employee at Joel’s software company, Fog Creek Software. Although I may comment on my experiences there from time to time, I want it to be clear that the words I write here do not necessarily represent the views of my employer. Fair enough?

How’d You Build this Thing?
This site is powered by WordPress.com. I’ve been itching to see if their community is as strong as they say it is, and figured that this was the best way to try it out. I’m going to give it a year. If things aren’t satisfactory, I’ll transfer the site over to my own server and go from there.