update: it appears that Experts Exchange will usually show you the answers at the bottom if the referrer was Google.  If you just copy and paste the url into your browser, the answers will not be there.  Strangeness.

Talk about a sensational headline, right? But it is true, all thanks to this nice forum post by OXY. The real title of this article ought to be “If You Would Like Your Data to Remain Private, Don’t Show it to Google”.

The premise is simple. Certain forums, Experts Exchange being a great example, allow Google to index their questions and answers. This gives them a high page rank for queries relating to technical questions. When folks like you or I go to Google to find help on a given network problem or operating system error, Experts Exchange always floats to the top. We click on the link, only to find that the answer is not available to the public – you have to be a paying member.

I ran into this earlier today while trying to find an answer on some error messages related to IIS. Google gave me this url. As you can see, there were no answers. After using the Google ‘cache’ directive, I was able to view this url. Scroll down, and you have answers.

It is quite simple to put into play: find a URL like this, as ‘http://www.foo.com/secretinfo.html’. Go to Google, and in the search box just type: ‘cache:http://www.foo.com/secretinfo.html’.

Not too bad, eh?

My morning inbox contained an email with the following subject: “We have hijacked your baby”.  

The body is as so:

Hey We have hijacked your baby but you must pay once to us $50 000. The details we will send later… We has attached photo of your fume    

And of course, a zip file was attached containing some sort of malicious executable archived within a zip.

This was certainly one of the more entertaining pieces of spam / malware that I’ve received.  To spammers everywhere: if you are going to do it, please use this as your model.  I want to see quality stuff!

It has been a while.  I’ve been busy, you’ve been busy.  You’ve called me, and all you got was my answering machine.  It is hard to put out good material while working and trying to spend time with your family.  I also adopted a pair of kittens that do their damnedest to keep my hands off of the keyboard and on them.  I am fending one of them off right now!  

But to all of those that still keep pumping out the posts despite the odds, I salute you!  Onward!

At the Fog Creek lunch table on Friday, a few of us were chatting about our past (pre-Creek) experiences with social engineering.  The consensus was that if you provide the right environment, a fair amount of people will fall for whatever trick you present.

I shared an experience in which I wanted to see this first hand, and therefore built an external website asking for users to enter their company passwords.  While that sounds silly, I dressed it up with a company logo, and sent out an email to a small sample of people claiming that the website will perform a test on their password strength.  By going through the test, they’d help protect us all.  This was ten years ago, which was well before phishing schemes became the cool thing.  Oh, I also told them that the test was mandatory.  35% of the sample went along with it, and only a few of the others contacted me to test legitimacy.  I guess the rest just didn’t want to deal with it.  :-)

I gathered the sample by Googling usenet and the web for email addresses from my company, as that is how any attacker would do it.  I wrote a Perl script and let it go to town.

In the process, I also discovered that our SMTP servers would allow external messages to arrive with a from address of the internal domain.  This would allow any joe on the Internet to connect to our SMTP servers and send email to internal user while masquerading as another internal user.  Not cool, my friends.

These results led me to believe that in order to exploit the trust of your average man, you only need to do two things:

1.  give them reasonable assurance that you are a person who has the authority to make the request (in this case, the attacker masquerades as a member of the internal IT staff)
2. make it known that by following your easy instructions, they are really helping you out

Anyone who has ever worked in the world of IT operations knows how hard it is to stay afloat. We are fighting entropy. We have a seemingly never ending todo list and people never stop interrupting. Ever. It’s the nature of this business, and we obviously like to hurt… err… like a good challange.

So how do you get things done? Everyone has their own system, and I’d like to share mine.

First, Come Up With an Organization System

I use a text file with vim. Seriously. It is located on a remote server that I can access from anywhere, and it has a very obvious format:

Apply patches to front-end IIS servers:
x verify that a policy exists
x schedule change
- apply patch on 2/20/2008 @ 00:01

It’s a no brainer, really. The first line denotes the goal, and the following lines are the tasks. After each task is complete, I prepend an ‘x’ and immediately write down the next action. If the goal is realized, then I cut and paste this to the todo.archive file. Simple as that. In the morning I scan the list and pick my target. Every Monday I go through and make sure that everything still fits with reality.

Eventually, I want to add some vim fanciness so that I can navigate the file easier or at least get some nice syntax highlighting. After that, maybe I’ll roll a simple Python cgi that can serve up the text file as an alternate interface. Hell, maybe it’ll even have a RESTful API of some sorts so that you can have all sorts of interface. Ahhh… to dream… to dream…

I use the unix utility remind to keep up with calendar items. I find the format clear and it is something I can bend to meet my needs without too much hassle. Google Calendar may be a better choice, though. I haven’t decided.

All of this data is backed up on a daily basis to Amazon S3 so I can rest easy.

Second, Keep Your Inbox Empty

Nothing raises anxiety like clutter. When I open my email inbox and see a bajillion read items, my mind races. Rather than do that, make your inbox serve your organization system. Every email is treated in one of three ways:

1. immediately answered
2. the goal is extracted and added to my todo list, and the email is moved to the ‘archive’ folder
3. the email is deleted

By doing this, I keep the stress level down and don’t have to dig through piles or email to figure out what I still have to do. That alone has made a big difference.

Wrapping Up

So, that’s how I do it. I’m no guru, and I’m sure it can be improved.

If you’d like to share your own system and you have a blog, I encourage you to write about it and trackback to this post.

If your IT operations team does not have a document library, I suggest you create one right now.  As mentioned earlier, it defines everything you do and takes very little investment to get started.  Install a wiki, make sure it is being backed up, and you are ready to start.

So how do you organize everything?

I’m one who hates complication, and try to stamp it out whenever I confront (or create) it.  Your front page of your library should contain links to all of your documents.  Don’t spread them out and hide them on separate pages – it just makes it harder to find the right one.  If you make it difficult, people stop using it.  We all know this from experience.

I recommend creating a heading for each service that you provide (Active Directory, DHCP, DNS, FogBugz On Demand, etc, etc), and then bullet point your policies and procedures below.  The first item under each heading can be the general policy for that service, and then order the other documents alphabetically.

Do not worry about getting it right the first time.  You probably won’t.  Since you’re storing this all in a wiki, you can easily refactor as things change.

All of this may sound simple and trite, but again, just make sure you have it in place.  If you aren’t writing it down, then you’re not doing a good job.

Microsoft has provided a preview for their patch release next week.  I counted a total of 9 remote exploitation vulnerabilities, so start making your plans.  Remember, if you don’t have one already, it is time to write up a patch deployment policy.  Early on, the goal is to be consistent so that you can start learning about the right way and wrong way to do things in your particular environment.

…are emails like this one:

Hi,

I look after about a dozen Unix boxes (mainly Fedora Linux, but also
Solaris, HP-UX and Digital Unix). I came to the conclusion after reading
your article that I need to formalize updates and document everything on
the wiki a bit more comprehensively. I also came to the conclusion that
anytime I update any system configuration files on any of the servers I
should check them in to the corporate CVS.

You can’t have too much documentation

Thanks again for a useful resource.

Cheers,
Jan.

You can’t beat that! It is fantastic encouragement, and really makes me feel like I may be able to do some good here.

Wanting to do things right around here, I have decided to summarize why I started this weblog and what I really want it to become. I have run a number of sites in the past, but have had a very hard time keeping them on track. This one is different, as the domain name itself doesn’t leave a whole lot of room for variance.

Every day reminds me that no matter how many years I can claim to have worked in the IT industry, I’m still a beginner. Twelve years ago I was a beginner to Microsoft Windows NT 4.0 and to the still young GNU Linux. Four years later, I was a learning how to keeping my head in a large environment. Five years from then, I had to figure out how to lead a large team of systems folk in an even larger organization and keep my soul. I’m still not sure I made it all the way out of that one.

Today, I’m the sole systems guy in a thriving software development company. I have my hands full with keeping up with day-to-day issues, managing risk, planning for the future, being continuously on call, and trying to build a capable operations team from the ground up. I am regularly amazed that a company so young and so small can challenge me like this – but it is doing a damned good job.

One of the most frustrating realizations is that there isn’t a guidebook to lend a hand. There are shelves full of operations management books out there, but none of them give you any idea as to what the hell you’re going to do come Monday morning. The same can be said about a lot of the technical books out there: it is easy to find a book on your favorite Linux distro or a guide to implement Active Directory, but that doesn’t solve the core problem of how you keep ahead.

Sometimes I think that the technical books on the wall behind me are just pleasant distractions from the mess I live in. It’s far easier for me to tinker with something new or arcane than it is to face up to the more immanent and difficult problems.

So this is what is going on: I am going to build a guidebook of sorts on how one can truly take charge of their environment. It will be concise, and it will be living. I’m going to use my posts to cover the topics, and I will write revisions as necessary. I expect to make mistakes. In the end, I’ll have a process that I trust and can use on a daily basis to not only keep things running in a safe and sane manner, but to keep making improvements.

Now, I will drift from time to time to keep things fresh and interesting. I’ll ramble about code and and related news here or there. As an example, I have a small series of articles detailing typical stack smashing techniques that will be fun to polish and publish. This is a blog, after all – it is allowed to wear different hats.

There you have it, folks. Now you know who I am and what I am up to here. Time to get back to writing.

I’m a proponent of centralized authentication mechanisms, as the complexity of managing the password database for each system can be overwhelming. A lot of organizations are already using Microsoft’s Active Directory product, and it has proven to be rather easy to perform simple authentication against. I’m posting this here because it is certainly security related, and I haven’t seen a quick howto out there covering this specific use case.

Prerequisites

I am testing this on a Debian Etch server, and only had to install the ‘python-ldap’ package.

Code

#!/usr/bin/env python

import ldap, sys

# fully qualified path to your ldap server
# if you'd like to use an encrypted channel, just use 'ldaps' instead of 'ldap'
LDAP_SERVER='ldap://mydomain.local'

# fully qualified AD user name
LDAP_USERNAME='myuser@mydomain.local'

# your password
LDAP_PASSWORD='mypass'

try:
        # build a client
        ldap_client = ldap.initialize(LDAP_SERVER)

        # perform a synchronous bind
        ldap_client.simple_bind_s(LDAP_USERNAME, LDAP_PASSWORD)

except ldap.INVALID_CREDENTIALS, e:
        print "Invalid credentials: ",e
        sys.exit()
except ldap.SERVER_DOWN, e:
        print "Your server appears to be down: ", e
        sys.exit()

# all is well
print 'connected!'
ldap_client.unbind()

If you are seeing this, then things are working correctly. I have moved this site over to WordPress.com. More details can be found here.

You can expect a new article in a couple of days!