ITIL, U-TIL, we all scream for…Configuration Management?

Ok, so the title is a little misleading. Configuration Management is a part of ITIL, however I’m not going to talk about ITIL, at least not directly.

As an administrator I’m responsible for multiple systems. Some of these are identical, e.g. Apache servers, MySQL servers, some of them provide unique, stand alone, services. However, they all have some things in common…sshd configuration, log rotation schedules (logrotated), etc.

It’s a PITA to keep up with all of these servers individually. A global change can take quite a bit of time, especially with our ever increasing number of ESX hosts. So, how do I make my job easier, myself more productive, and next year’s raise larger? Automated configuration management.

The (potential) solution

The *nix world has several open source tools available. Puppet and cfengine come to mind immediately, along with other projects from the likes of RedHat (cobbler), all of which make central management of many servers easy. For windows admins, I don’t know of any open source projects (I haven’t looked…), but Microsoft has their suite of tools…SMS, etc. There are also some packages that are distributed by independent companies…opsware (formerly LoudCloud, which developed in-house management software that was eventually bought by HP to become opsware) which is/are capable of interacting with many different OSs.

All of these tools make life easier for admins. Using puppet, you define the rules that a server must comply with…software package “A” must be installed, configuration file “B” must have “this line” in it, etc…and the management suite ensures compliance. This means that you configure one server (to act as your template/tester) and have the configuration applied to all servers. You can even categorize your servers…all of my Apache servers must have the apache rpm installed.

Configuration for things like security is also somewhat easier. Many of the packages listed above enable admins to be notified when certain files are changed. Want to know when the sudoers file was modified? You can be notified and verify that the modification was legitimate. Additionally, permissions and ownership rules can be implemented…/etc/passwd must be owned by root and have 640 permissions…if it doesn’t meet those criteria, make it that way.

Admins are no longer required to login to each host to verify potentially hundreds of permissions and ownership requirements (have you ever really read the Unix STIG?). You simply set the configuration on the master, tell it to enforce, and the work is taken care of for you.

What makes me enjoy tools like puppet even more is the ability to provision servers quickly. What happens if one of my Apache hosts suffers a catastrophic failure? I reinstall the OS and agent, then tell the controller to do it’s magic…the programs are installed and configuration files put in place.

What does it all mean?

Well, less administrator time per server. Which means (sorry admins) that each person is capable of managing more servers, and/or spending more time on problem management and root cause analysis.

But it’s not all rose colored glass and rainbows. Being able to configure servers en masse means that you can also screw up more servers, faster (I’ve seen admin shops refuse to use tools like opsware, flat out admitting they aren’t competent enough).

At the end of the day, assuming you have a release management process in place to ensure proper testing and verification, automated configuration management of servers can be a terrific time and money (if you’re the pointy haired boss type) saver.

In case your interested, there is some excellent reading on manging an infrastructure here. Additionally, a couple blogs I keep up with on data center management include The Hot Aisle and Datacenter Knowledge. They provide information on all aspects of managing a data center, from servers to power and cooling to data center migration.

5 thoughts on “ITIL, U-TIL, we all scream for…Configuration Management?”

  1. Sulli, Add em’ to the blog role… I couldn’t agree more. If you can automate you job you should. You won’t be fired or be made obsolete. Instead you’ll be free to work on the task’s that need work.

    Automate or DIE people!

    Oh yeah and I won free swag over at powerscripting.net check out there Upstream http://bit.ly/6ZZJH 9pm thus.

    ~Glenn

  2. The term “configuration management” now applies to several domains of knowledge. I agree that cfengine, puppet, bcfg2, etc are good at managing the configuration of sets of machines (via assertions). If you’re interested please see also [http://www.johnborwick.com/writing/system-automation.html].

    Having stepped over from the systems administrator side of the house to “process improvement,” “configuration management” now means something different: configuration management from ITIL’s perspective is understanding the important components of the overall infrastructure and services. For example, how can we figure out that the service “Email address lookup” depends on LDAP and DNS, and those services depend on servers X and Y?

    System automation and configuration tools a necessary, if not vital, component of effective administration, but it would be really nice if we could move towards an environment where systems are managed in terms of the overall infrastructure (e.g. that cfengine configurations could be written based on an overall configuration management database).

  3. John,

    First, thanks for your comment!! It’s good to know that I’m not talking to myself and Glenn all the time.

    I wholeheartedly agree that it would be wonderful to be able to provision servers based on needs that are determined by the CMDB. Especially if it could be done autonomously…in a world of virtual servers, wouldn’t it be great if the CMDB, in conjunction with capacity and performance monitoring, was able to determine that “Service A” needs the ability to handle more users, and then stand up a server, configure it with the necessary services, and let it go.

    My experience with CMDBs is that creating a CDMB that is capable of handling that amount and detail of information would be a nightmare (isn’t creating any CMDB an exercise in patience and hair pulling?)…from the creator’s standpoint and from a logistical standpoint. You now have to have documented knowledge of every dependency and interdependency that exists, both for hardware and the service provided (email, web, etc)…and in the case of circular dependencies which one to resolve first and how to do it. All that in addition to data on how to configure each service. Every CMDB that I have seen is hardware centric…this server has this processor, this amount of RAM, and these network cards. Those network cards are connected to this switch…etc, etc. Now we have to add on top of that information that the server is a member of this service provider group, which means that it has “program A” with “configuration A” installed to it. “Program A” must be defined, along with it’s dependencies. “Configuration A” must be created, and each element of it defined. We must account for any discrepancies, because they will always exist.

    And it must be done for all services on all operating systems.

    I’m not saying it’s not possible…I’m sure it is…maybe even the solution exists (please, please tell me it does!!!!), but I haven’t encountered it yet.

    I am, by no stretch of the imagination, an ITIL expert, and because of that I can’t speak on how to implement the ITIL definition of configuration management (or probably any part of ITIL)…I probably couldn’t even tell you the definition from memory. In my small and feeble mind, it seems like the ideal solution would be to separate the services from the hardware. Virtualization helps decouple service from hardware…server A is no longer solely an “Exchange server”, but can contribute it’s resources to many different parts of the organization, and new “servers” can be created quickly. It no longer matters what hardware is available, so long as there is capacity in the virtual infrastructure. I think using a configuration management tool, such as Puppet et. al., is necessary in this situation, as it ensures that all hardware, be it physical or virtual, is providing the service the same way, it is then up to administrators to integrate the newly provisioned resources with the existing.

    However, my intent with this post was much simpler than that. I only wanted to point out the availability of these applications and their inherent ability to provide (via the requirement to build the configurations) previously unknown and/or undocumented, data about systems under the purview of the organization.

    My recent experience has been with organizations that don’t use any kind of automation, configuration management, or even documentation, what-so-ever. Each server is configured by hand, almost always by different admins, and many core services were implemented years ago by admins long gone. There is little, if any, documentation about configuration, and even servers that provide the same services are configured dramatically different. Some servers are providing many different services, none of them related, and each configured by a different administrator. It’s basically a nightmare situation, and doing anything to one server can impact multiple servers and services spread throughout the organization…oh, and if you want to know which ones will be impacted, well, sorry, you’re S.O.L., cause no single person knows what’s installed to the server you’re on and what might be relying on it.

    So, by simply implementing the most basic Puppet setup utilizing the OS’s native package management, it’s a step forward…it’s a step towards documentation and standardization. And, as the saying goes, “A journey of 1000 miles starts with one step”.

  4. Yep, the CMDB is the “holy grail” of ITIL v2. In ITIL v3 they have created even more complex structures. Yay!

    Many vendors have created CMDBs designed specifically for hardware; I think that’s because it’s easier for them. Asset management has been around for a while, and why not just make a super asset management database and call it the CMDB?

    However, much of the value of a CMDB should be in documenting dependencies. CMDBs should be able to show how services and servers relate to one another. People want to put in all kinds of stuff in a CMDB–my rule of thumb, with the caveat that we don’t have a CMDB ourselves :-), is to put in as little data as you can stand.

    In our environment I don’t see any need for us to track any hardware beyond the server chassis. We might want to know all the virtual machines and what hardware they’re plugged into, but I doubt for us it’s worth the cost of tracking NICs etc.

    I think there is some neat overlap with cfengine, puppet, etc because CMDBs are the theoretical construct of an environment; cfengine and automation tools are all about taking a theory and putting it into practice. cfengine solves circular dependencies partly by “just trying again.” cfengine only guarantees configuration will converge to a state, not that your server will be totally set up after the first pass.

    In my dream world I would like to see a very basic CMDB with an export to cfengine configuration files. To my knowledge no such tool exists.

    Re: notifications about systems, at least one of the authors of tripwire now works at the IT Process Institute: http://www.itpi.org/home/default.php. They make the case that tripwire can be used as much as a change management audit tool as a security audit tool.

    Congratulations on implementing Puppet! There is a lot of really cool work to be done in this area, of getting a handle on how systems are configured, maintained, and how we figure out what’s going on with them!

Leave a Reply