The State of Open Source System Automation

The days of DIY system administration are rapidly coming to a close. Why? Because the open source tools available are just too good not to use. Presenting Bcfg2, Cfengine, Chef and Puppet.

Bcfg2

Bcfg2 was represented by its creator, Sr. Systems Administrator Narayan Desai.

Some notes on the name: pronounced “bee-config two”. The “B” in Bcfg2 originally stood for “bundle”; now it doesn’t stand for anything. The 2 indicates version 2.

Narayan thinks of configuration management as an API for programming your configuration.

Goals of Bcfg2′s design:

  • Efficient representation of diverse configuration.
  • Scalability to thousands of nodes.
  • Programmability (ability to insert code, not just declarative descriptions of desired configuration).
  • Model configuration in simple unambigious terms. (No way to end up with two different models of the same configuration.) “Closing the loops between goals and reality” is very important in Bcfg2. To enable this, there is a lot of reporting builtin; client reports its state upstream to the server to enable “gap analysis”. This is a very popular feature.
  • Support extensive configuration debugging. Help the sysadmin get to the bottom of things quickly. Bcfg2 has full system introspection capability(why is Bcfg2 making the decisions that it is). Many people think the debugger is the coolest feature of Bcfg2.
  • The Bcfg2 client can be run in dry-run (no changes), interactive (are you sure you want to do this?) and non-interactive mode, which is facilitates learning it.
  • Composition of information from a number of sources. (For example,put together different policies from different organizational sources such as organizational policy (FTP should not be running) and departmental policy(all user home directories must be NFS-mounted from fileserver HAPPYHOME).)
  • Expose plugin API to all aspects of the configuration process to enable handling of corner/edge cases. (Flexibility is an important part of Bcfg2′s design.)

Because Bcfg2 can capture the entire configuration of a system into a model you can diff the goals model (the desired state) and the current model and find out your degree of deviation, or you can diff two current models (Server A and Server B) and find out what’s different about their configuration.

Bcfg2 is very flexible – for example, you can designate a node as an exemplar, and the Bcfg2 server will copy it’s config to other nodes.

Bcfg2 is configuration “plumbing” (it just works). Here is how:

  1. Server probes client’s local state. (Client gets the probes from the server; client runs the probes; feeds data back to server.)
  2. Server builds configuration goals and sends them to the client.
  3. Client validates it’s local state and figures out what it has to do to meet the goals.
  4. Finally state information is sent back to the server and it is processed and can be reported.

Bcfg2 Policy: Install the Postfix Package

<Base>
    <Group name='Mail-server'>
        <Package name='postfix'/>
    </Group>
</Base>

Bcfg2 Policy: Install Multiple Packages

<Base>
    <Group name='Web-server'>
        <Package name='apache2'/>
        <Package name='apache2-mod_php'/>
        <Package name='php5'/>
    </Group>
</Base>

Bcfg2 Policy: Generate MOTD file using templating plug-in

Generate /etc/motd (message of the day) file that describes the system in terms of its Bcfg2 metadata and probe responses.

Here is the template (stored on the server):

 ------------------------------------------------------------------------
                    GOALS FOR SERVER MANAGED BY BCFG2
 ------------------------------------------------------------------------
 Hostname is ${metadata.hostname}

 Groups:
 {% for group in metadata.groups %}\
  * ${group}
 {% end %}\

 {% if metadata.categories %}\
 Categories:
 {% for category in metadata.categories %}\
  * ${category}
 {% end %}\
 {% end %}\

 {% if metadata.Probes %}\
 Probes:
 {% for probe, value in metadata.Probes.iteritems() %}\
  * ${probe} \
    ${value}
 {% end %}\
 {% end %}\

 -------------------------------------------------------------------------
                        ITOPS MOTD
 -------------------------------------------------------------------------
Please create a Ticket for any system level changes you need from IT.

This template gets the hostname, groups membership of the host, categories of the host (if any), and result of probes on the host (if any). The template formats this in with a header and footer that makes it visually more appealing.

Output

One possible output of this template would be:

 ------------------------------------------------------------------------
                     GOALS FOR SERVER MANAGED BY BCFG2
 ------------------------------------------------------------------------
 Hostname is cobra.example.com

 Groups:
  * oracle-server
  * centos5-5.2
  * centos5
  * redhat
  * x86_64
  * sys-vmware

 Categories:
  * os-variant
  * os
  * database-server
  * os-version

 Probes:
  * arch    x86_64
  * network    intranet_network
  * diskspace    Filesystem            Size  Used Avail Use% Mounted on
 /dev/mapper/VolGroup00-LogVol00
                        18G  2.1G   15G  13% /
 /dev/sda1              99M   13M   82M  13% /boot
 tmpfs                 3.8G     0  3.8G   0% /dev/shm
 /dev/mapper/mhcdbo-clear
                       1.5T  198M  1.5T   1% /mnt/san-oracle
  * virtual    vmware

 -------------------------------------------------------------------------
                        IT MOTD
 -------------------------------------------------------------------------
 Please create a Ticket for any system level changes you need from IT.

Reference: http://trac.mcs.anl.gov/projects/bcfg2/wiki/Plugins/TGenshi/examples/motd

Screenshots of Reporting System

summary_cal.jpg

node_dropdown.jpg

item_detail.jpg

Configuration Management Tips from the author of Bcfg2

Narayan pointed out configuration management can benefit from software engineering approaches:

  • Version control of configuration policy.
  • Testing and validation of changes to configuration policy.
  • Release management process for changes to configuration policy.

Narayan said, “It’s really hard to do roll backs due to lack of roll back support in package management systems”. His solution? The best way to roll back is a filesystem snapshot.

Next: Cfengine

Comments on "The State of Open Source System Automation"

redmumba

We run Bcfg2 pretty extensively at our offices, and it certainly has its pluses and minuses. However, one of the things that is a real stick in the side is TGenshi, the Bcfg2 templating system. One of the great things about TGenshi is, well, it allows you to add logic to your file–so you can generate files from the Properties plugin, dynamically encrypt passwords, etc.. Great feature, right?

Debugging is AWFUL. The errors TGenshi throws by defaulty largely generic; for example, if you have a 100 line Python file being run in the template, and an error occurs anywhere, you’ll just get a message saying “Could not generate this file.” No line number, no raising of the original Python exception, nothing. If you want to do any serious work, you’ll have to write your own wrapper to catch errors–or at least a line number for what failed.

Bcfg2′s strongest feature is keeping everything the same on every server, so I would consider combining this for day-to-day maintenance, and maybe Puppet or cfengine for deployment.

Andrew

Reply
jblaine

We’ve been using cfengine 1+2 for 11 years.

Reply
vicente

we use cfengine2 with some logic of our own to control around 130 computers, and is very nice and powerfull, when you get to understand it.

Now we are thinking to get in puppet. I’d like to post soon to tell you how it was.

Reply

    We just ran into the sccm cenlit performance issue. We had a script blow out the sccm cenlit to the domain, and the support team did this on sunday at about 1am. Every sunday at 1am since then our VMware farm grinds to a halt. Cpu spikes, storage spikes.We found that sccm was launching a dir /s inventory on all of its cenlits on the 7 day aniversary. About 200 vms.Still not sure how we will fix this, but ideas would be appreciated.

    Reply

Definitely would love to start a website like yours. Wish I had the time. My site is so amateurish compared to yours, feel free to check it out: http://tinyurl.com/o55af8p Alex :)

Reply

Helpful information. Fortunate my family I ran across your internet site by accident, using this program . surprised the key reason why this chance didn’t took place prior! I personally added it.

Reply

Blog looks nice. I’m still trying to make a blog but it won’t be as professional as yours /: Keep on blogging :) pirater un compte facebook

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>