In this new weekly column, longtime Linux guru Matt Frye gives readers the full monty, the real trials and triumphs of a system administrator. This week: The care and feeding of software developers.
Since the day I started my career as a system administrator, I have looked upon the profession with awe. As a system administrator, I am the custodian of the computing environment; the gate keeper to system resources; and the entrusted caretake of filesystems, backups, software, and more. In fact, I look at my job as a calling, an order of sorts, where I tarry every day in quiet devotions, learning and making my “Foo” stronger.
And I am not alone. Others feel the same way, and like an order, we convene and teach each other the ways of our craft. It might be called a user group meeting, but to dedicated practitioners, it is a place of worship—or at least an exclusive clubhouse, with arduous initiation rites, secret handshakes, and a cryptic esoteric vernacular. Call us the Brothers of Foo.
But also like an order, we brethren can look warily upon outsiders. Windows software? Heresy. The Sales department? Better lock down their permissions. Product development? What do they know? For political or practical reasons, technologies and colleagues can suffer our prejudices, and in the end, for better or worse, those biases affect the way we perform our jobs.
Now, I’ve mellowed a lot with age and experience and interpret the machine room more and more in shades of gray. However, business consolidation and the demand that IT deliver value to the enterprise still rankle my enlightenment. The bean counters treat system administrators as a cost center and, because we don’t bring in any revenue, we are forced to overtly demonstrate our value. More often than not, the math doesn’t add up for the CFO, and dedicated system staffers are let go or are rolled into engineering organizations. Recently, “Your local sysadmin” has taken on a mind-numbingly literal meaning and we’re often tasked with one thing: developer support.
Don’t get me wrong. It’s important to provide all the things that developers need: hardware changes, every possible environment to test with, tweaks, and new boxes. Virtualization has really helped. But we’re really mixing with another sect now.
Developers, though from the same proto-geek family tree, are now a distinct denomination from system administrators, and we don’t always integrate well. There was a time when developers had to be their own sysadmins and sysadmins were developers from days of yore. Early on, both developers and system administrators had to write code to make things work, but these days we are each used to our own practices and short cuts. The canon has diverged, and we are used to doing the things that define us as our own professions. Since both groups wear the battle scars of intense flame wars gone by, sometimes things get a little silly and sometimes things get a little tense.
So what do can we do about it? What can we learn from our heritage of system administration that we can apply here?
The most obvious point is that we don’t want to solve the same problem more than once. Instead of reverting to old, unproductive confrontation over doctrine, we can resort to maturity. One tip that I’ve given sysadmins for years is “Discard your technology bigotry and flame war debris of the past.” You know what you do and how to do it, so don’t spend a lot of time and energy proving it to other people.
Another important component of developer support is empathy. One thing I’ve often forgotten over the years while supporting developers is that they usually don’t have the visibility into system issues that we have.
Here is an example. Recently when trying to troubleshoot a directory issue in a severely stripped down Apache configuration, I asked a developer to tell me exactly what aliases needed to be mapped to what directories. It didn’t occur to me right away that he didn’t even have a login on this box, let alone that he wasn’t familiar with httpd.conf. Still, I asked him the same question three times before I realized that he wasn’t answering the question, not because he didn’t want to answer, but because he actually didn’t have any idea what I was talking about. When the light went on, it nearly blinded me. Have empathy with your developers.
There’s another angle to the empathy thing: When you look at a problem, find the most elegant and practical solution and then implement it. Don’t spend too much time with strong feelings over things that don’t matter in the long run. Going the long way around for a half-baked principle is a kind of ridiculous.
For example, if you like to use open source, that’s great. If it doesn’t fit, move on and do what makes sense, even if it’s a proprietary solution. Your developer may be on a different sort of timeline than you. They have crazy deadlines in release schedules and don’t get to take the Zen approach that sysadmins are lucky to afford. They will love that you got it done quickly.
And that leads to another point. Fundamentally, the developers we support are our customers. We are as responsible to them as a waiter in a restaurant is to a patron. We give them correct information about what is and is not available. We get things they need. We provide them with a generally pleasant experience. It’s all basic customer service.
It’s easy to point out how sysadmins are not like waiters. I can hear it now, “If I get bad service in a restaurant, I’d walk out. The developer I work with doesn’t have that choice.” Yeah, yeah. That’s great that you can look at the reasons why not to provide good customer service, but there are a few problems with that argument.
- It’s a particularly unproductive response. It doesn’t move the conversation forward or get any work done.Don’t be a harborer or a sandbagger. Get things done instead of complaining about how you have to do them.
- Further, you’re only hurting yourself. You have to work with this guy or gal for days, months, and possibly years to come. Take a confrontational approach, and you’re going to get the same thing back. If anything, you’re going to perpetuate a painful experience every time you have to deal with that developer—and his colleagues who share the office grapevine.
- Beyond that, it’s your responsibility. System administrators have customers, internal and external. There aren’t many roles within IT or even within the enterprise where one person has such a wide array of responsibilities. When our customers are satisfied, we get a better reputation and we deliver that all important value that keeps us off the chopping block for a while.
Like it or not, the word “social” has become part of the sysadmin’s job description. We may be technical genuises, but now we need to become customer service experts. We need to refocus our metrics and rewards systems, from how many file systems we built, to whether we helped the people to whom we are responsible succeed. All this, while we maintain our technical proficiency.
We need to understand that relinquishing control may yield better or more creative output. System administrators must build bridges with developers, database analysts, and non-technical people we support, because at some point down the road, we’ll want to cross those bridges and we’ll depend on the kindnress of all those people we met along the way.