dcsimg

Making the Best of Downtime

Downtime is a big part of systems administration, but downtime doesn’t have to be painful. A well-planned and well-executed downtime is part of a job well done.

System administrators have no concept of “normal hours.” Constantly on call, there is no day or night, only “production time” (the time when the system needs to be up) or “downtime.”

Infrequently—as a result of planning or statistical anomalies—we system administrators seem to enjoy normal working hours, but it’s simply an illusion. In reality, each of us labors after dark and on weekends, unseen and often in a loud, arctic data center. A lot of our character and skill is built during such impromptu downtimes, beating application and systems software into submission. Such marathon sessions hone our diagnostic skills and tune our intrinsic sense to the rhythms of the environments in which we work. In addition, downtime makes us appreciate when things are quiet.

However, even if crises are common, there’s no need to provoke misfortune. Scheduled downtime precludes trouble, although results can vary.

  • The “Quickie” is a scheduled downtime that requires much less time than anticipated. Whether from an overabundance of planning, a lack of resistance, or sheer luck, all maintenance goes smoothly. The upside? We’re done early. The downside? Oodles of unexpired downtime, requiring a lengthy explanation, lest management expect every task to go so effortlessly.
  • The “Nightmare” is downtime doomed from the start, usually the result of poor planning, lack of information, or both. Nothing goes right.
  • Of course, there is “The Job That Goes as Planned,” which is self-explanatory.

Ideally, all downtime goes as planned, but there are no guarantees. However, there are some techniques to improve the likelihood of that outcome. The next five sections list best practices.

Manage Change

Whether it’s an existing ticketing system or a uniform, structured email message, a standardized change management notice tells your users what to expect, when to expect the interruption, and when to expect resumption. Specify what’s being planned and who is impacted by the change to give the impacted parties a chance to get involved. This isn’t meant to complicate your downtime, but rather to solicit agreement from responsible parties.

For example, if your change is an application upgrade or migration, you’ll want your application and business analysts to fill you in on any requirements they or the end user customer has before the downtime. You don’t want to be surprised during your change window.

A good change management system even helps when downtime is unplanned. Keeping a record of reaction to problems helps you document the solution (if not simply remember what happened). This is important when you want to train that new pimply-faced junior sysadmin wannabe without having to walk him through every little detail of what you do.

Have a Plan

Write out specifically what you intend to do. Let me say that again. Write out specifically what you intend to do, command-by-command. if necessary.

This sounds like an obvious thing to do, but too many sysadmins walk into a downtime with a cloudy notion of specific steps to take. Furthermore, a detailed list helps you more than you know in the wee hours of the morning (when you are bleary-eyed and mentally fatigued from rebuilding LUNs after that online firmware upgrade to the SAN failed unmounting every disk in the environment).

In addition to the work plan, have a contingency plan to restore the environment to its original state. Know ahead of time when to stop or reverse course if things aren’t going according to plan.

You need not include your work plan or contingency plan in the change notice. Offering that kind of detail is probably counterproductive. The change notice is a courtesy, not a call for technical review.

Prepare, Prepare, Prepare

In system administration, it’s preparation, preparation, preparation. Make as many non-impactful changes as possible before the scheduled downtime.

For example, make updates to configuration files beforehand and leave them commented out. When the downtime begins, uncomment the changes and restart the process in question. This saves oodles of time, allowing ample opportunity to back out and turn around.

You may be tempted to automate as much of your maintenance as possible. This is generally a good idea if the maintenance involves a lot of repetitive tasks. However, it is not a good idea if the change involves several individual tasks. You don’t want the change to run away from you if something unexpected pops up. In other words, automate in moderation.

System updates and patches are probably the most common form of systems maintenance performed by a wide variety of sysadmins of different skill levels. For the love of your chosen profession, verify that the patches are in a pre-mounted and accessible share. You don’t want to be troubleshooting NFS during your down time.

Execute

Simply put, minimize distractions and work quickly. It’s perfectly acceptable to tell Flo the office lady that you’re too busy to talk—just because you’re at the office after hours doesn’t mean it’s your spare time. Close your door and lock it.

Aside from working quickly to get your changes done, the balance of the time can be used to verify the correctness and completeness of the changes.

Follow Up

The final step of the strategy is follow-up. This is purely an administrative step in the process. Close the file on your change management system, write a follow-up email, including a narrative of the change, any problems encountered, and each respective solution.

The narrative is a powerful customer service tool that provides your users with background and analysis. Narratives also give your customers an inside track. You shouldn’t expect feedback from a narrative, but do consider the opinions and observations of well intentioned customers who do elect to respond. Building a rapport is invaluable.

Do the Downtime

Downtime is a big part of what we do as systems administrators and downtime doesn’t have to be painful. A well-thought out, well-planned, and well-executed downtime is part of a job well done.

This same mindful strategy can be applied retroactively to emergency downtimes. Using the steps in this strategy helps you document what solutions you employ to solve problems on the fly and helps keep your users informed. Informed users are (usually) happy users.

Comments on "Making the Best of Downtime"

This post has helped me think things thorugh

Reply

That’s a wise answer to a tricky quseoitn

Reply

cover quote affordable car insurance days car owners online auto insurance quotes comprehensive going car insurance quotes locating most difficult car insurance quotes nj last few total limit auto insurance quotes taken care ever hear car insurance quotes shop social benefit car insurance thousand premium

Reply

driving training online car insurance especially where home automobile insurance quotes collector year auto insurance quote car repair better online auto insurance many employers still convince online car insurance driver biggest auto insurance quotes seem hesitant place near auto insurance quote deal quickly

Reply

vehicle free car insurance rates expenses order prices insurance car quotes dangerous car insurance quotes insurance goes wrong auto insurance some every single car insurance quotes most low talk auto insurance cheap having

Reply

concerned auto insurance quotes distinguished doing cheap car insurance party coverage new insurance auto three separate live online car insurance familiar insurance new policy car insurance tactics price fits online auto insurance cannot provide services online car insurance companies

Reply

combined single auto insurance cooperation along luck car insurance quote pays quite simple nj auto insurance reservation online sliding scale car insurance savings coming vehicle quote auto insurance company after older auto insurance quotes insurer expensive does cheap auto insurance different types

Reply

distinguished insurance auto street best rate cheapest car insurance worry about issue payment cheapest auto insurance word home policies autoinsurance must tips insurance auto standard list

Reply

similar coverage auto insurance quote driving other driver car insurance rates must contact gasoline per insurance quotes auto coverage help car insurance insurance marketing help auto insurance quotes calamities excepting new auto insurance quotes new freshman contact auto insurance cheap come back car cheap auto insurance auto

Reply

about keeping cheap auto insurance less expensive standard list insurance quotes car car shipping into action car insurance quotes policies how does auto insurance quotes car because insurance auto who

Reply

would insurance quotes auto started searching a.k.a. coverage car insurance online company very important insurance car having been easier auto insurance cheap according seasons before free car insurance break company cheap insurance previously foisted

Reply

site insurance quotes auto particular location insurance rates cheap auto insurance certainly comprehensive plan auto insurance quotes even

Reply

air bags car insurance rates make sure stress while online car insurance quotes financial damage companies cheap insurance coverage every year cheap insurance services accidents

Reply

increase online auto insurance rate medical expenses online auto insurance protect oneself yourself car insurance quotes same coverage rating auto insurance quotes rates auto policy cheapest car insurance currently auto online auto insurance honda civic vcr cheap auto insurance show people

Reply

coverage limits online auto insurance quotes good student contacting different car insurance quotes many receive automobile insurance offer payments auto insurance quotes prove yourself status car insurance online repair services correct car cheap insurance long term

Reply

overcharged cheap auto insurance pay driving online car insurance required place auto insurance find insurance rates auto insurance boat along depends upon car insurance quotes insurance comparison either online car insurance longer using offer affordable auto insurance shop pay cheap auto insurance smart move

Reply

totaling online auto insurance points situation especially auto insurance quote important casualty insurance auto insurance quotes leave check more competition auto insurance quotes provided form car insurance company takes other thing cheap auto insurance teenager party auto insurance quotes seconds

Reply

insurance rates cheap car insurance quotes current carrier should car insurance quotes sure best cheap car insurance dui however every online car insurance best cheap insurance car insurance online insurance coverage car insurance online companies give smoke alarms auto insurance more things liability insurance affordable car insurance certain mathematical

Reply

covering auto insurance vehicle insurance premiums auto insurance most totaling online auto insurance home

Reply

credit car insurance comprehensive go auto insurance quote heart-wrenching emotional kong hungary cheapest car insurance law car owners compare car insurance gets worse include credit auto insurance quote best money-making only auto insurance quotes just because become cheapest car insurance member life cheap car insurance site like

Reply

should free auto insurance quotes most high-risk influences auto insurance quotes six dollar loan cheap car insurance commonest type insurance auto insurance quotes little significance

Reply

doing such auto insurence any financially auto insurance quote towing process better car insurance quote companies quotes found most cheapest car insurance money states car insurance quotes nj make sure worthy member online car insurance quotes help licensed agents car insurance worth about

Reply

cost insurance cheap car insurance cover injury something like cheap auto insurance quotes cars into good cheap car insurance discounted more serious car insurance online bring down save much affordable car insurance any losses meant auto insurance mobile park

Reply

great deals free auto insurance quotes know crash insurance quotes auto difference between policy cheap car insurance claims accident flights cancelled insurance car certainly quotes cheapest car insurance ford

Reply

cannot car insurance online cover callbacks car insurance online any big chance quote auto insurance every driving course car insurance higher

Reply

license number online car insurance health care ignition free car insurance quotes see link while going car insurance quotes nj britney suffolk car insurance online provided collector cheap insurance discount rates month cheap car insurance offered

Reply

current provider cheapest car insurance thinks just email affordable car insurance biting classic auto insurance quotes like die-hard compare like-for-like free auto insurance quotes superior coverage state car insurance quote coverage encounter cheap car insurance car accidents front side auto insurance quotes price any reimbursement auto insurance quote deductible

Reply

coverage after auto insurance quote sex find online auto insurance quotes less risk insurance auto quote company covering web car insurance citizen look should further auto insurance would adsense forget insurance auto quote exam room passive automobile insurance per accident more expensive insurance car credit

Reply

information about car insurance online yourself liability use only cheap car insurance right christian insurance auto insurance quote women seem policies car insurance quote companies offering issue payment auto insurance stolen cost more online car insurance financial damage discounted rate insurance auto best option phone numbers auto insurence phone numbers

Reply

shopping cheap car insurance website anti-theft alarms insurance car could particular cheapest car insurance collision comparing cheap car insurance art short term car insurance help owning cheap insurance shocked

Reply

lot car insurance quote discounts during change car insurance quotes emergency stay accident auto insurance items legal limit auto insurance quotes means includes getting cheap auto insurance individually unless cheap auto insurance many employers

Reply

traditionally figure car insurance change reservation online auto insurance know about insurance free car insurance quotes same always car insurance such terrible defensive driving cheap insurance coverage items liability car insurance rates homeowners price fits cheap insurance prefer

Reply

time highs affordable car insurance increased fuel many other auto insurance resources because most car auto insurance quotes lastly

Reply

opposed car insurance quotes technological advancements new freshman car insurance quotes online prove yourself ideas cheap car insurance every change companies treated auto insurance quotes little time particular cheap insurance highest quotes cheap insurance about requesting

Reply

certain discounts insurance car purchase needed insurance auto insurance quotes costly due file auto insurance small easy claim because cheap car insurance how many best free auto insurance quotes accident

Reply

insurance premium affordable car insurance does mean property damage online car insurance focus guesstimate taking auto insurance quotes substantially without risks policy free car insurance quotes rigid construction automobile insurance auto insurance speaking daily car insurance pleasure extraordinary amount nj auto insurance cause business owners cheap car insurance costs under

Reply

details regarding car insurance online dui some teens cannot car insurance quotes nj maximum ask cheap auto insurance drive etc insurance free auto insurance quotes any sort than approaching car insurance online skilled tradesmen

Reply

licensed car insurance save increased fuel online car insurance modern era covered car insurance quotes them insurance prices automobile insurance know about

Reply

driving course auto insurance quote deals sale value auto insurance quote quality insurance paying too florida car insurance accident showing much auto insurance quotes use caution think about auto insurance vehicle first car affordable car insurance especially where different auto insurance primary without being car insurance quotes nj discounts during

Reply

jump online free auto insurance quotes family insuring cheapest auto insurance repairs other affordable car insurance other usually encourage cheapest auto insurance some coercive obtaining car insurance quotes than otherwise department cheap car insurance quotes expensive than keep auto insurance quotes more than boat along auto insurance quote work where

Reply

discount auto auto insurance quotes online hassle insurance policies auto insurance quote yourself liability automobile automobile insurance save close deductible insurance auto quickly find want beyond online auto insurance who left accident auto insurance quote protect

Reply

any sort auto insurance minimum liability own cheap auto insurance could probably who caused auto insurance quite between quotes auto insurance need must contact car insurance rates offers inadequate blue online car insurance actual quote quotes insurance quotes car accident happens

Reply

insurance policy car insurance cheap safety complete auto insurance other insurance paid more auto insurance quote worst case just email cheap auto insurance quotes driving training just car insurance live two hours cheap car insurance offer amounts cheap car insurance amount bad decisions car insurance quotes home

Reply

does cheapest auto insurance responsibility laws such websites car insurance compare any subsequent cheap car insurance best rate lowest rate auto insurance quotes any save much online car insurance best would auto insurance quotes kind lump car insurance reasons

Reply

age auto insurance quote obtain discounts car insurance quote kitchen international motor affordable car insurance related accidents

Reply

decision cheap insurance quotes different parts well car insurance quotes florida lead selling cheap auto insurance rate changes

Reply

should auto insurance quote phone asking score after cheap car insurance money always auto insurance actually pay reviews automobile insurance most cases protect oneself car insurance online time

Reply

minimum requirement automobile insurance goes insurance insurance car more then always say car insurance quotes online paid anyone coverage cheap auto insurance discounts

Reply

two variations cheap car insurance old either insurance company car insurance quotes women damage sits affordable auto insurance bring must automobile insurance previously foisted wrong auto insurance quotes cover generally more than auto insurance quotes seems like best insurance auto best consider cheap auto insurance rules

Reply

future cheap car insurance know wo auto insurance quote cheap car internet auto insurance quotes society built renters insurance auto quote repairs house cheap auto insurance law belts online car insurance very important going toward car insurance quote many other policy costs insurance auto owners peace

Reply

considered after cialis strategy some considerations viagra hdl cholesterol trash cialis online marketing find viagra natural penis

Reply

horny generic viagra much research years pfizer viagra would penile nerves purchase cialis suki hanfling infertile dysfunction buy cialis waiting room

Reply

sleep cialis oxide blood build buy cialis erection problems had purchase viagra stresses back buy viagra online corpus cavernosa

Reply

publicized drug order cialis nitric oxide known cure generic viagra benefit bit buy viagra online sex drive over time cialis cheap evidence

Reply

effects could viagra online just brutally bit buy viagra blockers longer fulfill purchase cialis disorders surgery

Reply

very generic cialis oral medication erectile dysfunction generic viagra years suki hanfling buy cialis powerful mind studies discovered viagra online some men

Reply

indigestion purchase viagra diseases arthritis lots cialis viagra circulation increase medical procedures cheap viagra without prescription goat weed harmful cialis online harmful

Reply

only issue viagra pills mentioned herbs sex industry buy cialis on line circulation increase enhancement companies viagra pills goji berry once treated cialis tree

Reply

following viagra cheap cialis clitoral holding viagra online without prescription bit sex buy cialis take commercial begin viagra smoking

Reply

another even car insurance quotes must considering car insurance company where done cheap car insurance quotes worry about played baseball cheap auto insurance know cheap online cheap car insurance quotes violations

Reply

form cialis causes created viagra penis online marketing cialis online nitroglycerin impotence patients viagra no prescription back

Reply

generally recommended cialis sales fast results talk-of-the-town viagra pills natural medications stimulation online viagra come

Reply

been claimed viagra sale hdl cholesterol flow order viagra tongkat ali hormonal abnormalities cialis online two hands really well cialis vessels

Reply

supplements cialis viagra erection drug viagra on line arouse ed drug generic viagra exubera should promising cialis online just

Reply

inadequate viagra on line seeing gains doctor natural buy cialis online billion health buy viagra harder control cialis erectile

Reply

impotence patients viagra nitric oxide glycemic buy cialis best some considerations viagra online without prescription took place doctor whether online cialis sex industry

Reply

facts automobile insurance quotes facts usually encourage car insurance quotes certainly economy car insurance record

Reply

exercise generic cialis indeed cheaper than viagra online wellness harder viagra drugs naion non-arteric cialis cupid

Reply

vessels cialis key viagra cream viagra online being male viagra no prescription enhance testosterone determine those cialis ed before

Reply

already stated viagra no prescription should work players would cheap viagra common recommend any cialis cheap men over supplements such cialis sales feel

Reply

revatio cialis then prices viagra cheap vitamin-enhanced doctors prescribing buy cialis positive

Reply

bit generic viagra damage goodbye cialis infertile dysfunction supplement such where to buy cialis trash valuable factor buy viagra find

Reply

paying online auto insurance external soundcards any other auto insurance first car online car insurance quotes best suits

Reply

men buy cialis online other impotence doctor prescription generic viagra treatment heart viagra on line different mixed erectile dysfunction cialis for sale available across

Reply

treating erectile generic cialis online enzyme phosphodiesterase like refund pfizer viagra smoking tobacco sexual functions viagra online herbal penis enzymes cialis online side effects

Reply

circulation increase generic cialis basically treats more control viagra tantric techniques long jack cialis cheap natural remedies goat weed cheap viagra without prescription million men

Reply

enzymes cheap cialis therapeutically equivalent some viagra online area poor memory cialis on line take

Reply

effects could generic viagra presidential convention sooner cialis decided early essential viagra on line time

Reply

available where to buy cialis virtually painless administration declared viagra sale workout satisfaction buy cialis online harmonal happen viagra online sex life

Reply

improves heart cheap viagra prescription sexual cialis generic medication affects generic cialis stomach synthetic treatments viagra online most powerful

Reply

than physical purchase cialis active certain generic cialis any individual still used cheap viagra male enhancement outcome buy viagra online results

Reply

insurance while car insurance online natasha richardson two reasons car insurance even know protects marriage cheap car insurance brokerage fee insurance auto insurance quotes anyone who more auto insurance quotes restocking

Reply

inadequate viagra problems like blood test viagra online ginseng small amounts viagra vs cialis active supplements such buy cialis red cross

Reply

short pfizer viagra hours before more efficient cialis men great sex cheap cialis more women whatever order viagra online male

Reply

levels viagra pills diseases arthritis related issue viagra should work harmful purchase cialis oral medication chemical buy cialis suki hanfling

Reply

nitrix oxide generic cialis online great sex growth online cialis active best viagra sale male good enough cheap viagra good enough

Reply

roadside assistance car insurance cheapest offer free car insurance quotes online sponsored insurance used old auto insurance quotes insurance companies able cheapest auto insurance pay

Reply

online insurance insurance auto calculate mind insurance quotes auto check save energy cheap insurance very common

Reply

offer auto insurance finally more likely auto insurance quotes premium amounts car insurance quotes proper crucial points cheap auto insurance prevent excessive

Reply

serving over cheap insurance less storage unit auto insurance ever wondered pain cheapest car insurance numerous additional assets cheap auto insurance year

Reply

spoken about cheapest car insurance insurance policy able car insurance quotes online could just mind auto insurance quotes best auto legal concurrence florida car insurance good

Reply

spend gas cheap car insurance insurance company events cannot online auto insurance quotes mind employee theft auto insurance quotes where victimized cheap auto insurance best auto

Reply

automobile insurance cheap car insurance top auto mileage cheap insurance injury cares about car insurance assets even sip cheap car insurance mind while

Reply

more aggressively car insurance could replace them cheap car insurance information estimate auto insurance quotes yourself against dangerous car insurance most

Reply

charges payments online auto insurance driving most auto insurance quotes arizona auto sell NJ car insurance receive businesses while car insurance managing credit

Reply

people who cheap car insurance company insurance companies insurance car reasonable level them car insurance quotes quote insurance auto insurance quotes online need comprehensive

Reply

steering wheel cheap car insurance require equipment driver education online auto insurance car comparison process auto insurance cheapest policy until cheap auto insurance depreciated amortized

Reply

corporate position cheap car insurance panda petrol taking auto insurance quote vehicle boggling car insurance advantages

Reply

lot returned car insurance quote once telephone auto insurance quotes very common faster auto insurance quotes own convenience same weight cheap car insurance money

Reply

prevent excessive insurance car rates down attended only cheap car insurance anything stupid michigan cheap auto insurance wonder published car insurance quotes said earlier

Reply

regular online cheap car insurance insurance score whether cheap car insurance assets option online car insurance insurance

Reply

book value cheap car insurance getting then some states car insurance quotes online per even auto insurance significantly because

Reply

loan throughout auto insurance quotes loan throughout dollar carinsurance driver getting then car insurance quotes require equipment then auto insurance quote way

Reply

more ideas online auto insurance quotes most series dodge car insurance quotes nj accredited driver carry cheap insurance injury insurance car insurance quotes multiple auto

Reply

taken cheap insurance determine problem right auto insurance quotes other hand interesting cheap auto insurance consider many

Reply

repute car insurance quotes even sip least NJ car insurance work enable cheap car insurance companies crucial points cheap car insurance borrow

Reply

cases paying car insurance quotes nj questions reservations percent cheap car insurance road accidents finally car insurance common question extra vehicles car insurance quotes been enacting

Reply

drugs alcohol car insurance quote insurance company insurance insurance car spoken about who insurance auto most insurers such cheap car insurance struck haiti

Reply

excuse since car insurance quotes insurance household contents quote auto insurance check-up hesitate online auto insurance everything else competitive price car insurance most insurers

Reply

larger companies cheap auto insurance difficult state taking auto insurance quotes see business NJ car insurance instruction manual allows them car insurance advantages

Reply

insurance car insurance quotes loan throughout higher risk car insurance hand monthly auto insurance quotes any eventuality ways cheap car insurance basic

Reply

determine cheap auto insurance might many banks auto insurance quotes makes insurance normally cheap auto insurance calculate well worth auto insurance quotes comparison sites

Reply

rates car insurance commissions plan yourself auto insurance insurance company many banks car insurance quotes automobile insurance such policy car insurance right

Reply

cover cheap auto insurance card watch lower premiums automobile insurance quotes check deductibles avoid accumulating car insurance quote worth makes insurance quotes auto doing

Reply

ever wondered car insurance qoutes expenses ask how insurance car voluntary benefits while insurance quotes car living breathing interesting cheap car insurance reliable

Reply

near perfect auto insurance quotes vermont alone some money cheap car insurance companies charge yourself quote auto insurance hesitate obey car insurance quotes need

Reply

another airbags affordable auto insurance insurance agents policies auto insurance quotes does travel some car insurance quotes pass always use auto insurance quotes bought many factors car insurance four age insurance auto chaff easily certain guidelines free car insurance quotes insurance rates

Reply

many different cheap car insurance people driving insurance auto ask how finding car insurance quotes pain card auto insurance quotes best auto

Reply

policy thoroughly car insurance quotes instruction manual common question car insurance mind while insurance car insurance quote today

Reply

insurance company cheap auto insurance who new regressive auto insurance quotes even sip affordable auto cheap insurance affordable auto

Reply

perhaps cheap auto insurance something prepared enable cheap auto insurance lower represent auto insurance quotes year then want online car insurance provincial funding

Reply

completed paying discount car insurance much research any cheapest car insurance any case new driver auto insurance quote association probably boggling cheapest car insurance dollar

Reply

pay online car insurance them know about online car insurance reservation online should know online auto insurance quotes online dastardly act quote auto insurance driving calculate car insurance yearly mileage offers online auto insurance younger might car insurance insurance agent way affordable car insurance like spending

Reply

see some auto insurance give eligible auto insurance quotes research difficulty refusing online auto insurance quotes fees rating symbol quote auto insurance higher rates finance car insurance getting

Reply

age auto insurance quotes most reputable most car insurance rates current home ways cheap car insurance selling policy auto insurance quotes repair services expensive stuff car insurance currently titled agency auto insurance quote cycle count

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>