Tuesday, December 14, 2010

Thoughts on Denver, the VA, and Mass Deployments

I kind of chuckled right after I typed the title line above.  I still have a middle school brain most of the time, so I'll leave it to you to figure out what alternate meaning that could have.

I've been fighting a Cold after I worked myself down on lack of sleep trying to get the book published.  I never learn.

Denver and the VA

I recently finished up a conference call with the Department of Veteran's Affairs.  They kindly asked if I would mind joining the call to answer a few questions about deploying Autodesk products with System Center Configuration Manager.  The call was fairly quiet, with many pauses waiting for the next person to chime in.  There wasn't a hard-written agenda of any kind, just a general topic, so we just went with the flow.

Now, this is the part where some of you might be thinking: "oh, here he goes bragging about his book and saving the world".  But in all honesty, I have no such delusions.  I was pleasantly surprised by the e-mail invite.  I have deployed to 5,000 computers in a single site, and 2,600 computers in 22 geographically dispersed sites, but these guys are well beyond that.  Yes, I was humbled.  I probably would have declined had it not been for the following key aspects:

  • It was the VA
  • The invite came from the Denver office
  • I was home with a Cold, sleeping all day, and awoke only minutes before the call started

Those of you that know me pretty well, also know that my father was a VA physician for 30+ years.  He retired from active duty in the Navy to work for the VA in 1957.  He passed away just a few months before retirement, after happily devoting his life to caring for veterans.  They were his extended family.  I cannot describe it with any other words. When my parents separated, I was in first grade, and he managed to secure living quarters on the Hampton, VA grounds while working as the Domiciliary Chief, and later as Chief of Staff.  I would spend weekends with him there and met many of his patients, other doctors and staff members.  In addition, my mother and father were from Denver, and there is a whole other, albeit distant and detached, branch of our family out in Colorado and parts of the southwest.  I simply couldn't decline this offer.

They have a fairly familiar environment and scenario: multiple, independent regional IT operations, each trying to race to find the cure for what ails their IT woes.  When one group breaks ground on something, they share with the others.  At least, that's what I could gather from our short conversation.  Each one supports a large number of customers and computers.

Unsung Heroes

Since I've been working for a municipal government IT operation, I can now see things from a different perspective.  One that few outside of this "world" can accept as even a remote possibility. That is, that there is a HUGE emphasis placed on efficiency and squeezing every drop from an ever-shrinking budget.  Most of the public at large is firmly convinced that government operations are drinking the finest champagne and eating caviar for lunch every day.  Au contraire!  Not only do most government IT operations have to "get by" with meager resources, they are constantly being forced to cut, cut, cut and cut again.  If you don't work in that environment, you cannot imaging the pressures it brings to bear on people struggling to meet regulatory demands, customer needs and just trying to keep all the gears meshed and moving properly.

I used to get upset when I overhear people make proclamations about how wasteful government operations are.  I'm sure there are some, but I can attest from personal experience that government IT endeavors are last in line for preferential treatment and funding.  They often get no credit for making things work (and they make things work very well), while getting the first fingers pointed at them when something fails.  A laptop left in the open by a government worker? IT gets blamed.  Someone opens an email virus and files get filched?  IT gets blamed.  Workers get caught surfing porn sites?  IT gets blamed.   Software gets packaged, staged and deployed to a million computers around the country? (crickets chirping…)

Now I just sigh and say "fuck it".  It's not worth trying to fight that battle.  You might as well charge into a redneck bar on a Saturday night and tell them "Ford sucks! Toyota rules!".  Their minds are made up and closed.  Don't bother.

Call of Duty

Anyhow, what are the challenges most LARGE SCALE software deployments face?

  • Packaging & Testing
  • Deployment Staging
  • Deployment Execution

I'll leave out the ancillary stuff like license servers, licensing, and custom add-ons for now.  These three bullets are enough already. Right?  I mean, I know quite a few people who immediately think about drinking as soon as you mention the word "deployment".  Or as one colleague likes to say: "gentlemen: it's time to pucker your sphincters and get to work!"

Packaging & Testing

As it pertains to building deployments, packaging (or re-packaging) and testing them: it varies widely from one product to the next.  There simply is very little consistency in some respects.  I have to commend Autodesk for putting a lot of effort and emphasis on achieving consistency and simplicity across their products as it pertains to building deployments.

Testing

My testing, actually, the testing procedure we use at work (I didn't invent this by any means), is as follows:

  1. Manual install from the source share to a test client (VM)
  2. SCCM deployment to a test client (VM)
  3. SCCM deployment to a physical client = UAT phase

After UAT (user acceptance testing, aka "customer acceptance") is completed, then we go to the next phase.

Deployment Staging

This is where you decide how and where (and when) you will get the package setup to make it ready to do the actual deployments (deployment execution).  For SCCM environments, this usually involves Distribution Points or Distribution Shares, as well as deciding (ahead of time) how you will tackle slow links, Branch DPs and Branch Caching (Windows 7 clients) and how the SCCM client will interact (download and run, or run from DP, fall-back on download/cache for slow links, etc.).

Advanced shops put a lot of work into automation of the staging process all by itself.  This may include clearing client caches or adjusting cache settings, pre-configuring branch cache options, BITS adjustments, adjusting maintenance windows (with respect to expected execution time of the package), phased scheduling and so on.  I'm not saying "un-advanced" shops don't do this.  I'm saying "advanced" shops put more dedicated time into these aspects independent of the staging and execution phases.

Disclaimer: I'm NOT an expert on SCCM concepts.  I have deployed SCCM sites.  I use SCCM daily at work.  But for a holistic view of SCCM concepts I would strongly recommend MyItforum.com or Microsoft TechNet, etc.

Deployment Execution

Deploying 50 targeted installs at once is usually not a big deal, unless your network is struggling already.  Even then, you can usually pull the trigger after hours or on a weekend, so as to avoid hurting anyone's feelings.  Going beyond a (relatively) small number of targets in a single "push", and you will want to plan for phasing the deployments.  Each environment has their own unique challenges and constraints, so there's no general guideline here.  Do whatever works for you and doesn't break things or create more work (unless you work on NMCI, in which case creating more work is part of the plan - just kidding).

Don't forget to include your FlexLM licensing environment when adding installations or moving them around.

No comments: