The irony of this is that this is about as fundamental to any medium-to-large scale IT environment as bricks are to building a big house. But even a small business can benefit from this, so don't scoff and walk away just yet.
What am I talking about?...
Manual versus Automated Software Product installation, and Repackaging.
Two things which fit together like peas and carrots. Or hookers and politicians. Or Florida and hurricanes.
Yes. It's 2014, and from what I can tell there is a frightening number of so-called "technology professionals" that sincerely believe that there is little or no difference in terms of cost or quality between these to approaches. These two diametrically-opposed approaches, that is. In fact, many think that manual installations are cheaper, even when dozens of installations are involved per product. I am not joking, nor am I exaggerating. Please read on.
Most of them, if fed enough beer, or Xanax, would bet their retirement interest on the assumption that the differences between these two are as close as driving a vehicle with stick-shift versus an automatic transmission. That this is a monolithic, linear-scale, pound-for-pound comparison.
It's a good thing for them that the retirement interest on their fortunes is almost invisible to their overall balance sheet. A few zeros can get lost in all that green I'm sure. When you dig into the real numbers, the comparison is about as close as a boxing match between Mike Tyson on his "best day ever" and Richard Simmons after getting a root canal.
All kidding aside, let's do some math. mmkay?
Quasi-Scientific Analysis
Let's say product "Fubar 2014", from a well-known vendor, is required by 500 of your employees in order to "do their assigned job duties". You have a minimum wage minion whip out a stopwatch and begin timing the installation on the first five computers. The minion tallies up the results and hands it to you. It goes something like this:
- Technician walks over to computer "123" in building 42, up on the 3rd floor, in room 112 and sits down. Time spent getting there by foot/bicycle/car/private jet/yacht or teleporter is, on average, 5 minutes.
- Technician then logs on and opens Windows Explorer. 2 minutes (waits for initial profile setup)
- Navigates to central file server share on the network (Active Directory domain environment) to locate the folder containing Fubar 2014 setup files and related files. 1 minute.
- Navigates to "prereqs" subfolder to install individual products which are required before installing Fubar 2014: Java Runtime 1.5 (vendor says it can't work with 1.6 or later), then Apple Quicktime 7.1, Adobe Flash Player 11.5 (Fubar 2014 won't work with version 12), and a few other items. 10 minutes.
- Double-clicks on Fubar 2014 "setup.exe" file to launch main setup. Clicks Next on the Welcome page.
- Accepts default for installation target folder path, clicks Next
- Checks the EULA terms and enters a license product key. Clicks Next.
- Waits for installation to complete. 8 minutes.
- Goes back into Windows Explorer and right-clicks on a folder under C:\Program Files\Fubar to open the Properties form. Modifies the NTFS permissions to allow members of the local "Users" group to have Modify/Change permissions on that folder and all sub-folders. This is required since the users do not have local Administrator rights, so UAC has been a problem. This has been known to resolve the problem, so your tech goes ahead with the routine modification. 5 minutes.
- Tech goes into Windows Services and disables a service that Fubar 2014 uses to check for periodic updates, which users cannot install without elevated permissions, so this is a standard practice at your shop to disable it. 2 minutes
- Tech opens REGEDIT, navigates down to HKEY_LOCAL_MACHINE\Software\Fubar\Fubar 2014\Settings\ and changes the value of "autoupdate" from 1 to 0. 1 minute.
- Tech reboots computer, and waits for login screen to log back on. 2 minutes.
- Tech logs back on (1 minute or less) and launches Fubar 2014 to confirm it works. While still opened, Tech navigates into settings to change the option (Tools / Options / Data) to set the "Default library location" to a central UNC server path where all the users share templates and common items to maintain standards. 2 minutes.
- Tech closes Fubar 2014 and logs off.
- Tech goes on to next location and repeats this process.
Paying the Bill
If you kept track of the time spent above, that's 5+2+1+10+8+5+2+1+2+2 or 38 minutes. That's without ANY interruptions or unexpected problems. And that's assuming the computers are relatively new and performing well.
In reality, from tests I have witnessed over the past 5 years alone, in various enterprise environments from 5,000 to 50,000 computers, the average time to perform an installation of this magnitude is roughly between 35 and 50 minutes.
When performed during business hours with people around in close proximity, the times averaged 45 minutes to 1 hour.
When additional problems had to be resolved, such as missing updates, recovering disk space, removing conflicting components, that range increased to around 1 hour 20 minutes to 1 hour 50 minutes.
I haven't even mentioned:
- Time spent deactivating old licenses
- Time spent activating new licenses
- Time spent dealing with device drivers
- Time spent dealing with custom network interface settings
- Time spent on the phone dealing with vendor support:
- Large vendor: waiting on line, listening to 70's pop music, interlaced with endless repeats of ads for their other products, like their "new cloud services". awesome.
- Small vendor: waiting for guy (company owner/programmer/tester/web admin/support rep) to move his cat off the desk so he can flip through his paper stack to find your purchase order.
- Impact on end-users while they wait for the tech to do their work
- Impact on production from unexpected conflicts with other line-of-business products which are only discovered after the installation because there was no lead-time testing afforded.
In situations where a previous version had to first be uninstalled before performing a new install of the later version (usually because the vendor didn't want to take the time to handle this within their new installation package) the time ranges increase to around 2 hours to 2 hours 30 minutes.
Simple: 35 - 50 minutes
Complex: 120 - 150 minutes
In beancounter English: that's a range of roughly 1 hour to 2-1/2 hours.
Repeat this times 500 and you get anywhere from 316 hours (simple) to 1125 hours (pain in the ass).
Multiply that times the technician labor of say, $9/hour (you're a cheap bastard, after all), and that equates to roughly $2,850 to $10,120 of labor. For ONE software installation.
I'd guess you probably have more than a few products that would be handled this same way across your organization.
Are you starting to see where this is going yet?
Sanity Check Time
Now, let's crank this puppy through ONE cycle of repackaging effort and see how this spews out the other end of the meat grinder.
- Software Package Engineer (hereinafter SPE) opens a CMD console within a guest virtual machine (VM) running inside of VMware Workstation or Microsoft Hyper-V (take your pick).
- Navigates to folder where Fubar files are stored.
- Launches setup.exe -r and completes a normal setup process.
- SPE grabs the resulting setup.iss file from C:\Windows and copies it into new folder along with the original setup files. 5 minutes total by now.
- SPE opens a text/code editor and loads a template script to enter some lines to handle checks for prerequisites like JRE, Silverlight, Quicktime and so forth.
- SPE enters code to invoke the setup.exe with setup.iss and redirect the output to a new log file. Total of 15 minutes by now.
- SPE saves script and puts all the files into the new deployment source folder. SPE launches a separate VM, which is configured to match the configuration of the computers in use by employees who will be getting the installation. SPE runs the install using a command shell to ensure it runs "silent" and requires no user interaction whatsoever. Total runtime, including launching the VM and logging on is now around 30 minutes.
- SPE emails or IM's the designated customer SME (that's subject-matter-expert) who was nominated to be the "test user" and asks them to log into the VM using Remote Desktop and kick the tires. Time spent contacting the customer about 1 minute.
- SPE moves on to work on other packages or tasks while waiting for customer to respond and attempt the testing (parlance: User Acceptance Testing, or "UAT") No time expended by SPE during this period by the way.
- Customer gives the package "two thumbs-up!" and the SPE moves it into staging for production deployment. SPE creates a new "Application" in System Center Configuration Manager 2012, creates a resource Collection with the computers to be targeted, and assigns the Application to the Collection using an Advertisement. 10 minutes (he's drinking decaf this morning)
- Advertisement is scheduled to run after hours, avoiding impact on production time against the customer staff who will receive the new installation. SPE does not have to wait for the installation because it is scheduled to run on its own, so he/she checks on it the next morning.
Total time spent: 5+15+30+1+10 = 61 minutes.
I realize I said "he" a lot, but "she" could do just as well obviously, so that's irrelevant.
Things I didn't include:
- UAT problems resulting in having to go back and make adjustments and retesting
- Pre-flight deployments to verify conflicts in production on a limited subset of computers.
- Next-day support calls for incidental one-offs like machines being offline or a service was stopped or an application was open and had a lock on a file that prevented the Advertisement from completing successfully.
- Cats walking across keyboards and causing a BSOD.
- Who knows what else.
Taking those things into account, the ranges can jump from 60-80 minutes for a simple scenario to 2 hours, for just a simple repackaging effort like the one Fubar 2014 involves.
In the "real world" some products can be much more difficult to repackage and may consume days or weeks of development and testing in order to get a rock-solid package into production. Those are rare, but even then, EVEN THEN, the savings when calculated across hundreds or thousands of computers, spread across multiple locations, states or continents, can be well worth the effort.
Think of "mission critical" applications, where the time window to get them into production, with 99.999% success rate, is only an hour or two, end to end, over 20,000 or 30,000 computers. That's not fiction. There are industries where this is not uncommon, and they rely heavily on this methodology to ensure:
- Highest probability of success
- Consistent and predictable results
- Minimized impact on production operations
- Optimum transparency of all moving parts (think reporting and monitoring)
Steak Dinner Anyone?
So, this SPE makes $75,000 a year in USD, roughly $36/hour, and spent an hour building this simple package. That's $36 to deploy one product to 500 computers over one evening without asking any users to step away from their computers during work hours.
The cheapest scenario in the first example was $2,850.
The most expensive scenario in the latter example was $1,442.
Even if the SPE had to devote an entire week to one product, or roughly 40 x $36 = $1,442, that's a LOT CHEAPER than a $9/hour tech running around to 500 computers, or 10 x $9/hour techs running around to 50 computers each.
That's Not All
Billy Mays homage: Now, if you go with the repackaging and automated deployment scenario, you have a mechanism in place that does the following for you without ANY additional cost:
- Provides automatic self-healing of deployments, to cover situations where a targeted computer is replaced or reimaged with a fresh Windows configuration.
- Provides simple, effortless scalability for future growth.
- Provides a robust auditing and reporting trail for accountability and compliance.
- Provides fault tolerance
- Provides coverage during non-production hours.
Still think that thumb drive is the way to go? Hmmmm?
No comments:
Post a Comment