Flashback: The Level 5 Sessions

I had to pause for a bit and reflect on 2008 and 2009 a bit.  2008 in particular was one of the worst years of my life aside from 1987 (father passed away) and 1997 (mother passed away).  I experience my first time being laid off from a job.  I have never been fired from any job in my life, so getting laid off, and on very short notice was not only a shock, but, well, I'm sure some of you are well aware of how that feels when you have a family and are barely keeping up on bills.  That put us so far behind that I'm sure we can ever catch up, but who knows.

In 2007, a mutual friend got me in touch with Brian S. Tucker.  We had a few conversations about some ideas Brian had and what he was hoping I could do to help him make those ideas work.  Mind you, these were all his ideas, and I was only one of quite a few other players involved.  Brian had a crazy idea (at the time) to start a web site devoted to subscription-based training videos for various Microsoft platform products.  The first phase was a free upload site called BlogcastRepository.com but his ideas was to develop a premium service called the "Level 5" Guides, which would involve paid monthly subscriptions.

2008 wasn't all bad, at least not after April. When Brian heard I was unemployed and having a tough time finding another job, he got in touch with me and helped push me to produce more content and help promote his site and become a win-win situation.  The combination of having someone keeping me involved and working towards a goal, kept me away from depression and solitude.  Those may sound trivial, but when you're laid off during cold weather months, and all your friends are busy at their jobs, your life can be come very lonely and scarry. 

I can't stress enough how important it is to just stay busy.  Especially if you can get outdoors as well.  I did that while still hunting for another job. But early-2008 was not a good time to look for a job.  Every single place I went to had a freeze on hiring due to fears of how far the economy was going to sink.  Everyone kept trying to encourage me with things like "with your skills" or "with your background" … "you should have no problem finding another job".  But that actually made it worse because the more rejections I got (or stark silence) the more I began to doubt my skills and marketability.  Human nature: Easy to study, much more complicated to experience.

Brian put himself, and his family's future on the line, to finance and push to make the dream a reality.  They endured a ton of shit with banks, developers, business people and trying to market their idea to both content authors and potential subscribers.  It was a stretch to assume it could be successful, and there were times when it didn't appear it would work out, but they never gave up or let pessimism creep in.

This happened within a few weeks of being laid off, and I used that down-time to gear up, ramp up, and man up to producing 17 sessions on WSUS 3.0 and 18 sessions on Windows Server 2008.  The latter of which was just barely out of beta/CTP when I recorded those.  Looking back they were very basic and almost embarrassingly amateur, but at the time it was all I had.  I needed to pay the mortgage, feed the kids and keep the electricity and water turned on.  Brian came through, and when I needed it most.  He didn't hand me any crying towels or pat me on the back either.  He knew my predicament and still insisted on me doing my best, just like everyone else who participated.  That gave me back much-needed respect and at least a little self-confidence.

Together with Edwin Sarmiento, Abhishek Joshi, Andy Dominey, Dennis Rietvink and a few others (very few), we pushed into what was at that time uncharted territory.  But it worked.  I was humbled by the credentials of these other guys, and Brian himself.  I wasn't sure if I belonged in that group, but Brian convinced me to step up.  The margins paid to us from the subscription sales weren't a lot but they absolutely saved me and my family.  That picture above was my rig and recording "studio" in our garage during that time.  It truly is worth a thousand words to me.

As the site began to grow in popularity, and more subscribers came aboard, we were soon facing the dilemna of what to do next.  What topics should we focus on?  What features should we dive deeper into?  We emailed subscribers to get feedback, but hardly anyone replied.  At that point Brian had to take on additional consulting work (which he had already been doing for quite a few years) in order to fund the business to a higher level.  In mid-Summer 2009, he took a job in Washington, D.C. where soon after he suffered a heart attack, and was recovering in the hospital, when he took a turn for the worse and passed away in August 2009.

Since Brian's passing, the site was not being managed and spammers began to victimize it badly.  Subscribers were asking what to do about cancelling payments and what the future was.  Unfortunately, I had no answers, and neither did any of the other authors.  We were as confused as anyone else.  In any case, I miss Brian and will never forget how much he helped me and many others in his short life.  May he rest in peace.

In May of 2010, Mike Rood contacted me to begin the process of finding who was managing or owned the site.  He was able to negotiate the purchase of the site from Julie Tucker.  I haven't heard anything since August 2010.  I e-mailed Mike to find out what the latest news is, so I hope to have something good to report soon - finger's crossed.

A Little Studying Goes a Long Way

It’s been one of those weeks again, and it’s only Tuesday.  Ooh boy.  One of the interesting patterns I’m seeing is an uptick in people hitting me up for help with various AD things.  I need help also, but I will get to that later.  One question was from a former colleagues trying to introduce a new 2008 R2 domain controller into his 2003 AD forest.  I had to help walk him through getting his ingredients in order before putting the pans in the oven. 

Here’s a tip on that end: don’t rush it!  When you bump your functional levels to native mode, give it a little time to settle out before bumping more things around.  In this case, he was running in 2000 mode, so he needed to get to 2003 native mode first.  Then run ADPREP /forestprep and then /domainprep and /gpprep, blah blah.  But whatever you do, don’t run ADPREP within a few seconds after having raised the functional forest and domain levels.  Not a good idea.  Thankfully he held off.  But if he hadn’t reached me by phone first, he likely would have.

Another issue was helping another acquaintance move FSMO roles around to avoid potential hardware failure impacts on older servers, while paying special care to put them on servers in well-connected sites to avoid replication problems.

Another issue, for yet another former colleague (I try to be careful with “friend” since most people I know are colleagues and acquaintances rather than true “friends”, but that’s not a bad thing) who was running some domain controllers on physical machines and others in virtual machines.  That can be interesting if you don’t read up on what to watch out for.

I suppose the case in point is that Google can be your friend.  Or it can be your biggest frustration.  But regardless of web searches (Bing, Yahoo! or whatever you prefer) having some books around can be a big help.  One of these days I’m buying a Kindle or some other decent (and affordable) book reader.  It’s MUCH easier to search through ebooks than a physical book shelf.

An interesting side note, and one I’ve commented on before, is the unique contrast between how much need there is for Active Directory troubleshooting knowledge, against the almost zero job potential for that skillset.  If you bundle it with Exchange, SharePoint and maybe IIS, it becomes marketable. But all the focused AD knowledge and experience in the world isn’t very marketable in and of itself.  It’s becoming a tier 1 skillset: Something tied more to sys-admins than to system engineers or systems architects, etc.  Even more ironic is that the vast majority of problems I’ve seen with AD environments can be nailed directly onto the foreheads of systems architects and systems engineers.  They often downplay the importance of AD in a logical, topological sense.  Especially as it relates to sites, site links and the modeling of security boundaries for policy-based management.

That’s where books are best suited.  Don’t buy just one.  I’ve never seen a single book cover enough of a given technology subject to really be helpful.  At least two or three on the same subject (by different authors) is what I recommend.  My wife hates that, of course, because my book shelf is a mess.  Hence the desire for a Kindle.  Maybe when I catch up on bills that’ll happen.  Oh wait a minute, I just told a joke?  Catching up on bills?!  Ha ha ha hahha aah oweor wefijoelkfjlfksdlksowierjwo…hrmph..uh.. (cough cough, gasp…)  Ok, I’m back.

Now, for the issue [I] need help with (anyone?):  One of my 2008 R2 domain controllers fails to display Roles and Features in the Server Manager console.  The MMC console opens and it says it’s “collecting data…” and it pauses forever, then says “error” and it can’t display them…

The event log shows the following error information…

Source: ServerManager
EventID: 1601
Level: Error
OpCode: Info
Log Name: Microsoft-Windows-ServerManager/Operational

General:

Could not discover the state of the system. An unexpected exception was found:
System.Runtime.InteropServices.COMException (0x800706BE): The remote procedure call failed. (Exception from HRESULT: 0x800706BE)
   at System.Runtime.InteropServices.Marshal.ThrowExceptionForHRInternal(Int32 errorCode, IntPtr errorInfo)
   at Microsoft.Windows.ServerManager.ComponentInstaller.CreateSessionAndPackage(IntPtr& session, IntPtr& package)
   at Microsoft.Windows.ServerManager.ComponentInstaller.InitializeUpdateInfo()
   at Microsoft.Windows.ServerManager.ComponentInstaller.Initialize()
   at Microsoft.Windows.ServerManager.Common.Provider.RefreshDiscovery()
   at Microsoft.Windows.ServerManager.LocalResult.PerformDiscovery()
   at Microsoft.Windows.ServerManager.ServerManagerModel.CreateLocalResult(RefreshType refreshType)
   at Microsoft.Windows.ServerManager.ServerManagerModel.InternalRefreshModelResult(Object state)

Clicking the Event Lookup link (TechNet) shows no help at all.  I’ve tried most of the suggestions I found from web searches, but nothing seems to help.  This one looked like it was close (http://social.technet.microsoft.com/Forums/en-US/winservermanager/thread/ae211676-4858-4b03-9360-280ab3dacd6f) but it didn’t help either.

However, this led me down another rat hole…

To replace in-process service packages you have to run the request in the SYSTEM context.  One of the age-old ways to insert yourself into the SYSTEM context is (or was) to use the AT command (example: at HH:MM /interactive cmd.exe) but this doesn’t work on 2008 R2.  You get this…

C:\Windows\servicing\Packages>at 20:42 /Interactive cmd.exe
Warning: Due to security enhancements, this task will run at the time expected but not interactively.
Use schtasks.exe utility if interactive task is required ('schtasks /?' for details).
Added a new job with job ID = 1

Bummer #1.  So I used SCHTASKS instead and…

schtasks /create /tn “CmdShell” /tr cmd.exe /ru “SYSTEM” /st 20:49 /sc ONCE

Bummer #2 = This will not open interactively on 2008 R2. 

Then came the suggestion to make a new service using the sc.exe command…

sc create cmdshell type= own type= interact binpath= “cmd.exe /k”

It prompts you to “View” the program’s message.  When you click that it opens a secure desktop shell and hides the previous desktop entirely.  You can interact through that, but oh man is that a pain.

Bummer #3

Which, as my blog’s name eludes to, reminded me to ask this question: Why the **** doesn’t Microsoft just stop and take a few weeks to clean up their command syntax structures?  Forget PowerShell.  I’m not running through a bunch of PowerShell to do things that utilities like SCHTASKS do very well as-is.  But note the spaces after the “=” in the SC command.  They matter!  Is this not retarded?!

Where was I?  Oh yeah, my stupid Server Manager console on this one DC refuses to be nice and help with roles and features.  I’m close to nuking the ****ing thing and replacing it.  I’m so frustrated, that I’m ready to just outright NUKE it, use ntdsutil to seize its roles and blow it out of the partition with adsiedit.  Yep.  I’m that pissed.  It’s annoying the crap out of me.  Any suggestions are welcome (besides Xanax and sleep).

Headaches: Windows Server 2008 R2 + WSUS + zzzzz

No sleep.  Been up beating my head against a wall but finally found some relief.

My lone WS08R2 domain controller/WSUS host was having issues.  It would go into perpetual "downloading updates" when doing either manual update check against MS-Updates, but the status always said 0%.  I could let it run for days and it wouldn't get beyond that.  So I ran across this article, and it helped.

Basically: Stop the BITS and Crypto services, then stop Update Service.  Then rename \Windows\catroot2 to catroot2.old and SoftwareUpdates to SoftwareUpdates.old.  Then restart the services and run an update check.  Ba-da-bing!  It worked!

However, I'm still having an issue with the Server Manager failing to display Roles or Features.  It tries to enumerate them and fails with a useless error.  The Windows Modules Installer service keeps stopping also for no apparent reason.   Still digging.  But I'm going to bed for now.  Screw this.

2008, 2009, 2010…

I felt like reflecting on the past year, if only for a minute.  Thinking back, it was a truly mixed bag of fortunes.

2008 was probably the worst year since my parents passed away in the 90’s.  I lost my job early in the year, and began a long a struggle back from financial chaos that ensued.  It was early in the Recession, so things were declining pretty quick.  It all started to turn around for me in June of that year, when I was hired back at a former employer, and so far has continued to be a very positive experience.  I was lucky indeed.

2009 was going pretty well, until mid/late Summer, when I lost two friends without any warning.  One died of a sudden heart attack, the other from suicide by cop (seriously).  Neither of them ever showed signs that would have indicated their fates were even remotely possible.  I still think about them often.  It’s a hole in my life that will take some time to heal.

During 2009, I was fortunate enough to be allowed to take some courses towards an MCSA and an MCITP certification, which I achieved.  My oldest daughter graduated high school.  We still have a house and electricity.  We still have a family.

I’m not sure what to expect from 2010.  The past two years have been rocky, but with bright spots as well.  If that means 2010 will be bad but will offer up more bright spots, then I suppose that’s not entirely bad.  Stay tuned…

Windows Server 2008, IIS, ASP, ADO and Access 2007

Ah, the joys of walking into a mine field alone. For those of you who don't know me, I've spent years writing web apps using ASP and a bit less (but way more often) on PHP+MySQL. Most of my ASP work was done using MS-SQL or Oracle, but lately these lines have blurred somewhat. I just finished up an application for a customer that uses ASP with MySQL. Talk about weird. It works fine though.

For those of you that do know me, how about a phone call once in a while? Eh? :(

So I'm building a web app to interface with Active Directory, which is a scaled down version of one I wrote for a previous employer. Instead of managing users, groups, computers, contacts and tracking all the crap that goes with that (for what else but SOX), this one is for home use. And while the big corporate-y version allowed me to control AD operations securely from my Blackberry, and provided our Deloitte and Douchebag auditors with more than enough reporting to make them grab a kleenex, I don't need that level of power at home.

I wanted to be able to empower my wife with the ability to inflict the ultimate torture on kids of cutting off their computer access. She's not up to dealing with ADUC or cryptic CMD shell commands. Scripts would be nice, but I don't want to dole out permissions like that (not that I don't trust my wife, but she might hit a malicious web site during her crazy surfing and it might leverage her elevated privs to do more damage than I would like - sorry honey). I also considered desktop shortcuts to scripts (one for each child) to stuff a record into the database or write a file to be picked up by another process, either by event sinks or by scheduling, but ah, yeck, too messy.

No, this little project will simply provide all the kids' user accounts with their ENABLED/DISABLED status and the option to toggle those individually. The site runs on ASP using IIS7 (Windows Server 2008) and only allows for toggling user accounts between the two given states. The site runs under Windows Authentication to allow my wife and I access. The toggle requests are queued in a database which is then queried on a schedule by another job that runs with elevated privs to execute the toggle requests and mark the database records as completed and timestamped. Very simple stuff - Nothing Earth-shattering here.

The challenge came with choosing a database. I thought about SQL 2005, but my AD DC server (my lone wolf) has three legs and piling another sack of goods on might break it's back. So I decided to go lightweight. I looked for SQL Express 2005. But when I get to the Microsoft site it goes right to SQL Express 2008 CTP/RC or whatever. Now is where the fun begins!

One of the pre-reqs for SQLX08 is to have .NET Framework 2.0 SP2 installed. But there is no .NET Framework 2.0 SP2. There is only .NET 2.0 SP1. The docs took me on a wild goose chase to discover a small post that .NET Framework 3.5 is supposed to rectify that, and allow a .NET 2.0 SP2 pre-req audit to pass. Not so! It does not! I have 3.5 installed and it still chokes during the SQLX install saying it needs .NET 2.0 Sp2 installed and stops there.

Frustrated, I did a Google search for SQL Express 2005 and found the download link on the Microsoft web site (funny how it still seems to be easier to find things on Microsoft's web site using Google). I installed SQLX 2005 and setup the table and permissions. I ran into so many stupid problems trying to get ASP to not only just read it (aka "SELECT"), but try to modify (aka "UPDATE") table rows. Gnashing my teeth again, I opted for MS-Access 2007.

More fun.

Google is the best. How can we live without it? I created an Access 2007 database with a table and all the required fields and datatypes, constraints, etc. Then I copied the .accdb file up to the ASP folder on my web server/domain controller/file server/print server/music and media server/foot-rest server, and linked it up to my ASP page using ADO and standard connectionless stuff. It failed with an expected 500 error. This page describes why and what to do about it.

Google led me to find the required Access Runtime Engine for MS-Access 2007. Even though there's no mention of it being required for WS08 and it's not on the system requirements list, it worked! Best of all, I didn't even have to reboot my server after installing it. And when it finished, I simply hit refresh on my ASP web page and it cranked up fine.

Now, my wife can go to our internal web site and lock the kiddies out and continue on with her chores without having to call me at work and whine about having her hands tied. I may scale this up if I have time and a need, but for now it's fine.

My 2008 IT Forecasting Crap

Oh, look, another person making predictions for the coming year. How amazing. How exciting. How stupid. Yes. It is stupid. But I have some time to kill, so why not add my 2 cents to the mountain of coinage already piling up on the web?

• Virtualization - Both operating systems and packaged/sequenced applications, will gain popularity in 2008. Look for an increase in the number of projects looking to deploy EMC's VMware, Microsoft's Hyper-V, Xen, and others in the data center. In addition, look for more cases of Microsoft Application Virtualization (formerly called "SoftGrid") deployments.
  ...
• Apple will March Onward - Look for a new iPhone in 2008 with many of the features current users are yearning for. Also, look for a new handheld/UMPC form factor device for mobile browsing or possibly something akin to the Sony Reader or Amazon Kindle products. The IPod product line needs some sort of boost, so look for increased storage capacities per line.
  ...
• Windows Server 2008 - What can I say. I love it. I love Linux also, but WS08 is one of the finest products I've seen come out of Microsoft in years. Very well designed and built. I would have liked to see a more thorough method of building "Server Core" than simply decoupling existing hooks, but it's a very bold and risky first-generation attempt. It works, so that's the biggest issue, and it will be accepted very well in 2008 along with the rest of the (GUI) WS08 product line.
  ...
• Vista will continue to struggle - Even with the added benefits of coupling it with WS08 (hence the marketing mantra "Better Together" you're going to start getting sick of hearing). There's still too thin of a rationale for justifying the expense, overhead, and deployment effort for most large environments. Most are going to defer their efforts towards further XP deployings using Service Pack 3. Unless Microsoft makes some dramatic changes in how Vista is packaged and sold, most companies are going to stick with XP until Windows 7 ships in 2009.
  ...
• Microsoft System Center will Do Well - Both Configuration Manager (SCCM) and Operations Manager (SCOM) will become more popular by Spring of 2008. Look for more projects involving those products.
  ...
• Microsoft SharePoint 2007 - AKA "MOSS" will continue to gain momentum in small to medium business environments in 2008. Larger environments will also see increased uptake but as a percentage of overall IT time/labor it won't compare with smaller environments who can adapt their workflows and methods to a new platform much more quickly and easily.
  ...
• Linux will Continue to Struggle - On the desktop. Yes. It will. As much as nay-sayers like to bash Microsoft over Vista and what-not, Linux is simply not an option for most medium to large environments. I won't go into the usual shopping list of reasons why, but suffice it to say, that running Linux with Wine and/or Xen to force-fit difficult Windows apps is just a dumb idea to propose in a boardroom meeting. It would get you fired immediately. The Linux world needs to take a lead for once, instead of continually saying "we can do that also!" and jumping up and down in the back of the room. Do something REALLY unique, not just different. There's a million ways to accomplish media streaming, web services, file sharing, authentication, mobile applications development, and virtualization. Those are done. Do something Windows and OSX haven't done.
  ...
• Google will try more Hail Mary's - Google has become laiden with overgrowth of corporate proportions. It has become increasingly similar to what it intended to avoid: Another Microsoft. Rapid growth, hiring, scaling out, and the overhead that automatically incurs, have all placed a massive drag on Google's ability to innovate rapidly. Most of 2007 has been evolutions and tweaks, as opposed to revolutions like we saw from them in 2004-2005. Do something with the Gmail interface for God's sake! Where's the huge lead you had over Microsoft? News flash: MS is catching up on you. Slowly, for sure, but they always do. Look for Google to make a few major releases or announcements in 2008 to help regain their lead in various areas. Hopefully, they do more than that.
  ...
• Microsoft will Continue to Expand - The Zune 2.0 is only a mid-term effort. Look for a more impressive push in that market to compete with Apple. IE8 will be another significant announcement, as will a bevy of "cloud" computing services for small-medium customers. Aside from the expected releases of Windows Server 2008, Visual Studio 2008 will also gain momentum in 2008, along with .NET 3.5 and 4.x later in 2008. Incubator products and technologies like Silverlight-oriented toolkits, tons of goodies that have come out of the birthing room known as MSR, and all sorts of things in between (XBox 360, MSN, Live, etc)
  ...
• Firefox 3.0 will do well - If Mozilla can manage to avoid stuffing too much bloat into 3.0 like they did with 2.0, and continue to impress with faster rendering performance and slicker features, it will do very well. Maybe well enough to push back against increased spread of IE7 and IE8. Opera will continue to decline. Safari will grow on OSX but decline on Windows platforms as a result of IE8 and FF3 popularity in 2008.

2008: The year of server flipping

We've all heard too much about "flipping houses" on various TV shows, magazines, web sites, etc. The practice of buying a "fixer-upper", fixing it upper, and then selling it for a profit, is a respectable effort (even a profession in some cases). But how does this fit with the IT world?

Simple: It fits with virtualization and existing hardware. A lot of the buzz being regurgitated on the web today is about how much virtualization will play in data center plans for 2008. There are all sorts of reasons and rationale provided. The interesting thing is how the various offerings will "fit" with respect to budgets and refresh cycles in most medium/large data center environments.

Many big spenders just finished a round of big spending, so it may be a tough thing for many IT managers to gently pry open the boardroom door again and kindly ask for more funding to chase after yet another enticing goal. The benefits are easy enough to justify and bulletize:

• Run more roles on less hardware
• Reduce electricity costs (hardware, cooling, etc.)
• Portability and failover simplification (aka Disaster Recovery)
• blah blah blah

Then comes the clincher: How much?

Well, the answer is, as always: "It depends".

On what? Well, you may be able to leverage your existing hardware, but that depends on which virtualization platform product line you intend to pursue. EMC's VMware ESX Server, Microsoft's upcoming Hyper-V and Hyper-V Server products, and open source (sort of) products by Xen and others, each have their own hardware limitations with respect to the physical host machine. That said, in a perfect world, many will be more than happy to buy new hardware with the latest and greatest goodies built-in, and begin consolidating their older machines into VM guest instances on the newer box. But in reality, many will be forced (by the evil budget folks) to make-do with existing hardware. In many cases that will mean hardware that won't run 64-bit products.

What to do?

Many will roll up their sleeves, drink some coffee, tell a few bad jokes, and while everyone is trying hard to laugh (like it was really funny), the IT tech will be switfly wiping a box and reloading with something like Linux and VMware Server 2.0, both of which are FREE and run just fine on 32-bit hardware. Ideal? Not really. But then again, define "Ideal". It is an entirely subjective term, like my other favorite terms: good, great, ok, fine, not too bad, and my favorite of favorites - so-so.

I will call this "flipping servers", since it's not moving things (servers) from older hardware to virtual instances on newer hardware. It will be ripping the walls and floors out and rebuilding the insides of the house and moving all new appliances and furniture (and tenants) in. Slick-n-reload as many refer to it.

So, for 2008, I see a considerable amount of server flipping in the pipeline. Everyone wants to virtualize and consolidate and gain from the benefits. But many don't have the budget or the cojones to ask for a budget increase to do it the way the vendors recommend. And is this really a failure? Hell no. Not by a long shot. It's getting the job done within imposed constraints. And in most cases it will work just fine. Maybe not for the likes of eBay, Amazon.com, or WSJ, but for most small-medium business data center environments (ok, "server rooms" for most of us), it will be what works and be affordable and attainable.