Mixing Oil and Water: Systems Engineering and Software Development

In all my years working this field we call "IT" I have never seen an environment where these two groups coexist cohesively and with a common goal.  Systems Engineering and Software Development.  Sure, they might get along well, and share working spaces, but as functional entities they typically exist in separate worlds, chasing their own beasts to slay.  And why not?  After all, they serve different purposes, for different masters (usually). They use different tools and techniques.  They often have very different reasons for doing things as well.

The Developers want to create and update things.
The Engineers want to implement and upgrade things.

Hmmm.  Those aren't that different after all.  Are they?  I know some folks would argue that engineers want to implement and "maintain" things, but maintenance is really an operations role, which in most cases falls into the hands of Systems Administrators (or "sysadmins" to use contemporary parlance).  Engineers want to engineer things.  Simple enough.  However, maintaining isn't really engineering, it's really maintaining or administering; keeping things moving along without interruption.

Developers want to create new things.  Write new code, new features, new widgets.  Add new capabilities or make the interface more "cool" and intuitive.  They don't want to maintain things any more than engineers do, even though they have to deal with bug fixes, patches, service packs and regression testing.  Life sucks.  Nobody said the gravy doesn't have lumps every now and then.  Engineers have to deal with those lumps too, especially when transitioning new things into the hands of the SysAdmins.  It's life (and it happens to pay pretty darn well, thank you).

So, if these two camps are so similar, and they often possess such awesome potential, when then are they not more often exploited towards a common goal?

Prologue

I'm writing this because this quasi-mythical world is precisely where I've found myself repeatedly over the past twenty-odd years.  You see, I started out as a draftsman (on the board, with sepia, paper and Mylar, thank you), and was part of the wave of folks in the 1980's who got corralled into the emerging world of CAD, or Computer Aided Design.  That led to customizing the CAD toolset, which sucked me into the world of software development and programming like drunk sailor in front of a Thai brothel (no, I was never in the Navy, nor have I ever been to Thailand, the phrase just seems to fit).

After years of that, I drifted into "mainstream" IT by way of helping an old friend implement SMS 2.0 in a large corporate environment (ok, he did 99% of the work, I just handed him some wrenches), but it was enough to turn on a light bulb in my head about the incredible potential of a networked environment, the tools, frameworks and protocols that make it possible to leverage a whole new level of capability and productivity on a much larger scale.  LDAP, ADSI, WMI, TCP/IP, SNMP, the Windows API stacks, Registry, Event logs, alert mechanisms, and so on.  Then came ever-increasing power and simplicity with enterprise databases (Oracle, SQL Server), and the arrival of web technologies.

Oh man, I was so there.  SO THERE. It was like a labor camp escapee wandering into a Ruth's Chris restaurant on "free dinner" night.

Even now, even when I get involved in another project to inject Development into Engineering, it's the exception, never the rule.  In most cases it creates such an oddity that neither camp gets offended, just more or less confused about the value.  That is, until the project starts to bear fruit.  Then I get Engineers asking about the development aspects, and Developers asking about the infrastructure engineering aspects.  It's kind of like the old Reese's Peanut Butter Cup ads from the 1980's.

Continuing On...

As Yogi Berra might have said, I've been doing the same thing over and over again, only differently each time.  I am handed some problems to solve, which span multiple systems, multiple departments, multiple environments, multiple business cultures, multiple security realms, multiple personalities, and asked to somehow build something of a bridge across all of it to get the traffic (information) moving.  I don't really have name for it.  Some call it BPA, or Business Process Automation, or Systems Automation, or Data Mining (not a really great term).  Some don't know what to call, so they just give a description like "that stuff Dave is working on".  But whatever it might be called, it's where these two worlds intersect.  And the result is quite a lot like the Super Hero Action League touching their power rings together (only without the "shazamm!!!" sound effect).

A lot of vendors would argue you can solve almost every problem with an out-of-the-box solution, but anyone who's worked in this realm for more than a decade knows that's as realistic as pocket-sized nuclear fusion power.  Maybe someday.  Maybe someday.

Putting this a slightly different way: Any time you implement something to abbreviate the steps required to perform a repetitive task, it most likely involves writing a script, a program, or configuring a bunch of options in a widget to enable this to happen.  This borders on Software Development.  With the advent of PowerShell, especially the concerted effort behind its proliferation, we're seeing a surge in the popularity and prevalence of SysAdmins writing scripts to handle chores that were once considered off limits.  Maybe it was because of a bad perception about VBscript or Batch scripting?  Maybe it's because PowerShell is becoming so intrinsically part of more and more Microsoft (and VMware) products?  Maybe it's because of it's Spartan syntax and brevity (at the command line, not always so within cmdlets)?

I've met so many people in this line of work who have strong feelings about "best practices".  Many are of the opinion that unless there's a ready-made product to do something, it shouldn't be done by other means.  This is not only short-sighted and foolish, it's downright disingenuous towards the employer.  If there exists a means to solve a problem now, even if it requires a little elbow grease, you owe it to yourself, your employer, and your customers to consider it.  Especially if this involves existing means which are "free" (provided with the products already purchased: Windows, etc.)

Microsoft didn't provide this insanely broad inventory of API's, protocols, command tools, and utilities for talking about at dinner parties.  They were provided for your benefit, and the benefit of their customers to build things to do more than they do "out of the box".   Don't be afraid.  Don't fear change.  The "Soft" in Software was Charles Babbage's gift to all of us that freed us of the Iron shackles to hardware.  It was intended, from the very start, to be "changeable".  To allow us to adapt processes and capabilities to meet newer challenges.

If you ever, and I mean EVER, enjoyed building things as a child, whether it be Lego's, model kits, model rockets, Linkin Logs, or even stacking wooden blocks, you should find a lot to enjoy building things on the Windows platform.  Heck, any platform to be honest, but for this discussion I'm thinking along the lines of Windows.  It seems that either a lot of grown-ups forgot how much fun that experience was, or they never got into it in the first place.  If you consider yourself an Engineer, and ever wondered what it was that Developers found appealing in their work, it's the fascination with putting things together from scratch and seeing them perform when finished.  Engineers get that sensation as well, but at a higher level in the logical technology stack.

In most cases, Engineers are working with components which were provided by Developers, but this is where it often stands and never changes.  There's still room for development during the Engineering phase.  When the out-of-box components don't hit every note in the song, it can do wonders to get a Developer involved.  In most cases, if the right people are involved, the song comes out even better than expected and ideas start popping out for new potential and new directions.

Conclusion

IT is all about change.  It's an intrinsic part of the fabric of technology.  To fear it is to refuse the absolute purpose in and of itself.  Nothing demonstrates, or proves, the value in embracing this more than seeking a convergence between Development and Engineering. Nothing.  The teaming of the "thinkers" with the "do-ers" is the ultimate way to push it as far as it can go.  As long as these two worlds are kept isolated, or at the very least, un-involved, the less we take this profession seriously and the more handicapped we keep our employers as a result.