change control

, 0 comments , , , , ,

How change breaks: four patterns to know about

I’ve been studying corporate enterprise systems for thirty years and have worked with about eighty different organizations in the process. Most of them had mission-critical applications that failed a lot. I was the software “jack of all trades” who came along to remedy those failures.

Out of that experience, some incredibly strong patterns have emerged, which have illuminated my problem solving this whole time. These patterns are themselves consistent, reliable, and solid—even though the phenomena they describe are complex, expensive, and often painful to deal with.

Pattern: Change is the enemy

Spoiler: alertness is key. (Photo by @Mawi on Twenty20.com)

First off, if a working system stops working, it’s because something has changed. That’s obvious, right? If your enterprise system is working, and if you’re able to keep everything around it from changing, you will have perfect reliability—so any failure must be a result of some kind of change.

That includes code, configuration, operating systems, APIs, and database schemas. But it also might include availability, which could change suddenly and without warning. And what about changes that are entirely unauthorized, such as malicious changes that should have been spotted by your intrusion detection system?

So we start with an incredibly basic principle: If you want a guarantee of zero risk, it means you have to have zero change.

More

, , 0 comments , ,

Change observation, not change control

A while ago I posted about Baqbeat, an enterprise systems management tool being developed by my college homie Dave. A lot of things have changed since then.

For one thing, the name has been updated to the less quirky but more descriptive “Congruence.” For another thing, the code is actually getting done. And, as we were discussing the other day, the minimum usable product concept has been pretty well refined.

Why this thing even matters

In a recent Facebook chat, Dave put it this way:

That is my real, underlying motivation… cutting back on the bickering and finger-pointing! It starts with replacing “But we didn’t change anything!” with “What changed when?” and getting a computed rather than human answer.

Then I ran into John, who runs a pretty substantial outsourced data center, at my favorite coffee joint in Lakewood and talked his ear off for an hour and a half. The thing he picked up on was the distinction between change control, which everyone wants to do but nobody has figured out, and change observation, which is actually feasible. I think it was John who said that trying impose a global solution to maintain all the database structures, configuration files, source code releases, and other knicknacks in your big corporate setting is “boiling the ocean.” He’s right. Strategically, you can try to control everything–which becomes a huge top-down endeavor that depends on someone making a big decision and making scores of underlings follow through–or you can take the gentler approach of watching what does change, and correlating it when something goes wrong.

I could go on for a while about the Congruence architecture that supports all this monitoring and correlation, but the cool part for me is that it has Agents and Minions and Dave doesn’t know very much about .NET. So in a couple months you may see me coding a base Congruence Agent for .NET.

For another perspective, Dave summarized his “aha! moment” on this point about observation.

What you can (and should) do now

If you have anything to do with changing or managing stuff in a network that has more than about three computers, go to the pretty placeholder site for Congruence right now and get yourself on the email notification list.

Start thinking about things in your network that change and break stuff. Think about how much it will help to have something that just lets you know what changed.

Get ready for a fall pre-beta thing.