Root Cause Analysis of IT Service Interruptions

I used to think about the day when I fixed everything so we would stop IT outages. Of course that is silly. Like other healthcare organizations we are adding applications to the portfolio every year as new solutions address previously under automated areas. Most of these are not core parts of the IT architecture, but they are supplemental such as documentation systems for clinical departments (e.g., rehab) and contract modeling systems.

With the increase in the number of applications in the portfolio comes complexity. In addition our infrastructure is becoming much more complicated including a more sophisticated network; changing virtualization technologies; and complex storage.

So, our IT Operations philosophy is to perform a Root Cause Analysis on every critical service interruption. Our Root Cause Analysis asks three things:

  • How can we prevent this type of outage in the future?
  • How can we detect this type of outage in the future?
  • How can we respond to this type of outage more quickly?

The second two questions are important. Even if the cause of the service interruption is s simple fix, sooner or later stuff is going to hit the fan. We want our IT folks to see when it does and already be communicating to our customers how we are fixing the problem before they call us.

Someone Else’s Meaningful Use Rant

Dr. Michael Koriwchak writing for the Wired EMR Practice blog:

“And our EMR use, our quality of patient care and our practice efficiency is for the most part no better.  In some ways it is worse.  As a result of MU”

I can see how that can happen. It is important that we hear the skeptical and the inspiring. The post is worth the read and the author’s candor is important.

Healthcare IT Consulting Is Broken

Somewhere along the way the word consulting in our field changed. Today consulting is about finding available freelancers on a just in time basis. The “consultant” is nothing more than a recruiter with a billing back office. Some consultants claim they screen the candidates, but there is no way that can be done effectively given the turnaround time to place people.

Furthermore, the consulting firms take very little accountability for the consultants they place. But, how can they when their experience is so varied and there is no standard for good service?

When I hire a consultant, part of what I am looking for is a well defined way of doing various types of work. I want the consulting group reviewing each engagement and revising their approach to work based on the lessons learned from each engagement. If I am going to hire a project manager, I want that person trained in the firm’s project management approach. If I hire someone to assist with a selection, I want that firm to have a clear written means to conduct IT selections. I don’t want someone that might have participated in one of these activities a while back and will try to mimic one the way a child mimics an adult.

Of course that means a large investment in people that develop these methodologies and take the time to train permanent staff. That seems to have gone the way of the dodo bird. Nobody has staff, they have home-based employee people working the phones looking for talent to place.

Update: In re-reading this post I recognize that it is too general. There are a lot of consulting groups that bring intellectual capital to the table. When I am introduced to a new consulting group the first thing I do is categorize them as a traditional firm with an investment in their staff, or a recruiter of free agents with no connection to the people they place.

Update 2: Too frequently someone claiming to represent a consulting firm, is really with a staff augmentation firm. There is a big difference between the two and I wish the staff augmentation firms understood this.

The Role of IT Engineering

A couple of years ago we separated our “technology division” into two groups: IT Engineering and IT Operations. The dividing line between the two is the production environment. Any new technology is architected by our Engineering group before it goes into production. Once something is in production it belongs to IT Operations and it cannot be touched without going through the change management process.

Here is an example of the IT Engineering group doing a good job:

All IT organizations are seeing a mounting desire for employees to use their own devices (especially iPads) in the workplace. When I recognized that this demand would be huge, I began advocating to connect Android and iOS devices to our Exchange Server via AtciveSync I went to the Engineering team, who is charged with evaluating new technologies before they go into production.

To their credit they said that the vanilla approach to device connectivity would not meet our security expectations. They told me that the only way we could safely manage employee owned devices would be through a device management system that would sandbox the organization’s data, protecting it from security flaws, malware and poor user security practices. They also told me that this would only cover the Exchange connectivity use case and that any other use cases would require further analysis (and perhaps additional expense).

I was disheartened to learn about the added cost, but I would much rather surface that with our executives so we can make a fully informed decision rather than spring a surprise expense on them later.

IT in Radiology Departments

Radiology IT is a more challenging area than other helathcare IT systems. The Radiologists (and cardiologists) rightly want to be very involved in the selection of the systems that they interact with. Many of them sit in front of these computer systems all day and something as nuanced as the placement of a button can have a great impact on their productivity and overall satisfaction. In this regard, trying to select a mutually acceptable Radiology IT system is much like standardization of orthopedic implants or surgical sutures.

eMail Stats for Our Health Care System

Over the last 3 years the number of email users at the organizations I serve (Ministry Health Care and Affinity Health System) has increased 17%. In 2008 we had 15,900 mailboxes. Today we have 18,578 mailboxes.

Over that same period our total email storage has increased from 1.5 Terabytes to 5.6 Terabytes. That is a 273% increase.

Interestingly, the top 7% of the mailboxes (1,300) account for 55% of the total storage. My mailbox is the 88th largest (5.3 Gb), this includes deleted and sent items, which I cling to.

These cool stats are courtesy of our Data Center Manager, Dave Roggenbauer and Pete Leonard, System Administrator.

Apple’s Fuzzy Healthcare Number

On October 4 Apple’s CEO said: “Over 80% of the top hospitals in the US are now testing, or piloting, the iPad.” This statement is a little misleading.

Firstly, he said “testing or piloting.” What is the number of hospitals that are actually buying and deploying iPads to use with their core clinical applications? I bet that number is less than 1%.

What exactly defines a “top hospital in the US?” Are those the ones that answered Apple’s calls? I have an Apple sales executive (Rachelle, who is a super nice woman) that periodically calls on me to keep track of what we are doing. When she last called I toled her we had a pilot taking place to test connecting employee owned devices (including iPads) to our Exchange Server so our employees can get email and appointments.

That’s it, just a pilot. No roll-out. No committment. Furthermore, it is just for Exchange connectivity. Still, I presume this means that our 15 hospitals are in that 80%.

But, the picture behind Cook is two men, with lab coats and stethescopes, staring into an iPad. This might give the world the impression that 80% of the hospitals have doctors and nurses accessing electronic patient records on an iPad.

Not yet.

My organization is not going to deploy the current generation of Windows client applications on iPads, just because they are cool. When core vendors produce native iPad apps (or properly formatted web apps) that will be the right time to look at radical device changes. And don’t talk to me about Citrix. Citrix on the iPad’s relatively tiny screen is a terrible user experience and not worth the cost to make a radical change to our device support model.

Doing Something Different (Go Brewers!)

I have been writing this blog for 8+ years. This is the first time I recall going completely off topic. I want to talk about baseball. Tonight all of us in Wisconsin are celebrating the Brewers advancing to the National League Championship Series.

There was a play in the top of the 9th inning that drives me crazy. The Diamondbacks had runners at the corners with 2 outs.  The Brewers Betancourt scooped up a slow bouncer and raced to 2nd base, ariving a split second before a sliding Justin Upton.

Why does Upton slide into second base? I would argue that sliding slowed Upton down enough to allow him to be forced out. What if instead, he were to run through 2nd base to avoid the force? Sure he would be tagged out before he could reach 3rd base. But the runner at third would have already crossed the plate before he could be tagged. I am gad that didn’t happen because that run would have likely eliminated the Brewers.

If a runner is forced out for the final out another runner crossing the plate will not score, even if the runner crosses home before the force out. If the runner is tagged out after reaching the base, then it becomes a matter of what happened first – the final out or the runner touching home plate. Because the runner at third usually has a bigger lead, they typically can cross home plate at the same time the force out occurs. Even if Upton were immediately tagged after running through second, the Diamondabacks would have scored the winning run.

Maybe someone will post a comment telling me why I am wrong. Maybe there is a rule that requires the runner to slide. But I think Upton slid because, for over 100 years, every player in the same situation has slid.

Is there something in your IT operation that you do because it has always been done that way? Is there a simple but unconventional change that you can make to get a better results?

Maybe this post wasn’t off topic after all.

The short list of great American inventors

Alexander Graham Bell (March 3, 1847 – August 2, 1922)
Thomas Alva Edison (February 11, 1847 – October 18, 1931)
Henry Ford (July 30, 1863 – April 7, 1947)
Steven Jobs (February 24, 1955 – October 5, 2011)

These people radically changed the lives of every American alive in their lifetime. It was awesome for me.

SaaS and the Enterprise

I hate desktop software. Every new application is a potential conflict with mission critical software. Then you have the effort of installing and updating. It is a terrible model for the enterprise. Wherever possible I am looking for Software as a Service (SaaS) option.

Adobe has a potentially great service in their CreatePDF SaaS offering (once they add the ability to manipuate PDFs online). The need to create and manipulate PDFs is wide-spread in my organization, but I don’t want to take on the burden of installing Acrobat or some similar app on hundreds of desktops.

Unfortunately Adobe is rolling CreatePDF as a consumer offering. A separate credit card for each use is not a model that works for the enterprise. People like me would be lining up for this service if there were a front-end where our Provisioning team could easily add and remove users and we could receive a quarterly invoice based on the number of users.

I think Adobe is missing the boat. DropBox is another cool app that does not have an enterprise model (box.net seems like a good enterprise alternative).