The AntiSyphus Effect

More RCA Resources

I started this series ten entries and nearly one month ago. I hope you've been able to take away some useful information; in particular, I hope you've gained an understanding of just how much your business can gain from using RCA to help diagnose business problems and prevent them from reoccurring.

I've developed a real passion for RCA, and although I work in an industry that requires its use, nearly any business can benefit from it. In fact, I suspect that anyone who goes through a few RCAs will find that s/he too develops a passion for it.

Let's recap, shall we?

• Root Cause Analysis is the art and science of determining how and why a bad event occurred.
• The goals of Root Cause Analysis include learning and preventative action, and not problem-solving or assigning blame.
• You can remember the characteristics of root cause using the FERCS test, and the PHaTS Domino Theory can help you remember to dig a little deeper as you conduct RCA.
• Conduct RCA by doing the following:
  1. Assemble a team to conduct the analysis, and agree on the problem.
  2. Identify likely categories of potential causes.
  3. Brainstorm on potential causes, and enter them into a diagram.
  4. Use the Five Whys (or Five-by-Five Whys) to squeeze as much information as possible out of every possible cause.
  5. Create a FERCS matrix to measure the quality of each potential cause.
  6. Identify and implement CAPA.
  7. Monitor the effect of the CAPA, and document your work.

Additional RCA Resources

• This article by James J. Rooney and Lee N. Vanden Heuvel is a terrific resource for RCA beginners. In addition to root cause theory and some background, Rooney and Vanden Heuvel offer example charts you can use as part of the process.
• If you're having a hard time selling your boss on RCA, this analysis by Arthur M. Schneiderman could prove helpful.
• Bill Wilson has a wealth of articles about RCA on his site.
• This article found on the University of Alabama Web site provides more information about how an organization's culture can create problems.
• For more information on RCA tools, read this article by Dr. Anthony Mark Doggett.

RCA can help identify and mitigate the habits, assumptions, and management philosophies that lead to problems in your business. A good RCA program is good for business in a number of ways: it can improve profits (by reducing errors); it can improve customer relations (as your customers watch you really fix problems instead of placing bandages on the symptoms); finally, it can increase employee morale (when your employees see that you use RCA to do more than punish them for their mistakes).

Analyze, learn, improve, repeat. Your business will love you for it.

Life after CAPA

Need to catch up? Read the entire series here.

Congratulations! You've analyzed your business problem, identified a root cause, and implemented a CAPA. So now you're done, right? Well, not so much.

Webster's Dictionary lists several  definitions for theory. And really, what you've done in the process of RCA is develop theories. You have a good idea of the root cause, and you've developed what seems as though it will be an effective CAPA. But like any scientific theory, your work needs to be proven. Implementing a CAPA is not the end of the process; it's the mid-point.

If you were wondering why I recommended that you focus on one root cause and one CAPA at a time, now you know. You need to test each CAPA to find out if it works; if it doesn't, then you can safely assume that either your root cause or your CAPA is wrong. Increasing the complexity by concurrently working on multiple root causes and CAPAs will make it far too difficult to see if your CAPA is having the desired effect.

I'm not going to get into a lengthy dissertation of how to measure your CAPA; you know your business better than I do. At its simplest, however, is this: if the problem continues, then you did something wrong. If that happens (and it will at some point), gather your team again and figure out the next step: do you want to implement a different CAPA? Was the CAPA you implemented essentially on the money, but needs a bit of tweaking? Or do you need to create a CAPA for a different root cause altogether?

This leads me to documentation. From programmers who don't like to comment their code to managers who fail to document personnel discussions, people hate to document their work. But documentation is a critical part of RCA (as we say in the compliance biz, if you didn't document it, it didn't happen). The reason documentation is so important to RCA is that you might have to revisit what you've done and make changes; if your documentation is poorly developed and maintained, you may find that the team has to recreate some of the work it's already done. And nobody wants to do the same work twice, yes?

In addition to the fact that the team may need to revisit the analysis, you need to consider something else: turnover. People may come and go, but the issues in a given organization tend to hang on; and while everyone will be gung-ho on the CAPA initially, folks tend to get a bit lax once a problem seems to disappear. If this particular problem pops up three years from now (and the original RCA team is gone), think of how helpful it will be for the new folks to have your experience to draw from when working on resolving the issue. The documentation isn't just for you: it's for everyone who follows after you've moved on.

At a minimum, your documentation needs to include:

• The fishbone (or other) diagram that illustrates all of the potential causes identified during your brainstorming sessions
• The FERCS scoring matrix
• Detailed notes from your CAPA brainstorming session

I would also suggest that you develop some sort of system for tracking and trending RCA. You might find that the same types of issues keep occurring; alternatively, you might find that a disproportionate number of negative events are coming from a particular workgroup. The more information you have at your disposal, the more effective you'll be in preventing problems in the long term. And if you can spot emerging problem trends before they become established bad habits, it'll be much easier to resolve them.

Tomorrow I'm going to wrap up with a variety of RCA resources you can visit for more information. And you're always free to e-mail me if you'd like to ask me any questions.

An ounce of prevention is worth... well, you know

Need to catch up? You can read the entire series here.

At long last, you've completed your analysis, and you're ready to talk CAPA (that's Corrective and Preventative Action, in case you didn't know). Before we begin, I have six words of caution for you: take one thing at a time. You may have several strong root causes that you want to eliminate; however, I recommend that you start with one. If you implement multiple CAPAs for multiple root causes, it's going to be difficult to gauge the effectiveness of each CAPA.

The term CAPA, although it's used as a single entity, refers to two different activities: corrective action and preventative action are not synonyms. Corrective action is problem-solving; it's fixing the immediate issue, and is often completed fairly quickly. For example, if you look at the order entry error I used as an example in an earlier entry, the corrective actions might be:

1. Apologize to the customer for the inconvenience.
2. Issue a credit for the amount overshipped.
3. Arrange for a freight carrier to retrieve the overshipped product and return it to your facility.

And just like that, problem solved. However, you've done nothing to keep the problem from happening again. That's where preventative actions come in. In our example, we concluded that the root cause of this error was the mistaken belief that "everyone" knows that we have to specify the unit of measure on the order entry form, and therefore it's unnecessary to modify the form to request the unit of measure.

Many businesses would suggest that the solution to this problem is simple: train all the employees so that our assumption that "everybody knows" becomes fact. Well, that's probably not the best answer, unless you want to re-train every time someone forgets (and they will), or with each new hire. Mind you, I'm a training professional, so it irritates me no end when folks think that training is the answer to every performance problem. Trust me, it's not.

So how do we prevent this from happening again? You need to think about systems; not software systems, necessarily (although sometimes a software modification is in order), but business systems. Once again, your team members need to leave their egos at the door. This is the time for management to 'fess up and admit that they've contributed to the problem by creating a system in which it's easy to perform the task improperly, and difficult to perform it correctly. If you're a manager, your goal should be to help your employees succeed by making it easy to do their jobs correctly, and difficult to do them incorrectly. Only after you've helped remove their obstacles to good performance can you reasonably hold them accountable.

You'll identify CAPAs in the same way you identified possible causes: brainstorming. If you were able to score the "fixable" quotient of the cause, then you've probably thought of possible CAPAs as part of that process. Now you'll flesh them out a bit more. When you're developing CAPA, there are a few questions you'll want to answer:

• Who owns the change(s) we're putting in place?
• Whose jobs will be impacted by the change(s), and to what degree? How do we communicate the change(s)?
• Do we need additional resources (human or capital) to effect the change(s)?
• Who will resist the change(s), and how can we overcome that resistance?
• How much will the change(s) cost, and how long will implementation take?

And the most important question of all: how will we know if the CAPA(s) work?

Tomorrow I'll post this week's favorites; Monday I'll cover monitoring CAPAs and documentation, and I'll finally wrap up this series on Tuesday. Have a great evening!

You take the good, you take the bad

Need to catch up? Read the entire series here.

Now that you have a comprehensive schematic of possible causes - and you've delved into each possibility to get to the hidden assumptions that may be driving those causes - it's time to start filtering your options. While it's entirely possible that the scenario you're analyzing could have more than one root cause, you can't fix everything at once; you need to prioritize. In addition, you need to throw out the red herrings. Remember, in the first phase of this process, you accepted any and all possible causes; now you have to take the opposite approach and make each possible cause fight for its life.

You may be wondering how on earth you're supposed to start filtering through the dozens of possibilities, and the answer to your question is this:

Put all the possible causes to the FERCS test.

You remember the FERCS test, don't you? It identifies the characteristics of a root cause.

"But Kathleen," I hear you cry, "how does this test help when there might be a lot of possible causes that meet all of the FERCS criteria?" Ah, grasshopper, what you must do is measure how well each cause meets the FERCS test. To do this, plug the possible causes into a matrix that lets you score them against the criteria.

Rather than make you come up with your own, I've designed a simple matrix that you can download and use freely. The file is an Excel workbook that uses a weighted average score, with the fixable, explains the bad event, and controlled by management scores weighted twice as heavily as the remaining criteria. I've weighted the scoring in that way because my experience has demonstrated that those criteria are the key players in RCA. The tool helps you measure the quality of each cause, and move forward accordingly. Identify a minimum passing FERCS Score, and then eliminate any causes that don't meet that minimum score.

By now, you should have narrowed down your list of possible causes to the mostly likely culprits that are also those you can reasonably expect to correct (and prevent from popping up again). The next post in this series will address developing CAPAs.

Channel your inner two-year-old

Need to catch up? You can read the entire series on RCA here.

At this point of RCA, you should be looking at a diagram filled with possible causes. But you're not done yet: this first pass of brainstorming tends to yield physical and human causes; in order to get to the top-secret causes, you need to dig into each of the causes you've identified thus far.

If you've ever parented a toddler, spent time with a toddler, or been a toddler, you'll know that the period known as "the Terrible Twos" is defined, at least in part, by what seems like constant curiosity. Annoying though it can be, the average toddler's tendency to ask "why" until your ears bleed is a habit you'll want to borrow in this phase of root cause analysis.

This technique is referred to as the Five Whys. Don't be misled - five is not a hard and fast number here; you might need to ask more than five times (or fewer). The point is to continue asking why until you get to what's really driving that particular cause. For example: you're analyzing an overshipment of widgets; among the possible causes is "Customer Service Rep mis-keyed the order quantity." Your mission, should you choose to accept it, is to figure out why.

So, why did the CSR mis-key the order? Because s/he thought the order was for 10 cases, when it fact it was for 10 eaches.

Why did the CSR think that the order was for 10 cases instead of eaches? Because the customer didn't specify the unit of measure on the order form he submitted.

Why didn't the customer specify the unit of measure? Because the order form in place doesn't have a space to enter unit of measure.

Why doesn't the current order form ask for unit of measure? Because it's an old form from our previous system (which used only one unit of measure for ordering), and the form hasn't been updated.

Why hasn't the form been updated? Because updating the order form is not considered a high priority.

Why isn't updating the order form a high priority? Because everyone knows that we have to specify the unit of measure in the order entry system, or all orders will default to case unit of measure.

Okay, so that took six whys, but we got to the top-secret cause, didn't we? If we can convince ourselves that "everyone" knows that they must specify the unit of measure (and apparently, that's exactly what we've done), then we don't need to go to the trouble of updating a form. These types of assumptions drive more business inaction than you might think - and consequently cause more problems than you might think.

If you want to take the Five Whys to a deeper level, you can use the Five-by-Five Whys. I find the Five-by-Five Whys to be a terrific tool in RCA, because they can help you both identify latent causes and filter your list of causes as you go, thereby helping the process move a little more quickly.

Let the ideas rain down

If you'd like to catch up, you can read the entire series here.

Now that you've established a loose framework of possible cause categories, it's time to continue your RCA with the third step:

Enter possible causes into the diagram of your choice.

To complete this step, you'll gather your team and conduct a guided brainstorming session. As with all brainstorming meetings, the only constraint is that all ideas are welcome. Nothing is too outrageous or unrealistic.; there'll be plenty of time for filtering later. For today, you want to include any and all possibilities, because each idea (however outlandish) could lead one of the participants to a breakthrough. I can't stress this enough: every idea gets written on the big board of possible causes.

As the team members suggest possible causes, the you'll place each suggestion into the correct category on the diagram. Diagram, you ask? What diagram? Well, you have a few choices:

• The Fishbone (aka Ishikawa, aka cause-and-effect) diagram
• The Interrelationship diagram
• The Current reality tree

You can do some reading on the benefits and challenges of each diagram in this article by Anthony Mark Doggett. However, my experience has shown me that the tools used in RCA are secondary; what's important is doing the RCA. And the best tool is the one that works.

Tomorrow: a primer on how to run effective meetings.

Building the framework

Need to catch up? You can read all the root cause analysis entries here.

The second step in conducting root cause analysis is:

Establish categories of potential causes.

This step provides you with a framework for the brainstorming that we'll talk about tomorrow, and it does make the process flow more smoothly. Without the kind of "jump-start" you get from establishing categories up front, you might end up with a conference room full of people, each staring at the other, with no idea where to start.

Some likely categories include:

• People: Human failures will fall into this category; so too would causes related to the number of people involved in a given procedure.
• Processes: Are there processes that are themselves flawed, and which may have caused the error? Those would be listed under this category.
• Standard Operating Procedures: If your business uses SOPs, any gaps or errors in the SOPs would be listed under this category.
• Equipment: This category will include any physical failures on the equipment used in the procedure where the failure occurred. This can refer to powered industrial equipment, powered assembly lines, or computer hardware.
• Environment: Any possible causes related to the layout of the facility, the temperature - or the cultural environment - will be listed in this category.
• Systems: Use this category to list systems causes such as software bugs or program failures.

Your particular business or the particular error might lend itself to more categories, fewer categories, or different categories. You don't need to use my list as the be-all and end-all of RCA categorization; my list is merely a jumping-off point you can use to get started.

Tomorrow: The heavy lifting begins when we start identifying specific potential causes.

What's your problem?

Need to catch up?

Part I
Part II
Part III

Today we take the first step in conducting RCA; but before we get to that, we need to attend to another issue. A crisis has occurred in your business (alternatively, there's a non-critical but significant error thats happened several times), so you decide to assemble a team to conduct RCA. Who's on the team?

The first team members should be the people who were directly involved in the crisis: the people who made the error and/or the people who identified the error. These folks are key to your success, because they're the ones who will help you identify what was going on before the crisis blew up; they're also your best bet for identifying the top secret factors that may have led to the crisis.

Your next team members should be business analysts or other subject-matter experts (SMEs) who have in-depth knowledge of the part of the business where the failure occurred.

Your remaining team members should be bright people from various departments. Don't limit the RCA team to employees from the department where the failure happened, as such a team will be limited in terms of perspective. Use RCA as an opportunity for your best and brightest to learn about a part of the business they don't ordinarily see. This gives your employees broader knowledge of your business, and also gives your RCA team the benefit of outsider perspectives. Try to keep the team to a manageable size - I typically use RCA teams of 6 people, give or take a few. If the group is too large, you may find that the process gets bogged down in too much discussion. This is definitely not a hard-and-fast rule, though; decide what size team works best for you.

You may wonder whether senior management should be involved in RCA. I tend toward leaving senior management out of RCA, for two reasons:

• The role of senior management is to plan strategy and direction; middle management's job is to manage the day-to-day.
• Line-level employees may be intimidated (by the presence of a director or VP in the conference room) and therefore less likely to speak freely.

You've assembled your team, and you're finally ready to begin with the first step of conducting RCA:

Agree on what the problem is.

Piece of cake, you say. We already know what the problem is - that's why we're here! Oh, if only it were that simple. I'd suggest a brief overview of the crisis or error that brought the group together, and then start working to define precisely the problem that the group is going to analyze. Do not assume that everyone is on the same page, and don't assume that the problem is sufficiently self-evident that discussion is unnecessary. It may be self-evident to you, and perhaps to your SMEs; but the folks from other areas of the business may need clarification.

The entire team must operate with a single definition of the problem. Why is this so critical? Well, if everyone doesn't have the same understanding of the problem, then you might waste a lot of time and effort analyzing a problem that isn't the problem at all. I once facilitated a session during which we got two hours into a fishbone diagram before we realized that the problem we were analyzing was the wrong problem. To say that the group was frustrated would be a gross understatement, and you can bet I didn't make that mistake again.

Take the time to discuss the crisis or error until everyone is in agreement and understands exactly what the problem is. The more time you spend defining the problem, the more smoothly the rest of the analysis will flow.

Tomorrow: step 2 of conducting RCA.

The Domino Theory

If you'd like to catch up:

Part I
Part II

When I was a little girl, I enjoyed playing with dominoes. I never wanted to play the actual game; no, I wanted to stand the dominoes upright in elaborate patterns so that I could knock 'em down in a line. I found this domino effect to be most entertaining, despite the fact that I really wasn't very successful in my attempts to create the most complex, beautiful domino pattern ever.

Given my appreciation for dominoes, it's not surprising that I discovered Bob Nelms early in my study of RCA. Bob uses the Domino Theory of root cause to explain how a failure or error occurs. The Domino Theory states that every failure is in fact a series of failures that play upon each other: a physical cause that's triggered by a human cause, which is in turn fed by a latent cause.

However, I'm not content using Bob's theory as is; when I develop training, I do whatever I can to develop mnemonic devices to facilitate data transfer from short-term memory to long-term memory. For example: remember the characteristics of a root cause that I defined in Part I? A root cause:

• can be fixed.
• explains the failure or event.
• can be identified with a reasonable amount of effort.
• is controlled by management.
• is specific.

Well, I've turned those criteria into the acronym "FERCS."  FERCS is a great way to remember the characteristics of root cause; for one thing, it's not an actual word, but it sounds vaguely as though it might be an obscenity. This makes it good for a laugh in training - and what people find funny, they remember.

Likewise, I've tweaked the Domino effect that Nelms describes; I call it the PHaTS Domino Theory (I'll give you three guesses what song I play during training workshops when we get to the PHaTS Domino Theory). In essence, the PHaTS Domino Theory renames one component - purely for the purposes of designing the acronym, you understand:

• Physical cause: this is the physical failure that led to the event. For example, let's say that the bad event is a car wreck. The physical cause might be that your brakes failed. It's important to know that this particular domino may not be present in every business problem per se. For example, with the bug in the order entry software I described yesterday, there's no true physical failure, although I would consider the software failure to be akin to the physical cause.
• Human cause: Nelms states that every physical failure is triggered by a human failure or cause. In the example of the software, the human cause might be that the programmer failed to account for international shipments. In the case of the brake failure, there are several possible human causes: perhaps the the technician who last worked on your brakes did a shoddy job; perhaps the brakes have been squealing for the last six months and you've put off getting the brakes done; perhaps the brakes were defective in the first place, but the quality control personnel at the manufacturer didn't catch the defect.

and..

• Top Secret cause: This is the latent cause that Nelms describes. At its most basic, Top Secret cause represents the 400-lb. gorilla that everyone pretends isn't there. Top Secret causes are those unspoken rules or policies that everyone follows even though those rules are not publicly defined. They are the assumptions and beliefs that drive business-as-usual for your business.

In RCA for business problems, it's usually not too difficult to isolate the physical and human causes; the top secret cause(s), however, are a different story. In order to identify the top secret cause, you need the cooperation of the people involved in the problem/failure, because you have to find out what they were thinking (remember, this is about learning rather than blaming, and there's an enormous difference between, "Why did you decide to do it that way?" and "Why on earth did you do that?). Management, too, may be somewhat wary of the notion of top secret cause, because when you identify those assumptions, beliefs and unspoken rules, you turn a big old mirror on the management team - and on the subtle (and in some cases not-so-subtle) messages that they're sending to the organization.

By the way, if you're interested in Bob Nelms (and his take on human nature as the real root cause of all failure is fascinating), I recommend his book, What You Can Learn From Things that Go Wrong.

Tomorrow: Step One of conducting RCA.

Root Cause Theory

If you need to catch up, you can read Part I here.

When you look at a tree, you can't see its roots. In order to do so, you have to dig beneath the surface. RCA is no different; in order to identify a root cause, you'll need to dig a bit. But before you can do that, you have to know what you're trying to accomplish. Today, I want to focus on what RCA is and is not. It's important to understand RCA - its intent and goals - if you're going to benefit from the work.

What Root Cause Analysis Is

The first goal of RCA is learning. When you analyze root cause, your goal is to look beyond the immediate problem and find the situation that's causing the problem. Think of it like the practice of medicine: in RCA, business problems are akin to the symptoms of an illness; they're merely signs of a deeper problem.

The second - but equally important - goal of RCA is eliminating the situation(s) that caused the problem(s) to occur. When you identify, treat and cure the underlying disorder (the root cause), then you cure the symptoms as well. Be sure to keep these goals in mind and in the proper sequence; you can't solve the deeper problem until you find it, so your first goal needs to be finding the disorder that's creating the symptoms.

What Root Cause Analysis Is Not

It can be easy to confuse RCA with problem-solving; after all, at the conclusion of RCA, you do develop CAPA(s). However, problem-solving and RCA - although they share some common threads - are not the same. Problem-solving is a sprint, whereas RCA is more like a marathon. If your order management software fails to generate a shipping label for an order that has to go out today, problem-solving methodology would lead your shipping department to create a label manually. While this solves your immediate problem, it doesn't tell you why the failure occurred (which means it could happen again); only RCA will do that. This doesn't mean that problem-solving isn't important; you need both weapons in your management arsenal.

Fault-finding, finger-pointing, and assigning blame are not among the goals of RCA, although some people do tend to associate RCA with negative consequences. You need to fight this perception; RCA cannot be successful if the people who made the error won't to speak up about what happened (because they're afraid of being punished for making the mistake). RCA is not about being punitive, and until everyone really believes that and acts on that philosophy, you're going to have a hard time getting the full story. True, sometimes managers will have to initiate disciplinary action at the conclusion of RCA; however, you can't know that until you've completed the process - and you can't complete the process if folks are afraid to 'fess up. Furthermore, I'd suggest that if many of your RCAs lead to disciplinary action, then you're probably missing something in your RCA (based both on my experience and Deming's theory that the majority of failures are caused by the business system itself rather than by individual employees).

Tomorrow: the three types of root cause, and how they work together to create problems in the business.