YES, THERE IS A CURE FOR AUDIT PARALYSIS
As you read this, thousands of organizations around the world are busy scurrying about collecting and updating health and safety policies and procedures, updating employee training and training records, reviewing job safety analysis or hazard assessment documents, and just generally getting their company prepared for what has become the most common health and safety management system assessment leading indicator. It’s called the “Safety Management System or Program Audit”. It goes by many other names such as the EHS Audit, Program Assessment, etc. but the processes for all of them are similar. A trained auditor or audit team, using a standard audit tool or audit protocol, assesses the health and safety management system of a company by interviewing employees, reviewing documents and conducting work site observations. Each question in the audit protocol is systematically scored and, depending on the score, the result is deemed as a pass or fail. Of these thousands of companies making ready for their audit, most of them suffer from a condition called “audit paralysis“.
I am an auditor. I have been in the health and safety business for forty-three years and have spent a great deal of that time working with companies as their health and safety auditor. I am semi-retired now which has allowed me more time to reflect on ways the profession can improve this leading indicator. Although I take a critical look at the audit process in this article, I am a strong proponent of the process. The purpose of this article is not to malign the auditor or the audit process. I am acquainted with a number of auditors that I would not hesitate to recommend to any of my past or present clients and would recommend the audit process to anyone looking to improve their health and safety management system. However, there are opportunities to improve the process. It is my hope that the following information inspires more people to look at the audit process critically and work toward improving it.
This article is about an audit a condition that I call “Audit Paralysis”. Audit Paralysis is a term used to describe what many companies suffer from when they undergo a safety management system audit. Here are some key symptoms of the condition:
- “Audit Fatigue” which is another related audit condition resulting in an audit report that reveals few new meaningful incites.
- Stagnant health and safety programs
- Inaccurate auditor findings that miss or conceal key opportunities for improvement.
- High Audit Costs
All of the aforementioned symptoms can lead to audit paralysis which is a condition that stifles continuous improvement. In order for companies to continue to benefit from the audit process, these conditions need to be addressed. Here is an explanation of the conditions and some suggestions on how we can address them.
Audit fatigue is a condition many companies suffer from after auditing year after year using the same audit protocol and often using the same auditor. Typically these companies are mired in a program that offers rewards or recognition for carrying out a standard annual audit. Every year it is the same questions and every year the audit reports read the same. As companies audit year in and year out, the useful information they obtain from the approach diminishes with each passing year. (This is assuming the company acts on the findings of previous audits). After awhile, the audit results become very predictable. Audit fatigue sets in and the audit process becomes a poor investment as it no longer yields a good return.
What is the cure for audit fatigue? The cure for audit fatigue is to offer more, or perhaps more challenging, measurement alternatives. One possible alternative is to offer safety perception surveys as a measurement option. Like audit programs, if this measurement alternative is to be successful, it needs to be properly supported. First the approach needs to be affordable. Many companies simply cannot afford to hire expensive survey consultants. In this case, a do-it yourself option may be considered. Second, Governments and program administrators need to set clear survey program standards. Surveys containing the specific survey questions or statements need to be established. Training programs have to be made available to help companies administer their surveys. Most important, is that a database has to be made available so that employees can complete the survey, preferably on line, and survey data can be properly collected and managed. The key to the success of any new measurement offering is to provide sufficient support so that it has a chance to succeed.
Stagnant Programs (Audit Paralysis) – Raising the Bar
Many companies are being audited against weak audit protocols with content derived from basic health and safety elements advocated by W. H. Heinrich 86 years ago. In his book (i.e. Industrial Accident Prevention: A Scientific Approach) Heinrich proposed that key elements such as hazard identification and assessment, inspection, safety meetings, etc. are requirements for safety success. Contemporary research does not disagree with Heinrich’s elements but it also reveals that there is much more to a well functioning health and safety management system than the original proposed elements. The research indicates we need to also assess our programs against other key safety determinants such as management credibility, trust, worker autonomy, work life balance, fairness, satisfaction, etc. There is now general agreement among health and safety professionals that many of these cultural, social psychological program factors are a pre-requisite to successfully implementing the original basic safety program elements.
What is the cure? Raise the bar. Users of these audit protocols and those responsible for administering them need to stop putting band aides on the outdated audit protocols and take them off of life support. Knowledgeable health and safety professionals with experience in managing health and safety programs need to be tasked with developing new assessment tools that reflect current research. Audit protocols should challenge companies to demonstrate improvement in health and safety against a strong standard. Too often the bar is set low allowing companies to develop cut and paste programs that meet current audit requirements but do deliver on safety.
Another possible solution is to tier the certification process. Companies whose programs are in early evolutionary stages such as the “Reactive or Traditional Stage” can be measured against a lower audit standard than companies at or near a “World Class Stage”. Certification rewards could also be tiered appropriately to recognize the program management effort.
Inaccurate/Imprecise Audit Scoring
I instructed in the Occupational Health and Safety Certification program at the University of Alberta for many years. The students were either health or safety professionals seeking credentials or were students hoping to complete the program and find work in the field of health and safety. As an auditing instructor, I frequently tested the groups of students in audit workshops. Some students, participating in the audit scoring workshop, indicated they had substantial experience auditing. The class was broken up into groups of four or five students. Attempts were made to separate the audit experienced students equally among the work groups. Each group was provided with the same workshop mock document, observation and interview information. From this information the groups were to score audit questions. Once completed, scores of the groups were examined question by question and were posted in a table on a white board similar to that shown in Figure #1. Many of the scores for the groups were consistent. For example, if the question asked whether or not there was a particular policy or procedure in place, group scores were generally pretty close. These types of audit questions are very clear cut and easy to score. There were significant differences noted however, on how these groups scored other types of questions which always lead to a great deal of discussion and debate in class. Scoring differences were most frequently noted on questions that required the groups to assess a process or a system For example, “Did the investigations identify root cause(s)?” Following discussion, it was determined the groups had significant differences in their understandings of root cause(s).
What is the cure for differences in audit scoring?
- Some protocol administrators do a very poor job of specifying in their scoring instructions what auditors need to see in order to award points. The cure is simple in this case; improve the audit protocol instructions.
- To become an auditor, some audit protocol administrators require perspective auditors to have secondary education, a number of years experience managing a health and safety program, five days of training on the audit protocol and undergo mentorship until the new auditor is determined competent. Others may require no experience and only two or three days of auditor training. It should be no surprise that these significant differences in certification requirements result in differences in auditor competency. There needs to be improved consistency in certifying competent auditors. Auditor competency is a factor that accounts for significant differences in audit scoring. If the audit process is to maintain credibility as a leading measurement indicator, the auditor competency bar has to be raised.
Another factor resulting in scoring inaccuracies is in how audit questions are scored. Typically audit protocols utilize a combination of two types of scoring, All or None and Range Scoring. With all-or-none scoring, the auditor basically has only two scoring choices (besides undecided). There is a NO response or a YES response. The audit protocol instructions generally outline at what percent positive point and auditor is to select the YES or NO score. For example, let’s assume that auditors are to assign a YES to any percent positive response above 50% positive. In an interview the auditor must determine to what extent the interviewees response is positive or negative. If the interviewee is unsure the auditor must probe deeper to determine if the response is 51% positive or greater for a YES (i.e.100% positive score) or less for a NO (i.e. 0% positive score). Assuming the interviewees response is just over 50% positive, the question is scored as 100% positive or YES. This method of scoring is very imprecise and the confidence in the score decreases with each subsequent interview. Figure #2 depicts the two scoring options of All-or-None scoring.
Figure #2 – All or None Scoring
For each All-or-None interview score, one is left wondering to what extent employee responses were actually 0% or 100% positive and to what extent they were somewhere in between. Let’s examine the scoring more closely using the auditor’s percent positive findings for each of the ten interviewee response shown in Figure #3. In the middle column employee responses are shown in terms of percent positive. The score necessitated by the audit protocol is shown in the Auditor Score column as the score must be either YES or NO.
Figure #3 – All or None Scoring
Once all of the interviews are completed, the auditor adds up the YES scores and the NO scores and comes to a conclusion on scoring the All or NONE points for that question. In this case, because eight of ten interviews score positively, ten out of ten, or full points would be awarded for the question. The issue with this scoring is that full points are awarded based on 68% positive responses. In fact, full points would have been awarded had the percent positive indicators been 51%. Some audit protocols attempt to mitigate this issue by instructing auditors to not award full points unless the positive indicators are higher such as 60% or 70% positive. This doesn’t solve the scoring issue but does improve it. If you are expecting your audit results to accurately reflect the state of your health and safety management system, you need to be aware of the number of All or None scoring questions in the audit protocol and the percent positive indicators required to obtain full points. Incredibly, there are significant consequences to companies that fail an audit as they may lose their ability to bid on projects because they cannot meet audit certification requirements.
The cure for this scoring problem may be to use range scoring. Range scoring is a better scoring indicator as it asks the interviewee (not the auditor) to assess the question based on a sliding negative to positive scale such as a 1- 5 scale where 1=0%, 2=25%, 3=50%, 4=75%, 5=100%. Figure #3 depicts the possible scoring options of range scoring. This type of scoring at least provides companies with a better estimated score of each audit question and directly/accurately, reflects the interviewees response to the question.
Figure #4 – Range Scoring
High Audit Cost
Throughout the world, there are programs administered by Governments and safety associations that certify companies that have passed a basic audit protocol. Certification often promises incentives for companies to participate such as lower compensation insurance premiums or the ability to meet the pre-requisites needed to bid on contracts. The ultimate goal of these programs is to see one hundred percent of the company membership participate in the program. Unfortunately, this goal may not be realistic given the high upfront costs associated with auditing. For some companies the upfront costs needed to participate in annual audits are just too high as some of them are struggling to make ends meet. If the audit process were less costly, more companies would be able to participate.
There is a cure for high audit costs? One way to cut audit costs is to reduce the high cost associated with gathering audit information. The audit process requires certified auditors to review documents, conduct observations and employee interviews. Generally, the highest cost in the audit process is in conducting employee interviews. Each interview is typically 30 minutes to 1 hour in duration. If there are 50 interviews to conduct, it may take the auditor four or five days to conduct them. Time needed to conduct interviews may represent 75% of the auditors on site time and 50% of the total time needed to complete all audit requirements including writing the report. Some audit administrators have attempted to make the process more affordable by allowing auditors to collect interview information using a limited number of worker (excluding supervisor or management) surveys. However, without a database to collect and manage all of the survey interview response data, little time is saved because auditors still need to take time and charge the client to review each of the completed respondent surveys and then consolidates all of the findings. In this case, the survey option offers little real cost savings as the option is not supported by a database capable of managing interview data.
What is the cure? The cure is to provide auditors with a proper web based survey database. The database should be capable of collecting employee audit question scores and comments to each question and also have the ability to manage and sort through all of the data (e.g. by position, location, department, etc.). If it were made possible to have all workers complete surveys electronically, auditor costs could be cut in half as a good web-based database would allow employees to complete the audit surveys anytime and anywhere and it would then consolidate all of the respondent data. This approach to soliciting employee perceptions is cost effective and it removes the auditor who could unintentionally bias responses. The reduced audit costs would be appreciated by all certificate participating companies and it would open the door for many more employers to participate in the audit process.
Audit Paralysis is a condition that is paralyzing many companies and preventing them from improving their health and safety programs. Their programs are stagnant because the bar has been set too low by an unchallenging and outdated assessment tool. A tiered participation approach would allow companies to achieve certification at an appropriate stage of program development. It is ironic that the measurement approach designed to help improve health and safety, is in many cases actually stifling safety improvement. Participants in recognition/certification programs are suffering from audit fatigue. Alternative measurement options are either not offered or not used. Introduction of alternate approaches need to be made available and properly supported. There are many competent auditors available to perform assessments but companies need to shop carefully to find them as the current system also fosters the certification of auditors that do not have proper experience and training. More consistency is needed in certifying auditors.
All of the issues presented here are readily resolved. Employers and safety professionals need to demand improved and alternative measurement systems. The systems need to be credible, flexible and affordable. They need to challenge all employers to continue to improve and not paralyze them by setting the bar too low. Employers need to demand a good return on their program assessment requirements just as they would require on other aspects of their business. Current research is telling us that there is much more to health and safety success than the basic elements advocated by W. H. Heinrich 86 years ago. It is time to act upon current research and provide participating companies with measurement approaches that challenge them and yield opportunities for continuous improvement.