Safety System Audit versus Safety Perception Surveys – Which One to Choose?
By: Dennis Ryan
Most safety professionals are aware of the importance of evaluating health and safety management systems. Evaluations help to assess the state of the management system and the subsequent findings and recommendations provide direction on how to improve. In this article we will examine two prominent methods of health and safety measurement: Safety System Audit versus Safety Perception Surveys.
Both are commonly used measurement techniques. Both have inherent strengths and limitations. It is important to know what they are in order to make sound measurement decisions. Some key strengths and limitation are discussed below.
What are the key differences between the two measurement approaches?
To answer this question we have to first go back almost a hundred years. In the early 1900s with the inception of workers’ compensation insurance, the idea of workplace health and safety really started to take off in North America. A short time later, safety leaders conceived some key elements that they felt were crucial to improving health and safety. They then established health and safety programs. The elements proposed (e.g. inspection, investigation and hazard identification.) were embraced and to this day they form the basis for many of our basic safety programs and audit instruments or protocols. System health and safety audits measure or assess these key elements.
Initially, it is believed this basic approach to health and safety helped reduced workplace fatality rates, but current national fatality rates indicate more work needs to be done. The safety profession continues to research what drives safety. Current research suggests it is the presence of a strong positive culture, not just the adherence to basic safety elements that influences employees’ behaviour and therefore their safety. The research goes on to say that safety programs cannot be implemented or sustained in an unsupportive corporate environment or culture. Many health and safety professionals’ have tried to implement basic safety elements against an unreceptive culture and have failed. Safety professionals who accept this new research first focus their safety improvement efforts on the culture of the organization. One of the best tools for measuring an organizations’ health and safety culture is a safety perception survey.
Surveys deal with employee perceptions and audits deal with facts.
Safety perception surveys solicit employee perceptions of the reality of the workplace’s health and safety culture. Some cynics suggest these perceptions are not as important as hard facts, as they can be inaccurate. To the employee however, these perceptions are reality. It is this reality that influences a person’s behaviour regardless of how accurate the perceptions actually are. For example, if employees believe senior management does not demonstrate a strong commitment to health and safety, that belief will influence them to focus less on safety and more on other aspects they see as more important such as meeting production quotas. Employee perceptions are important and should never be dismissed. Fortunately, once perceptions are known, they can be influenced and changed.
Some surveys do not solicit employee comments on the questions and only quantify the responses with a score. The scores do not provide rationale for why employees responded the way they did. This generally results in an expensive process of setting up focus groups to determine the rationale for the employees’ scores. Surveys that encourage respondents to comment as well as score questions provide some rationale for the score.
Audits also solicit employee perceptions through the interview process. The key difference in the two measurement techniques is that auditors filter or interpret employee perceptions and summarize them in their audit report. Which is better? It depends on whether you trust your auditor to accurately interpret and summarize the perceptions of your employees or if you would rather receive them directly in unabridged form.
Which form of measurement is more accurate?
Most safety perception surveys present a statement or question and offer a range of answer choices such as:
(1) – Strongly disagree
(2) – Disagree
(3) – Neutral – neither agree or disagree
(4) – Agree
(5) – Strongly agree
Employee survey scores, when combined with comments, provide the surveyor with a pretty good indication of the perceived strengths and opportunities for improvement within a company’s system.
Alternatively, system audits commonly employ two distinct types of scoring methods (often both in the same audit protocol). One type of scoring is called “range scoring” (e.g. 0 to 10 points). This method allows the auditor to award a point value within a range based on the assessed percent positive indicators. Therefore, if the percent positive indicators from interviews are 40% positive, the number of points awarded would be 4 points. This provides a level of accuracy that is equal to or better than the 1 to 5 Likert scale used on many surveys.
The other method used for scoring in auditing is often referred to as “all or none scoring” (e.g. the score is 0 or it is 10). The auditor must award either 0 or 10 points for the question based on the percent positive indicators and against a standard set by the audit protocol owner. For example, the protocol owner may establish that full points are to be awarded if the positive indicators are greater than 50% positive (i.e. 51% positive and greater). This is where the accuracy of the audit approach diminishes significantly. Let’s look at an example. Let’s say the auditor asks an interviewee the following question: “Does the company follow up on issues brought up at safety meetings?” The choice of answers is either “yes” or “no”. There is no sitting on the fence. If the interviewee feels the positive indicators are greater than 50% he or she must respond positively even if they feel follow-up is still somewhat lacking. The auditor is also obliged to register the interview response as positive. If every employee responded in a like manner, the collective response would be registered as 100% positive for full points. The reality may very well be that almost 50% of the employees interviewed feel there is a need to improve on follow-up yet full points are awarded. This method of scoring significantly reduces the accuracy of the approach and leads many employers into believing their system is better than it really is.
The same all-or-none approach to scoring is also offered by some survey consultants and therefore the same scoring issues may apply. From an accuracy perspective it is important for safety professionals to assess the scoring methods employed.
Reproducibility of results is another issue related to accuracy. This is primarily an audit issue as the survey consultant does not make any scoring decisions. A study conducted by the Institute for Work and Health suggests that if two auditors separately audited the same company at the same time, the results could differ significantly. How is it possible for there to be such inter-audit inconsistency? Here is how this happens. Let us compare a knowledgeable and experienced auditor to another with less knowledge and experience. When asked to assess an employer’s contractor safety management system or process, the experienced auditor, knowing what constitutes a complete process, may score it incomplete. The less knowledgeable auditor may unknowingly score the process as complete. To properly score a question with these descriptors, auditors have to first know what a good contractor safety management system should look like. Unfortunately, auditor training almost never includes this level of detail and clear guidance in audit protocols is generally lacking.
Employers can help improve the audit process accuracy by selecting an audit protocol that does not support all or none scoring and by ensuring the protocol auditors are provided with detailed interpretation guides to the audit questions. As auditors can be certified after only minimal training, comprehensive reference material should be provided to help them make an accurate assessment.
Which method is more valid?
In the aforementioned study conducted by the Institute for Work and Health, it was revealed that many of the audit protocols that were selected for the study had not undergone any validity and reliability testing. This study is likely as valid today as it was nine years ago when it was first carried out.
The elements on which the audit questions are based were originally conceived by safety professionals many years ago. There was no evidence then, or now, to confirm that all of the basic elements identified actually lead to safety excellence. Consider two commonly used audit questions:
Is the corporate safety policy posted in prominent locations?
Does management tour worksites at least two times annually?
First, we have to assume that posting the policy is an important. Some companies don’t think it is. They choose to expose their employees to the policy in other ways such as have senior management talk to employees at their new employee orientation. Similarly, management values could be displayed in other ways than touring worksites such as by management attending safety meetings or speaking at quarterly town hall meetings.
If the audit question specifies that the company health and safety policy must be posted, points cannot be awarded for any alternative methods of communicating the policy. Companies depending on the audit to guide them to improvement must be certain that the protocol offers the flexibility they need in order to maintain a health and safety program that is company specific and that works for them.
Surveys can also have issues with the validity of questions. If a similar study were conducted on safety perceptions surveys it would likely reveal similar results. Some survey consultants claim to have developed definitive statistically validated surveys. They imply they have developed a set of validated questions that will lead all companies down a path to safety excellence. But, if you were to ask for the evidence proving that these questions are the right ones to ask, it is unlikely any would be able to produce the evidence. Formulas cannot validate that a question is the right one to ask every company.
The health and safety profession has not yet established a credible baseline safety perception survey. There is no widespread agreement yet on what elements and questions a baseline survey should contain. There is clearly a considerable amount of work to do to improve the validity of both methods of measurement.
Safety System Audit versus Safety Perception Survey – Which method is most cost effective?
The costs associated with conducting safety perception surveys are the main reason why more companies have not conducted them. Safety perception surveys are most frequently administered by consultants. These consultants may charge very high fees as there are few consultants offering the services. There are currently no certification requirements to consult in this area of measurement.
Alternatively, auditors are abundant. In order to obtain a certification to become an auditor one typically has only to complete a two to five day auditing course (there may or may not be other pre-requisites such as conducting a qualifying audit). This certified auditor is now free to hang up their auditor’s shingle. However, as previously discussed, basic auditor training may not cover all of the necessary aspects of a successful workplace health and safety program or how to properly assess a program. If ever there was a case for being diligent it is in the selection of a competent survey consultants and auditors.
Which process is most cost effective? First it is important to know what the safety perception survey or audit is actually measuring. If you believe the audit protocol is needed to improve your program and if the auditor meets your scrutiny relative to competence, the audit may be your best selection. If you believe the survey is asking your employees the right questions, and the survey consultant is competent, the survey may be the best selection. Be prepared to pay a great deal for a survey consultant to administer your survey unless you use a do-it-yourself approach. A do-it-yourself survey option cuts your assessment costs down to less than half of what it would cost to hire a professional auditor or survey consultant.
Summary
For some, this article may have posed more questions than it answered. The writer never promised to provide all of the answers to all of the questions but did promise to heighten awareness of two of the most prominent health and safety system measurement tools available and in use today. The limitations for the auditing method of measurement is lengthy compared to surveying but that may be because the audit approach has been in use for a much longer period of time.
We do know with certainty that many audit protocols have not proven the questions asked are really indicative of a successful health and safety program. We know higher standards need to be established to ensure auditors are competent. We know that comprehensive scoring guidelines are needed to ensure auditors can accurately score the audit questions. Scoring methods need to be reviewed. Companies suffering from audit fatigue need to be offered alternative measures that will guide them to improve in other areas.
Surveys on the other hand, are a relatively new and essentially a little used method of measurement. The profession has yet to establish clear standards or guidelines on survey application or content. Perhaps it is too early to properly evaluate this measurement method as it is evolving rapidly. We do know that surveys tend to focus their evaluation on the cultural aspect of an organization. We know that a strong positive culture is a necessary prerequisite to a successful health and safety program. We also know that safety perception surveys are currently used most often by companies that already have advanced safety programs. Do these companies know something the rest of us do not?
With all of the issues identified affecting the usefulness of audits and surveys should we continue to use them? The purpose of this article is not to discourage the use of the instruments. Use them but recognize their limitations. Minimize those limitations by making wise selections relative to the assessment tool and person(s) conducting the assessment. It is hoped that this improved scrutiny will foster more research into the methods and force owners of the measurement processes to work harder to improve them.
References
-Petersen, D. (2000, Jan.). Safety management 2000: Our
strengths and weaknesses. Professional Safety, 45(1), 16-19.
-Bigelow P, Robson l, (2005) Occupational Health and Safety Management Audit Instruments: A Literature Review
Institute for Work & Health
-Ryan, D. (2008, Fall). Is Your Company Suffering From Audit Fatigue? CSSE Contact.
-Ryan, D. (2014, Spring). The Evolution of Safety and the Importance of Human Factors. CSSE Contact.
-Sheppard, A. (2001, Sept.). The perception survey: Measuring
the intangibles in workplace health and safety. Occupational Health and Safety.