Quieting noise: An opportunity to support great accreditation decisions

Lois Margaret Nora, MD, JD, MBA

I recently finished the book Noise, by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein.  While I generally agree with reviewer Caroline Criado Perez that this book was longer than necessary, the title of her piece, The Price of Poor Judgement, is apt. The book builds upon the authors’ prior work on improving decisions and sheds light on how a better understanding of what they call “noise” can help us evaluate and improve upon our decision-making.

The authors of Noise describe a body of evidence illustrating that circumstances having nothing to do with the choice at hand — from weather to the time of day to the performance of the local sports team — can influence the judgements we make.  The result can be substantial inconsistency among decisions involving similar or even nearly identical scenarios.  The authors cite evidence from criminal sentencing, hiring, foreign policy, business planning, and medicine, making the case that noise is far more pervasive than most of us realize, and that people and organizations should strive to manage noise to improve decision-making of all types.  While the book doesn’t discuss accreditation decisions, it is worth thinking about how noise and efforts to mitigate it might influence these processes.  

Many programs and institutions — including universities, hospitals, certifying organizations, and professional education programs (medicine, law, allied health) participate in accreditation programs.   Accreditation status is important to eligibility for federal funding, national rankings, attracting students, and maintaining public trust.  The stakes associated with these decisions are high, and the resources necessary to achieve and maintain accredited status are substantial.  While accreditation organizations make substantial efforts to ensure that their judgements are consistent and fair, specific attention to the issue of noise may be helpful.

Understanding noise

It’s reasonable to expect that similar circumstances will result in similar decisions.  However, that is not always the case – in fact, it is often not the case.  Using target-shooting as a metaphor, the authors note that multiple hits at the center of a target would be representative of consistent decisions involving minimal error.  Bias, which the authors distinguish from noise, would be represented by multiple shots that miss the center of the target, landing instead in a cluster to one side.  And noise is represented by a scattershot pattern: The shots are distributed in a random fashion across the target.  You will find a helpful visual in this HBR piece.

In practice, the scattershot scenario represents myriad conclusions, all different from one another for no obvious reason.  If the facts in a particular situation – criminal sentencing or hiring or accreditation – are similar, the decisions should generally also be similar. And yet, the authors present compelling evidence that such inconsistency in decision-making can occur at the individual level (when a person’s conclusions vary across multiple similar scenarios) and in group decisions (concerning a single case or multiple cases over time).  The variables associated with this inconsistency are unrelated to the circumstances of the decisions themselves. Examples include hunger, time of day, mood, and weather on the individual level and group dynamics and communication norms at the team level.   

To combat these influences, the authors urge individuals and organizations to prioritize what they call decision hygiene – a collection of tools and tactics intended to mitigate the effects of noise and otherwise improve decision-making. It is important to note that decision hygiene does not mean removing discretion from decision-making. In medicine, for example, even if two people present with similar symptoms and medical history, there may be important differences between their circumstances and preferences that are worth considering, and it may be appropriate for a healthcare professional to recommend different treatment plans to them as a result. 

Rather, the authors note, many of our heuristics or patterns of thinking — including relying on gut feelings, substituting easier questions for harder ones, halo effects, prejudgment, confirmation bias, and overconfidence — can contribute to noise.  Fortunately, it’s possible to mitigate these influences and improve our decision-making.

Assessing and minimizing noise in the context of accreditation

Accreditation decisions are complex.  Accreditors with organizations such as the Liaison Committee on Medical Education, the Southern Association of Colleges and Schools, the Joint Commission, and the National Commission for Certifying Agencies must assess compliance against many standards and elements. In many cases, those decisions draw on evidence from thousands of pages of documentation as well as site visits that can involve a week’s worth of meetings. Many perspectives are involved, including those of site visitors, staff, and committee decision-makers.

Recognizing the complexity of this work, accrediting bodies place great emphasis on bringing validated and trustworthy processes to their decisions. Throughout my own career, I have seen firsthand the efforts made by accreditation committees and staff members to ensure a consistent, fair process with integrity. Everyone involved, particularly accreditors themselves, wants these decisions to be the best they can be. Learning about and taking steps (or augmenting existing tactics) to mitigate risk of noise may be a worthwhile part of this work.

Building from the authors’ recommendations, here are some thoughts about how people involved with accreditation can work to mitigate the effects of noise in their own decision-making as well as when working as part of a team. It may be useful to think about these efforts in two buckets: prior to decisions (general efforts to understand and manage noise) and then during decision-making (procedural adjustments that may help).   

Prior to decision-making:

1.       Introduce the concept of noise. Because accreditation site visit teams and committees bring together people with diverse experiences and viewpoints, a discussion about best practices for decision-making is reasonable and expected. Orientations for individuals and groups working in accreditation provide an excellent opportunity to introduce the concept of noise. Simply learning about noise is powerful, and individuals who are exposed to the concept may proactively modify their own approach to decision-making in an effort to reduce the effect of noise.

2.       Make reducing noise and improving decision-making themes for professional development.  Consider gathering teams ahead of a site visit or program review for dedicated professional development on these topics. Scenario reviews can provide opportunities for discussion of how individuals arrive at their decisions and how groups can more effectively make decisions.  Teams can discuss how to promote expression of a variety of perspectives and explore ways of reducing noise.  Guidelines, structuring complex judgements, and deliberately incorporating counter-arguments are techniques that may warrant particular attention.  

3.       Conduct a noise audit.  Although it can be challenging to systematically examine the quality of judgments made by a group of people, noise audits involving hypothetical scenarios generate valuable information.  The process can help quantify whether noise is in fact affecting decision-making and to what extent, providing a data foundation for making improvements.  Although noise audits often involve external consultants, the expertise within accrediting organizations may mean they have resources that could allow for a thoughtful audit by an internal team.

During decision-making:

1.       Meet the group’s human needs.  Reducing noise in group decision-making begins with planning the meeting. Considering of time of day, scheduling regular breaks, and providing drinks and snacks that include protein will go a long way toward ensuring everyone is at their best. This kind of preparation ensures participants have what they need to act on their own awareness of noise — those who understand that hunger or fatigue, for example, may influence their decisions know they simply need to get a snack or take a break to improve their work.

2.       Ensure all perspectives are heard.  Use words and actions to demonstrate that alternate opinions will be welcome and listened to. Polling apps that allow people to register their opinions prior to discussion may also help ensure more perspectives are considered more fully before conclusions are drawn.  One tool we have used with success is Poll Everywhere, though there are many to choose from.

3.       Leverage decision hygiene practices. Using decision guidelines can reduce inconsistency among decision-makers by providing clarity around what constitutes acceptable performance and how to assess and weight various measures of compliance. Dividing complex decisions into component parts can reduce the likelihood that performance in one area influences perception of performance in other areas. And rotating the order in which people speak can help ensure all participants have an opportunity to influence the discussion.

4.       Actively monitor for noise. Consider tasking an independent person with watching for and speaking up as they see things that could be noise, and be sure to engage others in welcoming and legitimizing the feedback.  Questions like those listed in the bias observation checklist included in Noise may also help.

Supporting quality in all we do

Decisions are an important part of life, and we all have a stake in ensuring they are the best they can be. Practices to mitigate noise hold promise for enhancing accuracy and consistency in our own decision-making and in decisions made by groups of all types, including those involved with accreditation.