Quieting noise: An opportunity to support great accreditation decisions

Lois Margaret Nora, MD, JD, MBA

I recently finished the book Noise, by Daniel Kahneman, Olivier Sibony, and Cass R. Sunstein.  While I generally agree with reviewer Caroline Criado Perez that this book was longer than necessary, the title of her piece, The Price of Poor Judgement, is apt. The book builds upon the authors’ prior work on improving decisions and sheds light on how a better understanding of what they call “noise” can help us evaluate and improve upon our decision-making.

The authors of Noise describe a body of evidence illustrating that circumstances having nothing to do with the choice at hand — from weather to the time of day to the performance of the local sports team — can influence the judgements we make.  The result can be substantial inconsistency among decisions involving similar or even nearly identical scenarios.  The authors cite evidence from criminal sentencing, hiring, foreign policy, business planning, and medicine, making the case that noise is far more pervasive than most of us realize, and that people and organizations should strive to manage noise to improve decision-making of all types.  While the book doesn’t discuss accreditation decisions, it is worth thinking about how noise and efforts to mitigate it might influence these processes.  

Many programs and institutions — including universities, hospitals, certifying organizations, and professional education programs (medicine, law, allied health) participate in accreditation programs.   Accreditation status is important to eligibility for federal funding, national rankings, attracting students, and maintaining public trust.  The stakes associated with these decisions are high, and the resources necessary to achieve and maintain accredited status are substantial.  While accreditation organizations make substantial efforts to ensure that their judgements are consistent and fair, specific attention to the issue of noise may be helpful.

Understanding noise

It’s reasonable to expect that similar circumstances will result in similar decisions.  However, that is not always the case – in fact, it is often not the case.  Using target-shooting as a metaphor, the authors note that multiple hits at the center of a target would be representative of consistent decisions involving minimal error.  Bias, which the authors distinguish from noise, would be represented by multiple shots that miss the center of the target, landing instead in a cluster to one side.  And noise is represented by a scattershot pattern: The shots are distributed in a random fashion across the target.  You will find a helpful visual in this HBR piece.

In practice, the scattershot scenario represents myriad conclusions, all different from one another for no obvious reason.  If the facts in a particular situation – criminal sentencing or hiring or accreditation – are similar, the decisions should generally also be similar. And yet, the authors present compelling evidence that such inconsistency in decision-making can occur at the individual level (when a person’s conclusions vary across multiple similar scenarios) and in group decisions (concerning a single case or multiple cases over time).  The variables associated with this inconsistency are unrelated to the circumstances of the decisions themselves. Examples include hunger, time of day, mood, and weather on the individual level and group dynamics and communication norms at the team level.   

To combat these influences, the authors urge individuals and organizations to prioritize what they call decision hygiene – a collection of tools and tactics intended to mitigate the effects of noise and otherwise improve decision-making. It is important to note that decision hygiene does not mean removing discretion from decision-making. In medicine, for example, even if two people present with similar symptoms and medical history, there may be important differences between their circumstances and preferences that are worth considering, and it may be appropriate for a healthcare professional to recommend different treatment plans to them as a result. 

Rather, the authors note, many of our heuristics or patterns of thinking — including relying on gut feelings, substituting easier questions for harder ones, halo effects, prejudgment, confirmation bias, and overconfidence — can contribute to noise.  Fortunately, it’s possible to mitigate these influences and improve our decision-making.

Assessing and minimizing noise in the context of accreditation

Accreditation decisions are complex.  Accreditors with organizations such as the Liaison Committee on Medical Education, the Southern Association of Colleges and Schools, the Joint Commission, and the National Commission for Certifying Agencies must assess compliance against many standards and elements. In many cases, those decisions draw on evidence from thousands of pages of documentation as well as site visits that can involve a week’s worth of meetings. Many perspectives are involved, including those of site visitors, staff, and committee decision-makers.

Recognizing the complexity of this work, accrediting bodies place great emphasis on bringing validated and trustworthy processes to their decisions. Throughout my own career, I have seen firsthand the efforts made by accreditation committees and staff members to ensure a consistent, fair process with integrity. Everyone involved, particularly accreditors themselves, wants these decisions to be the best they can be. Learning about and taking steps (or augmenting existing tactics) to mitigate risk of noise may be a worthwhile part of this work.

Building from the authors’ recommendations, here are some thoughts about how people involved with accreditation can work to mitigate the effects of noise in their own decision-making as well as when working as part of a team. It may be useful to think about these efforts in two buckets: prior to decisions (general efforts to understand and manage noise) and then during decision-making (procedural adjustments that may help).   

Prior to decision-making:

1.       Introduce the concept of noise. Because accreditation site visit teams and committees bring together people with diverse experiences and viewpoints, a discussion about best practices for decision-making is reasonable and expected. Orientations for individuals and groups working in accreditation provide an excellent opportunity to introduce the concept of noise. Simply learning about noise is powerful, and individuals who are exposed to the concept may proactively modify their own approach to decision-making in an effort to reduce the effect of noise.

2.       Make reducing noise and improving decision-making themes for professional development.  Consider gathering teams ahead of a site visit or program review for dedicated professional development on these topics. Scenario reviews can provide opportunities for discussion of how individuals arrive at their decisions and how groups can more effectively make decisions.  Teams can discuss how to promote expression of a variety of perspectives and explore ways of reducing noise.  Guidelines, structuring complex judgements, and deliberately incorporating counter-arguments are techniques that may warrant particular attention.  

3.       Conduct a noise audit.  Although it can be challenging to systematically examine the quality of judgments made by a group of people, noise audits involving hypothetical scenarios generate valuable information.  The process can help quantify whether noise is in fact affecting decision-making and to what extent, providing a data foundation for making improvements.  Although noise audits often involve external consultants, the expertise within accrediting organizations may mean they have resources that could allow for a thoughtful audit by an internal team.

During decision-making:

1.       Meet the group’s human needs.  Reducing noise in group decision-making begins with planning the meeting. Considering of time of day, scheduling regular breaks, and providing drinks and snacks that include protein will go a long way toward ensuring everyone is at their best. This kind of preparation ensures participants have what they need to act on their own awareness of noise — those who understand that hunger or fatigue, for example, may influence their decisions know they simply need to get a snack or take a break to improve their work.

2.       Ensure all perspectives are heard.  Use words and actions to demonstrate that alternate opinions will be welcome and listened to. Polling apps that allow people to register their opinions prior to discussion may also help ensure more perspectives are considered more fully before conclusions are drawn.  One tool we have used with success is Poll Everywhere, though there are many to choose from.

3.       Leverage decision hygiene practices. Using decision guidelines can reduce inconsistency among decision-makers by providing clarity around what constitutes acceptable performance and how to assess and weight various measures of compliance. Dividing complex decisions into component parts can reduce the likelihood that performance in one area influences perception of performance in other areas. And rotating the order in which people speak can help ensure all participants have an opportunity to influence the discussion.

4.       Actively monitor for noise. Consider tasking an independent person with watching for and speaking up as they see things that could be noise, and be sure to engage others in welcoming and legitimizing the feedback.  Questions like those listed in the bias observation checklist included in Noise may also help.

Supporting quality in all we do

Decisions are an important part of life, and we all have a stake in ensuring they are the best they can be. Practices to mitigate noise hold promise for enhancing accuracy and consistency in our own decision-making and in decisions made by groups of all types, including those involved with accreditation.  

Best practices for making the LCME DCI part of your ongoing continuous quality improvement

Lois Margaret Nora, MD, JD, MBA

It’s no exaggeration to say that at many medical schools, there are few projects more challenging than preparing the Data Collection Instrument (DCI) ahead of an LCME accreditation site visit.  The full DCI questionnaire for the 2023-24 academic year numbers more than 150 pages, and it covers every aspect of medical school operations. Organizing, drafting, reviewing, and completing a DCI is a monumental job whose importance is difficult to overstate.

Updating such an important document after not touching it for years can be overwhelming. And yet, that is the experience at many schools.  However, it doesn’t have to be that way, and at MSAG we recommend a different approach. If schools reframe the DCI as a tool for continuous quality improvement (CQI) and then weave it into regular CQI processes, the document becomes not only easier to manage, but also a more accurate representation of the school and its medical education program. It becomes a tool for evaluating and memorializing institutional progress on strategic goals while supporting ongoing compliance with the LCME standards.  

While MSAG has long advocated that schools make the most of their work on the DCI, best practices for doing so come to us from medical educators and administrators.  I was fortunate to speak with two such colleagues recently about this topic.

Dr. Susan Perlis recently retired from her role as associate dean for medical education at Cooper Medical School of Rowan University, and Dr. Tim Gilbert is associate dean for accreditation and planning at the University of South Alabama College of Medicine. Both are seasoned administrators who have extensive backgrounds in education and multiple accreditation systems, providing a wealth of experience to draw on as they work with the LCME framework.

In a recent call, we discussed challenges, best practices, and lessons learned in their years of work with the DCI, and they shared many ideas that could benefit other schools. Here are some of the points I found particularly interesting.

Foster a constructive accreditation mindset

Accreditation is fundamentally about ensuring our institutions train physicians effectively, manage resources appropriately, and serve their communities well. What could be more worthy of our time? However, the challenging and time-consuming processes involved mean accreditation is sometimes viewed as an obligation, rather than an opportunity.

Sue suggests schools reframe their thinking. While pursuing her own research into assessment, she encountered an idea that transformed her thinking about the topic: Accreditation is a process that is done for a school, and not to a school.  “That creates a paradigm shift in the way we think about accreditation. Because if you think about accreditation that way … it becomes something we do for ourselves. For our quality, for our students, for our faculty, for our institution.”

There is a lot to the accreditation process, but one of the biggest hurdles is completion of the DCI. That’s why Tim and Sue have developed processes to keep the DCI up to date.  This work has enabled each of them to reframe DCI revisions and accreditation as a whole as more constructive, positive endeavors. And as Tim notes, this work has also fostered a wonderful spirit of teamwork as colleagues move toward their common goals.

Build a system – and a schedule – for keeping the DCI up to date

Sue and Tim have each created a system for revising the DCI on an annual basis, which they say enables ongoing CQI and streamlines the process of preparing for a site visit when reaccreditation is on the horizon.

When the new DCI is published, Tim has someone on his staff review the questionnaire to note any changes to the elements before turning to content experts for substantive updates. Then, his office works with stakeholders across the school to review one of the 12 standards each month. At any point in time, no part of the DCI is more than 11 months away from update, and someone is always working on it. “We have somebody who is in the DCI at least weekly, if not daily,” he said. The benefits are myriad, but one of the most obvious is simply keeping it current so anytime it’s needed, it’s ready.

Initially, Tim’s team worked on an academic-year schedule, starting with Standard 1 in July, but this year, as part of their own office-wide CQI, they are modifying the schedule to better align with the schedules of stakeholders across the institution. The revised schedule allows completion of Standard 5 (Educational Resources and Infrastructure) to better align with the fiscal year, and Standards 10-12 are completed during quieter times for colleagues in Admissions and Student Affairs. “We simply asked all those stakeholders, what’s the best time of year for you? And that’s how we’re doing it for next year,” Tim said.

Sue’s work on Standards 6-9 involved a flexible schedule as well, and she and her colleagues spent time copying information into the new DCI and highlighting changes and key questions before distributing the document to content experts for revision.  She included notes about what survey teams would be looking for, so contributors could think through how their responses would be read.  Timelines were set for making changes, but they were generous enough to accommodate other necessary work.

Lean on the right mix of expertise – and a teamwork approach

The breadth and depth of material covered by the DCI means no single team can handle it all. As Sue and Tim described their approach to the DCI, it became apparent that partnership is key.  They contribute deep expertise in education, accreditation, and assessment, and colleagues across the institution provide key subject matter expertise needed to fully populate the DCI.

At both medical schools where Sue worked, her primary collaborators beyond her team were faculty committee chairs, who had oversight over the activities documented in the DCI. While the faculty had ownership over the curriculum, Sue brought expertise in the LCME standards to help ensure conversations about changes to the curriculum occurred in the context of the DCI.  This collaboration ensured any possibility that plans might cause issues with compliance was dealt with before ideas were implemented. “Wherever I was, I would make sure I kept the element in front of us, and if a change was made — by the curriculum committee or by the academic standing committee — then I made sure we went back into the DCI and updated that.”

Tim also relies on a mix of accreditation and subject matter expertise for DCI updates. His process also engages reviewers from the CQI committee who bring a CQI lens and an objective perspective to the content. This system means the school has gone from a small handful of people reviewing the DCI to dozens of people offering input. “One of the unintended outcomes and real benefits we didn’t expect was a broadening of understanding of the DCI,” he said. As a result, when teams across the school consider making changes such as a schedule adjustment, implications for the DCI are top of mind. “We’ve got literally dozens of people who take ownership of the DCI, and it’s really reduced the complaining about the accreditation process because they understand it.”   

Leverage the support of leadership

One of the trickier aspects of building a system for regular use of the DCI is the extent to which people who don’t report to you must contribute. I’ve proposed that legislative leadership skills and systems thinking are an important aspect of gaining cross-departmental buy-in for challenging work. Sue and Tim clearly bring both to their efforts to make the most of the DCI.

However, strong support from institutional leadership is also important. Tim said his office enjoys clear support from his dean, which has made it far easier to build the team he needs and the influence to work effectively with other departments. Sue echoed that sentiment.  The importance of the dean’s support “can’t be overstated,” she said.  “It makes all the difference.”

Embrace the teaching and learning opportunity

My conversation with Sue and Tom underscored the deep expertise accreditation professionals bring to their work. Our discussion also served as a reminder of how valuable it can be for accreditation professionals to spend time sharing that knowledge and experience with others whose expertise may lie elsewhere.  When Tim joined his current institution, he saw an opportunity to build the team he needed and took it upon himself to train and promote a staff member who ultimately became an important voice of expertise on accreditation in her own right.  Likewise, Sue built expertise on her teams with a combination of direct teaching and comments on the DCI as the team worked to update it.

When I reflected on my conversation with Tim and Sue, I was reminded that an up-to-date DCI can be a great tool to orient new employees and for job candidates to learn about an institution. I have encouraged mentees to request and read the most recent LCME summary report as they consider job opportunities; a request for the DCI may also result in helpful insights.  As Sue notes, the DCI is really “the nuts and bolts of how we conduct business in a medical school,” making it a rich resource for anyone who needs to become familiar with the institution.

The accreditation process as opportunity

At MSAG, we believe thoughtful standards in medical education support the integrity of our educational programs and help improve the quality of care provided to patients. We also believe medical educators who embrace the CQI opportunity inherent in LCME accreditation will leverage that work to improve, grow, and achieve strategic goals that go well beyond compliance with standards.

Regularly reviewing and updating the DCI may sound like a lot of work, and it certainly does take time, intention, and commitment. However, schools can get much more out of the process than they put in. Accreditation becomes a more positive endeavor that continually benefits the school, rather than a process seen as a periodic and time-consuming burden.

Many thanks to Dr. Sue Perlis and Dr. Tim Gilbert for sharing how they have operationalized these ideas. Input from other colleagues is always welcome, so please reach out anytime with thoughts, questions, and ideas. We will continue to share insights and lessons here on this blog.

Putting accreditation on the faculty retreat agenda – why and how

Lois Margaret Nora, MD, JD, MBA

It’s common during the summer months and the early weeks of fall for medical schools to host annual educational retreats for faculty – both those who are employed and those who teach on a voluntary basis. These events provide a chance for attendees to connect and share information. They also provide an opportunity for leaders from across the school to express appreciation for the invaluable work done by faculty, bring attendees up-to-date on what’s new at the school, and provide faculty development opportunities.

Subscribe to our newsletter

A recent discussion about planning content for such a retreat reminded me that these gatherings offer a wonderful opportunity to help faculty learn more about LCME accreditation. Members of the faculty may not be aware of the ways in which accreditation expectations affect their work.  For example, a school faces real challenges from an accreditation perspective if narrative feedback is not provided or if evaluations are not submitted in a timely manner. Drawing a link between policies for student assessment and accreditation can help faculty understand the importance of following those policies. It can also help convey how expectations that may feel picky or onerous have real importance for the institution.

This is where the leadership of accreditation professionals comes in.  While accreditation is never far from the minds of those who deal with these processes regularly, it takes work to keep it on radars across the school and beyond high-profile periods like site visits. Talking with faculty about the role they play in accreditation and its importance to the school is one step in the right direction. And retreats, with their collegial atmosphere and attention to learning, may be just the right environment.

Integrating accreditation into retreat content plans

As retreat planners consider how to integrate accreditation into learning, it may help to draw cues from the accreditation cycle itself. Schools that are embarking on a self-study are at a very different place from schools that are years away from reaccreditation. The treatment of accreditation topics in retreat curricula should be adapted accordingly. Here are a few thoughts that may inspire ideas for your planning:

Considerations for schools in a self-study

If a school is poised to begin a self-study, consider focusing the entire retreat around this process.  Start by explaining how the school is accredited and outlining the timing and benchmarks involved (document submission, site visit timing, etc.)  It can be worthwhile to review past accreditation challenges, along with unique aspects of the school that may draw questions from the site visitors, such as distributed campuses or past issues with the learning environment.  Breakout groups can explore specific areas within student support, diversity, faculty affairs, and the educational program.  Involving key committee chairs and members can reinforce the importance of these discussions.

When schools are in the middle of the self-study process, consider sharing early findings from the process and discussing actions being taken to address these findings. Accreditation professionals can present topics themselves, but often this information is best presented by the faculty leaders and area administrators who work directly in these areas. 

Discussions about accreditation can help prepare faculty for conversations with the site visit team by helping them understand what they may be asked about and related expectations for compliance. These discussions also allow medical education professionals to draw a clear line between the everyday work of faculty and the accreditation process. And, importantly, they may open the door to a broader discussion of continuous quality improvement (CQI). The actions of noting red flags through student feedback, GQ and ISA responses or other sources; considering possible changes and then implementing them; and defining how the institution will follow up on outcomes are powerful illustrations of CQI.  Underscoring this message may help foster the collaboration needed for accreditation – and for improving the quality of medical education at your institution.   

Considerations for midcycle schools

Although accreditation is a common topic of concern at schools that are working on a self-study, most schools are elsewhere in the accreditation cycle.  While it can be tempting to set compliance considerations aside after reaccreditation is achieved, there are plenty of midcycle tasks —  CQI; ensuring policies and procedures are up-to-date; and building systems to support compliance — that are important and worthy of discussion at any point in the cycle.

Including an accreditation component in all annual retreats allows for review of this ongoing work.  Doing so also keeps accreditation top of mind. New expectations and standards, feedback about progress addressing past findings, and a conversation about the school’s CQI processes are all worthy topics for discussion.

There are numerous possible approaches realizing these ideas. Consider planning a short, focused session that covers accreditation “highlights.”  Conversely, accreditation could be woven throughout the retreat by asking all speakers to take a few minutes to provide an update within their area of focus (e.g., student support, faculty affairs, curriculum, etc.) Or, for retreats with a specific theme, look for ways in which that theme intersects with accreditation to inspire ideas.

Schools should pursue the approach that makes the most sense for their retreat and their institution. However they proceed, they will derive great value from involving the school’s accreditation professionals in retreat planning.  The unique lens brought by accreditation professionals is about more than preparing the school to do what is necessary for compliance. It’s also about helping the school leverage the LCME standards and processes for improving the quality of medical education at the institution.

Underscoring the value of accreditation, and the accreditation professional

At MSAG, we believe continual reinforcement of the value and principles of accreditation keeps the standards front and center at the school and enhances the quality of the education it provides.

Keeping accreditation top of mind also helps integrate accreditation professionals into the life of the school. As accreditation professionals are given (or create) opportunities to talk about how their work links to priorities across the institution, they clarify their role and value as helpful colleague, while positioning themselves as a resource on a topic of great importance to their institution.