There’s little evidence showing which police reforms work

When criminologist Robin Engel suddenly found herself leading the effort to reform a police department under fire after a white police officer killed an unarmed Black man in July 2015, she looked for some kind of road map to follow. Instead, she found herself in poorly charted territory.

A professor at the University of Cincinnati, Engel had been called on frequently to help police departments around the country manage their response to acts of police violence. This time, the call came from close to home. Campus Officer Ray Tensing, 25, had shot and killed 43-year-old musician Samuel DuBose during an off-campus traffic stop.

Engel recommended that the university hire a high-ranking official to oversee the police department and its immediate response to the crisis, and initiate longer term, comprehensive reforms to prevent future incidents.

Within days, Engel had become that official, reporting directly to the university president and outranking the university’s police chief, despite lacking police experience herself.

She sought input from various community stakeholders, many of whom had been rankled by her appointment to lead the police division. She also turned to her best-known tool — research. She began probing for studies to guide her on the sorts of reforms she could institute, ones with proven track records of changing police behavior in the field. Her search was unfruitful.

“I thought most certainly we would have an evidence base that I could follow,” Engel says. “I was incredibly disappointed at the lack of evidence that was available. I was really disappointed in my own field.”

Evidence gap

Among her efforts, Engel scoured the literature for so-called de-escalation programs with a history of success at defusing violence. Her review of that body of work, appearing in January 2020 in Criminology & Public Policy, found 64 de-escalation programs in the United States and elsewhere — but mostly administered to nurses and psychologists. She found no programs that had been tested among police officers. Just three studies showed cause and effect and included randomized control groups, and those showed that such programs led to minimal individual and organizational improvements.

In a February 2020 review in The Annals of the American Academy of Political and Social Science, Engel and colleagues discuss de-escalation trainings and four other reforms that tend to capture the public’s attention following fatal police-civilian encounters: body-worn cameras, implicit bias training (meant to reduce decisions and actions that arise from unconscious stereotypes) (SN: 10/26/15), early intervention systems that identify problematic officers before a crisis and civilian oversight of the police.

Engel was unable to identify a single police reform with convincing evidence of resulting behavior change among officers. Even studies on body-worn cameras, which are numerous, had mixed results. Engel cites a March 2019 review of 70 studies in Criminology & Public Policy by a team of researchers led by Cynthia Lum of George Mason University in Fairfax, Va., that gauged the link between camera use and a reduction in force. Just 16 of those studies looked directly at whether or not cameras reduced officers’ use of force; of that subset, some show that the cameras work as a deterrent to use of force whereas others reach the opposite conclusion.

Why no data?

The dearth of evidence stems from several factors, Engel says, but chief among them is the pressure for police departments to act fast when an instance of police violence captures national attention. Consider that less than two weeks following the death of George Floyd when white police officer Derek Chauvin kneeled on his neck for several minutes, the majority of city councilors pledged to dismantle the Minneapolis Police Department in response to activists’ calls to “defund the police” (SN: 6/5/2020).

Other departments around the country are likewise looking at ways to defund some police services, or reallocate to other agencies functions such as responding to mental health calls or monitoring safety in schools. Previous police brutality incidents have prompted calls for other sorts of reforms. For instance, a 2019 CBS News Survey of 155 police agencies found that almost 70 percent had implicit bias training with over half of those implemented after a white policeman shot and killed Black teenager Michael Brown in Ferguson, Mo., in August 2014. Some 60 percent of those agencies said they did not have a way to measure the success or failure of such programs.

“This year it’s defund; what is it going to be next year, five years from now?” says Renée Mitchell, a recently retired police sergeant with a Ph.D. in criminology. She’s also cofounder and president of the American Society of Evidence-Based Policing. Police departments are “flinging out interventions and having no clue about the effects, positive or negative.”

Police research is complicated by the fact that researchers who conduct the sorts of studies needed to evaluate reforms and police officials often have different priorities. Why would a police chief work with an academic who is going to publish papers about the department’s problems that may also receive considerable press attention, asks Erin Kerrison, an empirical legal scholar at the University of California, Berkeley.

What’s more, like some other areas of research related to violence (SN: 5/3/16), money for policing research is relatively limited. Consider that the National Institutes of Health invested about $39 billion for medical research in 2019, while the National Institute of Justice awarded far less than 1 percent of that amount — just under $214 million — for research that same year.

Yet researchers and police officials largely agree that rapid response is necessary to meet the demands of the moment. What research does emerge following George Floyd’s death won’t start coming out for two years, Kerrison says. “There will have been a thousand more George Floyds at that point.”

Needing to act

Which is why, back at the University of Cincinnati Police Division, Engel needed to act, evidence or no evidence. So on the de-escalation front, she selected a program run by Washington, D.C.–based Police Executive Research Forum, a nonprofit police research and policy organization. She was familiar with the organization, and the content — which emphasized training officers to recognize and effectively communicate with civilians behaving erratically and either unarmed or armed with something other than a firearm — looked promising. Engel then used the program to begin building her own evidence to help fill gaps in the field.

Engel treated reforms at her small university police department of 74 sworn officers as pilot projects that she could then test at larger police departments around the country. In February 2019, she and a team of researchers were able to conduct a larger study of the de-escalation program, when the Louisville Metro Police Department in Kentucky used it to train its 1,250 officers.

Engel randomized the order in which officers in the nine Louisville precincts were trained. That way, officers in each untrained precinct served as a control until they too underwent training. One benefit of this setup, called a “stepped wedge trial,” is that it doesn’t relegate one block of individuals into a control group that goes without training for the duration of the study. Stepped wedge trials have been used in other settings, such as health care and education. Officers were evaluated before and immediately after the training, and again, four to six months after training. Observations will continue for up to 12 months, with the team looking for changes in police behavior, and frequency and severity around the use of force.

Initial results will be out later this summer, says Engel, who is conducting an analogous study of an implicit bias training program. Also piloted at the University of Cincinnati, the program was rolled out at the NYPD, New York City’s police department.

Working together

Engel stepped down from her role overseeing the University of Cincinnati Police Division in January 2019, but the experience changed her thinking about criminology research. Academics tend to be interested in the philosophical, such as why officers use force, she says. But arguably more important are those nitty-gritty questions about how use of force can be mitigated in real life and in real time.

One challenge to understanding what reforms work is convincing police departments to collaborate with researchers, says Kerrison. She and colleagues outlined how academics can enter into ethical relationships with police departments in an August 2019 paper in Police Practice and Research. Crucial to such partnerships are clearly stated goals from the get-go, or airtight memoranda of understanding. That way, all parties agree in advance on the sorts of findings that will be communicated to the public and in what fashion, and everybody commits to helping police operations throughout the study process.

For instance, police departments can mandate that researchers anonymize their community’s identity in publications. Kerrison herself can’t talk about her relationships with police departments she’s working with due to such agreements. “Everybody has got to have skin in the game,” she says.

Given the challenges with funding and creating such academic-police partnerships, sometimes the clearest path forward may be to train the police officers in how to do science, Mitchell says. At the American Society for Evidence-Based Policing, she and colleagues are launching a four-week training course in 2021 for police officers similar to one already available in the United Kingdom. “Nowhere have our police leaders been taught how to interpret data and how to interpret statistics and how to interpret a research article,” she says. With such training, police departments will be better positioned to collect and evaluate data on their own.

Mitchell likens the model to medicine, where, for example, it would be a breach of ethics for doctors to advise patients with cancer without knowing about relevant evidence-based treatments. “[Policing] should be held to the same standard as the medical field,” she says.

However such research comes about, without it, police responses to crises will default to the quickest solutions, Engel says. “That is a very dangerous position to be in.”