FMCSA Drops IRT Model and Proposes Overhaul of How It Ranks Carrier Safety

In February 2023, the Federal Motor Carrier Safety Administration published a notice proposing major changes to the Safety Measurement System (SMS), the tool it uses to rank motor carriers and decide which ones to prioritize for interventions. The notice appeared in the Federal Register on February 15, 2023.

IRT set aside

The agency said it had studied Item Response Theory (IRT), an advanced statistical model recommended by the National Academy of Sciences, and found it "overly complex." Instead, FMCSA said it would improve its existing SMS methodology.

"Safety is FMCSA's core mission," Administrator Robin Hutcheson said, according to FreightWaves. "The proposed changes are part of the agency's continued commitment to enhancing the fairness, accuracy and clarity of our prioritization system."

The proposed changes

The notice proposed reorganizing the safety categories into seven: Unsafe Driving, Crash Indicator, Hours-of-Service Compliance, Vehicle Maintenance, Vehicle Maintenance: Driver Observed, Hazardous Materials Compliance and Driver Fitness. The Unsafe Driving category would absorb the controlled substances and alcohol violations along with violations for operating while out of service.

Other proposals included:

• Combining the 959 violations used in SMS into 116 violation groups.
• Replacing the 1-to-10 severity weights, which FMCSA acknowledged had been criticized as overly subjective, with weights of 1 or 2.
• Using proportionate percentiles in place of safety event groups.
• Revising intervention thresholds, including raising the Hazardous Materials Compliance threshold to 90%.
• Assigning a percentile in a category only if the carrier had a violation in that category within the past 12 months.
• Extending the utilization factor to carriers running up to 250,000 vehicle miles traveled per power unit.

FMCSA said the recent-violation change would mean 1,081 carriers no longer had a category at or above the intervention threshold, and that those carriers had a crash rate 13% lower than the national average.

The agency also said it considered but would not propose adjusting results for differences in enforcement among states, arguing that doing so would discourage state partners from enforcement initiatives.

Reaction

P. Sean Garney, co-director of Scopelitis Transportation Consulting, told FreightWaves that "what surprised us was the foundational nature" of some of the changes. "Ultimately, the mark of success for any of these changes is how it impacts crash rate. Time will tell," he said.

The Owner-Operator Independent Drivers Association was critical. "The overwhelming majority of data the agency collects and analyzes has no connection to crashes – none," OOIDA President Todd Spencer said, urging the agency to move ahead with the large truck crash causation study authorized by Congress.

FMCSA set a 90-day comment period ending May 16, 2023, and offered a preview website where carriers could see how the proposed methodology would affect their own results.

What it means for drivers and fleets

The February 2023 proposal did not change any regulation that drivers must follow, but it would change how violations they receive at roadside are scored against their carriers. Fleets were able to use the preview site to compare current and proposed results and to see whether the new categories, simplified weights and 12-month recency rule would move them above or below intervention thresholds. Safety managers also had the comment window to argue for or against specific changes before FMCSA decided on a final methodology.