National Academy of Sciences Releases CSA Study
June 27, 2017 – The National Academy of Sciences (NAS) who was tasked by Congress in the FAST Act, to conduct a thorough comprehensive examination of the Federal Motor Carrier Safety Administration’s (FMCSA) Compliance, Safety and Accountability (CSA) initiative, has released their much-anticipated report. Congress commissioned the NAS to conduct this study because of concerns from motor carriers, shippers and brokers about the validity of the system, and the impacts of the scores on third-parties. In summary, the NAS found that the CSA and Safety Measurement System (SMS) is:
structured in a reasonable way, and its method of identifying motor carriers for alert status is defensible. However, much of what is now done is ad hoc and based on subject-matter expertise that has not been sufficiently empirically validated. This argues for FMCSA adopting a more statistically principled approach that can include the expert opinion that is implicit in SMS in a natural way.
Additionally, the NAS based on the current information available to them, was unable to make a recommendation if a motor carrier’s SMS results and scores should be made publicly available. The NAS recommends that the Agency conduct a detailed review and analysis of the potential impacts to industry and the public of making those scores publicly available. Ultimately, this means that the scores will likely remain hidden from view until further analysis is conducted.
Specifically, the NAS makes the following recommendations to the FMCSA for improvements:
TIA participated and provided testimony during one of the NAS public meetings, where we advocated on behalf of the third-party logistics industry to keep the data from being displayed on public view, and outlined the liability concerns of having the scores publicly available.
- FMCSA should implement an item response theory (IRT), which is a paradigm for the design, analysis, and scoring of tests, questionnaires, and similar instruments measuring abilities, attitudes, or other variables. It is a theory of testing based on the relationship between individuals’ performances on a test item and the test takers’ levels of performance on an overall measure of the ability that item was designed to measure. Several different statistical models are used to represent both item and test taker characteristics. This would:
- move towards leaning on data alone, and not expert opinions,
- enhance transparency of the evaluation system,
- support the direct estimation of variability of scores and rankings, and
- Allow the Agency to adapt to future changes.
- FMCSA should continue to work with State Agencies and law enforcement to improve the quality of data in MCMIS. The two areas in need of immediate attention would be: carrier exposure and crash data and fault. The NAS calls for the development of a National Minimum Uniform Crash Criteria to be implemented nationwide.
- FMCSA should investigate new ways of collecting data that will likely benefit the recommended methodology for safety assessment. This would include data on carrier characteristics – including information on driver turnover rates, type of cargo, method and level of compensation, and better information on exposure.
- FMCSA should structure a user-friendly version of the MCMIS data file used as input to SMS without any personally identifiable information to facilitate its use by external parties, such as researchers, and by carriers. In addition, FMCSA should make user-friendly computer code used to compute SMS elements available to individuals in accordance with reproducibility and transparency guidelines.
- FMCSA should undertake a study to better understand the statistical operating characteristics of the percentile ranks to support decisions regarding the usability of public scores.
- FMCSA should decide on the carriers that receive SMS alerts using both the SMS percentile ranks and the SMS absolute measures, and the percentile ranks should be computed both conditionally within safety event groups and over all motor carriers.
GAO Releases CSA Report on Improving Information
October 27, 2016 - The Government Accountability Office (GAO) released a report entitled: “Better Information Needed to Assess Effectiveness and Efficiency of Safety Interventions”, addressing the CSA initiative.
In July 2010, the Federal Motor Carrier Safety Administration (FMCSA) chose to delay nationwide implementation of two of the eight interventions that FMCSA uses to address motor carrier safety concerns under its Compliance, Safety, Accountability (CSA) program. This delay is linked to continuing delays in developing the software needed to support the two interventions, offsite investigations and the use of cooperative safety plans.
The software under development is intended to help FMCSA overcome some of the information challenges it faces due to its reliance on legacy information systems. FMCSA estimates that the software development project will be completed by April 2017. FMCSA has conducted evaluations of the effectiveness and efficiency outcomes it established for the CSA program. However, GAO identified several limitations in FMCSA’s approaches that impact the usefulness of the evaluations:
Intervention effectiveness: FMCSA has developed a statistical model to annually evaluate the combined effectiveness of interventions. Although the model has some key strengths, such as accounting for a broad range of external factors, GAO identified a number of design and methodology limitations that reduce the usefulness of its results. For example, the model does not include an assessment of individual intervention types. Without this type of specific information, FMCSA is hampered in its ability to identify the circumstances under which different types of interventions are effective. Similarly, these types of limitations affect FMCSA’s ability to accurately draw conclusions about intervention effectiveness across all intervention types.
Intervention efficiency: To assess the efficiency of CSA interventions, FMCSA has relied on a study that it sponsored and that was published in 2011. This study estimated the average cost of conducting interventions in four states from October 2008 through May 2009. However, FMCSA has not taken steps to update its cost estimates for interventions since the 2011 evaluation, despite changes since that time in the resources needed to conduct CSA interventions; nor has it taken steps to develop additional information that is representative of the costs in other states. Without current cost estimates that are representative of all states, FMCSA cannot appropriately assess the efficiency of its interventions. FMCSA has taken some actions to improve the effectiveness and efficiency of CSA interventions but lacks measures to monitor progress.
In April 2014, FMCSA established a working group to assess CSA interventions and make recommendations for improvement. As of April 2016, the group had made 20 recommendations, of which 12 had been implemented. However, GAO found that while FMCSA has established some performance measures for its effectiveness outcome that are appropriate, it has not established similar measures for its efficiency outcome. FMCSA headquarters officials told GAO that effectiveness and efficiency are complementary outcomes that FMCSA strives to balance. Without a complete set of measures for both outcomes, FMCSA lacks benchmarks needed to regularly measure progress to achieve these outcomes.
As fatalities involving commercial motor vehicles have increased, rising from about 4,200 in 2011 to almost 4,500 in 2015, Agency interventions can play a critical role in reversing this troubling trend. FMCSA aims to reduce such fatalities by using a data-driven approach that identifies and intervenes with the highest risk motor carriers earlier. However, the GAO has identified important limitations in the information that FMCSA uses to evaluate the effectiveness and efficiency of interventions.
To address these deficiencies, GAO recommends:
FMCSA identify and implement, as appropriate, methods to evaluate the effectiveness of individual intervention types or common intervention patterns to obtain more complete, appropriate, and accurate information on effectiveness of interventions in improving motor carrier safety performance. FMCSA should incorporate accepted practices for designing program effectiveness evaluations, including practices that would enable FMCSA to more confidently attribute changes in carriers’ safety behavior to CSA interventions.
Update FMCSA’s cost estimates to determine the resources currently used to conduct individual intervention types and ensure FMCSA has cost information that is representative of all states.
Establish and use performance measures to regularly monitor progress toward both FMCSA’s effectiveness outcome and its efficiency outcome.
For more information, please contact TIA Advocacy at firstname.lastname@example.org or 703.299.5700.
FMCSA Seeking Industry Feedback on Proposed Changes to CSA Scores
October 5, 2016 - The Federal Motor Carrier Safety Administration has published a website that allows fleets and owner-operators to preview potential changes to their Compliance, Safety, Accountability scores based on proposals to reform the program issued in the last two years.
Motor carriers can preview their altered scores here starting on October 4, should any changes be reflected from the 2015- and 2016-proposed reforms. The agency is accepting public comment for 60 days based on the new preview tool. The public comment period opens October 5, and closes Monday, December 5. The hope is that the methodology changes to CSA’s SMS will better tie carriers’ safety to their rankings within the system, FMCSA says, and allow it to better target at-risk carriers, the agency argues.
The key changes proposed to CSA and its Safety Measurement System percentile rankings (and now available for preview) include:
- Lowering intervention thresholds in some SMS categories/BASICs,
- raising the threshold in one BASIC, and
- making the Hazmat BASIC public and splitting it into two segments.
Other proposed changes available for preview include increasing the minimum number of crashes needed before a carrier receives a score in the Crash Indicator BASIC and shortening the time period for which violations lead to BASIC percentile ratings.
Changes issued this year are also available for preview on the agency’s new site. Those changes include raising the score threshold for the Crash Indicator BASIC from two crashes to three crashes and removing carriers with no violations in the past year in several BASICs from its intervention prioritization.
The Crash Indicator BASIC changes mean carriers only involved in two crashes will no longer register a ranking within the Crash Indicator BASIC. Carriers must now be involved in three crashes before registering a score.
FMCSA plans to maintain the 65 percent intervention threshold for the BASICs with the highest correlation to crash risk: Unsafe Driving, Crash Indicator and Hours-of-Service Compliance.
With the transportation industry’s advocacy efforts, a stipulation within 2015’s FAST Act highway bill, the percentile rankings within the SMS BASICs are no longer available for public view. Only carriers and enforcers can see the ratings, even though the underlying data used to formulate the BASIC is still publicly available within the SMS portal.
The basic intervention thresholds now available for preview include:
- Lowering the intervention threshold of the Vehicle Maintenance BASIC to 75 percent from the current 80 percent for general property carriers, thus targeting more carriers for intervention.
- Raising the intervention threshold for the Controlled Substances/Alcohol BASIC from 80 percent to 90 percent, thus targeting fewer carriers for intervention.
- Raising the intervention threshold for the Driver Fitness BASIC from 80 to 90 percent, thus targeting fewer carriers.
In addition to proposing to go public with the Hazmat Compliance BASIC (which has now been effectively barred by the FAST Act provisions), the agency has proposed splitting the BASIC into two segments — one for cargo tank carriers and one for non-cargo tank carriers. These proposed changes will also be available for preview for hazmat haulers on the agency’s SMS preview site.
Now also available for preview are the proposal to reclassify violation of an out-of-service order to the Unsafe Driving BASIC (instead of whichever BASIC the original violation that caused the OOS order is in) and upping the maximum vehicle miles traveled figure used in CSA’s so-called “utilization factor” to better account for carriers who run in high-mileage operations.
Carrier, especially smaller carriers with fewer inspection, will now be assigned BASIC percentile rankings based on violations from their most recent inspection within the previous 24 months, the agency will now only assign BASIC ratings for carriers who have had an inspection violation within the past year. The change applies to four BASICs: HOS Compliance, Vehicle Maintenance, Hazmat Compliance and Driver Fitness.
If you have any questions, please contact Chris Burroughs at 703-299-5700 or email@example.com.
FMCSA to Examine Preventable Crashes in CSA
July 13, 2016 - The Federal Motor Carrier Safety Administration (FMCSA) announced that it is proposing to develop and implement a program, which would allow motor carriers and drivers to contest certain crashes within the Safety Measurement System (SMS) that generally are not necessarily the fault of a driver.
Based on industry stakeholder’s comments to a previous Federal Register notices, FMCSA considered examples of non-preventable crashes and has decided to develop a demonstration program. The program would accept requests for data review (RDRs) that seek to establish the non-preventability of certain crashes, through its national data correction system known as DataQs. The Agency would accept an RDR, as part of this program when documentation established that the crash was not preventable by the motor carrier or commercial driver.
A crash would be considered not preventable if the CMV was struck by a motorist who was convicted of one of the four following offenses or a related offense:
- Driving under the influence;
- Driving the wrong direction;
- Striking the CMV in the rear; or
- Striking the CMV while it was legally stopped.
The Agency proposes that evidence of a conviction must be submitted with the RDR to document that the crash was not preventable by the motor carrier or driver. In addition to documentation of the conviction, these RDRs should include all available law enforcement reports, insurance reports for all parties involved in the crash, and any other relevant information. RDRs could also be submitted through DataQs when the crash did not involve other vehicles, for example animal strikes.
Upon the receipt of a complete RDR, FMCSA staff or a contractor would review the submission using the preventability definition in 49 CFR part 285. The Agency proposes that the RDR would result in one of the following three decisions and actions:
- Not Preventable – In these cases, the crash is removed from SMS;
- Preventable – In these cases, the crash is not removed from SMS for the purpose of calculating the Crash Indicator BASIC percentile. FMCSA is looking at rating these crashes for severity, and a crash considered “Preventable” could carry a higher severity rating.
- Undecided – In these cases, not enough evidence was presented and the crash remains in SMS, but a severity rating would not apply.
The Agency proposes that the minimum time period for this test would be 24 months.
On January 23, 2015, FMCSA announced the results of the Agency’s study on the feasibility of using a motor carrier’s role in crashes in the assessment of the company’s safety. This study assessed:
- Whether police accident reports (PARs) provide sufficient, consistent, and reliable information to support crash-weighting determinations;
- Whether a crash-weighting determination process would offer an even stronger predictor of crash risk than overall crash involvement and how crash weighting would be implemented in the Agency’s Safety Measurement System (SMS); and
- How FMCSA might manage a process for making crash weighting determinations, including the acceptance of public input.
If you have any questions, contact Chris Burroughs (firstname.lastname@example.org, 703.299.5705).
CSA Reform within the Transportation Bill
The “Fixing America’s Surface Transportation Act” or FAST Act (Transportation Bill) contains extensive reform language dealing with the Federal Motor Carrier Safety Administration’s (FMCSA) Compliance, Safety, and Accountability (CSA) initiative. A major provision within the CSA reform package of the deals, removes from public view all information regarding analysis of violations, crashes in which a determination is made that the motor carrier or commercial motor vehicle driver is not at fault, alerts, or the relative percentile for each BASIC until corrective action is taken. All this information will be removed from public view within 1 day of signing this report into law. With that being said, all inspection and violation information submitted to FMCSA by law enforcement and inspectors along with out-of-service rates, and “absolute measures will remain publicly available.
The FAST Act will require the National Research Council of the National Academics to conduct a study within 18 months on the following key aspects of the CSA initiative:
- Accuracy with which BASICs:
- Identify high-risk carriers; and
- Predict or correlate with future crash risk.
- The methodology used to calculate BASIC percentiles and identify carriers for enforcement, including weights assigned to particular violations and the tie between crash risks.
- The relative value of inspection information and roadside enforcement data.
- Any data collection gaps or data sufficiency problems.
- The accuracy of safety data, including the use of crash data from crashes in which a motor carrier was free from fault.
- Whether BASIC percentiles for motor carrier of passengers should be calculated separately from motor carriers of freight.
- The difference in the rates at which safety violations are reported to FMCSA by various enforcement authorities, including States, territories, and Federal inspectors.
- How members of the public use the SMS and what effect making the SMS information public has had on reducing crashes and eliminating unsafe motor carriers form the industry.
- Whether the SMS provides comparable precision and confidence, through SMS alerts and percentiles, for the relative crash risk of individual large and small motor carriers.
- Whether alternatives to the SMS would identify high-risk carriers more accurately.
- The recommendations and findings of the Comptroller General of the United States and the Inspector General of the DOT, and independent review team reports, issued before the date of enactment.
Once the report has been completed and sent to the FMCSA and Congress, not later than 120 days after submission, the FMCSA will have to develop a corrective action plan that responds to deficiencies and opportunities identified in the report and how FMCSA plans to fix those problems.
After the corrective action plan is submitted to Congress, the Inspector General (IG) of the DOT must review and the plan and ensure that it address all the issues and problems identified within the study. If the DOT IG certifies that the corrective action plan is sufficient, FMCSA must begin implementing all necessary changes. Until that time the information removed from public view will remain hidden, and the Agency will not be allowed to complete a safety fitness determination (SFD) rulemaking that would tie the CSA scores to a motor carrier’s safety rating.
Additionally, the FAST Act requires FMCSA to implement within 18 months a beyond compliance program within CSA that gives recognition, including credit or an improved SMS percentile, for a motor carrier that:
- Installs advanced safety equipment;
- Uses enhanced driver fitness measures;
- Adopts fleet safety management tools, technologies, and programs; or
- Satisfies other standards determined appropriate by the Administrator.
Finally, the FAST Act requires the FMCSA’s Motor Carrier Safety Advisory Committee or MCSAC not later than 1 year after the certification of CSA improvement has been completed to examine and develop a plan of action for the treatment of preventable crashes under the SMS.
If you have any questions, contact Chris Burroughs (email@example.com, 703.299.5705).