ROP Evaluation

icon/hyperlink for: Evaluation of the ROP page; consisting of a circle target image with light green outline and red and white alternating rings with a large white arrowhead pointer pointing in the center

On this page:

This page includes links to files in non-HTML format. See Plugins, Viewers, and Other Tools for more information.

Annual Self-Assessments

On February 24, 2000, the staff issued SECY-00-0049, "Results of the Revised Reactor Oversight Process Pilot Program". The resulting staff requirements memorandum (SRM), issued on March 28, 2000, approved initial implementation of the ROP as recommended by the staff. The initial implementation of the ROP began on April 2, 2000. In a follow-up SRM issued on May 17, 2000, the Commission directed the staff to report on the ROP results after the first year of implementation. The staff did so and documented the results in SECY-01-0114, "Results of the Initial Implementation of the New Reactor Oversight Process", issued June 25, 2001. SECY-01-0114 also noted the staff's intention to continue to perform an annual self-assessment of the ROP.

The objectives and details of the ROP self-assessment program are contained in IMC 0307, "Reactor Oversight Process Self-Assessment Program" and its appendices. ROP self-assessment activities leverage data analysis and trending tools to (1) assess program-wide uniformity and effectiveness of ROP implementation, (2) evaluate the effectiveness of recent significant changes to the ROP, (3) evaluate the NRC's response to significant licensee events and declining licensee performance, and (4) perform periodic in-depth reviews of specific areas of interest, including the baseline inspection program. These ROP self-assessment activities help the staff determine whether the ROP was implemented in accordance with current governance documents, whether it was implemented uniformly and effectively across the regions and headquarters, whether it met its pre-established ROP goals and intended outcomes, and whether it was implemented consistent with the NRC Principles of Good Regulation. Taken in aggregate, these determinations form the basis for the staff's annual evaluation of whether the ROP remains effective. Finally, the ROP self-assessment approach ensures that the staff appropriately invests resources in addressing lessons learned that improve the efficiency and effectiveness of the ROP.

Each year, the staff issues an ROP performance metrics report and an ROP self-assessment Commission paper in support of the Agency Action Review Meeting (AARM). The staff also briefs the Commission on the ROP self-assessment results following the AARM. The results of these activities are summarized in the table below.

Time Period Commission Paper Commission Meeting Commission SRM Metric Report
CY 2019 SECY-20-0040 TBD TBD CY 2019
CY 2018 SECY-19-0037 June 20, 2019 Meeting SRM M190620 CY 2018
CY 2017 SECY-18-0045 June 19, 2017 Meeting SRM M180619 CY 2017
CY 2016 SECY-17-0049 June 15, 2017 Meeting SRM M170615 CY 2016
CY 2015 SECY-16-0047 June 2, 2016 Meeting SRM M160602B CY 2015
CY 2014 COMSECY-14-0030 May 21, 2015 COMSECY SRM  
CY 2013 SECY-14-0047 June 3, 2014 Meeting SRM M140603 CY 2013
CY 2012 SECY-13-0037 May 29, 2013 Meeting SRM M130529 CY 2012
CY 2011 SECY-12-0055 June 1, 2012 Meeting SRM M120601 CY 2011
CY 2010 SECY-11-0054 May 27, 2011 Meeting SRM M110527 CY 2010
CY 2009 SECY-10-0042 May 27, 2010 Meeting SRM M100527 CY 2009
CY 2008 SECY-09-0054 May 14, 2009 Meeting SRM M090514 CY 2008
CY 2007 SECY-08-0046 June 4, 2008 Meeting SRM M080604 CY 2007
CY 2006 SECY-07-0069 May 31, 2007 Meeting SRM M070531 CY 2006
CY 2005 SECY-06-0074 May 16, 2006 Meeting SRM M060516B CY 2005
CY 2004 SECY-05-0070 May 25, 2005 Meeting SRM M050525B CY 2004
CY 2003 SECY-04-0053 May 4, 2004 Meeting SRM M040504B SECY-04-0053, Attachment 3
CY 2002 SECY-03-0062 May 15, 2003 Meeting SRM M030515 SECY-03-0062, Attachment 3
CY 2001 SECY-02-0062 May 1, 2002 Meeting SRM M020501 SECY-02-0062, Attachment 2
Initial Implementation SECY-01-0114 July 19, 2001 Meeting SRM M010719B N/A
Pilot Program SECY-00-0049 March 7, 2000 SRM 1 / SRM 2 N/A

Independent Evaluations

In addition to the ROP self-assessment program, several independent evaluations have been performed since the inception of the ROP to analyze its effectiveness and recommend improvements. Over the past several years, the Government Accountability Office (GAO), the Office of Management and Budget (OMB), the Office of the Inspector General (OIG), the Advisory Committee for Reactor Safeguards (ACRS), the Davis-Besse Lessons Learned Task Force (DBLLTF), and the SDP Task Group have all performed evaluations of the ROP. Early on, two evaluation panels performed detailed reviews of the pilot program and initial implementation of the ROP. These evaluations have generally provided favorable results, but have also suggested potential areas of improvement. Several recommendations from these independent evaluations have been or are being incorporated into the ROP as detailed in the annual self-assessments above and staff responses below.

Entity Final Report Staff Response/Program Improvements
OIG audit of Support to Resident Inspectors OIG-14-A-12  03/2014 OIG-14-A-12, Appendix C
Commission-Directed Independent Review of ROP per SRM to SECY-12-0081 Working Group Report  02/2014 SECY-14-0047, Enclosure 2
OIG audit of Active Component Aging OIG-14-A-02  10/2013 OIG-14-A-02, Appendix B
GAO report on Regional Differences and Access to Information GAO-13-743 09/2013 Response  12/2013
Independent Evaluation of the Reactor Oversight and Incident Response Program 12/31/2008 Action Plan 6/09
NRC Cross-cutting review and Palo Verde Lessons Learned 12/2008 Draft dispositions current as of April 17, 2008
GAO report on Nuclear Reactor Oversight GAO-06-1029 09/2006 Response 11/06
OIG audit of the Reactor Program System OIG-05-A-11 04/2005 Status 11/05 (closes items 5 and 10)
Status 02/06 (closes remaining items)
OIG audit of the Baseline Inspection Program OIG-05-A-06 12/2004 Status 09/05 (closes items 1, 5, 6, 9, and 10)
Status 12/05 (closes items 7 and 8)
Status 06/06 (closes items 3 and 4)
OIG audit of the Significance Determination Process OIG-02-A-15 08/2002 Status updates in annual SECYs
SDP Task Group Task Group Report 12/2002 Status updates in annual SECYs
Advisory Committee for Reactor Safeguards Latest Letter 03/2003 Latest response 04/2003
Davis-Besse Lessons Learned Task Force DBLLTF final report 09/2002 Final Report
Significant Improvements
OMB Program Assessment Rating Tool Evaluation Final Report & Status of Performance Measures and Follow-up Actions 09/2003 Final Report & Status of Performance Measures and Follow-up Actions
Initial Implementation Evaluation Panel IIEP final report 05/2001 SECY-01-0014, Attachment 5
Pilot Program Evaluation Panel PPEP final report 12/1999 SECY-00-0049, Attachment 3

Stakeholder Feedback

External Surveys/Feedback

The staff biennially publishes a survey in a Federal Register notice to obtain external stakeholder input regarding ROP effectiveness. The frequency of the survey changed from annually to biennially in CY 2007. The surveys request responses to questions corresponding to specific ROP performance metrics, as defined in IMC 0307, and also provide an opportunity for open comments on the ROP that may not be directly captured by the specific questions. In an effort to actively solicit feedback on the implementation of the ROP, the staff (1) mails surveys directly to hundreds of stakeholders, (2) places a direct link to the survey information on both the ROP Web page and the "Documents for Comment" page of the NRC Web site, and (3) issues a press release and posted it on the NRC Web site. Since 2002, the NRC acknowledged receipt of each survey response by correspondence indicating that the staff has considered and generally addressed the comments during the annual ROP self-assessment, as appropriate. However, the NRC received feedback in the CY 2004 survey indicating that it was unresponsive to survey comments and decided to consolidate the comments by question and provide a comprehensive response to each question in addition to the acknowledgment letters. The staff received positive feedback on the consolidated response from several stakeholders in the CY 2005 survey, and plans to continue this practice.

External feedback is also solicited through monthly public meetings, special meetings/workshops to discuss specific issues, public meetings in the vicinity of each operating reactor to discuss the results of the NRC's annual assessment of the licensee's performance, and at the annual Regulatory Information Conference.

  Solicitation Date Responses Received Staff Evaluation Staff Response
CY2013/2012 Survey not issued for CY 2013* N/A SECY-14-0047 (discussion of survey suspension) N/A
CY2011/2010 FRN 11/29/2011 15 SECY-12-0055 Consolidated Response
CY2009/2008 FRN 09/25/2009 5 SECY-10-0042 Consolidated Response
CY2007 FRN 10/11/2007 7 SECY-08-0046, Enclosure 3 Consolidated Response
CY2006 FRN 10/10/2006 16 SECY-07-0069, Enclosure 3 Consolidated Response
CY2005 FRN 10/21/2005 21 SECY-06-0074, Enclosure 6 Consolidated Response
CY2004 FRN 10/25/2004 21 SECY-05-0070, Attachment 6 Consolidated Response
CY2003 FRN 11/05/2003 18 SECY-04-0053, Attachment 4 Letters
CY2002 FRN 11/22/2002 19 SECY-03-0062, Attachment 4 Letters
CY2001 FRN 11/21/2001 13 SECY-02-0062, Attachment 5 N/A
First Year FRN 01/10/2001 14 SECY-01-0014, Attachment 2 N/A
Pilot Program FRN 07/26/1999 21 SECY-00-0049, Attachment 1 N/A
Pre-ROP #2 FRN 01/22/1999 28 SECY-99-007A, Attachment 1 N/A
Pre-ROP #1 FRN 26 SECY-99-007, Attachment 7 N/A

*As discussed in SECY-14-0047, the staff has noted in the past several self-assessments that the level of participation and the number of new insights from the ROP surveys have been limited. Further, the staff has noted its intent to explore ways to improve or replace the survey tool to improve objectivity in the measurement of ROP performance and minimize the reliance on more subjective measures such as stakeholder perception. In addition, the staff has experienced challenges in obtaining the necessary clearance from the Office of Management and Budget to issue the survey to external stakeholders. As a result, the staff did not issue an external survey in CY 2013 and has suspended the use of surveys to assess ROP effectiveness. Those ROP performance metrics that are evaluated based on the survey results have been characterized as not applicable for the CY 2013 self-assessment and will be removed from or replaced in the self-assessment process guidance going forward. The staff discussed its intent to discontinue the survey with both external and internal stakeholders, and stakeholders cited both the Commission-directed independent review and the ROP enhancement project as significant opportunities for stakeholder engagement and feedback in CY 2013. The staff continues to value and encourage stakeholder feedback on potential improvements to the ROP and is exploring alternate avenues for obtaining that feedback.

Internal Surveys/Feedback

Internal surveys are conducted on a biennial basis to solicit and analyze stakeholder feedback from NRC staff regarding the effectiveness of the ROP. All respondent replies are anonymous and each specific question has five possible answers/ratings. Many of the questions correspond to specific ROP performance metrics as defined in IMC 0307. The final each section of the survey provides space to expound or make additional comments. There have been four internal surveys to date as noted in the following table.

In addition, Office of Nuclear Reactor Regulation (NRR) staff conducts biweekly conference calls with regional division- and branch-level management to discuss current issues associated with the ROP, and meets periodically with regional managers to discuss more complex ROP topics and issues. NRR staff also participates in each regions’ resident counterparts meeting and makes site visits to give regional staff and management the opportunity to discuss ROP implementation and provide feedback. The internal feedback process, as described in Inspection Manual Chapter (IMC) 0801, “Reactor Oversight Process Feedback Program,” also provides a useful means for the NRC staff to identify concerns or issues and recommend improvements related to ROP implementation.

  Solicitation Date Responses Received Staff Evaluation/Response*
CY 2012 & 2011 11/2012 128 SECY-13-0037
CY 2010 & 2009 11/2010 168 SECY-11-0054
CY 2008 & 2007 11/2008 159 SECY-09-0054, Enclosure 2
CY 2006 & 2005 11/2006 266 SECY-07-0069, Enclosure 3
CY 2004 & 2003 11/2004 209 SECY-05-0070, Attachment 6
CY 2002 & 2001 12/2002 236 SECY-03-0062, Attachment 5
First Year 03/2001 234 SECY-01-0014, Attachment 1
Pilot Program 11/1999 102 SECY-00-0049, Attachment 1

*Also evaluated in annual metric report, as applicable

Page Last Reviewed/Updated Thursday, March 25, 2021