Revised Reactor Oversight Process

March 15, 2000

The Honorable Richard A. Meserve
Chairman
U.S. Nuclear Regulatory Commission
Washington, D.C. 20555-0001

Dear Chairman Meserve:

SUBJECT:  REVISED REACTOR OVERSIGHT PROCESS

During the 469^th and 470^th meetings of the Advisory Committee on Reactor Safeguards, February 3-5 and March 1-4, 2000, we discussed technical aspects of the revised reactor oversight process, including the technical adequacy of current and proposed performance indicators (PIs) and the significance determination process (SDP).

This report responds to the Commission request in the December 17, 1999 Staff Requirements Memorandum, that the ACRS evaluate the extent to which the PIs, collectively, provide meaningful insights into those areas of plant operations that are most important to safety. Our Subcommittee on Plant Operations met on January 20, 2000, to discuss these matters. We also had the benefit of the documents referenced.

Conclusions and Recommendations

1.	The Revised Reactor Oversight Process (RROP) makes NRC assessments and actions more objective, predictable, and understandable to both the public and industry.
2.	Although the RROP is a work in progress, it is ready for initial implementation at all power reactors. Further adjustments in the process may be needed as more experience is gained with a larger base of plants. Because changes are expected after the initial implementation, staff should look for methods to implement the process in ways that it can be easily changed.
3.	The choices of the PIs and the associated thresholds remain controversial. Alternative views of ACRS members regarding the choice of thresholds are offered in the discussion.
4.	The SDP is incomplete. Further development of this process and the analytical tools it uses is required for full implementation.
5.	Additional PIs will be needed for full and effective implementation of the RROP. In particular, PIs are needed to characterize the licensee's problem identification and corrective action program (CAP), human performance, safety culture, and low-power and shutdown operations.

Discussion

The RROP pilot program was completed in November 1999 and lessons learned have resulted in changes that have improved the process prior to its initial implementation at all power reactors. The process is intended to ensure that plants continue to perform at an acceptable level and to provide early warning of adverse trends.

We recognize that the RROP is a work in progress and that certain aspects could not be fully exercised and evaluated during the 6-month pilot program. We agree that the overall process, the concept of the cornerstones, and the associated framework are sound. The new process will make NRC assessments more objective, predictable, and understandable to both the public and industry and should be approved for initial implementation at all plants. The staff has stated that continued development and implementation of the process will not adversely affect initial implementation. The staff plans to assess the effectiveness of the entire process after the first year of initial implementation.

The staff has selected a set of PIs to be used as part of the RROP, which is intended to be risk informed and performance based. The PIs are defined in the expectation that they are correlated with risk, even though in some cases the implied correlation cannot be explicitly defined or quantified. Without such an explicit connection to risk, it is difficult to determine which and how many PIs are sufficient or to determine quantitative threshold values. An added practical constraint to the selection of a set of PIs is the limited ability of the staff to obtain data from the licensees.

Recognizing that there are unavoidable limitations in the chosen set of PIs, the staff has developed a baseline inspection program for each cornerstone to complement and supplement the PIs. We agree with the staff that the technical adequacy of the proposed PIs should be evaluated in the context of the overall assessment process.

Another key element of the RROP is the licensee's problem identification and CAP. A basic tenet of the RROP is that the licensee's CAP should be relied upon to correct issues that do not result in crossing safety performance thresholds. This is based on the assumption that the improved overall industry performance over the past 10 years has demonstrated the general robustness of the CAPs. Confirmation of this assumption for individual plants requires that NRC periodically assess the effectiveness of each CAP as part of the baseline inspection program.

We believe that additional PIs will be needed for full and effective implementation of the program. In particular, PIs are needed to characterize the licensee's problem identification and CAP, human performance, safety culture, and low-power and shutdown operations.

The proposed green-white PI thresholds have been selected as the 95^th percentile of the values for the whole population of operating plants. Some ACRS members believe that this approach has led to the selection of PI thresholds that are too high to provide early warning of adverse trends in performance. The proposed values are such that most indicators will always be in the green, therefore, the PIs may not contribute meaningful information to the oversight process. Because performance in the green may be interpreted as good performance, there will be a reduced incentive for improved performance by the licensees.

Some ACRS members find the staff's approach to the selection of the green-white thresholds acceptable. Current industry practices and regulatory requirements, along with the previous inspection and oversight process, have resulted in acceptable overall industry performance. Therefore, the set of current values for the PIs does represent the range of acceptable performance values, and the 95^th percentile values are to identify outliers. Obviously there is some degree of arbitrariness involved, but it is an acceptable choice for initial implementation.

Some ACRS members believe that there is a fundamental flaw with the process of selecting the PI thresholds. As noted in our report dated June 10, 1999, a lesson from the probabilistic risk assessments and Individual Plant Examinations is that the risk profile of each plant is unique. The PIs and the thresholds should reflect this finding and should be plant specific. This means that the threshold for a specific PI should be selected from a distribution of values that reflects past performance with respect to this PI at that plant. A typical value that is usually selected is the 95^th percentile of this plant-specific curve. The current process, however, selects the thresholds from distributions that include plant-to-plant variability. A plant-to-plant variability curve represents the distribution of the past values of a PI across all plants. The selection of the 95^th percentile of these distributions could have two significant consequences. First, the thresholds are too high for the plants whose past performance placed them below the chosen threshold value. Second, the few plants with past performance above the selected threshold value may be in the "white" category without credit for other compensating features. This situation would create pressure on those licensees to "improve" their performance with respect to the PI, thereby ratcheting up the expected performance of the plant.

The same ACRS members believe that the establishment of plant-specific thresholds is feasible. The staff has agreed that, ideally, plant-specific thresholds would be desirable, but that they cannot be established at this time. An example of such an exercise, however, was the implementation of the maintenance rule and the proposed plant-specific performance criteria by the licensees. The staff has collected and published plant-specific data, including those from studies by the former Office for Analysis and Evaluation of Operational Data, e.g., NUREG/CR-5500, Volumes 4-8, and associated updates. Alternatively, it may be possible to identify groups of plants with similar design and operational characteristics that could share the same PI threshold values.

Some ACRS members are concerned that the high PI thresholds focus on equipment performance only. The staff has stated that cross-cutting issues involving human performance and safety culture will manifest themselves through the PIs or the baseline inspections. The baseline inspections may lag adverse human performance trends and not trigger action until some PI thresholds are exceeded. PI thresholds do not appear to provide timely warning of negative trends.

The SDP is designed to provide guidance for the risk characterization of inspection program findings so that the overall licensee performance assessment process can compare and evaluate the findings on a significance scale similar to that established for PIs. The SDP is still incomplete. Findings from workshops and lessons learned on the pilot program have not been accounted for in the SDP. Because of limitations in the staff's analytical tools, very approximate risk assessment methods are used for some SDP evaluations.

It is expected that the overwhelming majority of SDP findings will be "green." We are concerned that such an outcome could mask programmatic problems. For example, weakness in a maintenance program that was manifested by the failure of an unimportant component would result in a "green" finding, but the same programmatic weakness could result in the failure of a safety-significant component. The staff recognizes the potential problem but believes that such programmatic weakness will be reflected in the PIs or identified through inspection of the problem identification and CAP. More experience with the process is needed to validate this assumption.

Notwithstanding these concerns, we believe that the staff has developed a comprehensive oversight process, which is a significant improvement over the previous one. The staff's request to proceed with initial implementation should be approved, recognizing that changes will be made to the RROP, including the SDP; that research should continue to identify better choices for PIs and associated thresholds; that the current PIs are limited in scope; and that any reduction in the baseline inspection effort will require more realistic PIs.

Once the RROP has been implemented, substantial resistance may arise toward any changes. Because changes are expected after the initial implementation, staff should look for methods to implement the process in ways that it can be easily changed.

Sincerely, /s/ Dana A. Powers Chairman

References:

1.	Memorandum dated February 24, 2000, from William D. Travers, Executive Director for Operations, NRC, for The Commissioners, Subject: SECY-00-0049, Results of the Revised Reactor Oversight Pilot Program.
2.	Memorandum dated December 17, 1999, from Annette L. Vietti-Cook, Secretary, NRC, to John T. Larkins, ACRS, Subject: Staff Requirements - Meeting with Advisory Committee on Reactor Safeguards, November 4, 1999.
3.	Nuclear Energy Institute, NEI 99-02, Draft Revision D, "Regulatory Assessment Performance Indicator Guideline," November 1999.
4.	Letter dated November 23, 1999, from Samuel J. Collins, Director, Office of Nuclear Reactor Regulation, NRC, to Dana A. Powers, Chairman, ACRS, Subject: Advisory Committee on Reactor Safeguards Review of Revised Reactor Oversight Process Technical Components.
5.	Memorandum dated June 18, 1999, from Annette Vietti-Cook, Secretary, NRC, to William D. Travers, Executive Director for Operations, NRC, Subject: Staff Requirements - SECY-99-007 - Recommendations for Reactor Oversight Process Improvements, and SECY-99-007A - Recommendations for Reactor Oversight Process Improvements (Follow-up to SECY-99-007).
6.	Letter dated June 10, 1999, from Dana A. Powers, Chairman, ACRS, to William D. Travers, Executive Director for Operations, NRC, Subject: Pilot Application of the Revised Inspection and Assessment Programs, Risk-Based Performance Indicators, and Performance-Based Regulatory Initiatives and Related Matters.
7.	Report dated February 23, 1999, from Dana A. Powers, Chairman, ACRS, to Shirley Ann Jackson, Chairman, NRC, Subject: Proposed Improvements to the NRC Inspection and Assessment Programs.
8.	U. S. Nuclear Regulatory Commission, NUREG/CR-5500, Vol. 4, "Reliability Study: High-Pressure Coolant Injection (HPCI) System, 1987-1993, September 1999.
9.	U. S. Nuclear Regulatory Commission, NUREG/CR-5500, Vol. 5, "Reliability Study: Emergency Diesel Generator Power System, 1987-1993, September 1999.
10.	U. S. Nuclear Regulatory Commission, NUREG/CR-5500, Vol. 6, "Reliability Study: Isolation Condenser System, 1987-1993," September 1999.
11.	U. S. Nuclear Regulatory Commission, NUREG/CR-5500, Vol. 7, "Reliability Study: Reactor Core Isolation Cooling System, 1987-1993," September 1999.
12.	U. S. Nuclear Regulatory Commission, NUREG/CR-5500, Vol. 8, "Reliability Study: High Pressure Core Spray (HPCS) System, 1987-1993," September 1999.
13.	U. S. Nuclear Regulatory Commission, NUREG/CR-xxx Vol. X, "Reliability Study Update: High-Pressure Coolant Injection (HPCI) System, 1987-1998" (Draft), October 1999.
14.	U. S. Nuclear Regulatory Commission, NUREG/CR-xxx, Vol. X, "Reliability Study Update: Reactor Core isolation Cooling (RCIC) System, 1987-1998" (Draft), October 1999.
15.	U. S. Nuclear Regulatory Commission, NUREG/CR-xxx Vol. x, "Reliability Study Update: High-Pressure Coolant Injection (HPCI) System, 1987-1998" (Draft), October 1999.