Issues Associated With Industry-Developed Thermal-Hydraulic Codes

January 11, 2001

The Honorable Richard A. Meserve
Chairman
U.S. Nuclear Regulatory Commission
Washington, D. C. 20555-0001

Dear Chairman Meserve:

SUBJECT: ISSUES ASSOCIATED WITH INDUSTRY-DEVELOPED THERMAL-HYDRAULIC CODES

We are responding to the Staff Requirements Memorandum dated November 7, 2000 [Reference 1], in which the Commission requested the ACRS to provide a more detailed discussion on how the perceived weaknesses with industry-developed thermal-hydraulic codes may adversely affect the NRC's regulatory role and to provide more specific recommendations on how these weaknesses should be addressed.

In the Background Section we explain why thermal-hydraulic codes are of high topical interest to the ACRS and the Commission. We then respond to the Commission's question about the effects of thermal-hydraulic codes on NRC's regulatory role. Finally, we list several specific recommendations which are cross-referenced to a discussion in the appendix. The bases for the recommendations are also included in the appendix along with specific items on slides cited by the Commission in its SRM [Reference 2].

Background

Thermal-hydraulic codes have been used for decades. Most of these codes have included deliberate conservatisms or bounding assumptions to address uncertainties in the models. The codes have proven to be adequate to satisfy regulatory requirements, when used with appropriate conservatism and judgment and when extensively examined by the staff. The efforts by the industry to improve the performance of its plants (by power uprates for example) and the efforts to reduce unnecessary burden pose new challenges to the use of these codes. It is no longer sufficient for a code to make conservative predictions; there is an increasing need for "realistic" predictions. In the case of new reactor designs, a lack of confidence in the proper application of first principles may require additional testing and benchmarking to establish regulatory confidence.

The replacement of conservative codes by realistic (best-estimate) codes raises a number of questions such as:

• How is the degree of realism to be measured?
• How do approximations, models, and assumptions affect the results?

• How are these approximations related to measures of realism such as bias and uncertainty?

• What other qualities, such as completeness and flexibility, are desirable in these codes?

When codes are relied upon to predict success criteria for risk analysis and to justify changes in regulations, further questions arise such as:

• Is the code giving the wrong answer and what is the impact of the result?

• If conservatism is reduced, what measure of confidence in code predictions is needed to ensure that adequate safety margins are preserved?

• Should more rigorous comparisons with data (assessments) be required? How much do these comparisons depend on the nature of the regulatory decision to be made?

• What is the effect of model uncertainty in the thermal-hydraulic codes on conclusions to be drawn from PRAs?

The Commission needs to be assured that the staff can respond to these sorts of questions and has the insight to pose others that may be important. The agency also needs to have a good perspective of the capabilities and limitations of codes and their modes of use so that it is on sure ground as it makes decisions.

We have recently been involved in reviewing realistic codes submitted by developers and these reviews are continuing. We have also reviewed a new Draft Regulatory Guide (DG-1096) [Reference 3] and Standard Review Plan Section [Reference 4]. We have learned some lessons and made some observations that were the background for our presentation to the Commission on October 6, 2000. At that time, we did not make specific recommendations. The recommendations that we now make are mostly for staff actions in response to the situation as we see it today. The final recommendation addresses the changing regulatory environment in a more general way, but is perhaps the most important of all.

Before presenting our recommendations, we address the question of how the perceived weaknesses of the thermal-hydraulic codes may affect the agency's regulatory role.

Effects of Codes on NRC's Regulatory Role

Traditionally, codes have been used to predict the behavior of plants under design basis and beyond-design-basis conditions. Because of the limitations of these codes, there is a probability that their predictions may be wrong; in other words, a better code would lead to a different answer to a safety question. The traditional approach has been to account for uncertainty through conservatism and additional safety margins, thus giving added assurance that the prediction of success is valid.

In contrast to the traditional approach, risk-informed regulation is based on realistic analysis and seeks to avoid undue conservatism. Codes are used to evaluate success criteria for risk analysis. With this approach, the uncertainties must be explicitly taken into account to provide a desired level of confidence in the results. Similarly, power uprates will depend on the ability of more realistic codes to justify the reduction in conservatisms required to provide the necessary margin.

Because the theoretical basis of two-phase flow thermal-hydraulic codes is incomplete, significant uncertainties in predictions are expected. They can be reduced by developing better models, and better quantified by more extensive experiments. If the theoretical basis is weak, more experiments are needed to support empirical correlations and to establish the uncertainties.

The key question is: "What quality of codes is needed to support regulatory decisions?" The rational measures of "quality" are the uncertainties in the code predictions and the influence these uncertainties have on decisions.

Our concern is that codes may not have sufficient quality to support decisions that the Commission has to make. More specifically, poor quality codes may restrict the degree to which the regulations may confidently be risk informed and the extent to which conservatism may be reduced.

The use of thermal-hydraulic codes affects the performance goals of the agency in the following ways:

• Maintain Safety

If uncertainties in code predictions are not adequately understood and addressed, safety may be (unwittingly) compromised.

• Increase Public Confidence

Excessive uncertainties, errors, and unjustified assumptions in codes reduce public confidence, and more importantly the confidence of the informed technical community, including the NRC staff and the users of the code.

• Increase Efficiency and Effectiveness

Weaknesses in code documentation and validation lead to lengthy negotiations between the staff and the applicant and substantially increase the time and effort for decisionmaking.

Poor theory leads to requirements for more extensive experimental evidence.

• Reduce Unnecessary Burden

Large uncertainties in code predictions lead to conservative decisionmaking in order to maintain assurance of safety. This imposes burdens that better predictions might show to be unnecessary.

Recommendations

(The item numbers in the appendix to which these recommendations respond are given in parentheses.)

To ensure that the codes can meet present and anticipated standards of quality:

1.	The staff should make clear that standards to be applied to documentation of proprietary codes are the same as for codes generally accessible to public scrutiny (Items 2, 10, 11, 14).
2.	The staff must continue to require that vendors and licensees supply working versions of the codes for internal NRC use and evaluation (Items 6, 7, 12, 18).
3.	The staff should recommend how improvements can be more readily incorporated into codes (Items 1, 20, 21).

To promote more effective evaluation of the codes:

4.	The staff should continue developing its own thermal-hydraulic code, making it more reliable, flexible, and easy to use (Items 1, 9, 17, 19).
5.	The staff should examine ways in which the process of evaluation and assessment of proprietary codes can be made more publicly accessible and scrutable (Items 2, 11, 12, 14, 19).
6.	The staff, perhaps in cooperation with an industry-supported entity such as the Nuclear Energy Institute (NEI), should undertake an authoritative study assessing when, how, and why codes produce reasonable results despite numerous assumptions and simplifications. This study should include measures of code strengths and weaknesses and include an assessment of circumstances under which the shortcomings of the codes may have significant influence on regulatory outcomes (Items 1, 3, 4, 5, 19).
7.	The staff should take steps to ensure that the existing data base for thermal-hydraulic code evaluation is preserved in accessible form (Items 4, 8, 12, 16).

To ensure that the codes meet anticipated regulatory requirements:

8.	The staff should consider how definite measures of code quality, such as bias and uncertainty in predicting significant phenomena and success criteria, can be more specifically required as outputs from the code assessment process (Items 8, 12).
9.	The staff should investigate and recommend how uncertainties in code predictions can be best quantified to be suitable for incorporation into risk-informed regulation (Items 8, 12, 13, 15).
10.	The staff should reevaluate the design specifications for the outputs of codes and their relationship to present and anticipated regulatory requirements (Item 6, 7, 8, 13, 22).

Sincerely, /RA/ Dana A. Powers Chairman

References:

1.	Memorandum dated November 7, 2000, from A. L. Vietti-Cook, Secretary, to J. T. Larkins, ACRS, Subject: Staff Requirements, Meeting with Advisory Committee on Reactor Safeguards, October 6, 2000, Commissioners' Conference Room, One White Flint North, Rockville, Maryland, (M001006B).
2.	Slide presentation during ACRS Meeting with NRC Commissioners on October 6, 2000, "More Realistic Thermal-Hydraulic Codes."
3.	U. S. Nuclear Regulatory Commission, Draft Regulatory Guide, DG-1096, "Transient and Accident Analysis Methods," dated July 18, 2000.
4.	U.S. Nuclear Regulatory Commission, Standard Review Plan Section 15.0.1 (subsequently changed to 15.0.2), "Review of Analytical Computer Codes," dated April 14, 2000.

Attachment: Appendix

---

Appendix

Discussion of the specific points raised in the ACRS slides addressing issues associated with industry-developed thermal-hydraulic codes is provided below.

1. Many codes have the same ancestry, including a 30-year old foundation

Several codes are derivatives of the RELAP series, going back to developments at Aerojet Nuclear Corporation (ANC, now INEEL) in the late 1960s, with influence from the Bettis FLASH code. Several assumptions were made at a fundamental level in order to produce equations that could be solved with the computer technology of the time. These assumptions introduced errors and uncertainties that have not been quantified for general purposes, though for some applications the results appear to be adequate. For example, the staff responded [Reference 5] in 1972 to a "controversy concerning the momentum equation" by saying:" We find that the relatively few assumptions made by ANC (in the RELAP3 code) can all be justified in LOCA analysis. There may be further simplifying assumptions which could be justified. However, there is no adequate basis now available to us for judging the adequacy of the assumptions implicit in the momentum flux representations used in the four vendor evaluation models."

In the Water Reactor Evaluation Model report [Reference 6], the staff lists several options of the momentum equation, each of which is unrealistic to some degree, as a basis for modeling real reactor components. Dr. Novak Zuber, an ACRS consultant who was an AEC staff member at the time, informs us [Reference 7] that in 1974 he reviewed RELAP4 and concluded that its momentum equation was incorrect. It was not corrected at the time because the effect on peak clad temperature for a large-break LOCA appeared to be small. Emphasis in the subsequent decades seems to have been on getting codes to run and introducing correlations to describe the details of the phenomena without reviewing the basic equations and their limitations.

Some of the original controversial, or at least approximate, equations are still in use, sometimes expanded or rederived by unconvincing methods. Some of the examples used to illustrate them appear to be incorrect, decreasing the confidence of a reviewer in the approach used.

The point here is that there is a long history of decisions having been made to accept what was perhaps the best available or "adequate" model, despite what appear to be fundamental errors in the basic formulation; the consequences, as the uses of codes expanded, were not well understood.

One consequence of this common ancestry is that if the NRC code has the same basic assumptions as a vendor's code, and no attempt is made to investigate alternatives, then a dimension of "independence" may be sacrificed.

2. Designed specifically for nuclear applications. Not commercial or academic.

Computer methods have evolved rapidly in recent decades in all branches of science and engineering. An example is computational fluid dynamics (CFD) codes that are now widely used by industry and universities. Feedback from users and continual upgrading by code vendors spurs development, corrects faults or limitations, and improves capabilities and accuracy. Since the basis of these codes is transparent, or is available to the more sophisticated user, errors and shortcomings are discovered and improvements are made at many levels of detail.

In contrast, nuclear codes, particularly those that are treated as proprietary, have not been subject to such public scrutiny. The industry has been reluctant to improve the codes, even when this is desirable, perhaps because of the requirements for regulatory review and the cost of licensing changes.

3. Contain many assumptions, idealizations, "best-shot" estimates, and user choices.

Code documentation is usually presented so that the derivations appear to follow a logical thread based on sound fundamentals. Despite increases in computing power, it is still necessary to introduce assumptions. Sometimes assumptions are explained, but often they are not justified by further rationale. Other assumptions may be inherent in the derivations but are not acknowledged and are only apparent if a reviewer makes the effort to understand what is being done. It may then be discovered that the derivations have a much less authoritative base than is implied by the text.

For example, most codes make use of a flow regime map defining how the two-phase flow pattern changes from "bubbly" to "annular" (a liquid film on the wall) to dispersed droplet regimes as the fraction of vapor increases.

These flow regimes are dependent on the geometry of the devices through which the fluids flow. For instance, they are different for bends and inclined pipes. They are also considerably influenced by conditions upstream and downstream of the component under consideration. For example, if fluid is introduced into a pipe along the wall, it may take a considerable distance before it is entrained into an established droplet pattern.

Even the many flow regime maps available for two-phase flows in long straight pipes are often not compatible with each other, and data can always be found that disagree with the best versions.

There is little record that flow regimes have actually been measured and the results compared with predictions for reactor system components such as rod bundles, downcomers, or the upper and lower plenum of a reactor vessel. The sensitivity of predictions to the choice of flow regime boundaries is usually not investigated. The choice of a simple universal flow regime map as the basis for a code represents a huge simplification that can only be justified by arguments that it "works" well enough for the purpose of predicting some overall parameter of interest under some specific circumstances. Under other circumstances, such as a design change or a new concern (such as boron dilution) in which other regimes could play a significant role, the code must again be shown to "work" or be modified.

Even after a flow regime is selected, additional simplifications are necessary. Often these simplifications are then applied in a questionable manner. Derivations may be made for "one-dimensional flow" in a straight pipe and then applied to bends or other reactor components in ways that are not explained or justified. Correlations of wall friction factors or forces between the vapor and liquid that were developed as very rough approximations for air-water flows in straight pipes are adopted to represent steam-water flows at high pressure in a host of different geometries. Sometimes the user is given a choice of correlations without clear guidance about which one to select, and this introduces a "user effect" whereby different answers can be obtained to the same problem.

When no precedent exists for modeling a component or situation, the code developer may make an estimate based on an idealized view of what is happening. Without independent experimental validation of the individual model, the reviewer cannot tell how reasonable the estimate is except in the context of the code's apparent success at a global level, which could be the result of compensating errors.

We do not mean to imply a need for excessive detail, if this can be shown to be inappropriate. Professional judgment must guide choices among imperfect, but adequate, analytical models. Engineers who need to get results with limited knowledge, resources, and time must often make such choices. The point is that it is unwise to rely on the resulting structure without a broad base of methodical assessment against the full range of conditions for which the code will be applied. It also means that restrictions must usually be placed on the use of a code in unevaluated situations, or for new or modified designs for which a sufficient basis for assessment does not exist, or to answer questions for which greater accuracy is required than has been demonstrated. These characteristics of codes demand detailed staff review, supported by knowledge, persistence and sufficient resources.

4. Codes have evolved, but the development process is hard to trace.

The codes have been developed over the years by various authors, many of whom are no longer active. The rationale for the features and assumptions in the codes, their advantages and limitations, and why a certain approach was chosen may have been forgotten. When the rationale is lost, the present generation of code developers, users, and reviewers has to rediscover and reevaluate the technical bases of these features. More detailed explanations and justifications in the original documentation might avoid the need for this extra work. This situation supports the need for the staff to have its own code and to maintain a clear record of why design choices were made in its development.

5. Codes may "work" but they are not based on a mature, secure science.

We discussed some of the rationale for this statement under Item 3. The science of multiphase flow and heat transfer has not reached a point where predictions can be made solely from a basis of secure fundamentals (as they can for many viscous single-phase flows, for example). Codes have evolved as an elaborate tapestry of interwoven working assumptions and approximate equations and correlations that have proved to be useful. Longevity of these engineering methods is no assurance of maturity, nor does it guarantee that the codes need no further development and improvement as new questions arise. Nevertheless, a series of demonstrations of applicability to reactor transients has established confidence in their utility as one input to regulatory decisionmaking, as long as their limitations are adequately understood and evaluated.

6, 7. Pro: For several accidents (e.g., small-break LOCA) a few phenomena appear to dominate and the calculated output figures of merit, such as peak clad temperature, appear insensitive to the details. Con: Some phenomena that could be important are poorly modeled (e.g. two-phase level in core boil-off).

We present the following examples to make the point that the precision of code predictions, and their sensitivity to details, depend on the question being asked. After the water level drops during the initial transient, small-break LOCA becomes "a pot of boiling water with steam escaping from a small hole." A relatively simple model may be quite successful in representing the main phenomena. The steam release rate is given by an energy balance for the vessel and the system pressure is governed by single-phase steam flow out of the break. On the other hand, the swelling of the two-phase mixture in the core and the upper plenum is a complex phenomenon. In the analysis of one experiment, for example, it could only be represented accurately by changing the interphase friction factor by a factor of five from the value typically used in the code chosen for the analysis. Such discrepancies are not unusual when dealing with two-phase phenomena. The net effect on peak clad temperature was smaller than this discrepancy suggests, but the result could have a considerable impact if the success criterion was that the core be completely covered by the boiling pool. We note that boilup and steam cooling are considerations in licensee proposals for changes to the current licensing basis. For example, boron mixing and the recent proposal to use low pressure systems as an alternative strategy in response to a fire. These examples point out the importance of the staff having the wisdom and experience to know when to ask for more information and when to accept a calculation that is offered. This decision may involve input from the research branch of the agency.

8. Code predictions have to be assessed for each application and extensive sensitivity checks performed.

Because of the uncertainties already mentioned, a code cannot be approved a priori for all applications. The staff must be satisfied that the range of data for which the code "works" covers the range that is needed to provide adequate confidence that the results can be extrapolated to predict the performance of a full-scale system during the relevant scenarios. If conservatism is to be reduced, better confidence in the code is needed, in the form of reduced uncertainty in its predictions. One of the ways to assess this uncertainty is by assigning distributions to the assumptions and parameters that go into the code and propagating these through the calculation of the desired result. Another way is through quantitative evaluation of the accuracy and uncertainties in the code by systematic statistical comparisons with data. We perceive a need for the staff to be more specific about what are acceptable methods of deriving and expressing the uncertainties in codes and how these methods are to be used in the regulatory context.

9. The staff's knowledge, experience, and thoroughness are key.

For all of the above reasons, decisions about the acceptability of a code and its range of validity depend heavily on the competence and thoroughness of the staff. Steps are currently being taken, through the preparation of a Regulatory Guide (DG-1096) and Standard Review Plan Section (15.0.2) to develop more explicit explanations of the expectations of the staff and criteria to be used for evaluating codes. This should reduce variability in the extent and depth of reviews. It should also help submitters to understand the standards to be used. We have commented on draft versions of these documents.

The Commission needs to be assured that the staff is sufficiently consistent and thorough in its reviews and has sufficient resources and capabilities to complete them.

10, 11. Some code documentation is poor. The physical basis for analytical models is often incomplete and poorly explained.

The above, more general observations that we cited under Item 3, did not prepare the ACRS for what it found in some of the code documentation. Examples are given below. The specific submittals in which these shortcomings were observed will not be identified, but no code was without some of them.

• Basic equations, of the type to be found in standard textbooks, contained many major typographical errors. These should be immediately apparent to a knowledgeable observer.
• Equations were so garbled as to require lengthy rederivation by the reviewer.

• Simple algebraic errors appeared to be made in deducing one equation from another. The result was an equation which could not have been derived from the given starting point.

• Scalar and vector quantities appeared to be intermixed in inappropriate ways.

• Examples intended to illustrate the method being used actually served to discredit it.

• Terms in equations were insufficiently defined. Examples of the use of these equations suggested a lack of understanding of how the terms should be evaluated.

• General derivations were made that appeared incompatible with specific cases. Such derivations were usually unique to the particular code.

• The logic in developing coefficients or expressions for use in the code was unclear. Equations were written down without an explanation of how to use them or where the terms come from. A reviewer could not tell what process was actually being followed.

• Assumptions were made on an ad hoc basis without supporting evidence.

• Equations were derived for straight pipes with no indication of how they might apply to the real shapes found in actual systems.

Shortcomings of this sort would be much less likely if the codes were prepared and reviewed for open publication. The ACRS would prefer to see the same standards applied and the same quality achieved whether or not the code will be publicly available.

Despite these deficiencies, the codes have proven to be acceptable to satisfy regulatory requirements. However, an important aspect of the review process is the establishment of confidence in the approaches taken by the code developer. Weak documentation undermines this confidence.

12. Assessment is unfocussed and insufficiently extensive.

Assessment is the comparison of predictions with data to determine how well the code works. Because of the many assumptions and approximations made in the codes and the numerous empirical coefficients and correlations imported from other applications, it is important that comparisons be made with data from actual or scaled nuclear components. Some codes are presented for review with few, if any, such assessments.

Other code predictions are compared with a small selection of data. For instance, predictions are compared with some results from a Loss of Fluid Test (LOFT) facility test. It is not explained why this test was chosen and why predictions were compared with only a small group of data when other results were also available.

Code assessment should be a more complete and logical process. Sensitivity studies can reveal which parts of the code it is most important to evaluate. Comparisons can then be focused on those parts and a thorough evaluation made, covering the whole range of parameters expected in practice, with an emphasis on discovering limitations, rather than showing some limited general agreement. A logical road map should be provided, explaining what features of the code are to be assessed by a particular comparison and how they are being evaluated. For the purposes of risk-informed regulation, code assessment should result in quantitative measures of uncertainty so that the risk that the code will give an unacceptable answer can be determined.

Code assessment is presently one of the weakest links in the NRC's review process. Insufficient thought has been given to demonstrate acceptability, leaving too much up to the judgment of the NRC reviewer.

We believe that a logical assessment review process can be developed. It should start with identification of which features require assessment and to what degree. This might resemble a Phenomena Identification and Ranking Table (PIRT) process, but, unlike PIRT, it should be carried through to the end of the assessment review to determine not only what a code should do but how well it does it. A procedural framework should be set up explaining how specific features are to be assessed and by what measures. At present, in some assessments, there is only a tenuous association between the selected comparisons with data and specific questions to be answered.

The assessment process described in the Draft Regulatory Guide (DG-1096) is qualitative and leaves considerable latitude for interpreting how to measure success. This essential process would be considerably strengthened, without being unnecessarily prescriptive, if specific quantitative outputs were identified. For the purpose of regulation, including risk-informed considerations, the most significant of these are measures of bias and uncertainty in predicting significant phenomena and success criteria.

13. Methods for calculating uncertainties are primitive and not comprehensive.

The regulations require an assessment of uncertainty for all "realistic" codes. We have encountered a wide range of approaches to estimating uncertainties. At the crudest level, a predicted curve is shown to pass in the vicinity of some data points and agreement is characterized by terms such as "good" or "acceptable." This is not a basis for quantitative assessment of model uncertainty and its effect on safety margins.

Another approach is to vary key parameters in the code and determine the effect on the success criteria, such as peak clad temperature. This gives useful information about sensitivities of the code to these specific parameters. It does not capture additional uncertainties due to the overall structure of the code and the particular form of equations and correlations chosen. Moreover, without comparison with data, the actual likelihood that parameters will have specific values within the chosen range is unknown.

The most thorough approach we have seen to date methodically compares predictions with many data points to quantify bias and uncertainty in predicting specific phenomena. The uncertainties are then combined to quantify the likelihood of meeting success criteria in the regulations. This impressive achievement follows the intent of the Code Scaling, Applicability, and Uncertainty (CSAU) Evaluation methodology [Reference 8]. However, questions can be raised about the completeness of the implementation. For example, errors in void fraction, traced to uncertain forces between the steam and water phases, may be evaluated from data taken predominately in a large vertical duct or vessel. How does this apply to flows in more tortuous passages, as in the reactor core, in horizontal flows, or flows in bends such as the loop seal in a PWR primary circuit?

Because of the large variability in approaches to the evaluation of uncertainty, and the deficiencies in present methods, the staff should conduct a comprehensive study of ways to quantify and assess uncertainty, the measures to be employed, and their adequacy for meeting regulations. This will require innovative research to show how to evaluate uncertainties when formulating important equations and correlations so that the uncertainties are incorporated into the solution routine of the code itself.

In discussing the uncertainty methodology, draft Regulatory Guide DG-1096 refers to a previous Regulatory Guide, 1.157 [Reference 9]. Regulatory Guide 1.157 covers many features of realistic calculations, including the evaluation of uncertainty and bias, but its guidance is very qualitative and leaves considerable latitude in interpretation. Typical phrases seen are "performs adequately," "acceptable provided their technical basis is demonstrated with appropriate data and analysis," "uncertainties and biases in the parameter should be stated," and "sensitivity studies should be performed." The meaning of these requirements depends on how the code will be used in making regulatory decisions. Without a clear connection between the outputs from the code and their regulatory function, there is no basis for deciding when the modeling techniques are good enough. This is why we believe that the staff needs to clearly tie its safety evaluation findings to the specifications these codes need to satisfy, particularly to support risk-informed regulations.

14. Documentation should be acceptable to knowledgeable, impartial observers.

When a code is proprietary, the vendor and the NRC staff are usually the only people who have access to its documentation. The ACRS may be the one external group that provides a third-party review; it becomes the sole independent guarantor of public confidence. At the same time, the ACRS does not have the resources to investigate all the details in these codes.

As already described, we were surprised to find that some documentation had not been prepared to standards appropriate to inspiring confidence in impartial observers. There are various reasons for this situation, some historical and traceable to the way the regulatory process has functioned. Eventually, perhaps through published papers based on the work, or through a relaxation of proprietary sensitivity, the documentation will be seen by knowledgeable outsiders. Moreover, it is professionally demoralizing to both industrial engineers and to the NRC staff to discover uncorrected errors, some of which are obvious from a cursory review, in approved documentation. The staff should make clear that standards to be applied to documentation of proprietary codes are the same as for codes generally accessible to public scrutiny.

15. Risk-informed regulation will require more quantitative evaluation of model uncertainties and their consequences.

Shortcomings in the codes may have been allowable in the past when "engineering judgment" imposed appropriate conservatism and safety margins to compensate for lack of confidence in accurate predictions.

The basis of risk-informed regulation is different. Predictions are "realistic" but not exact. The probability of errors in code predictions must be assessed. Safety margins require a more precise definition so that the probability of exceeding them can be evaluated.

Just as in the case of PRAs, the quality of thermal-hydraulic codes must be compatible with the purposes for which they will be used to make decisions. The need for increased code quality measured by decreased uncertainty will grow, because requests for reduction in regulatory burden, power uprates, digital instrumentation and control, and the synergistic effect of such changes on aging plants may erode margins of safety.

16. The data base for assessment must be preserved and, in some cases, expanded.

The decisions that went into formulating deterministic regulations, such as Appendix K to 10 CFR Part 50, were confirmed to a great extent by the experience with specially designed experimental facilities such as LOFT.

The Westinghouse AP600 passive plant design differed substantially from previous designs. Confidence in its characteristics could not be obtained from theory alone but required tests in several facilities, notably the scaled APEX facility at Oregon State University (OSU).

As new questions arise or old questions become more critical, the data base is the arbiter. When the data base is inadequate new experiments are conducted, as in the thermal-hydraulic test program related to pressurized thermal shock being performed by the NRC at OSU.

We are concerned at reports that parts of the thermal-hydraulic data base, developed at considerable expense, that might be sufficient to answer questions raised by suggested changes in regulations or plant characteristics, are unavailable or have been lost for reasons such as changes in methods of computer storage. Steps should be taken to inventory and maintain this data base and ensure its preservation in usable form.

17. A base of experts needs to be maintained.

We consider three groups of experts to be important: the code developer, the NRC staff, and the experienced public professionals who may be called upon for independent review, consultation, or participation in expert panels.

Our experience with reviewing codes suggests that a very small group of developers is sufficiently knowledgeable to answer technical questions. Some of the original developers have left the industry. Perhaps the notion that codes were mature and needed little more attention has taken resources away to the point where some vendors may be insufficiently prepared for the effort required to upgrade codes to the "realistic" level. The NRC might consider conducting periodic audits of industrial support for codes of the type that were performed by the agency in the past. These audits focused on such important issues as code quality assurance and the adequacy of documentation.

The NRC needs two sets of experts on its staff. The first group reviews developers codes and runs them with sufficient insight and informed curiosity to make regulatory decisions. The second group, in the research category, makes independent assessments with NRC's own code, investigates "what if" scenarios, anticipates future uses that may stretch the capabilities of existing tools, and develops new capabilities when these are required. Our impression is that these teams are presently close to minimal strength, which may delay efficient processing of applications involving new codes.

The third group of experts has been well served by those who were active in developing thermal-hydraulic theory and codes in the 1960s and 1970s. Many of these people are now retired and may soon no longer be available. The NRC should encourage the development of a new generation of experts in this field.

18. Staff should run and evaluate vendor codes independently.

The former process by which the NRC staff evaluated code predictions submitted by the developer, had many limitations. Obtaining results other than those selected by the developer, exploring the effect of different assumptions, determining limitations on the range of variables or conditions for which the code performed well, and getting other information needed for a thorough assessment, was laborious and sometimes frustrating.

In the past two years the staff, with ACRS encouragement, has adopted a policy of obtaining working versions of these codes and running them. This is a much more efficient way of discovering the strengths and robustness of a code, as well as its limitations and weaknesses. It also allows the staff to check what could not be checked previously, that the code, as programmed, actually follows the documentation. This sort of openness vastly increases the effectiveness of the review process and should benefit both parties, as well as enhance public confidence.

19. Staff should maintain in-house code competence, including an NRC-developed code.

There is no standard code for thermal hydraulics. Each code has its individual characteristics, even peculiarities.

The ACRS strongly supports the current efforts by the NRC research staff to combine its present suite of thermal-hydraulic codes into one, and to improve the models, functions, flexibility and speed of operation. For the foreseeable future, there will exist manifold uncertainties about the effects of models in vendor codes, their limitations, phenomena that may be insufficiently explored and undiscovered implications. The NRC needs its own tool it is familiar with, can experiment with, and use to anticipate new questions and resolve them. The staff will then be able to accept or question features of the developers codes based on independent knowledge. It will also have a tool readily available for assessing new concerns, major plant events, and operating transients.

The informed technical community has long been aware of shortcomings in thermal-hydraulic codes, even though these codes have proven adequate to satisfy current regulatory requirements. A prime example is the various approximations and shortcuts that are needed to manipulate the momentum conservation equation into usable form.

With access to vendor codes and intimate knowledge of its own code, the staff is in the best position to obtain a reasoned perspective on this issue. With the enormous advances in speed of computation, including parallel processing and improvements in code architecture, it should be possible to run a set of carefully designed computer experiments to assess the practicality and limitations of existing methods. For example, the consequences of using alternative forms of the momentum equation, doubling or halving approximate terms, and including or neglecting various effects could be realistically assessed. The aim would be to produce an authoritative document that would enhance public confidence and provide a landmark source of reference in future decades. This could be a major contribution to the effectiveness and efficiency of the NRC review process and the preparation of documentation by industry.

A possible impediment to this sort of comprehensive evaluation is the industry wish to protect its proprietary codes. This may inhibit the public availability of independent assessments of technical details. Perhaps this impediment could be overcome by a cooperative effort between the staff and an industry-supported entity, such as NEI. There are recent precedents for this sort of cooperation.

20. Regulatory processes should encourage code improvements.

Several elements of codes are based on ideas that originated 30 or even 40 years ago. They were tentative at the time and it is remarkable that they have not been replaced or improved. We are not sure why this is. There are mechanisms for modifying codes, but industry must usually sense some benefit before doing so. Perhaps the regulatory process itself has an inhibiting effect on modifications, once approval has been obtained. We recommend that the staff suggest effective ways to encourage the development of code improvements. The measure of success would be a reduction in uncertainty in the output of the code and in the resulting conservatism in decisionmaking.

21. NRC should be preparing for an eventual new generation of thermal-hydraulic codes.

Computer methods now permeate society and are one of the most rapidly evolving technologies. Computational fluid mechanics, which was the subject of research a couple of decades ago, is now a flourishing industry. Continually evolving commercial codes are commonplace in the arsenal of industrial engineers. Grids involving over a million nodes are routinely used to evaluate air flows over aircraft and automobiles or over the internals in a vehicle engine compartment. The NRC has a small research effort to use these codes for single-phase flow problems relevant to safety.

Comparable multiphase industrial thermal-hydraulic codes have also been under development but they are not as mature or reliable as their single-phase parents. However, commercial needs are stimulating a rapid evolution. The NRC clearly needs to keep abreast of these developments and decide which commercial codes, or which features of them, might profitably be incorporated into nuclear safety assessment in the future. The NRC was a leader in code development two decades ago; it now needs to adjust to a technological climate in which the most significant developments originate outside the agency.

The rewards from use of a new generation of commercial codes would be greater confidence in the results and a more efficient review process. If a code is widely used, it should limit the necessity for each licensee to subject the code to broad-based assessment for each application. The NRC might eventually be able to give generic approval to codes, reducing the time needed to examine the details of every new application.

22. Specifications for codes

Finally, we explain Recommendation 10, that the design specifications for codes be reexamined. We cite two examples of future anticipated needs. The examples are not intended to be exclusive.

Past codes were used to evaluate design basis accidents, particularly LOCAs, as described in 10 CFR 50.46 (the ECCS Rule). If the ECCS Rule were to be risk-informed, what would be the appropriate output from codes in order to support this new regulation? What would be the functional requirements of these codes in terms of quantitative measures of accuracy and uncertainty that are now left vague? What measures would be required to demonstrate that the codes were "realistic"? How would model uncertainties in the codes be incorporated into risk estimates in order to make them more realistic?

The Commission is receiving requests for power uprates on the order of 20%. This significant increase in core power must have some impact on safety. Can present codes quantify this impact? What maximum power uprate is tolerable and on the basis of what criteria? What must codes be able to do in the future to provide realistic measures of safety margins rather than the conservative prescriptive criteria used in the past?

We foresee that the resolution of these sorts of questions will be held up by inconclusive arguments about code quality and credibility unless the agency clearly defines what the specifications for codes must be to support anticipated regulations. This should be a creative and ongoing activity, involving both the research and the regulatory offices of the agency.

References:

1.	Supplementary testimony of the AEC Regulatory Staff, Public Rulemaking on Interim Acceptance Criteria for Emergency Core Cooling Systems for Light-Water Cooled Power Reactors, Docket No. RM 50-1, dated October 26, 1972.
2.	U.S. Nuclear Regulatory Commission, WREM: Water Reactor Evaluation Model, NUREG-75/056, May 1975.
3.	Memorandum from N. Zuber, ACRS Consultant, to G.B. Wallis, ACRS, Subject: The Effect of Deregulation on NRC's Capabilities in the Field of Thermo-Hydraulics, dated April 6, 2000.
4.	U.S. Nuclear Regulatory Commission Report, "Quantifying Reactor Safety Margins - An Application of Code Scaling, Applicability, and Uncertainty Evaluation Methodology to a Large-Break Loss-of-Coolant Accident", NUREG/CR-5249, December 1989.
5.	Regulatory Guide 1.157. "Best-Estimate Calculations of Emergency Core Cooling System Performance", May 1989.