Home
> NRC Library
> Document Collections
> ACRS
> Meeting Schedule and Related Documents
> 2000
> 476th Meeting - October 5, 2000
476th Meeting Advisory Committee on Reactor Safeguards - October 5, 2000
UNITED STATES OF AMERICA NUCLEAR REGULATORY COMMISSION ADVISORY COMMITTEE ON REACTOR SAFEGUARDS ***** 476th MEETING Two White Flint North Room T2-B3 11545 Rockville Pike Rockville, Maryland Thursday, October 5, 2000 The committee met, pursuant to notice, at 8:30 a.m. . MEMBERS PRESENT: DANA A. POWERS, Chairman GEORGE APOSTOLAKIS, Vice Chairman MARIO V. BONACA THOMAS S. KRESS GRAHAM M. LEITCH ROBERT L. SEALE WILLIAM J. SHACK JOHN D. SIEBER ROBERT E. UHRIG GRAHAM B. WALLIS . P R O C E E D I N G S [8:30 a.m.] DR. POWERS: The meeting will now come to order. This is the first day of the 476th meeting of the Advisory Committee on Reactor Safeguards. During today's meeting, the committee will consider the following: Discussion of Union of Concerned Scientists report, "Nuclear Plant Risk Studies: Failing The Grade"; NEI 00-02, "Industry PRA Peer Review Process Guidelines"; Staff views on the ASME standard for PRA for nuclear power plant applications; And pressurized thermal shock technical bases reevaluation project. We will also discuss proposed ACRS reports, and we will discuss the topics to be raised in our meeting with the Commissioners tomorrow. Our meeting today is being conducted in accordance with the provisions of the Federal Advisory Committee Act. Dr. John D. Larkins is the designated Federal official for the initial portion of the meeting. We have received no written comments from members of the public regarding today's sessions. A transcript of portions of the meeting is being kept, and it is requested that the speakers use one of the microphones, identify themselves, and speak with sufficient clarity and volume so they can be readily heard. I want to bring to the members' attention that Jim Lyons is now on-board as our Associate Director for Technical Support. Welcome aboard, Jim. MR. LYONS: Thank you. I'm glad to be here, happy to serve and see what we can do to make everything work as well as possible. DR. POWERS: And we'll all do our best to try to really confuse him in his first few days. DR. SEALE: We'll see what we can do. DR. POWERS: Now I have some good news and some bad news. Let me start with the bad news. The bad news is Lilly Gaskins is leaving us for greener pastures. She's going off -- Defense Intelligence agency? MS. GASKINS: Yes. DR. POWERS: Lilly, I want to say we've very much appreciated having you here, and we're going to be disappointed, but our loss is their gain. [Laughter.] [Applause.] DR. POWERS: Now for the good news. Our own Carol Harris, Miss Carol Harris, has now become Ms. Carol Harris. [Applause.] DR. POWERS: Very best wishes from the committee, Carol, and we'll harass you like crazy. MS. HARRIS: I count on it. [Laughter.] DR. POWERS: Members do have a hand-out called "Items of Interest." I'll call to your attention that the water reactor safety information meeting is coming up, and I also call to your attention that the staff has issued their report on the Indian Point 2 incident. With that, I'll ask, are there any opening comments that members would like to begin? [No response.] DR. POWERS: Seeing none, I think we can move to the first of our sessions, which is a discussion of the recent report by the Union of Concerned Scientists concerning nuclear plant risk studies. Professor Apostolakis, do you want to lead us in this session? DR. APOSTOLAKIS: Okay. Thank you, Mr. Chairman. This report was issued last August, and it is very critical of the NRC's risk-informed initiative, and this committee has been very supportive of that initiative. So, naturally, we were interested in understanding better what the arguments of the report were. Mr. Lochbaum happened to be in the building last time we were here and very graciously agreed to come and chat with us in an informal way, but we felt that the more formal forum was needed, so we scheduled this hour-and-a- half today, and Mr. Lochbaum is here. So, how would you like to proceed? Would you like to make some opening statements? You have transparencies, I see. MR. LOCHBAUM: I have approximately 10 or 15 minutes of stuff, and then I'd be glad to entertain any questions or comments. DR. APOSTOLAKIS: Why don't we go that way, then? MR. LOCHBAUM: Okay. Roughly two months ago, we released a report called "Nuclear Plant Risk Studies: Failing The Grade." Almost immediately, we heard criticism that the report was flawed because we relied primarily on data from the individual plant examinations that had been done and not information from the more recent plant safety assessments that each plant owner does. The reason we used the IPE results is because those are really the only results that are publicly available. So, we didn't have access to anything other than that information. We also found it curious that the NRC was among the ones criticizing us for using the IPE results. Yet, in the site-specific work-sheets for use in the significant determination process, it's the IPE data the NRC uses, not the PSA data. So, you know, on one hand, we're criticized for using the data by the NRC that the NRC itself uses. So, I guess I didn't realize it was proprietary in that nature. We also heard that some folks argue that our evaluation, specifically the case studies that we did in our evaluation, were flawed because our conclusions were that the case studies prove that the risk assessments were bad because the results are so different for plants of similar design. In fact, I met, within the last week or so, with one critic of our report, who said that the plants are different, even plants at the same site are different, and those differences drive the risk assessments results to be different, so it's more surprising when the results are the same than when they're different. So, we heard that criticism, and we wanted to look into it a little bit. So, we went to the NRC's IPE database that's on the internet, and these following three sheets are from that database. I just printed out the overall core damage frequency from the IPE database and had sorted it from high to low, and I notice, with the exception of a few plants, Hatch 1 and 2, St. Lucie 1 and 2, Indian Point 2 and 3, Salem 1 and 2, and Beaver Valley 1 and 2, all multiple-unit sites that have similar reactor designs have the exact same core damage frequency reported for all units. So, if, indeed, the plants are different enough to drive the risk results -- assessment results different, then that's not done on a consistent basis. Sometimes it is, sometimes it isn't, and apparently, the NRC is happy whether the results are the same or whether the results are different, and like I say, that IPE database has all the plant results, even for plants that are now closed. DR. APOSTOLAKIS: So, are these so-called sister plants, or it's just a listing of the plants? MR. LOCHBAUM: This is a listing of the IPE results that all the plants submitted. Some of them are sister units and some of them are not. For example, Millstone -- obviously, they're at the same site, but they're different reactors. I wouldn't expect those numbers to be the same, and they're not. DR. APOSTOLAKIS: But your argument in the report is that sister plants report different results. DR. UHRIG: Well, St. Lucie -- they have different cores. One has 14-by-14 and one has 16-by-16, and there's a lot of other design features that are different. MR. LOCHBAUM: The same thing is true with Brown's Ferry. Even plants at the same site operate differently, whether it's the core design or the cooling water system design or whatever, because I don't think there's any -- it's like snowflakes. No two nuclear power plants are identical, even at the same site, yet the results are the same and are not the same. I understand the criticism. I don't know -- based on the data, I can't confirm or refute it, because we have some in both categories. DR. APOSTOLAKIS: Would you like us to raise questions as you go along or wait until you're done? MR. LOCHBAUM: Either way, whichever is easiest for you. If it makes more sense to ask me a question about it as we're going -- DR. APOSTOLAKIS: Okay. Let's pick up this issue. The issue -- I guess you are making the point that the PRA is such a weak technology that, when applied -- or immature, perhaps -- that when applied by two different groups to two plants that are very similar, it produces different results. MR. LOCHBAUM: That's correct. DR. APOSTOLAKIS: This is an issue, as you probably know, that the staff identified in its report on lessons learned and insights from the IPE reviews. So, it is really nothing new there except that you are taking a little more extreme position, but it's really the methodology that is the culprit here and not so much the design differences. The staff's conclusion was that, from the evidence, they could not conclude whether it was really the methodology or the design features, although they do say somewhere there, as I recall, that the differences were primarily driven by design and operation differences. So, I mean there are differences, and it's not just -- I mean it also depends very much on what you call sister plants, and my colleague, Dr. Bonaca, has more experience in that, and maybe you can say a few words. MR. BONACA: Oftentimes, we talk about sister plants on different sites, and when you look at them, really, oftentimes they are different on the secondary site, because the AE was different, because they were implemented in different ways. So, the sister plant connotation is one that relates more to the NSS island than to the balance of plant, and yet, the balance of plant is critical in determining some of the results in the PRA. For example, the layout of the auxiliary feedwater system is a fundamental element looking at the results. Now, on the same site, you have some plants that probably are sister plants, and maybe the case was made by the applicant, like Calvert Cliffs, that they're identical, therefore we submit only one IPE. I will expect that probably was the approach taken. On some sites, there may significant differences and you have different values. But one point I wanted to make is that that's why we don't like to see a bottom line number for a PRA. We're looking at uncertainty. If you take two teams doing a PRA for the same plant, you will get different results, no question about it. If you get two different vendors doing a LOCA analysis for the same plant, the same fuel, we get different results. Nobody expects to get the same value. What you expect to see, in fact, is a reflect of the uncertainty in the whole evaluation reflected in the two assessments, and I don't think it's surprising. I think it actually would be a healthy insight to have two different estimates of the same matrix for the plant, so you could understand what the subtleties are and what the effects are. MR. LOCHBAUM: I don't disagree with that. In fact, you know, the plants are different, and therefore, the results should be -- if the result -- different results are due to plant differences, that's one thing. If the different results are due to different methodologies and both methodologies are the same, six of one, half-a-dozen of the other, then that's fine, too. What we're concerned about is the lack of controls over the methodology and the assumptions in inputs that would allow a plant owner to -- if they're contemplating a modification to the plant or a procedure change -- for example, putting kerosene in the fire water headers -- to go back and adjust the inputs to compensate for the actual increase in risk from the proposed change, make some methodology hand-waving to make it look -- the net effect to be the same or even an improvement in safety. We're concerned that the controls over the methodology and the assumptions wouldn't prevent those kind of abuses, whether intentional or mistaken. DR. APOSTOLAKIS: I believe you are absolutely right on that, and this committee has been concerned about it, and the staff has made it clear that they are concerned about it. So, I think there is no question that, in some IPEs, people, whenever they found an opportunity, they were more optimistic than the general state of knowledge would justify, but it seems to me that the real question -- and I think that's an issue that one can raise in several places in your report -- is not so much whether there are studies that are not really very good out there. The question should be, at least in my view, has the NRC used any of these studies in its decision-making process and was led to incorrect conclusions because the NRC was unable to identify the sources of the differences, perhaps? I mean it all comes down to decision-making. I think we should make it very clear that, you know, the industry -- it's a private industry that can do studies for themselves, and if they want to use optimistic numbers, that's up to them, but when it comes to the agency, does the agency accept these analyses, and do you have any examples where, indeed, the agency used what, in your view, were inadequate PRAs to make a decision, because that's really the heart of the matter, the decision-making process. MR. LOCHBAUM: I guess we didn't, because the agency is clearly moving towards risk-informed regulation, and the risk assessment results are going to be an input or a factor in that regulatory decision-making. We wanted to try to prevent some bumps in that road. We're not saying that the road is wrong. There are some problems along the way that we wanted to try to address in our report and get fixed before we proceeded too far down that road. So, the answer to the question is, no, we don't have any examples, but also, we didn't look, because we were trying to prevent mistakes in the future, rather than flag errors of the past. DR. APOSTOLAKIS: This answer, I must say, is really surprising to me, because by reading the report, I didn't get the impression that you were trying to prevent mistakes from happening. I mean the report gives the impression that things are very bad. So, I must say your statement is very welcome, to me, at least, because the committee, of course, does not have a position at this point. So, if that was your intent, I think you're succeeding, but it doesn't come through by reading the report that you are really trying to prevent mistakes from happening. I mean "Failing The Grade" is a pretty strong statement to put in the title. MR. LOCHBAUM: I think we have some data to show why we think the risk assessments are bad. We didn't look for any examples where those results have been used yet, but the agency is moving in that direction, and that's what troubled us. DR. APOSTOLAKIS: But you do agree that this is really what the issue is, I mean the decision-making process. I mean, you know, a private company can do whatever they like, if they want to kid themselves that the risk is 10 to the minus 20, but when they try to use it here, then it's a different story. MR. LOCHBAUM: Well, I think the related question -- and I agree that that is the question, but the related question that Commissioner McGaffigan poses is, would this information or this approach lead the NRC to a different answer than it would using the current prescriptive approach, you know, because errors can be made on either side. DR. APOSTOLAKIS: And that brings up another major question that I have. I don't know whether I should raise it now or later. MR. LOCHBAUM: One of the other criticisms we had was our concern about the risk assessments not accounting for design basis problems, and the bulk of that criticism was that, yes, design basis problems have been identified, but they haven't been risk-significant; they've been much ado about nothing, if I can characterize the criticism, if I understand it correctly, and to investigate that criticism, we went to a recent -- a May 2000 draft report that the Office of Research prepared on the design basis problems, and this is Figure 22 from that report that looks from 1990 to 1997. The number -- the percentage of LERs with design basis issues that have been classified as accident sequence precursor events -- and while the trend is, indeed, downward, the important part that we think is that none of the years is non-zero. So, not all of design basis problems that have been reported have been able to be dismissed as non-safety- significant. DR. POWERS: I guess I'm a little bit confused. Would you expect it to go to zero? I mean I can hope it goes to zero, but I wouldn't expect it to. MR. LOCHBAUM: That's true. I wouldn't expect it to, but since the risk assessments basically assume that it's zero, then there's a disconnect between reality and the risk assessment assumptions, and that disconnect is what -- DR. POWERS: I don't think -- I mean it doesn't seem to me that the risk assessments assume this is zero. They assume these things actually occur. Most of the accident events result in nothing happening. In a typical modern PRA, there are, what -- I think, for 1150, the average one had three million accident sequences, of which nearly all of them ended just like these precursors ended, no consequences. I mean I can go through the 1150 and actually give a prediction on how often those things that are recorded there should occur. MR. LOCHBAUM: I guess the point we look at is, for example, the stand-by liquid control system we talked about in our report -- the Big Rock Point plant operated for 13 -- the last 13 years of its 39-year life with that system not working quite right. The risk assessments don't reflect -- DR. POWERS: Nor does the design basis analysis reflect -- nor does anything -- if you don't know that something is not right, there is no amount of analysis you could ever do in your entire life, by the most intelligent people, impossible, that will discover that if it's not discovered. I mean it's a non-criticism. MR. LOCHBAUM: But once something becomes a reported event, it doesn't seem that design basis events are factored back into the process, like human errors. There's a human error database. There's an equipment failure database. There doesn't seem to be a design deficiency database or -- a widget just doesn't work because it's designed improperly. If the widget doesn't work because somebody mis- operates it, that seems to be captured. You can argue whether it's right or wrong, but at least it's captured. If the widget doesn't work because it fails on start or fails on demand, then that seems to be in there. But if the widget is designed improperly, that doesn't seem to be in the risk assessments, and you know, any one of those failures in any one of those columns can cause something from working properly. DR. SEALE: I'm curious as to what's driving that curve down, then. MR. LOCHBAUM: Well, I would hope one of the factors would be, as you find things and fix them, you have a work-off curve. MR. BONACA: I would like to make a comment about this point. This is the trend, and that's the trend, but we have to recognize that we didn't look -- I mean one thing we found is, the more we look, the more we find, and we looked the most between 1995 and 1997. To me, it's comforting that that number of precursors is so low in the very period in which we looked so much, and there was a limit to how much we found. The other point I would like to make is that, again, for those precursors there, you know, there wasn't a terminicity evaluation or system failures that did not represent the range of operation in which the system should operate. There were some conditions that, in the deterministic space, says the system is not functional or is not operable. So, anyway, that's a different issue, but the point I would like to stress here is it's -- this trend -- it's encouraging in my mind, because we looked so much in '95-'97 timeframe, and we found, you know, we didn't upset that curve. MR. LOCHBAUM: I guess you could -- statistics can be looked at any number of ways. You could look at -- with all the things that reported in the '95 to '97 timeframe, the percentage would go down, because this is not absolute number, this is percentage, and there were so many less significant items found, as all those problems were flushed out, that the percentage would have gone down even if the absolute numbers stayed the same. MR. BONACA: There was also a finding that, again, the more you look, the more you find, and there was a lot of looking, and so, many of these issues were to do with original design. MR. LOCHBAUM: Seventy percent of them, according to this report, by the way. MR. BONACA: That's right, and it seems to be that as you -- the plant ages and these efforts are undertaken and so on and so forth, and the SSFIs took place in the early '90s and so on and so forth, it's an encouraging trend. I think we are getting to the point where, probably, most of this design -- original design defects are not there anymore. There will be always some, and we cannot eliminate those. MR. LOCHBAUM: That was the end of addressing the criticisms, so far, at least in the presentation. I'd like to turn now to some of the information we gathered as we researched the report that didn't -- wasn't in the final report, but we think that this information supports the conclusions that we drew in the report. This is an NRC study -- I forget the exact date -- it's on the isolation condenser system reliability at boiling water reactors. This is a figure showing the unreliability for the systems from actual plant events compared to what the IPE results were for -- that were used for these various plants, and you can see, for every case, the IPE result was to the left of the actual plant data or actual operating experience, although the error bands and the uncertainty bands did cover all the data. DR. APOSTOLAKIS: So, let's look at Dresden 2. Can you show us -- I see the PRA on reliability. Dresden 2 is the first one. Where is the operating experience? MR. LOCHBAUM: The last two lines are taking operational experience with and without credit for recovery. MR. BONACA: But that's an industry mean. I can tell you it was based on data, actual data. MR. LOCHBAUM: Right. MR. BONACA: So, I'm saying that that's a mean down there for the industry and doesn't represent, necessarily, individual -- DR. APOSTOLAKIS: So, you don't have the operating experience number for Dresden 2 in the figure. MR. LOCHBAUM: Right. DR. APOSTOLAKIS: I see. MR. LOCHBAUM: The NRC report said not all the plants had enough operational data in order to do individual plant comparisons. I do have some charts that do have information in that regard. DR. POWERS: I guess I still don't understand the figure. It seems to me this is a ringing endorsement of what they've done. The data are plotted, or the number used in the analysis is plotted. In some cases, they used point values, and I have ugly words to say about point values, but they get used, and I grow more tolerant with age, I suppose, and then you show this industry mean with a range, which is good. What's wrong? MR. LOCHBAUM: I didn't mean to trap anybody, but the next few figures, I think, will show what the problems are. This is the same approach applied for the high- pressure coolant injection system on boiling water reactors. The black closed circles with the bands are the operational data for each plant, with the uncertainty bands. The white circles are the IPE data, without uncertainty bands, and you'll notice, in this case, every single one of the IPE results is to the left of the actual performance. Most -- or some of the IPE data is not even inside the uncertainty bands for the actual operating experience, and with this unreliability curve on the axis, being to the left with the IPE result means that the IPEs assumed more reliability than what operational experience showed. The other thing we thought was -- we noticed as these curves came out was the IPEs were submitted in like the '91-'92 timeframe. So, they would have used -- if they used anything, they would have used operational data from the '80s, early '90s, which is a little bit earlier than the data used -- the operational data plotted here, and everybody keeps conceding that operating performance is getting better and better. So, you would have expected the IPE results, perhaps, to be closer to today's operational experience or perhaps even to the right of today's operational experience, and that wasn't the case. DR. APOSTOLAKIS: I'm getting two messages from this. First of all, I really think that PRAs and the standards that are being developed for PRAs now should insist on using plant-specific data. This is an excellent argument here, and we'll have an opportunity to discuss this during this meeting with -- when we discuss the standards, because this clearly shows that you have to do that. I mean you really have to use plant-specific data. I don't know what the basis of the other curves -- the other estimates was, but clearly, it was not plant-specific data. So, that's an excellent argument for that. And second, again, I will come back to the point I made earlier. I mean it is the NRC staff that supplies this figure. It is the NRC staff that makes decisions using PRAs, using the integrated decision-making process. So, I would expect the staff to raise this issue if a particular licensee comes, say, like Dresden, with a number that is way down to the left and say, no, this is not acceptable, I mean you have to do something or your PRA is not very good. So, that is the right context, in my mind, of this figure, which I think is very enlightening. MR. LOCHBAUM: I think what concerns us about this data -- these reports are put out by the NRC's Office of Research, and it seems like they go up on a shelf without the rest of the NRC staff relying on those, because when we talk about these numbers to the regions, about why Nine Mile Point seems to have more trouble with RCSI, they've never heard of this stuff. DR. POWERS: You're preaching to the choir. DR. APOSTOLAKIS: You are telling the truth, David, and this committee has expressed disappointment many times that a lot of the results from the former AEOD do not seem to be used. I think that's an excellent point. MR. LOCHBAUM: I got one. Okay. I'm on a roll now. I guess I don't want to -- since I'm on a roll, this is some more data. This is the reactor core isolation cooling system for boiling water reactors. I think this is even more striking than the last chart in that seven of the 30 results or plants that are reporting, the uncertainty bands for the IPE data don't even overlap at all with the uncertainty bands for the actual operating experience data. In none of the cases do the upper end uncertainty bands for the IPE data match the mean for the operational experience data. So, again, there seems to be a very strong bias towards using IPE results that are more reliable than actual operating experience would show. Now, those are boiling water reactor examples. It's totally different on the pressurized water reactor side, totally different. This is the same -- I blew it up a little bit. That's why it doesn't -- it's right out of the figure, but this is the auxiliary feedwater system reliability study that the NRC issued, and you can see that it's totally different, because the NRC put the unreliability axis on the vertical axis. So, instead of being to the left or right, it's above or below. There is one case for design class 4 where the operational data -- the IPE data is actually below the operational experience data. In all of the other cases, it's the same as the boiling water reactor data. The IPE data is shown to be more reliable than the operational experience data. And I agree with you, the bias can be handled as long as the staff recognize that and handles it right. What we're concerned about was the controls or the standards or how the NRC makes that -- how it factors that into its regulatory decision-making process. Once we saw the system bias or what we perceive to be a system bias, we wanted to try to figure out why that was happening, and we haven't conclusively determined why that's happening, but we think -- we eliminated one suspect, and that was that, you know, these biases being introduced at the component level and then just being rolled up into the overall system reliability number. So, this is a chart -- and there's about six or seven different charts in this NRC report. This is component level for the turbine-driven feed pumps -- turbine-driven pumps, not necessarily feed pumps. It's the probability of failure on demand, and you can see there, you have the operational experience, and then you have what the individual plants reported, and it's some above, some below, which is what you would expect if best estimates were used or realistic data was used. So, it didn't seem that the bias was being introduced at the component level. It was being introduced somewhere else. In eliminating one candidate, we didn't identify what it was, but what we think it is, but we can't prove, we think it's somehow related to this, and this figure appears in every one of these NRC research reports, perhaps a little bit easier to read. It's got three round things. Region A is all inoperabilities that are reported by license event reports. Region B is the ones that have been -- of those inoperabilities that have been classified as failures, and region C, the smaller of the round things, is the ones that have been -- failures that have been considered countable, and I think, you know, if the industry has a different definition for region C than the NRC does, then that could explain why the bias at its system level, because a component-level failure could be perceived as not causing the system to not do its function. If the NRC uses a different definition than the industry, then that could explain the two curves or the two -- the results being so disparate, and again, going back to your comment, that doesn't necessarily mean it's wrong. It's a bias that has to be recognized and factored into a decision-making process. What our concern is, is while the research is cranking out report after report identifying and classifying and labeling this bias or these differences, the rest of the NRC staff isn't accounting for that in its decision-making process, and that's why, in our report, we didn't say this was the wrong road for the staff or the industry to be on; we thought that those standards, those controls needed to be done before anymore progress was made down this path. I mean that seemed to be an important part that was not in the right order in our mind. MR. BONACA: Probably I'm not the person to answer that question, but the question I would have is, has the NRC asked the licensees why there are these biases? I mean, here, in the test, it implies there have been calculations done using raw data and using actual operating experience and using data from the IPEs, and I'm not sure there has been a communication back and forth to understand where this bias trend comes from. DR. POWERS: I guess maybe this is the point to interject an issue that comes up here, and I'm not being critical of your use of the IPE, because I think I'm very sympathetic, in fact, using the IPE results. That's all you have. The situation is, the NRC asked the licensees to conduct the IPE studies. They didn't ask them to use PRA. In many cases, the analyses that were done to support the IPE submissions were the first attempts to do a PRA of that plant, and the intentions of the NRC was not to have the licensee do a very rigorous, careful PRA analysis. It was to familiarize themselves with the vulnerabilities of their plant and to gain some experience in the area, and so, they didn't review them in exhaustive detail. The question that comes up, I think that gets raised by all of this, is if we use a risk assessment, which now is rigorous and carefully reviewed as part of a licensing action, and the public has an interest in that particular licensing action, how do they routinely get access to these PRA analyses? Well, clearly, one way is to put them on the docket, but if you put them on the docket, you ossify them, and it becomes -- and that almost defeats the purpose of having them, and so, the question I put to you is, have you thought of any way to make these PRA results accessible to the interested parties without ossifying them? MR. LOCHBAUM: We've thought about it a little bit, because we are on the outside trying to get at information, and the IPEs themselves are very large, but there are summary sections and tables that basically provide the bottom line to the results fairly quickly, and I think you could have that on the docket, the summary and the results and the basic description of the approach taken and a brief description of the methodology, without having all the component failure data, tables, and all that stuff on the docket. DR. POWERS: The problem I see with that is that it doesn't take a genius to figure out how to make bottom lines look very, very supportive of one position or another, and in fact, in the regulatory actions, you very seldom use those bottom lines, and the regulatory actions tend to be insensitive to them. In 1.174, you use that bottom line number to position yourself on a horizontal axis, which is plotted in a log scale, so being a factor of 2 or 3 one way or another hardly makes a difference at all. It is the differentials of those results that get used in deciding whether they're licensing actions, and those differentials have to -- you now have to plunge in the details. Almost, I'm willing to give people the bottom line results that they come back with. I can almost guess the number that they'll come back with. What I'm looking for is vulnerabilities in the system, and those vulnerabilities are discovered only by plunging into the details. MR. LOCHBAUM: I guess the other answer to that question would be, if the NRC had some standards or controls over the methodology that plant owners did, inputs, so on, and had some -- so that the NRC's methodology was publicly available and the NRC's verification that plant owners met or exceeded those standards, then I'm not sure that the public needs -- or even I -- I wouldn't want full access to everybody's PSA. I want, and I think the rest of the public wants, some assurance that the NRC is making sure that people making decisions on risk analysis -- that those risk analysis is a solid foundation for making those decisions, and I think there's -- either make the risk assessments and all the attendant details available or make the NRC's role in verifying that that's good, make that publicly available, and that might be the easier of the two or the more proper of the two paths. DR. APOSTOLAKIS: One of the fundamental points, I think, that needs to be made here is that the decision- making process that utilizes PRA is, as you know, an integrated process. As Regulatory Guide 1.174 describes it, there are five inputs to the process, and four of them are really traditional deterministic considerations, like preservation of the defense-in-depth philosophy, safety margins, and so on. So, in that context, it seems to me that your criticism in the report acquires a different flavor. It's not in the report, though. In the report, you just mention in passing the risk-informed initiative on page 22, giving it one paragraph. You give the impression that decisions rely -- you use the verb "rely" too many times -- rely on PRAs, and surely you're aware of the fact that decisions do not rely on PRAs. I mean it's just one input, and we have figures with shades of gray, we have long discussions in Regulatory Guide 1.174 about model uncertainty, about sensitivity studies that sometimes drive the industry crazy. So, do you think that perhaps you did not present the actual decision-making process as accurately as you could have? MR. LOCHBAUM: I think I understated it. DR. APOSTOLAKIS: You understated what? MR. LOCHBAUM: The reliance on the risk assessments. DR. APOSTOLAKIS: You think you are relying too much on risk assessments? MR. LOCHBAUM: In certain cases. I think the recent example is Indian Point 2, the Inspector General's report on how the staff handled that steam generator inspection, where the NRC staff thought that the whole thing was of low safety significance and just basically put it up on the shelf. That wasn't made by any of these five factors and weighing all this information. This was based on one person's shooting from the hip, deciding that something didn't warrant further NRC resources. So, I think there are two many cases like -- even one case like that's too many, and I think that's not the isolated case. DR. APOSTOLAKIS: So, we should be vigilant, then, to make sure that what we call risk-informed approach is actually risk-informed and not risk-based. You're saying there are examples where the decision was risk-based, and that was inappropriate. MR. LOCHBAUM: Well, it was considered -- that decision, technically, was considered risk-informed, because all factors were done, but one of them was given 99 percent of the weight, and the other four added up to maybe like 1 percent. So, that was, technically, risk-informed, but it really -- it was an abuse -- DR. APOSTOLAKIS: -- a mis-application of the intent. MR. LOCHBAUM: Right. DR. APOSTOLAKIS: But a broader issue, though, is the following, in my mind. One -- and I've said that in many other contexts, not just in the context of your report, because many times we've had staff come here and give us a million reasons why a risk-informed approach to a particular issue is not appropriate. People take the PRA and they scrutinize it to extreme detail in the absolute. They say, well, gee, you know, there are all these problems, therefore you can't use it. It seems to me that's the wrong way to approach it. The way to approach it is to ask yourself, I have now a deterministic system in place. Am I better off if I supplement what I'm doing there by using risk information or not? That's really the proper context to evaluate the PRA, and you can apply that to specific issues. For example, you raised the issue of design issues that have not been found and so on, and well, if I decide not to use PRA, does the existing system do a better job identifying those design issues, and if I use a PRA, do I have a better chance of finding them and evaluating them in the proper context, and I think you can practically pick any of the issues you raise, and if you place it in that context -- now, you probably reach a different conclusion from mine, but I believe that the system is better if it blends both. I grant you that there may be exceptions where people mis-use the process. You know, it's a big agency handling many, many situations. I mean we're all human. But I think that's the proper context, and just to say, gee, PRA has all these deficiencies, therefore it should not be used, is really an evaluation in vacuum. PRA is not proposed as a methodology that will replace the existing system. It will add information to it. I was wondering what your thoughts were on that. MR. LOCHBAUM: I agree with that. Again, if we thought PRA was the wrong tool to be using, or if it was going to replace the deterministic, the recommendations would be to stop and fix the standards. It would be just stay where you are and stop wasting all those resources. But that wasn't the conclusion we reached in the report. We said fix the problem. I think where we see the problem is that, in the example being risk-informed regulation or risk-informed inspection, clearly the industry leaders in that approach have looked at plant-specific inspection results, identified areas where inspections are not finding problems, and have prudently chosen to redirect those resources into areas where the problems are being found, and to me, that's a perfect example of what you explained about deterministic and now factoring in risk information to be smarter, do things better, and we attended some of those meetings and thought that was fine. Our concern is, without the standards that the NRC applies, there are other plant owners who didn't spend all the time to really understand the subtleties of the issues that the leaders have done, just are going to get on that highway and go down the same road and might make the wrong decision, and the NRC, by not having established standards, doesn't have the proper tools or infrastructure to prevent those subsequent abuses. You know, the first -- South Texas and commercial- grade classification or whatever -- those guys spent an awful lot of time and an awful lot of effort to make sure they fully understand it. So, it's not the leaders, it's the ones that then jump on the highway down the road, and we're concerned that NRC's not policing against those. DR. APOSTOLAKIS: Have you had the chance, by the way, since you mentioned the standards, to look at the ASME standard, and would you care to make a comment or two about it? MR. LOCHBAUM: I haven't looked at the ASME standard. I have looked at NEI's -- what is it -- 0002? DR. APOSTOLAKIS: The industry certification process. MR. LOCHBAUM: The peer review process. We talked earlier about the need for using plant- specific data. NEI's peer review process does include that, if -- depending on what grade you're trying to get your risk assessment classified as. I forget whether 1 or 4 is good or bad, but if you're just trying to use it for run-of-the-mill -- DR. APOSTOLAKIS: Four is good. MR. LOCHBAUM: Okay. If you just want a grade 1, you don't have to use plant-specific data. So, there is recognition for things like that. The one criticism we have at this point -- it's preliminary review, because I only got the thing Monday -- DR. APOSTOLAKIS: Sure. MR. LOCHBAUM: -- so I haven't -- is that those checklists, the things that have to be looked at, while they're very thorough, they seem more like inventory checks than they are quality checks. You know, the example we could cite would be -- you could ask the captain of the Titanic if you have life- boats, and that answer would be yes. If the question was, do you have enough life-boats for all passengers and crew, that's a different answer. So, they seem to me more questions of the first category than the second. DR. APOSTOLAKIS: There was an example of that in Greece last week. DR. SEALE: Two examples. DR. APOSTOLAKIS: What do you think of this idea of grades? MR. LOCHBAUM: Well, I think the whole concept of having -- I don't think very plant owner needs to have the same standard risk assessment, it depends on the application, and that, again, goes back to your point about -- you know, deterministic base is one thing. If you opt for more and more risk insights, then you should have a stronger foundation for supporting those moves. So, I don't think you have to have -- it's a one- size-fits-all approach, and it makes sense that there should be varying degrees. Whether it's grades or -- you know, the actual mechanism for classifying that is -- I don't have a strong comment on, but I think it is good to have those tiers and to be used in that way. DR. APOSTOLAKIS: As long as the tiers can be defined clearly, so everybody agrees, right? MR. LOCHBAUM: The concern we have also goes back to the old SALP process, where they -- there were tiers in that approach, too, but they seemed to be somewhat subjective. If the NRC thought you were a good plant owner, you tended to get a 1, and if they thought -- if you were in the regulatory dog-house, you got a 3. So, we're trying not to get the peer review or any risk assessment grade also be a reflection of how much the NRC likes you. That fondness should not be a factor in the overall result, whether it's a grade or anything else. DR. APOSTOLAKIS: You make a statement in your report which I find strange, and I'm not sure that -- and I'm curious to see how you would react to that. On page 21, you say, "But it is not possible to properly manage risk when only reasonable instead of all possible measures are taken to prevent and mitigate events unless the probabilities and consequences are accurately known." Are you seriously proposing that all possible measures be taken? I mean what does that mean? You know, as you know, this is a very ill-defined concept, all possible. I mean we don't do that with automobiles. We don't do it with airplanes. Is this an exaggeration to make a point, or should it be taken literally? MR. LOCHBAUM: It can be taken either way. It is true, if you take all possible measures, then it doesn't really matter to know what the probability or consequences of any event are, because you're doing everything that's possible, and those risk insights wouldn't change what you're doing, because you're already doing everything you can. I wasn't advocating doing everything that was possible. DR. APOSTOLAKIS: I see. MR. LOCHBAUM: The point I was trying to make -- I've had several comments on this paragraph. The comments I had were from people who didn't like risk assessments at all, and they were conceding that things could be done. They thought I understated the point. But what I was trying to say is, if risk insights are then being used to draw a line between what you do and what you don't do, then you need to understand the consequences and the probabilities well enough to draw the line and decide not to do things that are on the wrong -- on one side of the line, and that was the point I was trying to make, and it clearly didn't come across, because I've had several comments on that. DR. APOSTOLAKIS: I have one last question. Looking back the last 25, 30 years, do you think that the use of risk information has made plants safer or not? MR. LOCHBAUM: I think it has. I think the IPE program itself identified vulnerabilities and led to several real changes, not paperwork changes, actual plant modifications or procedure changes to improve safety. I think the extension of that effort was the maintenance rule. A lot of that research or activity led into the maintenance rule. I think the maintenance rule -- the emphasis on both the safety system reliability and also what were traditionally considered BOP systems and the increased attention on those has led to overall safety improvements. So, I'm not saying that risk insights have been a net loss. There clearly have been some gains, and important gains. DR. APOSTOLAKIS: Do any of my colleagues have any comments or questions? DR. POWERS: I'm still coming back to the access to information. One of the studies -- PRA studies that was fairly extensive that is publicly available was the reactor risk reference study. MR. LOCHBAUM: Is that in NUREG-1150? DR. POWERS: I think that's 1150. MR. LOCHBAUM: Yes. DR. POWERS: Did you consult that at all in preparing this document? MR. LOCHBAUM: In earlier discussions over the last year, as this report was being researched, we referenced 1150 quite a bit, and NEI was the obligatory counterpoint on each of those arguments, and it was that 1150's out of date, and you really need to understand the inputs that drove the numbers, you can't just rely on the end points, which is what I had been doing when I was citing 1150. So, we decided to go back and look at the individual IPEs to try to respond to that criticism. So, that's why we didn't use 1150 in this report, although we were aware of it and had used it previously. DR. POWERS: Well, the upshot of that is that your report now harkens back more to the old WASH-1400 and Army studies and things like that, which are really geriatric. I mean that's when the technology was really in the developmental stage, I would say. So, you end up abandoning one study, because it's out of date, in favor of some that are really old. DR. SEALE: A whole array of studies -- DR. POWERS: -- that are really old. MR. LOCHBAUM: That's true. DR. SEALE: And again, the quality control, if I may use a phrase, the legal authority for the IPEs was not what we would expect for any PRA that we would use today. It was vulnerability identification, and we all know that you can do a risk assessment which can be very coarse in certain areas and very fine in other areas to identify particular vulnerabilities. DR. POWERS: I think that's especially true when you're trying to look for vulnerabilities that you don't know about. DR. SEALE: That's right. DR. POWERS: And you say, okay, well, gee, I know that I'm vulnerable in this particular area, in my piping system, so I'll just put in some numbers there, because I already concede that point, I'm looking for something else, and so, you do strange things on your inputs there. DR. KRESS: If I boil this down to a few words, it seems to me like your problem is that NRC doesn't seem to have good quality standards that it enforces in PRAs and that PRAs are shortcoming in that they don't deal with latent design defects. If we had those two things fixed and the movement towards risk-informed regulation, I think you might be in favor of. MR. LOCHBAUM: That's basically our conclusion. You know, those standards need to be there and enforced, and then the move to risk-informed regulation would be -- could be a good thing. Not every plant owner is going to do that, but those objections would be removed. DR. SEALE: Yeah, but then again, you run into the same problem again. Suppose you want to look at the difference in two alternatives, and those two alternatives could be very specifically defined. Now, I can do that within the context of a very detailed, highly formalized, high QA, overall PRA, perhaps even levels 1, 2, and 3, and the whole school yard, or I can do a modified PRA process which treats those particular detail differences in considerable detail, and the rest of it rather coarsely, and come up with a number that may not be valid in terms of actual risk for the overall plant but will tell me that the difference in risk between this alternative and that alternative is a factor of whatever. Now, under certain circumstances, if that's to determine how I'm going to conduct a maintenance process and what equipment I will have in service and what equipment I won't have in service, then clearly, the focused assessment tells me all the answer I need to know to make that decision. MR. LOCHBAUM: Well, conceivably, both of them could. DR. SEALE: Yeah, but one of them is so long to do that I won't have the answer when I need to do the appropriate maintenance, or it's so expensive that I don't have the people on the floor to do the work, I have them in the PRA group doing the assessment. MR. LOCHBAUM: I agree with that, because again comparing it to what we're doing today, in 50.59, you have to make a determination like that, if a modification or a procedure change or configuration affects the margin of safety, and you have to do some evaluation, whether it's PRA-based or not PRA-based. DR. SEALE: Yeah. MR. LOCHBAUM: So, if the approach that's selected, whichever of those two is done, meets or exceeds the decision that would have been made to the old non-PRA 50.59, then either of those approaches should be acceptable. DR. SEALE: Yes. MR. BONACA: I have a question regarding those two plots you showed, HPCI and unreliability, and the question I have is, did you find these trends consistently for other systems, or are these the grossest examples? MR. LOCHBAUM: Every volume that I received from the research that was sitting on my desk had this trend. The only one I didn't use was the reactor protection system study that recently came out for B&W and Combustion Engineering, because I didn't see a plot in there like this. I think there was some text to that effect, but there wasn't a plot, and I tried to illustrate with a plot. So, it wasn't that I only picked the ones that supported my argument. MR. BONACA: Okay. So, the trend is there, you say. Okay. MR. LOCHBAUM: I didn't look at all the reliability studies, but the ones that were on my desk that I received recently, I did, and every one of them supported it. MR. BONACA: One thing we found -- and I'm not sure this is the answer to the problem -- when we reviewed the experience of station blackout, we found that -- the NRC found that the industry and the NRC were counting unreliability a different way, and that was because regulation in different forms allows different ways of counting, and I'm just wondering -- because I mean, this consistent bias seems to be -- you know, certainly is a concern. MR. LOCHBAUM: That's why I think it's some of an accounting system, because if it was a methodology or if it was a personnel problem, people just weren't getting all the data, then I would expect to see some plants perhaps to the right of the operational data, whereas there seems to be a fairly consistent bias for all plants, all methods. So, I think there's something more generic than just -- DR. APOSTOLAKIS: On the good side, of course, the same office issued reports that showed that the frequencies of initiating events were, in fact, over-estimated in some instances by the PRAs. So, it would be, in fact, interesting to find out why these things are happening and whether it was just a matter of judgement or accounting or whatever. So, the evidence is mixed. But the main message that the PRAs we're using now should reflect this experience I think is a good one. MR. BONACA: That's the most troublesome part to me, that there are these biases, and we don't know why. DR. APOSTOLAKIS: We can investigate that. Any other questions from the members? Comments? [No response.] DR. APOSTOLAKIS: Does the NRC staff have any comments that they would like to make? MR. BARRETT: I'd like to make a few comments. DR. APOSTOLAKIS: Please identify yourself first. MR. BARRETT: My name is Richard Barrett. I'm Chief of the PSA -- Probabilistic Safety Assessment branch in the Office of Nuclear Reactor Regulation. I just would like to make a few comments based on what I heard today, and I think, to some extent, I'm repeating what some of the members had to say. I believe there's a lot in this report that we would say is technically correct. I think there are some things in this report that we would take issue with, technically, but I think primarily what we would be concerned about is the implication that PRAs have not been used in a judicious fashion by the staff in our decision-making process. We feel that, throughout the history of the use of PRA by the NRC, which goes back 25 years, we've been very cautious in using it, and we have used it with full knowledge of the weaknesses, limitations, and uncertainties in the methodology. There are some, like myself, who have felt that we could have been much more aggressive over time, but here we are now 25 years after the Rasmussen study and we're now moving into an arena where we are beginning to use PRA in a way that will challenge the methodology. I think that what you'll find, though, is that, in our decision-making process to move into risk-informing Part 50, both option 2 and option 3, we are taking into account a lot of the lessons learned from our own experience and some of the ones that are pointed out in the UCS study, and we feel that we have defined -- and I would refer everyone to SECY 00-162, which I think is an excellent description of the decision-making process that we tend to use, because SECY 00-162, what we say is that it's the quality of the decision that counts, not the quality of the PRA. PRA -- as with Reg. Guide 1.174, PRA will be used in conjunction with other methodologies, with other sources of information, and with other consideration. We will look at generic PRA results, as well as plant-specific PRA results. We will look at the results of deterministic studies, and we will also look at considerations of defense in depth and safety margin. And having looked at all of those, the staff will decide what trade-offs have to be made between quality of the analysis that's submitted and the quality and depth, scope of the staff's review of individual applications. We know that PRAs, in the past, have had their weaknesses. I think my favorite example in the UCS report is the discrepancies between the Wolf Creek and Calloway plant, because Wolf Creek and Calloway were standard plants, designed to very similar specification, and yet, they not only had different numerical results, they came up with different dominant contributors. We know that, today, those two PRAs have very similar results both in the numerical results and in the dominant contributors, and the reason for that is that there has been an effort over time on the part of the owners groups, on the part of those licensees to compare the results, to compare the plant designs, and to find out what are the reasons for these discrepancies. Now, you could say that, over time, these groups, working together, have converged on the same answer, possibly not the right answer. We believe that the opposite is true, that these efforts to compare the bases of these PRAs and to challenge each other through these peer processes actually leads to more correct answers. So, we believe that, over time, this peer review- type of process will give us better PRAs, PRAs of higher quality. We are currently reviewing the industry's peer review process, and we know that, in fact, the peer review process does ask the question, how many life-boats you'd have to have on the Titanic, not just whether or not you have any, but we've asked the industry to document so-called sub-tier criteria, asked them to document them so that it's not only clear to the NRC staff but it's also clear to our other stakeholders, and we think that's important. I think I agree with the assertion that the UCS study had to be done on the basis of IPE results, because that's all the results that were available, and I think, as time goes by, we're going to see more and more information available on the public record. A couple of specifics: With regard to the use of -- DR. POWERS: Let me interrupt you on that point. You're persuaded we don't have to do anything special here, that this is just a natural evolution, we're going to have publicly available data that would allow people like Gary to make comparisons that didn't have this question of whether he's comparing data that nobody would stand behind or something like that? MR. BARRETT: I don't want to speak for the industry, and perhaps someone from the industry would like to speak, but there has been a lot of discussion on the part of the industry, and I think, to some extent, motivated by the UCS report, to make a lot more information available, publicly available in a scrutable way so that more accurate public discussion of these PRAs can be held. I won't say anymore than that, but maybe the industry would like to say something. I'd like to speak to a couple of specifics. One is the use of the operational experience reports that were published by EOD, now published by Research. We in NRR get those reports in draft form, we review them. We are aware of the results, and we use the results, as applicable, in the reviews of license amendments and other regulatory actions. I'm not familiar with this particular view-graph about the high-pressure coolant injection systems. I am surprised to see that it indicates the unreliability of this safety-related system is in the 30- and 40-, even as high as 70-percent range for BWRs in this country. I think my recollection of the report was that -- and the other operational data reports -- was that the unreliability of most of the systems on a train basis is in the few-percent range, not the few-tens-of-percent range. This may not be representative of the bottom line of these operational experience reports. The other thing I'd like to point out is that we are using -- the example of Indian Point was brought up, and I think that we now have a couple of examples of how risk- informed regulatory action has been taken in the review of inspections like the Indian Point case. I would point you to the Farley case and the ANO case, and in those two cases, one of which we approved a licensee continuing to the end of the cycle and one in which we did not approve it, the full risk-informed Regulatory Guide 1.14 process was used, and I think that, if you review those two SERs, you'll get an example of what happens when this risk-informed thinking is applied. So, in summary, I'd like to say we do see a lot of things in the UCS report that we agree with. We think that PRAs have to be continuously improved. We also think that there is a limit to how much they can be improved. We have to have a regulatory process that accounts for these limitations. We think that, in the past, we've had such a regulatory process. We're committed to having such a regulatory process in the future. DR. APOSTOLAKIS: Comments? MR. LOCHBAUM: My recollection on the Farley and the ANO uses of Reg. Guide 1.174 is Farley was approved before the IP-2 accident and ANO was denied after the IP-2 accident. If those were flipped, I'd question very strongly whether the approval/denial would have been reversed. I think it's more a function of the accident at IP-2 than the technical merits of the two cases, but I could debate that till the cows come home. DR. APOSTOLAKIS: Any other comments from the staff? Dr. Parry? MR. PARRY: This is Garth Parry from NRR. I just want to make a kind of clarification about the use of the IPE results in the significance determination process. There are two points to be made here. The first is that the IPEs were taken primarily as the first step and that the process then is to send that out to the licensees, who will review it, together with NRC staff and contractors, to reflect the most up-to-date results, but I think the more important thing about the use of the IPE is that the results that are being used, the significance determination process, are probably among the more robust results from the PRA in the sense that all that's being used is the structure of the event trees in terms of the systems and functions that are required to respond to the different initiating events. I think I've made this statement before in front of this committee that I think those results are unlikely to change very much. The IPEs differ largely in the level of detail and in the numerical results, which are not directly used in the SDP process, and remember, too, that the SDP process and the way that the work-sheets are used is really only a screening tool. It's not the last word. It's just intended to be a conservative screening tool to act as a filter. MR. LOCHBAUM: I'll look into that, but this is the site-specific work-sheet that went out to Indian Point 2, dated January 3rd of 2000. The pages aren't numbered, so I don't know which page it is, but it says that the human error probability assessed in the IPE, page 3-371, is 5.62E to the minus 2. So, you know, it seems to be more than just structure and things like that. MR. PARRY: Yeah, I'll make comments on that, too. In the very original SDP process, effectively, human error probabilities were given a choice of .1 or .01 across the board, depending on an assessed level of stress, and I think we've designed it that that really isn't appropriate, because in fact -- take boiling water reactors as an example. Every sequence ends with initiating suppression pool cooling. If we gave that a 10 to the minus 2, then every transient would turn out to be -- any change that related to a transient would turn out to be a red, which really doesn't make any sense, so it defeats the object. So, what I think is underway is that we're trying to get all the results on the AGPs from the suite of IPEs that's out there, and on the basis of that, a conservative estimate for a particular function will be chosen to represent the screening tool. So, yeah, the plant-specific numbers are in there, but they're being used at the moment as -- they're just being collected. DR. APOSTOLAKIS: Any other comments from the staff? [No response.] DR. APOSTOLAKIS: Any comments from the public? [No response.] DR. APOSTOLAKIS: Hearing none, thank you very much, Mr. Lochbaum. MR. LOCHBAUM: Thank you. DR. APOSTOLAKIS: Back to you, Mr. Chairman. DR. POWERS: Thank you, Professor Apostolakis. You've given me an extra 15 minutes that, alas, I cannot use, so we will have to take a -- we will recess until a quarter after 10. [Recess.] DR. POWERS: Let's come back into session. We're going to continue our discussion of PRA with an examination of the industry PRA peer review process guidelines, and again, I'll turn to Professor Apostolakis to provide leadership in this area. DR. APOSTOLAKIS: Thank you, Mr. Chairman. We have at the table Mr. Fleming and Mr. Bradley, who will take the lead in this. I have a question before we start, because -- and it's probably a naive question, but in the Executive Summary of NEI 00-02, it says that one desired outcome of having a PRA review process is to streamline regulatory review of risk-informed applications. In other contexts, we've said that, you know, this would expedite reviews, will make the life of the staff easier, of the industry, of course. If that is the case, why do you need the NRC to approve anything? I mean if you have a process that you think will do that, won't that happen de facto? I mean the NRC will have to review, no matter what, whatever submission is made. Now, if you follow a process that you think is reasonable, then the NRC staff naturally will do this in a more efficient way. Why do we need to go through this painful process of getting the blessing of the NRC in advance? Do you anticipate that there may be a situation where a licensee comes in there and say, oh, my peer review certification process which you have approved, let's say, says that this PRA is grade 3, so you should accept it's grade 3. I mean, clearly, that cannot be the case, because the staff will still have to review it. So, I don't know why we have to go through this and have the staff approve anything. I mean isn't it de facto, something that will happen de facto? MR. BRADLEY: Okay. I'll try to answer that. I think we will attempt to answer that in more detail in today's presentation. What we requested was a specific NRC review with regard to option 2 of the Part 50 regulatory reform effort, and maybe I'll just go ahead and put my first slide up, since that's on there anyway. We've had discussions with the staff regarding the use of the peer review process to facilitate review, focused NRC review of applications for some time now, and there have been continuing questions about aspects and details of the process. Submitting the process for NRC review was intended to give NRC the opportunity to look at the process in detail, ask the questions they need to, and basically try to achieve the comfort level we believe they need in order to successfully use this process for regulatory reform and provide focused NRC review based on their knowledge of the process itself, as well as what we'd have to submit in terms of the option 2 application. So, it was specific. You're right, it's going to be somewhat of a painful process. We've already gotten the first RAI, and I know what South Texas feels like now, and we'll have to kill a few trees to respond to that, but we believe it's a necessary thing that we need to go through, and it puts it in the public record, and it's, you know, for everyone there to look at. DR. APOSTOLAKIS: But the question is, do you expect -- let's say that you finally agree on something, you know, you modify your process, the NRC staff is happy, and so on. Do you expect that the licensee may come to the staff and say, look, the supporting documentation went through the peer review process, which you guys have approved, therefore you should accept it, or is the staff going to review it anyway? MR. BRADLEY: No, we're not asking for some type of carte blanche acceptance based on the fact that it's been peer-reviewed. We're asking for a focused -- using that result, to focus the review and streamline the review, not to obviate the review. DR. APOSTOLAKIS: So, that is my question. MR. BRADLEY: Right. DR. APOSTOLAKIS: If that is the case and you do a good job, is there any need for -- okay. The staff passes judgement now and you say, well, gee, we really would like to see that, too, and leave it at that, without asking them to actually bless this. MR. BRADLEY: I think both pieces are necessary. Given the nature of PRA, we're trying to get at this from all possible directions and establish a successful framework to get applications to go forward. So, it seemed appropriate to put this on the docket and get NRC to have a look at it. MR. FLEMING: If I may add to what Biff said, I think another motivation for that statement that George is referring to is the fact that, if utilities follow the certification process -- and that identifies strengths and weaknesses in their PRA -- those strengths and weaknesses can be addressed in the application, and as part of the application process, as it's submitted to the NRC, that information can be presented as a way to build confidence that strengths and weaknesses in the PRA have been identified and they've been addressed for that particular application. DR. POWERS: But the NRC staff will have to make that determination anyway. So, if you have done it already, the staff will breeze through it and say this is good. Why is there a need for the staff to bless the process in advance? I mean they will just see the results of it and say, gee, those guys really know what they're doing, and then after they do that three or four times, they will start saying, oh, yeah. I mean if they submit a grade 3, chances are we'll review it very quickly. MR. BRADLEY: I guess our view is, if NRC is familiar with the process, if they've reviewed it and have confidence in it, that will make it that much easier. I mean that might be a leap to expect them to be able to reach that kind of conclusion if we haven't asked them to review the process. DR. APOSTOLAKIS: But you're not expecting that somebody will come in here and say this is grade 3, you have approved it -- MR. BRADLEY: That's correct. DR. APOSTOLAKIS: -- therefore we need an answer by Monday. MR. BRADLEY: That's correct. That's not what we're asking for. We recognize -- I mean, if regulatory reform succeeds, we're going to get a fairly large number of applications in process concurrently, and we have to have some method to use NRC's resources in an efficient way to approve those, and we're all trying to achieve some reasonable middle ground as to how we can do that, and this is part of that. DR. SIEBER: Is it a factor that the NEI document, by itself, is not a mandatory thing for utilities to use, and therefore, they could pick and choose whether they would use it at all or what parts they would use by having some kind of regulatory blessing that provides the incentive to use it all the way it stands? Would that be part of the reasoning? MR. BRADLEY: I guess I never thought of that as part of the reason. I don't know, Karl, if you want to elaborate, but I guess I'm not aware of utilities that are just using portions of it. I mean we have funded through the owners groups the application of the process to essentially all our plants by the end of next year, and it's the whole thing. It's the whole process. DR. SIEBER: Okay. MR. FLEMING: It may be a rather moot point, because all the owners groups have committed to completing a peer review process, and we're more than halfway through all the plants in the country applying it. MR. BRADLEY: Why don't we go ahead and try to start here? I have taken the bold step of putting a fair amount of detail in Karl's presentation, and I guess we may beg the committee's indulgence to maybe let him try to get through as much of that as possible. We did want to give you a pretty good sense of how the peer review process can be used to support an application, and that's what Karl's presentation is going to get at. I just wanted to set him with just a few brief remarks here on -- and I think we already covered some of them. We are talking about a specific application here, option 2 of the Part 50 regulatory reform effort. NEI has developed a guideline which we've given to the staff for review on that whole process. It's an integrated decision process. It's sort of a classic 1174 application. It also uses sensitivity studies and other methods to -- as part of the process to check the overall result and the categorization. Therefore, the use of the PRA in that application is specific, and there's specific things about a PRA that are important for that application. DR. APOSTOLAKIS: NEI 00-02 doesn't say that it's only for -- MR. BRADLEY: No, no, no. 00-02 was developed for broader use. DR. APOSTOLAKIS: Isn't that what we're reviewing today? MR. BRADLEY: We want to talk to you about how we want to use 00-02 to facilitate NRC review of option 2. That's what we're going to talk about today. We've already briefed the committee a number of times on the general process of 00-02 or the peer review process, and wanted to move beyond that today and talk about specifically how we would focus the NRC review of option 2. DR. APOSTOLAKIS: Well, I have some specific comments anyway. MR. BRADLEY: Okay. That's what we're intending to do. These are some of the slides we've used in some of our recent discussions with the Commission and with our own working group at NEI. We do believe that a peer review will always be required, that you can never reach a point where you can have a checklist that would give you the necessary information to use a PRA for an application without any further advanced review. DR. WALLIS: What is the second bullet here? What do you mean by that? MR. BRADLEY: That there is a certain amount of engineering judgement inherent in the process. DR. WALLIS: Well, I'd say there's judgement in how extensive it needs to be or how much evidence you need, but it's not judgmental. It's based on evidence. It's based on scientific reasoning. It's not inherently judgmental. MR. BRADLEY: Well, there are some judgmental aspects. MR. FLEMING: I think what we meant by that -- and maybe the choice of words could have been refined -- is that PRA is not a tangible, measurable type of thing, it's based on a state of knowledge, and of course, the state of knowledge -- DR. WALLIS: All science and engineering is, obviously. MR. FLEMING: Right. DR. WALLIS: That's the kind of remark I would have expected from UCS. MR. BRADLEY: Well, I'm not UCS. DR. WALLIS: I'm sorry. You'll have to clarify what you mean by a statement like that. DR. APOSTOLAKIS: I think the word "inherently" is a bit too strong. DR. WALLIS: Much too strong. MR. BRADLEY: The point we were trying to make here is that, regardless of what requirements you write into a standard or to a process, that you need a peer review of a team of experts to look at it. DR. WALLIS: Well, let me try this. If I said the thermodynamic analysis is judgmental in the sense that you have to use judgement in how far you're going to go in detail, that's true, but it's not inherently so. It's inherently scientific. MR. BRADLEY: Okay. DR. WALLIS: Is that what you mean? That's what you mean, isn't it? MR. BRADLEY: Yeah. DR. WALLIS: Okay. DR. SHACK: What you mean is that you don't believe you can do a design to PRA standard. Is that what it means? MR. FLEMING: Yes, I think that's what it means, and I'd just amplify on it. The "judgmental" refers to the fact that, right off the bat, to select the initiating events for a PRA model involves many, many judgements about what's important, and those judgements are inherent into the process of selecting initiating events. DR. WALLIS: Well, that's inherent in the thermodynamic code. I mean how much detail are you going to go into? MR. FLEMING: But in a thermodynamic code, at least I can design an experiment to go out and benchmark my computer code against some actual measurements. DR. WALLIS: You can do that with PRA if you have enough time. DR. POWERS: What you say is perhaps true, but since no one ever has enough time, I think the distinction has to be drawn here. But maybe, Karl, it would help if you gave us a few examples of where this judgement has to be made, because clearly, I could say things in a standard like, you shall use plant-specific data for the reliability of valves, okay. I may not want to, but I could. MR. FLEMING: I think where this comment was heading is that, no matter what you write in a book in terms of criteria for performing a PRA, a standard for a PRA, or a process for doing a review, it requires expertise to perform the PRA and to review the PRA, and no matter what you put in there, judgements will have to be made about whether the state-of-the-art has been applied in an appropriate way. So, it's just an acknowledgement of that, and I think it's more important than the thermodynamic example -- I have to take issue with that -- because the quantities that we calculate in a PRA, core damage frequency, are not observable quantities. The temperature of a mass is an observable quantity. So, we don't have the scientific ability of experiment to match up the theory to the same extent we had with thermodynamics. DR. APOSTOLAKIS: I think there is a place in the PRA where judgement is crucial, and this is in the collection of data, where you have to decide whether a particular occurrence in a failure or not. You really need experienced analysts to declare that something is a failure or not, especially when you go to common cause failures, you know, whether that failure applies to your plant and so on, as Karl knows very well. So, I think this is a judgement that you don't really find in other more deterministic -- DR. SEALE: PRAs are probabilistic, but they're not actuarial. DR. APOSTOLAKIS: They're not actuarial, and again, they agree that the word "inherently" perhaps was too strong. I mean there is always an element of judgement. So, let's go on. MR. BRADLEY: I think I've already covered this slide. Basically, what we're asking for with option 2 and with any application is the ability to focus NRC's review. We also -- I want to make the point that -- as you'll see, I think, in Karl's presentation, the peer review process does elucidate the areas in a PRA that need improvement to support an application, and we're going to illustrate how it does that and what some of the improvements that have been made in some PRAs are as a result of peer reviews, but we do recognize that a number of the existing PRAs will need improvements to support the option 2 application. We've already provided the NRC staff with the schedules for the upcoming peer reviews. They're being conducted by all four owners groups, and we believe the process itself, in order to understand the process and appreciate its value, it's not simply reviewing the process guidance, it's a dynamic process, and it's a very intensive process involving the team and the plant site and includes preparation prior to the review and meetings every day and a final interaction between the peer review team and the utility. And the whole process -- it really needs to be observed to be appreciated, and we've extended the invitation to NRC staff to observe some of these upcoming reviews, and I'm going to extend that same invitation to the members of the committee, if they're interested. The staff is working now with the schedules, and you know, please let them know if you're interested in observing. We strongly believe this is a credible, good process, and we want as many people as possible to get out there and observe it firsthand. DR. APOSTOLAKIS: Could you have a Heisenburg effect here? MR. BRADLEY: Well, at some point, that's possible, you know, if we had 20 observers. There obviously are some practical limitations, but we certainly would work with the schedules we have and with the interest to facilitate as many people as we could. Another thing we recognize is that the process right now is a one-time review, although there are a few sites that have been through or will go through it twice, but in order -- there may be a need to develop a closure mechanism to address, one, whatever deltas may come out of the NRC review, if there's a belief that certain technical elements need to be improved or whatever, or if there are substantial improvements made to a plant PRA. There may be cases where you need to do a second - - some type of a streamlined peer review to help close that loop, and we're looking at mechanisms that might be available to do that. Also, with regard to facilitating NRC review of option 2, ISI is the example we use. We're going to have 78 units coming in to apply for risk-informed in-service inspection over the next two years. I mean that is, by far, the most successful of the optional type applications we've achieved yet, and in order to have NRC review that many applications, we developed a template for the review, and that template includes a summary of your results, as well as a discussion of the peer review results and what improvements you may have needed to make to the PRA to allow you to do the ISI application. DR. APOSTOLAKIS: As you say repeatedly in the report, you don't give an overall grade. You grade individual elements. MR. BRADLEY: Right. DR. APOSTOLAKIS: So, when the licensee comes for an ISI application, then the licensee submits the PRA and says, look, for the elements that are really important to ISI, the peer review process ended up with the highest grade. For others, where the grade is lower, they are not really that relevant to this particular application. I can see that happening, but I don't see a licensee coming in here and saying the result of the review process is that, for this element, we got a low grade, and that's important to ISI. Is that correct? MR. BRADLEY: That's correct. DR. APOSTOLAKIS: Okay. So, it's for the elements that are not relevant to the particular application where a lower grade would be tolerated. MR. BRADLEY: That's right. DR. APOSTOLAKIS: Okay. DR. WALLIS: Can you explain the third bullet? MR. BRADLEY: The template? DR. WALLIS: That sounds to me as if you're telling NRC what to do. MR. BRADLEY: No. What we want to do is work with the staff, just like we did on ISI, to find what elements are important to their review of the application and make sure that we can capture those in a streamlined practical way, and remember, NRC set out to implement option 2 with no advance review, which I think is an incredibly ambitious undertaking, and I guess industry's view is that that's really not feasible, that there will have to be some review. Obviously, the entire detailed documentation, models, and everything else are going to be available for inspection and assessment. DR. WALLIS: Doesn't NRC develop its own templates in the form of standard review plans and things like that? MR. BRADLEY: This is simply a review template. This isn't an inspection template or anything else. This is a template where we can agree with the staff that, if a licensee puts this set of information into the application, that will help NRC to be able to do an expeditious review. That's all we're trying to achieve here. DR. WALLIS: This is a question, perhaps, of getting the words right, but it doesn't look right, as if you're developing a template which looks as if you're telling NRC what to do. MR. BRADLEY: No. Finally, the -- in listening to Dave Lochbaum's presentation, there was quite a bit of discussion of the need for updated risk information, and industry is aware of this need. I don't think it serves us to have to continually defend studies of IPE results that are 12 years old, and really, we've moved well beyond those for the majority of plants, and there are two elements I think we're looking at. One is, for those plants that actually would undertake regulatory reform, like an option 2 application, we're looking at developing some type of summary description that would go into the FSAR for that plant, because at that point, you really are putting the licensing basis more firmly into a risk-informed arena, with option 2 or option 3, for that matter, and those plants, we believe, would need to develop something along those lines, and so, we're working with that as part of the option 2. Now, there's another question of general industry risk information and making summary information available for all plants, you know, in terms of updating the type of information that's out there for the IPEs now, so that there's publicly available, current risk information. We're not talking about docketing the models or anything but coming up with some reasonable high-level summary that we could -- probably in some kind of matrix form or something. And those are the two things we're looking at now as an industry to try to get updated risk information to the forefront. I think of the -- you know, I would agree that all the stakeholders would be served by having more current information. DR. APOSTOLAKIS: So, when you say that this is to support the implementation of option 2, essentially you're talking about the classification of components -- MR. BRADLEY: Yeah. DR. APOSTOLAKIS: -- risk 1, 2, 3, 4, which essentially means Fussell-Veseley and risk achievement worth. Okay. But the first bullet also asks the ACRS to comment on the whole document, not just option 2. I mean you have briefed us before, but we never really reviewed it as such. MR. BRADLEY: If you want to comment on the document, that's great, and I'm sure you will, but what we're really asking for here is your interest in observing the actual process. DR. APOSTOLAKIS: Okay. That clarifies it. MR. BRADLEY: Okay. At this point, I'm going to turn it over to Karl, and he's going to go into a little more detail about how we would -- I guess the other thing I'd mention is we haven't yet sat down and developed all the details of how the option 2 review template would work, but we have done for other types of submittals that we're doing now, such as tech spec AOT extensions, and we're going to talk about just using that as an example to show how we're thinking here. DR. SHACK: Will the peer review results be publicly available in any sense? MR. BRADLEY: The peer review results -- to the extent that you make an application for, say, an option 2 or ISI or whatever and you go on the docket with a summary of your strengths and weaknesses and how you disposition those to support the application, the answer is yes. Whether we would docket the -- you know, the lengthy detailed peer review report, probably less likely we would do that, although at this point, we are still looking at the level of detail of information, but I think the answer is generally yes, in some form. MR. FLEMING: What I'd like to do in the next few minutes -- I apologize for the volume of material in the hand-out package. When I was warned about having too much material for an ACRS presentation, I told my colleagues at ComEd that I wasn't ambitious enough to believe that I would get through all my slides, even if I had a single slide, but I did want to kind of point out some examples of how recently, especially in the ComEd case, how very recently the certification process was used to help support a specific application, and the purpose of this is to try to bring out, I think, a better understanding of how the industry or at least one element of the industry plans on using this process and maybe clear up some possible misconceptions that I think may have arisen on the certification process. So, in this particular example, we're talking about the processes that apply to ComEd. A few things that I wanted to point out -- I don't want to go into the details. I know you've had the NEI 00- 02 report to review. Just a few key highlights I wanted to bring out. I've been involved personally on six of these certifications, three on the BWR side and three on the PWR side, but in this particular example I'm going to go through, I was on the end supporting the PRA team and using certification findings done by others. But what these consist of is a team of six, sometimes seven people who spend about two to three person months total reviewing the PRA documentation in a very structured process. They do homework, quite a bit of homework before the actual site visit, and they probably spend a good 60 hours in actual on-site review at the site. DR. APOSTOLAKIS: Does this include a walk-down? MR. FLEMING: Yes. There is a focused walk-down made by a subset of the team to look at specific issues that have come up in the PRA. What's important to understand is that there's a very structured process. I mean every minute of every day of the on-site review session is structured in terms of identifying assignments for different people to take the lead on different parts of the review, and what's also important to recognize is that there is a very important set of products that are produced in the certification team, and I want to make a little bit of a comment about the certification team itself. The certification teams that are being put together include at least one or two people who are recognized experts in PRA that a lot of you would probably have seen in the PRA community before but, importantly, include members of the same owners group utility PRA staffs, which provides some very useful features of the certification process. One of them is that the people that are participating on these reviews know plants, and they know plants of the same owners group vintage that is being reviewed, and they bring into the certification team insights from their plants in terms of the plant features that are important to risk, as well as the features that they brought from their overall PRA program. So, they leave on the doorstep of the certification process recommendations on how the PRA could be enhanced, and they also have a capability of going in and finding weaknesses, especially in the area of looking at the plant fidelity -- the plant model fidelity issue, and I think that's very important. You can have people that are very, very experienced in PRA come in and not really know the plant very well and not necessarily do a fruitful review. For each -- in the process that we come up with, there's a grading system, 1 through 4. There's a lot more information in the report for that. This grading process is an important part of the certification process, but it's not the most important part, and I think its uses maybe have been somewhat overstated. What really is the most valuable aspect of the process, from my point of view, and all the people that are participating on these, is a detailed set of fact and observation forms that identify the strengths and weaknesses of the PRA. These are very, very specific issues that the team comes up with that are put into different classifications of priority. The most important categories are the A and B categories, which could have a significant impact on the PRA results. Category A are recommendations for immediate attention and update of the PRA before it's used in decision-making. Category B are other issues that are considered to be important but could be deferred until the next update. The C issues are issues that probably don't impact the baseline PSA results but could be important for specific applications. Category D issues are the ones that are basically editorial comments that are recommendations for cleaning up the documentation, and a very, very important category is category S, where the team is identifying particular strengths, where this particular element of this PRA is recognized as an industry leader in that type of an activity. DR. WALLIS: This is like the inverse of the academic grade. I mean A means bad and D means good. MR. FLEMING: That's right. DR. WALLIS: It's a bit unfortunate. DR. APOSTOLAKIS: They keep their distance. MR. FLEMING: That's right. It probably reflects the fact that so many of us have been so long out of school. DR. WALLIS: Don't come round saying all the plants got A's, therefore it's good. MR. FLEMING: That's right. Now, the other -- DR. LEITCH: Before you move too far away from the certification team, it seems to me that there's a measure of subjectivity in this peer review and that that subjectivity is largely tied up in the team. Is the team always independent of the plant that is being evaluated? MR. FLEMING: Yes. There are requirements for independence, and as part of the documentation for the peer review is basically an affidavit, a statement by each team member, who identifies his independence from the PRA team. Now, there have been a few cases where a particular team member has been involved, for example, in the level 2 aspects of the PRA, where that person basically excuses himself from any of the consensus sessions, and that's a reality, is that if you force -- if you want to force too much independence, you may not have adequate expertise to do the review. DR. LEITCH: Yeah. MR. FLEMING: So, there have been a few exceptions like that, but there's an affidavit, a statement made in the documentation, a declaration of independence or a declaration of whatever involvement they did have, so it's on paper and documented, and that's part of the process. DR. LEITCH: The other side of independence, as you point out, is having an adequate knowledge base, and you obviously need people that are well versed in the issues. So, it's kind of a two-edged sword. MR. FLEMING: That's right. In the formulation of the team, there's quite a bit of effort that goes together by the owners group chairman -- for example, Barry Sloan on the Westinghouse owners group, takes the lead on that, and Rick Hule on the General Electric one, and so forth -- to make sure that the specific team that has been put together covers all the expertise needed to review the elements of the PRA. DR. LEITCH: Might a particular BWR, though, have a set of six or seven people that are totally different than those that are evaluating another BWR, or would there be some commonality between those players? MR. FLEMING: They worked hard to have some common players on the certification team to make sure that there's a carry-over of consistency, and that's the main ingredient that's put into the process to try to improve the certification and certification consistency. An important product, though, is the third item, which is, having identified strengths and weakness of the PRA, very specific recommendations on what could be done to get rid of or to resolve the issue, and a very important part of this is that, for the most important categories of issues, the A/B issues on the negative side and the F's on the positive side, it's a requirement that there's a consensus of the entire six-or-seven-member team on these grades, because they're very important. We realize that we're leaving on the doorstep, in the case of A and B, issues that have to be addressed, and we want to make sure that it's not just the opinion of one person. In the consensus process and the participation of Bruce Logan of INPO, for example, he recognized that to be a very, very important part of this. It's not just a bunch of opinions that are rolled together; it's a consensus process. DR. KRESS: Just for clarification, when you talk about each PRA element and sub-element, just exactly what do you mean by that? MR. FLEMING: What I mean by that -- if you look at the NEI 00-02, the PRA is divided up into -- I can't remember the actual number -- about 10 or so elements. Initiating events would be one, and then those are further broken down into sub-elements, and there's a total of 209 of these, and these are just simply the elements within the elements. So, for an initiating event, there would be sub- elements for identifying and for grouping and for quantifying and frequency and so forth, and it just provide -- that checklist is simply a way to add structure to the process to make sure that we're looking at, you know, all the same things in each one of the reviews. It's not intended to be a comprehensive or complete all-inclusive list, but it's enough of a structure to provide some consistency for the reviews. DR. WALLIS: How does this affect the uncertainty in the results? It seems to me that you can keep on improving the structure, keep on updating, but it doesn't mean to say that the certainty or the confidence you have in the answer that's being given is necessarily increased as a result of all this. MR. FLEMING: Well, I'm going to give you an example in a second that I hope to address that, but one of the elements that is looked at is quantification, and as far as the confidence in the overall results of the PRA, whether they make sense, that's looked at in great detail in the quantification element, and if there is believed to be technical issues, A and B issues, in particular, that could impact the quantification, those are brought out in the review. I'm going to walk through an -- DR. APOSTOLAKIS: I have questions. When you say 1 is IPE and 2 is risk-ranking, 1 means identifying the dominant sequences, vulnerabilities, and 2 means the option 2 application? MR. FLEMING: This is better explained in, actually, the NEI report. It's recognized that there's a continuum of quality levels for each element of the PRA, and arbitrarily, those were broken up into four levels, and the general definition is that 1 represents a PRA that just meets the requirements of the IPE, and it's just a anchor point, historical anchor point to be able to take reference to what's already happened in the industry. A 2 means that the PRA is capable to support application involving screening, you know, screening kind of applications. We call it ranking, but what it really means is being able to screen SSEs into broad categories of high, medium, and low safety significance, where you're not really worrying too much about the absolute number. Three is the risk significance determination, which would be like a Reg. Guide 174 kind of an application, and 4 is something to capture, you know, a state-of-the-art level treatment of the actual element. DR. APOSTOLAKIS: Well, the impression I get from reading the report was slightly different, that 1 was really identifying -- being able to identify the dominant sequences, 2 was importance measures, and then 3 and 4, I agree with you. MR. FLEMING: Yeah. DR. APOSTOLAKIS: This is very important, because if you're going to the body of the report, there are some questions that are raised, and I think I should raise one now. It is stated on page 9 and then on page 18 that you don't need to do a time-phased analysis -- is that what they call it? -- for 1. I lost the page now. In other words, if you have a certain window of time and the operators have to do something, that you don't need to do that if you do a 1, grade 1, I guess, and then that common cause failures, on page 18, are not needed. It says explicitly are not needed for risk ranking, okay? Now, the note is "not required for successful ranking or dominant contributor determination." Now, there was a PRA which you are extremely familiar with where the number 1 accident sequence was loss of off-site power, loss of all the diesels, and failure to recover power within the available time before core uncovery occurs. That's the number one contributor. If I am not -- I mean I don't know how you lose the diesels there, but I'm sure common cause failure played a role. If I don't do common cause failure analysis and if I don't do this time-dependent human probability for recovery of power, I will never be able to identify this dominant sequence. So, my grade 1 and 2 will really not give a reasonable result, unless I go to 3. MR. FLEMING: Yeah. I don't have the document in front of me, but I think that may reflect a poor wording in the document. I can assure you that, if someone came and presented a PRA for the certification process that didn't model common cause failures, they could not get a grade higher than 1. DR. APOSTOLAKIS: Well, it's very clear in the note. MR. FLEMING: Okay. DR. APOSTOLAKIS: "Not required for successful ranking or dominant contributor determination." MR. FLEMING: Okay. Well, yeah, I can't account for that. DR. APOSTOLAKIS: Do you remember, perhaps, where the definition of the grades is given in the document? MR. BRADLEY: I think if you could let us proceed, you'll -- what we're trying to make the point here is that we're not trying to hinge the review on the grades, and all this discussion of the grades is really a little bit tangential to our intent here of trying to show how we're going to use the process, certainly not our intent to go in and say I got a grade X and therefore it's okay. That's not how we're doing this, and obviously, things like common cause and time dependencies are going to be important for most applications that we're going to apply this to. But I think if you could possibly let Karl proceed, we might answer some of these questions. DR. APOSTOLAKIS: Well, it seems to me that this is such an explicit instruction here -- it says there is a note, "not required for successful ranking or dominant contributor determination," and here is a major plant where the number one sequence involves these things. I mean that's a little troublesome, isn't it? MR. FLEMING: Well, I can assure you that it is required. In the certifications that we're doing, if you would come in with a model without common cause failures in it, it could not get a grade higher than 1, and 1 is just a lower bound that the system -- we don't have a zero. Probably, we should get a zero in that case. I can't explain this particular aspect of the document, but the document is a problem, and I can't recall seeing a PRA that does not have common cause failures of diesels modeled. I don't think there's any out there. So, I don't know whether this is really an operational issue or not. The document may have some flaws in it. What I'd like to do now is to basically walk through a little case study of how, at the ComEd Byron and Braidwood stations, this certification process was used in an actual successful application, and if I can walk you through the -- in this particular case study I wanted to walk you through, back a few years ago, ComEd decided that they want to pursue a risk-informed application involving a 14-day diesel generator AOT, and that was done in conjunction with a decision to upgrade their PRAs, to take into account advances in PRA technology and design changes since the original IPEs were done. So, they're in the process of doing a major upgrade to the PRA, and they also had decided to pursue a Reg. Guide 177-style submittal to request a 14-day extension on the diesel generator allowed outage time. So, the Westinghouse owners group certification was scheduled for September 1999. That was scheduled during the period of the PRA update, and it was scheduled, actually, to provide an opportunity to get some input while the PRA upgrade was being actually completed, and then what happened was that, on the basis of the fact and observations, strengths and weaknesses that were identified during the Braidwood certification, there was a continuing process of upgrading the PRA, and then, in September of 1999, there was a submittal to the NRC staff requesting diesel generator AOT extensions for both Byron and Braidwood. Byron is a sister plant to Braidwood, and the PRA models are very similar, but there are differences, as well. In the submittal itself, it was a Reg. Guide 177- style submittal. There was information submitted to summarize the updated PRA results, and there was also a representation that a certification process had been done to support the basis for the quality of the PRA supplied in the application. After that and while the NRC was in the process of reviewing the submittal which was made in -- actually, the submittal was actually made in January 2000, this year. Later on, in the summer of this year, there was a followup certification on the Byron plant, and that was -- provided a special opportunity, since the Byron and Braidwood PRAs were being done concurrently, because of the similarities in the plant. There was an opportunity for basically the same certification team to come back and take a look at how the issues identified for Braidwood had been resolved that applied to Byron, as well, which was essentially 98 percent of them, and at the same time provided sort of a confirmation that the strategy taken by ComEd to resolve the technical issues that came up had been satisfactorily addressed, and that was reflected by a significant improvement in the results of the certification process. DR. APOSTOLAKIS: Now, when you say Braidwood was doing a PRA, what do you mean? They were doing what you and I would understand as a PRA? MR. FLEMING: Right. They were in the process of upgrading their PRA from soup to nuts, you know, converting the software, going back over the initiating events, constructing new event trees, success criteria, the whole aspect, and as you may recall, the original IPEs submitted for the ComEd plants were subjected to a lot of issues associated with a very different set of success criteria and so forth. So, there was just a lot of background in terms of lessons learned from the original IPE process that ComEd wanted to take advantage of, and they basically have completely updated all the PRA programs at all five of their plants. DR. APOSTOLAKIS: So, they were not upgrading their PRAs, the parts of the PRA that they felt would be useful to this particular application. They were upgrading the PRA, period. MR. FLEMING: Well, they were upgrading the -- I'm glad you mentioned that. They were upgrading the PRA for a range of applications that they had planned to pursue, which included risk-informed tech specs, included risk-informed ISI, included supporting their configuration risk management program, and others. DR. APOSTOLAKIS: Okay. MR. FLEMING: So, they did have a specific package of applications that they wanted to pursue, but the first one -- the first like Reg. Guide 174 application that was launched as a result of this upgrade was the diesel generator case. DR. APOSTOLAKIS: So, would you say, since you are very familiar with the process, that they were coming close to having a category 2 PRA of the ASME standard? MR. FLEMING: I'm going to get to the details of that in a -- I'll answer your question in a second, if I might indulge -- have your indulgence on that. Now, what happened was, in the course of the NRC review of the diesel generator tech spec submittal, they asked for some additional information on the results of the certification process, and as a result of that, ComEd submitted a summary of the category A and B issues, the ones that were given the highest priority, together with what had been done to address each one of the issues, and that then led to the final completion of the NRC review, and just recently, the NRC has issued the safety evaluation report granting the risk-informed tech spec. So, that's sort of a synopsis of how the process was used, and I want to get back into some insights came through the overall process. What's been happening here is that it sort of illustrates in one particular case study that a decision had been made to use the PRA, a certification process was identifying specific strengths and weaknesses of the PRA, and through this overall process, there were a number of risk-management insights that -- we can argue about certification processes and standards and things like that, but the bottom line is that the risk-management process was working and working very well. The two big issues that had been identified in the Byron and Braidwood PRA -- one involved vulnerability due to internal flooding scenarios in the auxiliary building, where there was a possibility of floods that would take out the safety-related service water pumps located in the basement of the aux building, and loss of service water, of course, was a very serious event at this plant, Westinghouse plant, which would lead to a reactor coolant pump seal LOCA problem. The other issue was that there was a very large contribution due to reactor coolant pump seal LOCAs, and the first insight was that, actually through Westinghouse's -- I'm sorry -- through ComEd's participation on the Westinghouse owners group certification process, they became aware of different strategies that were being used by different Westinghouse plants to reduce the risk of reactor coolant pump seal LOCAs. And they actually decided to implement one of these changes, which has to do with providing a way to use the fire water system to provide an alternate component cooling pathway for the charging pumps so that, in the event that you would lose service water and you had the charging pumps available, you could maintain a path of seal injection. It turns out that a very large number of the Westinghouse plants have been using, you know, techniques like this to reduce the probability of the conditions for the pump seal LOCA sequence. So, the actual -- you know, the participation in the certification process actually led to this insight, led to a decision to change the plant design and improve the risk with respect to this aspect. The second one was that plant modifications were made to address the internal flooding issue, which was the dominant contributor in the PRA reviewed by the certification team, and plant modifications were identified to also reduce this risk contributor. The other thing that was discovered through this process was that, in going through the evaluations required by Reg. Guide 177 and Reg. Guide 174, the risk metrics that we were using to evaluate the acceptability of the 14-day allowed outage time turned out to be not affected by either the flooding risk or the modifications that were put in place, and what was a little bit difficult about this application was that ComEd was in the midst of managing the risk of flooding, managing the risk of reactor coolant pump seal LOCA during the course of making this submittal to the NRC. Another risk-management insight that we found was that, when we tried to calculate the incremental risk metrics that Reg. Guide 177 calls for for evaluating tech specs, we discovered that just a straight application of those risk metrics led to problems meeting the acceptance criteria. And that led to insights back into the PRA to determine insights from the configuration risk management program on what compensatory measures needed to be taken while you're taking a diesel generator out of service to be able to justify risk acceptance criteria being met, and as a result of this process, it was determined that the risk acceptability or the risk insights that bear on the question of acceptability from this overall process actually was dictated by how the plant configuration was managed during the 14-day diesel generator outage time, and these insights were actually reflected in the license amendment request and in the NRC safety evaluation report. Now, how did the -- just want to talk a little bit -- how did the certification impact all of this, and this happens to be a roll-up of the grades that were obtained in the original Braidwood IPE or PRA review process, and as noted in NEI 00-02, there's grades given at the sub-element and element level but not on the overall PRA. This is a rack-up of what grades were given by the team for each of the elements of the PRA. The parentheses (c) means that the grade level 3 was provided under specific conditions that specific issues that came up in the PRA were identified, and those issues are identified in the specific fact and observation sheets that are sort of tallied here in this table. So, the overall flavor of the certification review process was that they either got 3's or condition 3's but the conditions were conditioned on very, very specific issues that the certification team took issue with that didn't think were quite adequate for supporting the application. MR. BRADLEY: So, that would be conditional on those being resolved. MR. FLEMING: Yeah. So, what this suggests here is that, you know, it wasn't so much the grades themselves that were important, was the specific things that had to be done to be able to support the risk-informed application. DR. APOSTOLAKIS: Let me understand the initiating events, the first row. MR. FLEMING: Right. DR. APOSTOLAKIS: You have A, B, C, D, S are the possible grades. MR. FLEMING: No, the grades are 1, 2, 3, 4 for initiating events, and what's listed in the rest of the table are basically a frequency distribution of the number of fact and observation issues that came up for initiating events. So, a total of nine comments were made or technical comments were made for initiating events by the whole group, and they are distributed according to priority. You know, there was one A issue, two B issues, and four C issues and so forth, and each one of these is documented in the form of here's the technical issue, here's what we think ought to be done to resolve it, and so forth. DR. APOSTOLAKIS: And what you call a grade was derived from those how? MR. FLEMING: The grade was derived by looking at all the sub-elements, the grades for the sub-elements, which I haven't showed you here -- DR. APOSTOLAKIS: For initiating events. MR. FLEMING: -- for initiating events, the specific fact and observations -- in other words, the technical issues that were identified for that, and then there was -- those were weighed against the overall criteria for grades 1, 2, and 3. So, what you don't see here -- there's a big detailed checklist for initiating events that has grades for maybe 25 or 30 sub-elements for initiating events, identification, grouping, support system initiators, and so forth, and what this table simply shows is that the certification team gave an overall grade for initiating events of a conditional 3, meaning that, if these three items in column A and B, if those issues were resolved -- DR. APOSTOLAKIS: If they are resolved. MR. FLEMING: If they are resolved, they would qualify for 3. They're effectively a 2, with the path to get to a 3 by meeting these particular issues. DR. APOSTOLAKIS: There is a 3 and a C. Three means it can be used for risk-informed applications? MR. FLEMING: Under conditions. DR. APOSTOLAKIS: Under these conditions. MR. FLEMING: Under these conditions. DR. APOSTOLAKIS: And C means desirable for applications? Why is the parentheses in a C? MR. FLEMING: I'm sorry, that's a different C. The C in the grade column simply means that there is a condition on meeting the grade, whereas C in the other column means it's categories -- those are different C's. I'm sorry to confuse you. DR. APOSTOLAKIS: So, the grade, then, coming back to your second slide or so, is 3 refers to risk-informed -- MR. FLEMING: That's right. DR. APOSTOLAKIS: -- could be used for risk- informed applications -- MR. FLEMING: Yeah. DR. APOSTOLAKIS: -- according to 1.174. MR. FLEMING: And the C means that you don't get the grade 3 unless you meet specific -- if you address specific issues, and I'm going to give you what those issues are in a second. MR. BONACA: The A is significant. There is no modeling of the ABG design feature, but there is a 3 without a condition. DR. APOSTOLAKIS: System analysis? MR. BONACA: System analysis, for example. MR. FLEMING: Well, in the case of systems analysis, the team did not feel that the issues in this case were significant enough to affect the grade of 3, but keep in mind that the utility is still left with A and B issues, and that's one of the points I want to get here. They're not just going to stop because they get a 3. They've also got to resolve their category A and B issues. MR. BONACA: I think the issue -- the second issue in the next table -- it was resolved. MR. FLEMING: Yeah, right. DR. APOSTOLAKIS: Why can't the industry do this for every single unit and have a grade 3 PRA so we will not have to argue about 1 and 2? MR. FLEMING: I think the industry wants to get there, but they want to get there along an optimal path of allocating resources. They want to be able to see where they are, measure where they are right now, see what kind of applications they want to do this year, next year, and the year after that, and they want to advance towards quality in the most cost-effective strategy. I think that's what they want to do. DR. APOSTOLAKIS: That's my fundamental problem. Can one identify the dominant contributors without having at least a grade 3 PRA? That's my problem, because there is an allowance for that. You know, in category 1, all you're looking for is vulnerabilities. Can you really do that without having a level 3 PRA? That's my problem. I agree with you that they want to follow the optimal path and get some return for their investment on the way, but it seems to me this is the baseline PRA that we should have, and this process is very good. MR. FLEMING: What I believe is, in this scheme, the category 2, 3, and 4 all have to be able to identify the dominant sequences. Category 1 is simply a historical milepost. DR. APOSTOLAKIS: Okay. MR. FLEMING: Okay? DR. APOSTOLAKIS: But again, you dismissed the document earlier, but we have to go by the document, and if the document says that, for category 2, I don't have to look at the time calculations, like, you know, recovering AC power and then I know that the PRA you managed came up with a number one sequence that involved that, I'm having a problem. MR. FLEMING: I think what the time comment referred to in the document is the level of detail in the calculation of these time-sensitive sequences. In other words, in category 2, you could roll up your frequency of loss of off-site power and probability of non-recovery in a very simplistic time-independent model, whereas for the category 3 and 4, you'd have to basically be able to delineate how much was happening in the first hour and second hour and third hour. I mean it's a question of level of detail and simplicity. It's not things that are missing. DR. APOSTOLAKIS: But it seems to me that -- you know, later on, we're going to discuss, also, the ASME standard. A lot of the disagreement comes from the fact or from the apparent claim that you can do certain things by doing a limited number of things for a PRA. I mean if you do what you're describing here, I think a lot of the controversy would go away, but that's not what the documents say. MR. FLEMING: The document is the document. The document is part of the process, and that's one of the things I wanted to try to get off in this presentation, is to walk you through in a complete soup-to-nuts application to show you how this is actually being used. The document is one part of it. It may be an imperfect document. DR. APOSTOLAKIS: My problem is not what you're describing, because you see you are ending up with a grade 3 PRA. The question is, will there be other licensees who will be happy with a 2 there, instead of a 3, and they would demand risk-informed decisions from the staff? MR. BRADLEY: It depends on what the risk-informed decisions are and how you're using the PRA to support those. It's conceivable there could be certain risk-informed decisions. We're doing it today with a number of things that we're doing. You can't distill it down to a black-and-white line. With regard to 00-02, which you have in front of you, try to look at that in the context of the letter with which we submitted that to NRC, and we are asking for a review with respect to a specific application and also with regard to the ability to focus NRC's review using the facts and observations, not to obviate their review in a specific application. We're trying to get this down to some pragmatic thing we can do to get option 2 implemented here, and I think a lot of the questions you're asking have to do more with the general approach of 00-02 and specifically the four categories, which were developed a long time ago, and you know, we've been through this on the ASME standard ad nauseam with trying to define what fits into what category, and we're really trying to just sort of get away from that here with regard to how we would use this in option 2. It's a specific application. DR. APOSTOLAKIS: I understand that, and the question is, then, is the methodology required for identifying the dominant sequences different from the methodology required to classify systems, structures, and components, and if so, why? Do you require a simpler methodology to place SSCs in those four groups? MR. BRADLEY: It depends on the details of the categorization process. That's why we're asking for these things to be reviewed in concert. In depends on how -- where you draw the line, what sensitivity studies you use, and all kinds of other aspects, and how this feeds into the integrated decision process of the option 2 application. That's why you have to look at this in context with the categorization. DR. APOSTOLAKIS: And that's the problem. Can I use a methodology that will miss a major accident sequence and yet will give me satisfactory results? MR. FLEMING: Let me see if I can address that. First of all, the only of these four grades that have any practical significance is grades 2, 3, and 4. Grade number 1 is basically for historical reference purposes, and I don't think anyone in the industry would be satisfied with a grade level 1 anything in their PRA. They're getting some grade level 1's at elements and sub-element levels, and they're fixing those and getting up at least to grade level 2, but for grade level 2, 3, and 4, it is necessary to be able to identify the dominant sequences, and for grade level 2, whatever else you have to have in order to be able to do risk screening, and there are utilities that are happy to use a grade level 2 PRA to be able to identify that something is not important, to be able to say that this set of motor-operated valves is definitely not important. You don't have to have a lot of detailed PRA information, necessarily, to be able to do that. If you have enough -- a minimum threshold or critical mass, if you will, to call this thing a PRA, you need to be able to identify the dominant sequences and at least be able to put components, SSCs, into broad categories of safety significance. DR. POWERS: I get the impression most people understand that. The question is that you do need some details PRA results to do that categorization. MR. FLEMING: Yes. DR. POWERS: The question is which detailed ones? Professor Apostolakis has found a contradiction that he brings to your attention here in discussing level 1. You don't want to discuss level 1, because it's meaningless now, but the same contradictions are potentially available to us in levels 2, 3, and 4, aren't they? MR. FLEMING: I guess I don't really appreciate what the contradiction is. DR. APOSTOLAKIS: I followed what the document said, and it specifically says, for 1 and 2, in fact -- I think it includes 2, and it's a very important point, so bear with me for a second. I'll find it. DR. POWERS: While you're looking, I will comment you're succeeding well on not getting through your view- graphs. DR. APOSTOLAKIS: So, on page B-9, designated AS- 13, time-phased evaluation is included for sequences with significant time-dependent failure modes; for example, batteries for station blackout, BWR RCPC LOCA, and significant recoveries." And then it says, for PSA grades 1 and 2, you don't need to do this, and I'm telling you, the number one sequence in the PRA you managed a number of years ago was this. MR. FLEMING: I don't think that's what's intended. DR. APOSTOLAKIS: There may be an error. MR. FLEMING: There's a level of detail. It's not that you don't have to include it. I think you have to include it for the lower grades. It's a question of whether you have to include it using a time-dependent model or a simplified model. I think that's what's intended for that. DR. APOSTOLAKIS: Okay. MR. FLEMING: That's the way we are using it. DR. POWERS: You pose a challenge to understanding your document, then, because a blank no longer means a blank, it means kind of a blank. MR. BRADLEY: I think maybe you're answering the question you asked earlier about why we submitted this for NRC review. We'll go through this thing in detail and specifically ferret out any issue that's going to impact option 2. DR. APOSTOLAKIS: I'm willing to accept Karl's point that maybe there are some mistakes here, but the intent was not to do that. Now, the fundamental question in my mind is, is there a difference in the methodology that identifies the dominant sequences from the one that identifies the significance of SSCs? MR. FLEMING: No. DR. APOSTOLAKIS: There shouldn't be. MR. FLEMING: I don't believe there is. DR. APOSTOLAKIS: You might argue that I can do a cruder analysis for the classification, because I will be very conservative in placing things in boxes. MR. FLEMING: That's what's intended. DR. APOSTOLAKIS: But in principle, there shouldn't be a difference. MR. FLEMING: No, there isn't. DR. APOSTOLAKIS: I agree with you. Dr. Bonaca. MR. BONACA: I just had a question. If you put back the previous slide and you take off the C's, you will have a number of areas where you call it a 2 right now. MR. FLEMING: That's right. MR. BONACA: And it will be mostly 2's. MR. FLEMING: Right. MR. BONACA: And here it seems to me that the effort required to go from a 2 to a 3 is really a minor effort, seems to be, almost. MR. FLEMING: Excuse me? It's a minor effort? MR. BONACA: Yeah, it seems to be. I mean there are a few issues -- granted, these are only the A's, but there are, you know, a number of B's. The question I'm having is that I could say, well, this PRA was already almost a 3, had just a minor number of issues that made it a 2, and I'm trying to understand what is the range of quality in a grade 2, for example. MR. FLEMING: Well, first of all, let me clarify something that I didn't want to mislead you on. The effort it took ComEd to go from the three C's to 3 for the aspects of the PRA that were important for the diesel generator AOT was a major PRA update. MR. BONACA: Okay. MR. FLEMING: It wasn't just going in and doing a few things. In fact, that's what I wanted to -- let me see if I can get through this key slide, because that's really the one I was trying to get to. In the way in which this particular certification was used in this particular application, the grades themselves were not used directly, and what I mean by that is that ComEd didn't come in and say, hey, we got grade level 3's or, you know, we got grade level 3's subject to these conditions and, therefore, you know, we're a grade level 3. The grades were in the process and they're an important part of the process to try to ensure consistency, but you know, the way the process was used is that they found specific technical issues that stood in the way of getting a grade level 3, and they figured out a resolution strategy to get rid of those, most of which involved updates to the PRA. So, there was a substantial improvement in the quality of the PRA driven by the need to get approval for a particular application. The second thing that ComEd does is that all the issues identified -- A, B, C, and D -- the S's are only retained to not lose things that were successful -- are put into an action tracking system, and it's ComEd's commitment to address all these issues, you know, in some priority, but they're going to try to roll in the schedule for addressing the issues in areas that are significant for the given applications, because they don't have unlimited resources to do this. And the final point I wanted to make here is that, you know, the submittal was made in January 2000 for the 14- day AOT for four reactor units at two stations. The safety evaluation report was granted in September of this year, and in that process, NRC was given sufficient information to review this submittal and address quality concerns by information that was presented in the licensing submittal itself, which included summary information of the PRA. An RAI process extracted the A and B issues and what ComEd was doing about them, and that happened about a month before the SER was issued, and the process was successful. DR. APOSTOLAKIS: Is it possible for me to get copies of the submittal and the SER? I really would like to read them. Are these public documents? I would like to have those. I would appreciate that. Karl, the point is -- I mean you are arguing very forcefully about how good this was. I'm with on that. I agree with you. I think what you're presenting is very good. The problem I'm having is -- and maybe it's a misunderstanding on my part -- is the lower grades. I'm under the impression that both the ASME standard and the NEI peer review process allow a licensee to petition for something by using only limited parts of PRA, without having a level 3 PRA somewhere else, just to support that application and then demanding that the NRC staff not look at other things, because you know, some guide says there is a blank there. That's my problem. MR. BRADLEY: It's not the purview of the licensees to demand anything of the NRC staff. DR. APOSTOLAKIS: I'm sorry? MR. BRADLEY: The NRC staff can certainly look at any aspect they want in reviewing any application. DR. APOSTOLAKIS: Yeah. MR. BRADLEY: We would never demand that they not look at -- DR. APOSTOLAKIS: If they have blessed something, you know, either the ASME standard or this, then you can come back and argue very forcefully that, gee, you know, you guys are going beyond the rules of the game. Why do we need category 1 in the ASME? Why do we need category 1 here? And 2. Why don't we all agree that 3 makes sense? Let's do it and use pieces of it as appropriate in applications, which is what you're doing now with the extension of the AOTs for diesels. I think this is great. You told them what is required to come up to standards of level 3 or grade 3, they did it, now they're going to use it in a number of applications. That's beautiful. MR. FLEMING: Well, let me give you an example. In the particular example that I gave you here, what ComEd needed to do is to identify the issues that stood in the way for grade level 3 for those portions of the PRA that were important to the diesel generator AOT submittal, and that turns out to be a rather narrow range of sequences that involve extended maintenance on the diesel generator that don't involve issues of LOCA and ECCS and switch over to recirculation and things like that. DR. APOSTOLAKIS: So, they went to level 4 for those? MR. FLEMING: No. What I'm trying to say is that they only had to make the case that the A and B issues that had been identified had been resolved to the extent needed for that application. Now, the next application, there's another set that are going to become important. I think eventually -- I think that eventually we'll get there, George, but -- DR. APOSTOLAKIS: That's a good point that you're making. MR. FLEMING: I think eventually we'll get there, but I think one of the reasons why the industry wanted to do this certification process is that, you know, if I start with South Texas -- the South Texas experience, South Texas had gone down a pathway, they had invested a lot in their PRA, they paid the NRC for a detailed nuts-and-bolts review, much more than the IPE submittal, and they went down the particular path that was successful for them, and they're industry leaders in that process. One of the things that the industry wanted to do in the certification process is to say let's see what we've got now, let's benchmark what's out there right now and clarify what current applications the utility could do now and delineate which ones he has to defer until he invests the resources necessary to bring the PRA up, as opposed to going to a situation where the industry has to go off and spend millions and millions of dollars to get everything up to grade level 3 and now start applications. DR. APOSTOLAKIS: I think that's a very reasonable approach. MR. FLEMING: So, it's a question of allocating resources. DR. APOSTOLAKIS: What you just said I cannot find written anywhere, and I agree with what you said. I think that, if I look at this table you just had there, you know, with the ABC's and so on -- MR. FLEMING: Yeah. DR. APOSTOLAKIS: Would you put it back on? MR. FLEMING: Sure. By the way, I wasn't originally planning on even presenting this. DR. APOSTOLAKIS: If everyone wants to do this, it seems to me that's great, and then individual pieces, you know, for particular applications, can afford to wait until they fix the A's and B's. That's good, but that's not what's in the document. So, let's go on. MR. FLEMING: I think part of that is that the document doesn't really describe the application process. DR. APOSTOLAKIS: Can you accelerate your -- be more efficient? MR. FLEMING: I'm just going to come to the conclusion right here. MR. BRADLEY: Coming from you, that's an interesting request. MR. FLEMING: I just wanted to summarize. From this particular example, I just wanted to, you know, get across a few points, that for those of us who participate both on the reviewing side and the receiving review comment side, the most important results of this peer review process is, first of all, the delineation of specific strengths and weaknesses of existing PRAs, and a clear road map that results from that on what exactly does the PRA team have to do to bring his particular PRA up to the level needed for a given application. Over time, as the certification process continues, because of the participation of owners group utility representatives from different plants and the information that they carry back to their PRA programs, this will eventually -- and it already has increased the level of consistency across the PRAs, and I think where we were a few years ago -- or at the IPE stage, we had -- most of the variabilities in PRA results were driven by assumptions, judgements, scope, and things that had nothing to do with the plant. I think that we're going down the right path towards getting more consistent application, and the main thing that's contributing to that is the make-up of the teams from the owners group plant PRAs. The grades were good in the sense that they provided an element of consistency from certification to certification. I don't want to discount that, but we're not using the grades in the sense of trying to abuse them by saying, hey, we got a grade level 3, leave us alone, don't bother reviewing our PRA. That's not what we're saying. We're saying we went through the process, we identified the strengths and weaknesses, here's what they were, tell the world what they were and what you did about them, and I think that's the most valuable part of the whole process. So, that's the summary of my presentation. MR. BONACA: Do you have any idea when this Braidwood PRA will be a 3? MR. FLEMING: What happened was that, in the Byron -- the Byron PRA was reviewed approximately a year later, and all of these C's but one were eliminated, and we're in the process right now of trying to figure out what it takes to get that up there. So, I think the Byron and Braidwood PRAs are at grade level 3 right now. MR. BONACA: Okay. Thank you. MR. FLEMING: Or if they're not, there may be one or two specific issues that need to be resolved. This was a year ago, and today, we're much further than that, and that's another key point, is that the certification process by itself does not produce quality, the standard by itself doesn't produce quality, but what does lead to quality is exercising the PRA in actual applications, trying to make decisions from the PRA, and as a technical analyst, we ask ourselves the question, how does the technical conclusion derive from the analysis that I did, does it logically flow to be able to support a decision, and it's through exercising the PRA in specific decisions that lead to quality. DR. LEITCH: You mentioned earlier that, within the next year, some 78 or 79 ISI applications might be received. It seems to me, just on a -- thinking about it for a few minutes, that perhaps all of those PRA elements would be in some way tied up with the ISI program. Might I then imply that, by that time, those 78 units would all have grade 3 PRAs, or am I getting something mixed up there? MR. BRADLEY: I don't think ISI necessarily would exercise all those elements. As risk-informed applications go, it has a fairly limited scope of PRA elements. DR. LEITCH: Which ones would you think would be involved? MR. BRADLEY: You're getting a little bit beyond my expertise here. DR. LEITCH: Okay. MR. FLEMING: I think they are involved, but I think one of the things that reduces the -- I don't know -- the anxiety, if you will, about the PRA quality issue and the risk-informed ISI process is that the individual decisions are done on sort of a weld-by-weld basis. There may be thousands of welds that you're processing this evaluation, and when you start looking in detail about how much the risk associated with a pipe rupture at a weld is going to change because I have it in or outside the inspection program, you come to get an appreciation that the changes in risk that are at stake here are very, very -- they tend to be very, very small changes, because whether you have something in or outside the inspection program doesn't mean that the probability of failure goes from high to zero. There's a very indirect effect of doing an inspection and whether the pipe's going to rupture in the first place, because you can't inspect for all the damage mechanisms that could occur in the pipe, for example, and the localized effects of one weld failing, either as an initiating event or a consequence of initiating event -- you tend to come up with very, very small numbers. I don't know if you've looked at the risk-informed ISI thing, but the delta risks that we're calculating are very, very small, which gives us -- you know, we're so far away from the decision criteria that, you know, we don't have a lot of anxiety, but if you do a risk-informed tech spec, you take the diesel out for 14 days, you can see some very, very real potential effects on the PRA. So, now, we get much more anxious about how well we're calculating diesel general failure rates and common cause failures and these time-dependent issues that were brought up. There can be big swings in the results. DR. LEITCH: Let me just ask my question another way, then. Regardless of what the grades are, then is it reasonable to conclude that these 78 or 79 units would have this peer review process completed in a year, when they submit these? I mean are we that far along? MR. BRADLEY: The 78 units is over the next two years, and given the schedule we have for peer review, I think it is reasonable to conclude that they will have been through that at that time. It's possible that some of the plants that submit will not have completed their peer reviews yet, in which case they can -- there are other ways to get -- you know, NRC is reviewing these applications, and there are other ways to identify the PRA information if you don't have the peer review results. I'm sure there are probably a handful of sites that will be ahead of that curve. DR. SIEBER: It would seem to me that PRAs don't model welds. What's important is an importance measure for the system or some portion of the system that would reflect a chance of rupture, which you could do with a level 2 PRA. MR. FLEMING: Right. DR. SIEBER: Okay. So, the demand on the PRA quality and content is not as high as it would be for other kinds of requests. MR. FLEMING: Having been involved to some extent in the risk-informed ISI arena, in the risk-informed application, one of the steps in the process is to -- having recognized that you don't have the welds in the PRA, you don't have the identity of the welds in the PRA, is to exercise the PRA models that you do have so that you can simulate what the effects of a postulated weld failure would be. So, you end up getting to the end point, and eventually, what you end up doing is developing the capability to do risk significance on welds. DR. SIEBER: Right. DR. POWERS: I'm going to have to cut this interesting discussion off. I thank you very much. Thank you for the view-graphs, because I think they do merit study beyond the lecture. MR. BRADLEY: I'd appreciate it if you would look at those, because we put a lot of effort into putting those together, and we didn't get through all of them today. DR. POWERS: We'll continue with the Professor Apostolakis show into the staff views on ASME standard for PRA for nuclear power plant applications. DR. APOSTOLAKIS: Okay. We have reviewed the ASME standard. What we have not had the chance to do is to review the staff's comments on the standard. So, today, Ms. Drouin and Dr. Parry are here to enlighten us on that. Mary? DR. DROUIN: Okay. Mary Drouin from the Office of Research, and with me is Gareth Parry from the Office of Reactor Regulation. We're going to go through today to talk about the recent activities that have happened since the issuance of Revision 12 of the ASME PRA standard. Back in June, on the 14th, ASME issued what they called Rev. 12 of the standard for probabilistic risk assessment for nuclear power plant applications. This was the second time for public review and comment. The NRC spent quite a bit of time during the public review and comment and went through Rev. 12 in quite detail, and we provided substantial comments that were a combined effort between the two offices, and we provided those in a letter to ASME on August the 14th. In doing our review of the ASME standard, there was SECY-162, which provided a lot of the guidance that we used in coming up with our comments, using Attachment 1, and we also went back and looked at our comments that we had made on Rev. 10 to see if we still had some of those concerns, if they were still valid, and looked at that in terms of Rev. 12. In our letter to ASME that was submitted on August the 14th -- and these four bullets are lifted verbatim from the letter. I did not try and paraphrase them or anything. There were four points that the staff concluded: One, that Rev. 12 was not a standard that addresses PRA quality. It's difficult to use in determining where there are weaknesses and strengths in the PRA results and, therefore, will have limited use in the decision-making process. It only provides limited assistance to the staff in performing a more focused review of the licensee PRA submittals, and the last conclusion, it provides minimal assistance in making more efficient use of NRC resources, and those were the four conclusions, even though there was backed up with the letter, I think, about 70 pages of comments of why the staff came to those conclusions. What I am going to do at this point -- because we don't have time to go through all 70 pages of comments but try and give you a general feeling at a high level from each of the chapters where our major concerns and comments were. Starting with Chapter 1, the biggest thing that you see in Rev. 12, in Chapter 1, was the definition of the categories, and our main concerns there is that, when you look at the single categories and you look at applications, there's no single application that fits under each category. So, from that aspect, we felt that the categories were not very useful or very helpful, and the categories also were being defined more from an application process, and since you don't have a single application that fits under any category, we felt that was the wrong way to approach defining the categories. When you went to Chapter 2 and looked at the term -- DR. SHACK: Presumably, you would have the same objection to the grades and the peer review process. DR. DROUIN: In terms of defining those, yes. DR. SHACK: Yes. DR. DROUIN: Yes. DR. SHACK: And I think everybody sort of agrees you probably can't do that, that we really, really shouldn't be focusing on -- you know, that's somebody's dream of how it can be done. I mean they can propose, but you dispose. DR. DROUIN: Yes. [Laughter.] DR. DROUIN: That's one way of saying it, yes. Jump in any time. [Laughter.] DR. DROUIN: In Chapter 2, the definitions, when we looked at these, I think the words here really captured our feelings that many were inaccurate, many were not written for the context in which they were used, and many of them just simply unnecessary, we didn't see why there was a definition there proposed. Maybe there were already well-known definitions for these and it wasn't necessary to come up with one. Chapter 3, the risk assessment application process, we had several concerns in this chapter, but the biggest ones is the way it was written is that, one, it doesn't provide any requirements in there, and then, also, because of the way it was written, it sort of exclude any minimum requirements, so that when you get to Chapter 4, which was the technical content, Chapter 3 almost came in and said you don't have to meet anything in Chapter 4, because it always allowed you to do supplementary analysis that were equally acceptable. So, in essence, you ended up without a standard because of the way Chapter 3 was phrased. MR. PARRY: Also, I think, in that chapter, there is somewhat of -- the logic isn't quite right in the sense that either you meet the standard or you don't meet the standard, but you present reasons to the decision-making panel why you didn't, why that doesn't matter, and I think, instead, the documentation seems to suggest you do something else and you get around the standard and say you've met it. So, it was a little -- the logic was a little strange. DR. POWERS: The committee certainly commented on precisely that unusual feature of the standard. You cannot get an N-stamp, but you do get an N-stamp if you do something that's undescribed. DR. DROUIN: Okay. Section 4, which some might say is the heart of the standard, because it gets into the technical content, and in our 70 pages of comments, probably at least three- fourths of our comments were on this particular chapter, and in summation, where we had problems was a lack of completeness, in many places just a lack of accuracy. We felt that it was inaccurate in terms of some of the technical requirements. The logic, the organization, and the structure, the supporting requirements against the high-level requirements, we saw lots of problems in those area, and this was probably the main one where it led back to our conclusions that we had in our cover letter, were the problems associated with Chapter 4. DR. APOSTOLAKIS: Now, the long table you have in your comments on data analysis, comparing Rev. 10 to Rev. 12, is under section 4, right? DR. DROUIN: Is under section 4, lack of completeness. DR. APOSTOLAKIS: That was really a very good table. DR. DROUIN: Thank you. DR. APOSTOLAKIS: And in light of what Mr. Lochbaum said this morning, it acquires even greater significance, because I notice there was an effort in Rev. 12 to get away as much as possible from using plant-specific data, and you point that out in several places -- DR. SEALE: Yes. DR. APOSTOLAKIS: -- and you know, that's the complaint from UCS, that the numbers that are being used for plants are generic, non-conservative, and so on. DR. DROUIN: I think in our -- DR. APOSTOLAKIS: But that was really a good table. DR. DROUIN: In our Executive Summary, we gave, I thought, two good examples of things that were in Rev. 10 that were not in Rev. 12 that got into that -- appropriate plant-specific estimate of equipment unreliability shall be developed. DR. APOSTOLAKIS: Yeah, yeah. DR. DROUIN: That was missing in Rev. 12. DR. APOSTOLAKIS: There's a broader issue here, though, and I mean I realized it when Karl Fleming was making his presentation. A standard is not a procedures guide. DR. DROUIN: We agree. DR. APOSTOLAKIS: So, a standard cannot tell you which method to use, I suppose. DR. DROUIN: Right. We agree with that. DR. APOSTOLAKIS: So, your first statement that this standard does not address PRA quality -- you didn't really mean that it had to tell you had to do certain things. DR. DROUIN: No. It was getting into these problems here, because of these problems. DR. APOSTOLAKIS: But how to do it becomes a factor -- is Fleming still here? -- becomes a factor when, for example, in their peer review process, the peer reviewers say this is an A or B or C. In other words, you're relying now on the peer reviewers to know the methods and see whether the appropriate method was used for a particular requirement. Is that correct for both ASME and the peer review process? DR. DROUIN: Yes. DR. APOSTOLAKIS: And everyone is happy with that. DR. DROUIN: Yes. MR. PARRY: That gets a comment on Chapter 6 that you'll see in a minute. DR. APOSTOLAKIS: Okay. DR. DROUIN: I mean, from the very beginning, ASME, with NRC, support that the peer review was an essential ingredient of the standard, because we're never going to be able to get prescriptive in the standard. DR. APOSTOLAKIS: What was an essential ingredient? DR. DROUIN: A peer review. That's why a peer review was part of the standard. DR. APOSTOLAKIS: There's something that bothers me about the peer review. Is this an appropriate time to raise it? DR. DROUIN: Well, I'm going to get to it in two more bullets. We didn't have a whole lot to say on Chapter 5. We felt that that was a strength of the standard. DR. APOSTOLAKIS: You didn't have a whole lot bad to say. DR. DROUIN: That's right. The comments we provided on Chapter 5 were more editorial, but we felt this was one strength in the standard. In Chapter 6, we had several comments. The most significant was the one I put here, where we felt that the focus was not on the need for reviewers to make value judgements on the appropriateness of the assumptions and approximations and an assessment of the impact of their results. This was -- should be an essential part of the peer review, and we didn't see that coming out when you read Chapter 6 of the standard. Do you want to elaborate on that, Gareth? MR. PARRY: Yeah. Really, if you read Chapter 6, it almost sounds like it's a QA check of the calculations, rather than an assessment of how well the assumptions have been justified and how well the thing has been modeled, basically. It had the wrong focus, I think. DR. APOSTOLAKIS: The thing that struck me as odd in the peer review process, certification process, was that the criteria for selecting the peers were formal. In other words, does a guy have a Bachelor's degree or doesn't, and if he doesn't, does he have so many years of experience. DR. DROUIN: That's not in the ASME. That's in the NEI 02 document. DR. APOSTOLAKIS: What are the criteria for choosing the peers here? Is it experience again? DR. DROUIN: It's experience, and in Rev. 10, it read more like the certification, and we were -- ASME was heavily criticized for that. So, we tried to -- now, I'm speaking more as one of the ASME members -- to approach it differently, and it got into, you know, of course, independence, which we agreed with, but we tried to move away from saying number of years, because a lot of people can have a long number of years, but it doesn't necessarily make them an expert. So, it read more, be knowledgeable, the requirements, have demonstrated experience, have collective knowledge of the plant design. That was at the general requirements, and then it went on to be more specific about what it meant by that and not come in and say, you know, you have to have five years. You could have somebody who could have two years who could be an outstanding person. So, it tried to get more into explaining what we meant by the word "expertise." DR. APOSTOLAKIS: Okay. DR. DROUIN: Whether it accomplished it well enough could be argued. Okay. I'm sure, as you're aware, the NRC letter came out, a lot of other public comments, but the NRC was probably -- their letter was probably the catalyst for some very recent activities, and that's what I'm going to speak to. ASME did appoint this task group to look at Rev. 12, to provide advice back to ASME, and participating in this effort, the staff did come in and propose a set of principles and objectives of the standard, and these were through different phone calls, came to a consensus on these principles and objectives between NRC and industry, and this is what was used by the ASME task group. This task group then met on September the 19th and 20th, and immediately thereafter, this task group did brief the NRC peer -- the NRC PRA steering committee and NEI's risk-informed regulation working group and ASME on September the 21st. So, these next slides -- this task group did issue a report on their finding. I've tried not to paraphrase any of it but lift the words directly from the report. I wasn't going to go over these, but I did put them in the hand-out. These were -- there's two pages of them, and that's pretty clear, actually. I'm impressed. But they were high-level objectives and principles, starting off -- if we just look at the first one -- DR. WALLIS: Are these written after the work was done or before? DR. DROUIN: This was written before the task group met. DR. WALLIS: So, they should have done all these things. I mean if they've produced a good standard, it would have met all these requirements? MR. PARRY: That would have been the conclusion, yeah. DR. WALLIS: So, these were the objectives before they started out, and somehow they went astray? MR. PARRY: These are the objectives before the task group started. DR. DROUIN: After Rev. 12. DR. WALLIS: That's what surprised me, is it would seem to me this would have been written right at the beginning as the objectives, specifications, and standards - - DR. DROUIN: That probably would have helped. DR. WALLIS: -- and you wouldn't have had 12 revs that didn't meet the objectives. MR. PARRY: I think it's taken time to develop. DR. WALLIS: But isn't this the design process? I mean they were just learning the design process? DR. DROUIN: Fair comment. DR. WALLIS: Somebody's just learning the design process. DR. APOSTOLAKIS: Mary, in connection with this, is the basic position of the staff that the quality of the PRA -- there is a minimum standard for a quality of a PRA that doesn't belong to any category. You start talking about categories when you talk about applications, as opposed to having categories that have different quality requirements for different applications, and it seems to me that the staff wants to have a good baseline PRA -- DR. DROUIN: Yes. DR. APOSTOLAKIS: -- where the quality is not questionable. DR. DROUIN: That's correct. DR. APOSTOLAKIS: And then, if you want to apply it to risk-informed ISI, then you do more or less, right? You take the appropriate pieces of the PRA that apply and you say, for this category, this is what I need. DR. DROUIN: Let's get to the word "minimum." I don't know that you need to have a minimum. I think what you -- I would tend to use the word "benchmark." You want to have a set, whether that set is the minimum, but where do you line up to the left and right in terms of your weaknesses and strengths of that. DR. APOSTOLAKIS: Good PRA practice, let's say. DR. DROUIN: Yes. DR. APOSTOLAKIS: Present good PRA practice. DR. DROUIN: You know, what are your good current PRA practices? DR. APOSTOLAKIS: So, you're not going to tie that to the application. DR. DROUIN: That's right. DR. APOSTOLAKIS: And that's what the ASME standard does right now. It says, for different categories, the quality can be different. DR. DROUIN: Did Rev. 12 do that? DR. APOSTOLAKIS: I thought so. For category 1, here are the requirements; for category 2, here are the requirements, different requirements. DR. DROUIN: I think it attempted to do it. I don't think it was successful. DR. APOSTOLAKIS: No, I'm talking about the approach, and this was also different from what Mr. Fleming presented. What he said was that everybody's striving to go to grade 3, but then, for different applications, maybe, you know, if you have a comment A that is irrelevant to this application, you don't take care of it for this application, but you are trying to get there. I think this is an important point, because it's really at the root of the disagreement, I think. It's one thing to try to define quality according to application and quite another to have a PRA of certain quality and then, depending on the application, I may do more or less or use pieces of the PRA. DR. DROUIN: Okay. I'd like to get to that. There was a finding on that by the task group, and I'm not trying to put you off, but I'd like to answer it when we get to that point. DR. APOSTOLAKIS: One last point. On 3, to facilitate the use of the standard for a wide range of applications, categories can be defined, it seems to me that, instead of trying to define categories, if you give a set of examples, you avoid a lot of the debate you're having right now. DR. DROUIN: I think that's the same point. DR. APOSTOLAKIS: Okay. DR. DROUIN: I'm going to skip the next slide, which is just the rest of the principles and objectives, quickly show you who was on the task group. DR. APOSTOLAKIS: That's interesting that Mr. Fleming is not there. DR. DROUIN: I cannot say why who was on this side. Industry proposed their people, NRC proposed their people. MR. PARRY: I think Mr. Fleming was not available that week. DR. DROUIN: I don't know. This is who industry proposed. DR. APOSTOLAKIS: Who's the chairman of this group? DR. DROUIN: There was not a chairman. There was what we call a facilitator. DR. APOSTOLAKIS: Who is that? DR. DROUIN: Syd Bernson was the facilitator. DR. APOSTOLAKIS: Okay. DR. DROUIN: In the task report that was issued by the task group -- and again, I lifted these words verbatim from the report -- this is what was stated. Let me rephrase that a little bit. This is what was given to the task group by ASME as the charge for what the task group was to look at, and of course, they're restated in the report as what the charge was, and the task group was asked to evaluate the principles and objectives that we were given and provide conclusions and recommendations on the following. The first one was, you know, is it possible and/or appropriate for the standard to meet each objective, to what extent does draft 12 of the standard meet each objective, identify the critical technical issues associated with as many technical elements as possible, and propose resolution for the issues identified in 3 above and provide examples of changes that could be made affecting the structure and organization of the technical elements. So, those were the four specific things that the task group was directed to do during the two-day period, in looking at Rev. 12. When you just look at the charge there, just at a very high level, again, as stated, I did not rewrite anything by the task group. The general conclusions they came to is that, when you looked at the principles and objectives, they felt that the standard was appropriate and it was possible to meet all those objectives and standards. DR. WALLIS: This is rather strange in light of your -- your conclusions are very strong about what the standard doesn't do, and yet, you end up here saying that they can now be modified to essentially meet all the things you didn't meet before. It looked to me as if it would need drastic surgery, not just be modified. DR. DROUIN: I don't mean to -- from a personal opinion, when you look at the third one, where it says it should and can be modified, to imply that that's a trivial process to get there. DR. WALLIS: No, it's a big modification you're asking for. DR. DROUIN: It depends on -- DR. WALLIS: Your criticisms really implied that it hasn't taken the main thrust, the main thrust was wrong, not the details. Modification, to me, means details, but you're essentially attacking the main thrust of the standard in your critique. That would seem to me that they have to really revise their approach, not just modify it. DR. DROUIN: I don't think the approach was revised, but the -- MR. PARRY: I think it really was the structure, certainly of Chapter 4. It's not logically structured, and I think what we felt was that the lower-level requirements were addressing the issues we'd like to address in a standard. It's just that they were not in a format that would make the standard itself a quality document and that could be easily used. So, you could call that major surgery. It's sort of shifting things around. DR. WALLIS: So, it's a reorganization of the material? MR. PARRY: And some rewriting of the objectives and high-level requirements. DR. DROUIN: The next slides get into the details of it, but I agree, it wasn't a trivial thing to do. In going through the task report, I will say I did take a little bit of literary license, because I wanted to match up the recommendation to each of the observations. When you read the task report, I think it came out with 12 detailed observations, and then you went to another chapter, and for each of the observations, there was a recommendation. So, for the sake of -- to try to put it on as few slides as possible, in many cases the recommendation just rewords the observation, so I didn't exactly quote on the recommendation always. But when you go to the detailed observations, the current objective statements for the technical elements do not always provide a clear description of the overall objective for each element. When you looked at Rev. 12, what the task group was getting into is that, at the beginning of each technical element, there was a set of bullets that got into the objective. MR. PARRY: We're talking specifically about Chapter 4, which is the technical requirements, for all these detailed observations. DR. DROUIN: Thank you. Good point. When you looked at these bullets that were the objectives for that particular element you never could find a clear statement in there of the objective that was unique and specific to that element, and we thought that that was - - the task group felt that that was something that was important and that was missing, because then to go there to the high-level requirements, you didn't see the tie in the relationship, and they just weren't always consistent. So, the recommendation from the task group was, you know, go fix it, essentially, provide these objectives, and try and make them clear. Then, when you went from these objective statements to the next part in Chapter 4, with the high- level requirements, the task group also felt they were not logically related, and they should be logically related. So, of course, that was the recommendation that came out. When you go into the support requirements, the task group felt that the supporting requirements should fully implement the high-level requirements, and what they mean by that is there seemed to be an interpretation by some members of the task group that, when you read the high-level requirement, that there were supporting requirements that were missing. So, you could meet the supporting requirements and you didn't necessarily meet the high-level requirements, and the task group felt that that should be the other way around, that if you've met the supporting requirements, then you should, by definition, meet the high-level requirement. So, that was one recommendation that came out, and also, the supporting requirements should be your minimum set. The next one that came out in terms of the supporting requirements was that, when you went from technical element to technical element and disregarding whether the logic was appropriate or the organization but just looking at the supporting requirements themselves, the task group did feel that it went, for the most part, to the right level of detail. The two exceptions that the task group came to was the data section was incomplete, but the quantification section tended to be too detailed. So, the recommendation there -- this one went hand in hand with the above bullet. So, when you looked at the recommendation, it was written as one across those two observations, wanting to pay particular attention to data and quantification as you went through and looked at the supporting requirements. The next one was getting into particular issues or topics that can have a major influence on your results but also where there's not usually a consensus on how to approach it, and the task group felt that those should, as best as possible, all of them -- you should be as complete as you can be, and those should be addressed in the standard. Some examples there were BWR ATWS, the consequential steam generator tube rupture, dual unit initiators. An example of one that is in the standard would be RCP seal LOCA. The recommendation that came out from the task group is that, in addressing it, we did not feel that you needed to give an accepted methodology but to come in, and it should be part of the standard to require what approach you used, document what assumptions were done, and what was the significance of it. MR. PARRY: It's not that we expect people not to deal with BWR ATWS, but it was specific issues related to that, to the timing issues and the interrelation of various operator actions that need to be addressed. DR. DROUIN: The next one got into the clarity of the supporting requirements need to be improved. We had quite a few recommendations, I just pulled out the main ones. We saw a lot of places to the extent necessary to support category X application. We felt that was inappropriate and should be replaced and explained what you meant. The word "may" we felt was inappropriate, because it's totally permissive, so you don't know what they're going to do, and the term "consider" also was another place that brought a lot of ambiguity to the process. Now, getting to the categories, getting back to your comment, George -- now, in hindsight, I wish I had copied some other things from the task group report, because this was an area that the task group did spend a lot of time on. The conclusion coming to the task group was that the current definitions of the categories were not clear and not adequate enough to help formulate the supporting requirements, and it went further to say that the specification applications, since they may span categories, therefore, categories cannot be defined by applications. And I think this is a very important point, is that when you look at Rev. 12 and you go into Chapter 1 and you look at the criteria that are used to differentiate the application -- I mean the categories -- they were more application-driven, but since you don't have a single application that goes across a -- sorry -- since you don't have an application that stays within one category, then it doesn't make sense to use that as your criteria to differentiate. The task group then went the step further and spent a lot of time and have proposed criteria to be used to differentiate the categories and then have also come up with a set of words to define each of the categories. The three -- they came up with three criteria -- please help my memory here. The first one got into the scope and level of detail of your PRA, the second one dealt with how much plant-specific information should be included in the PRA, and then the third one was the level of realism you should be bringing into the PRA, so that when you go from -- and Rev. 12 only has three categories, doesn't have four. So, as you go from category 1 to category 3, you're going to -- if you look at the first category, which is scope and level of detail, you're going to increase your scope and level of your detail of your PRA as you go from category 1 to category 3. When you look at your degree of plant-specific information, again as you go from category 1 to category 3, you're going to increase the amount of plant-specific information you bring in, and the same thing on the degree of realism as you go from category 1 to category 3, you're going to increase the level of realism. MR. PARRY: Another way of saying "the degree of realism," I think, is a reduction in the conservatism as you go from one end to the other in terms of modeling. DR. POWERS: When you say the scope varies as you go from one category to the next, is there a category in which it is not necessary to consider common cause failure or not necessary to consider time-dependencies? DR. DROUIN: No. DR. APOSTOLAKIS: I think the issue of categories really can be bypassed completely. I mean if we come back to the basic idea that you should include in the argument the things that matter to the decision and things that don't matter can be left out, I think if you provide, as the committee has recommended, a number of examples where decisions were actually made and elaborate on those, you know, what insights we gained, what was important from the PRA, what was not, I think that should be sufficient, in my view, and that way, you avoid debates, again, as to whether category 2 makes sense, has it been defined correctly, and so on. I think if we see examples -- and maybe down the line, after we have sufficient experience, we will be able to define categories, but I really don't see the value of trying to define categories. It's really case by case. Karl gave us a few examples earlier. You have a number of examples in your earlier -- DR. DROUIN: You're not going to get any argument from us on this point. DR. APOSTOLAKIS: Yeah, but I mean -- DR. POWERS: Maybe I'll throw up an argument if you're not going to get an argument from them. [Laughter.] DR. APOSTOLAKIS: Okay. DR. POWERS: I mean it seems to me that we're going to have people who would like to know what is the minimum that I can do and still use my PRA for some applications that look attractive to me, and rather than having to plow through all these requirements and make some judgement on which ones are applicable for some minimalist activity, they'd like somebody to tell them. DR. APOSTOLAKIS: My point is that you will have to plow through. There is no way you won't. And all you're doing now by trying to formalize it is create this debate. DR. SHACK: I would look at the categories and the grades -- I think you should divorce them completely from applications, because I think that's a decision you make on a case-by-case basis. I think it's convenient to have categories or grades as a shorthand description for how complete and how much detail this particular PRA has gone into this element. DR. DROUIN: And that's what task group did. DR. SHACK: And I think that's a useful purpose to have categories and grades, because PRAs, element by element, will differ in those things. You could divorce it completely from deciding what application -- trying to determine a priori what application -- DR. APOSTOLAKIS: But that was Dana's argument. DR. SHACK: Well, I don't like Dana's argument. I want categories for a different reason. DR. APOSTOLAKIS: Let me put it a different way. I think defining categories right now is premature, and we've seen the debate, we've seen the agony. Why don't we look at a number of -- DR. SHACK: It's a useful way to describe the level of detail and completeness of a PRA, at least to the utility, so he knows what he needs to -- the most useful thing out of all of this is to identify weak spots in existing PRAs, so the guy can go off and do something about it. DR. APOSTOLAKIS: But if, per chance, I have four or five analyses and studies like the ones that Fleming presented, don't I get that feeling? I mean he had nice tables, he told you that for risk-informed ISI there is a comment A, but it's irrelevant to this, so, you know, we're not going to take care of it. If I look at a number of those -- DR. SHACK: It isn't up to Karl to define how good it has to be for risk -- you know, he can suggest that maybe this is good enough, but these people decide how good it is. DR. APOSTOLAKIS: Sure. DR. SHACK: And I really think you should get away from using those grades as applications and think of them more as -- DR. APOSTOLAKIS: I fully agree with that, but what I'm saying is that it's a risk-benefit calculation here. Attempting to define categories will create more headaches than the benefit you get from them, at least right now. Was it option 2 where you have Appendix B with a number of examples where, you know, in some cases, you needed this kind of thing from the PRA. We commented on it in our letter last time. I thought that was great. So, let's build up that information base first and then worry about the categories. The categories have been a problem with the ASME standard; the grades have been a problem with the PRA certification process. The way Karl presented it, though, makes sense to me. DR. DROUIN: What the task group did was to define the categories without any -- totally divorce it from applications. DR. APOSTOLAKIS: And quality. You can't define that a priori. DR. DROUIN: You have quality in all categories. DR. APOSTOLAKIS: Yeah, right. DR. DROUIN: So, that's what I'm saying. It got into the scope and level of detail, the amount of plant- specific information, and the amount of conservatism or realism. We felt that those were the three criteria that you can use from a PRA perspective to define it. Now, I said I wasn't going to present an argument for the categories, but I will pick up on Dr. Powers' argument, because I have been -- without mentioning names, several utilities who have explained to me why they want at least the category 1, and that is to have a minimum, because when we sat down and we went through with the task group and came up with -- I just gave you the criteria. We actually then came up with a definition for each of the categories, and we're going through the elements, but our guideline that we were using for category 2 in determining what should be the scope and level of detail, what should be the amount of plant-specific information, what should be the degree of realism, it was what is the current good practice. So, as we went from element to element, you know, as the eight people -- we all put in our views, what do we think is the current good practice, and then we said, okay, now, stepping aside from the current good practice for category 1, what's the minimum that we think is acceptable, so that if you don't meet that, you just don't have a PRA that's of anything. DR. APOSTOLAKIS: But the minimum for that application or the minimum for a PRA? DR. DROUIN: Minimum for a PRA. DR. APOSTOLAKIS: PRA. Oh, then I'm with you. DR. DROUIN: Minimum for a PRA. DR. APOSTOLAKIS: I'm with you. And I think that's where Fleming was going. DR. DROUIN: And that's -- you know, if you're going to argue categories, I think that is a good argument, if you want to know what the minimum is. DR. APOSTOLAKIS: My basic objection is I don't want everybody to get the impression that, boy, I don't have any PRA now and the NRC is telling me, if I do A, B, C, I can have a category 1 application, and I don't think that will ever work. You have to have a baseline PRA, and then a piece of that may be appropriate only for category 1. DR. DROUIN: And that's been our criticism of Rev. 12 from the beginning. They've been trying to do it on an application basis, which is not appropriate. Okay. Moving on. MR. PARRY: In terms of section 6 on the peer review, the major thing, I think, was to try to, again, emphasize that what the peer review team needs to do is make a value judgement about essentially the quality of the analysis, first of all see that it's met the requirements, to see that it, indeed, does meet all the requirements of a PRA, and then to provide an assessment of how well that's been done, over and above that. In terms of the application process, section 3, what we felt about that was that, in terms of describing an application process, it was too short, and really, to define that in terms of a standard, I think you'd have to make it much, much more detailed. The alternative, I think, is to have a chapter that defines how you would use this standard in a decision- making process or in an application process, what role the standard has in that process, and allow that process to be described in another document, such as -- one of the ideas that we threw out was an update of the PSA application, for example. We made one additional comment -- editorial comment, really, is that additional references in the document would be useful. It currently has very few references, and again, the reference, though, would not be to acceptable methods but be to documents that were used to explain why the requirements were necessary, because again, I want to get away from defining acceptable methods in the standard. Finally, the definitions -- well, we've already mentioned that they're pretty poor and they need a lot of work. I think they haven't been given much attention. Everybody's just blown over that chapter. DR. DROUIN: The last part in the task report by the task group was what future actions that would be undertaken by the task group. Some recommendations were also made in that area. Most of the recommendations had to be where the task group would provide support on the previous recommendations. The task group undertook to write objective statements for each element, modify the high-level requirements, to go and identify where we thought there were missing technical topics, not to then go through and write the requirements for that, the project team, but just to identify to the project team what the topics were that were missing, define the categories. They did that at the high level, and right now, the task group is going through and writing for each of the technical elements, and then to identify suggested references. The project team is also recommending that -- and that all pertains to Chapter 4, and while the task group is doing that, there's no reason why they shouldn't initiate the review and resolution of the public comments on the remaining chapters, and so, the recommendation was to move forward with that. The last recommendation from the task group is they felt a small group should be organized to come through -- and we used the word "organize and edit," but again, it's not quite a simple as that probably implies, to go back and fix that according to the principles and objectives. There was several reasons why the task group felt the small group from the task group ought to be formed, was because, one, to approach it in your holistic manner so that you were looking at all the elements together, so you could deal with the consistency and make sure you had the right organization and logic, instead of piecemealing it out. When I say piecemeal, you know, have one group go off and do one element and another group another element. That's part of the problem, so to keep it to the small group that did it all together, and because the task group had undertaken to do the objective statements and modify the high-level requirements, that consistency could be carried on and just to re-clean up this part and then turn it back over to the project team to go through the public comments then and to resolve the public comments. And that's -- I hope I've characterized correctly what came out of the task group. DR. POWERS: Maybe I missed it. Did they speak to this issue of the supplemental analyses? DR. DROUIN: I'm trying to remember. Indirectly. MR. PARRY: I don't think specifically, but certainly we did in the NRC comments, anyway. So, that's a public comment that's going to have to be dealt with. I guess, in a way, it is, by virtue of the fact that it really is -- the supplemental analyses are really to do with how you make decisions, and I think we were recommending that the standard be somewhat divorced from the decision-making process in this document but that Chapter 3 should make it clear how the standard would be used in such a process. DR. POWERS: Somebody put in the discussion of supplemental analyses in Rev. 12 for a purpose. They didn't succeed in whatever that purpose is, but I don't understand what that purpose was, nor do I understand how the technical group addressed that. DR. DROUIN: As I said, I don't think we explicitly addressed it. To talk about Rev. 12, I think what happened is you have to look at the history in terms of some evolution between Rev. 10 and Rev. 12, and in Rev. 10, the same words were in there, except there were a few others that said you were then outside the scope of the standard. When Rev. 12 came in, those words got dropped, but they didn't get dropped with the intention of meaning that you were still in the standard. I wish I could remember some of the discussion, but it had to do with the way ASME writes a standard. So, the intent was never, I don't think, to say that you were still -- that if you went off and did some supplementary analysis, that you had met the requirements, for example, of Chapter 4. DR. LEITCH: Mary, maybe I have a semantic problem, but I'm a little confused with what you're calling categories and grades. That is, ASME has set out to write a standard, and in terms of the previous NEI discussion on 00-02, are they trying to write a standard for a PRA that would be grade 1, 2, 3, or 4? DR. DROUIN: ASME elected to adopt the word "category" versus the word "grade." DR. LEITCH: Okay. So, you're using those terms kind of interchangeably. DR. DROUIN: Yes. We just thought the word carried meaning to it that was not truly should have been there. DR. LEITCH: Okay. So, then, using the term "category," then, you're aiming at category 1? In other words, this is what you call the benchmark, a minimum standard to be a PRA. DR. DROUIN: Okay. The task group feels that category 1 should be your minimum DR. LEITCH: Uh-huh. DR. DROUIN: I also think that was also the intent in ASME. DR. LEITCH: Okay. DR. DROUIN: Has the NRC had an opportunity to comment on a number of previous revisions, or is this the first opportunity? DR. DROUIN: In terms of ASME? DR. LEITCH: Yeah. DR. DROUIN: Yes. There has only been two that have gone out publicly for review. The other ones were just -- every time we make a little change, as someone who's on the project team, it's probably misleading to the public to say there have been 12 revisions. There's only been, really, two revisions. DR. LEITCH: Okay. Because I'd be a little discouraged if I saw 71 pages of comments on the 12th revision. DR. DROUIN: And what happens is, you know, Revision 1 of the ASME maybe only had two chapters in it. DR. LEITCH: Okay. DR. DROUIN: So, every time we put out a new one internally to ourselves, just to keep track of where we were, we kept calling those a revision when it wasn't truly a revision to the whole standard. So, the first revision that came out was Revision 10, and then, when the ASME team got together, you know, we tweaked some things and we tweaked some more things, and then we came out with the next revision, which we call 12. That's an internal counting. DR. LEITCH: Do you have any knowledge about the schedule from here? When might Revision 13 hit the streets? DR. DROUIN: They're looking to try and do it within six months, and there has been a proposed schedule, but that is still under refinement. That's why I didn't want to get into details, because it's still proposed, but I think that the goal they're looking to is to have it ready for balloting within six months. DR. LEITCH: So, our role here is just information, discussion? What's the ACRS doing with this presentation today? DR. DROUIN: I don't know. We were asked to come and present. DR. LEITCH: Okay. DR. APOSTOLAKIS: Are you requesting a letter? DR. DROUIN: Are we requesting a letter? I don't think so. DR. APOSTOLAKIS: Okay. Would it be beneficial to structure the peer review process in the standard along the lines of Karl's presentation, forgetting for a moment that there is a NEI 00-02, because you may disagree with what's in there, but the idea of identifying elements that need to be done right away or others, you know, are good to do but you can wait, so that you can eventually reach category 2 -- I thought that was a good idea, and maybe the peer review process in the standard can follow something like that, instead of saying, you know, just make sure that it's okay. MR. PARRY: I think that's also true of -- if you are familiar with the IAEA review guidance, they do a similar thing. They categorize their comments in things that you need to do straight away, you can leave till later. DR. APOSTOLAKIS: I think that's a good idea, and I don't see why the standard cannot adopt something like that, because it's also more specific that way, and the comments, of course, will refer to what's in the standard, not what's in the NEI document. That's why I'm saying divorce the two. DR. DROUIN: Yes. DR. APOSTOLAKIS: But the table that Karl showed was very good, because if you do all these things, then you will have a category X. Any other comments from the members? DR. WALLIS: I'm trying to get an overview of what's going on here, and maybe I don't know enough history, but the value to having an ASME standard is that ASME is an independent body and it gives some authority. It's not being biased by NRC habits and so on, and therefore, it has some sort of authority out there of representing the public or some other group. Now, the impression I get is that it's being wagged completely by NRC. So, if NRC is influencing the specifications and objectives of the standard, it's no longer an ASME standard; it's more a kind of NRC standard. It's like an SRP or something. DR. DROUIN: My question is what gave you that impression? DR. WALLIS: Just the way things have been described and in the recent documents I've read. DR. DROUIN: We were asked to give a presentation on the activities. I tried to, as best as possible, quote directly from the task group, and the task group had four NRC people on it and it had five industry people on it, and the recommendations and observations that I presented here -- I mean I happen to work with NRC, but these were observations that unilaterally and unanimously were derived by those nine people. MR. PARRY: There's one thing that could be confusing in terms of one of these bullets here. It says the staff proposed a set of principles and objectives. Okay. That was really -- to be honest, that was a negotiated set of principles and objectives. Somebody had to start it, and I can't remember which -- whether it was the industry side or ourselves that started it, but it was negotiated over. DR. WALLIS: ASME could come back and say that we're an independent body, we know what we're doing, why should we respond to all these staff things. MR. PARRY: It's not just the staff. It's also industry. And you could argue, in fact, that the industry had a major influence in changing from Rev. 10 to Rev. 12. DR. DROUIN: But everything I've presented here is the ASME task group. DR. POWERS: Well, I think, in fact, Graham, that what I've come to learn -- I, like you, naively assumed that there was some body of people called the exalted ASME that produced this boiler and pressure vessel code that was the fount of all wisdom, and they were independent and unassailable, and but that turns out not to be the case, that in fact, any time they write these standards, they solicit volunteers, and in this case, they solicited volunteers from the staff and the industry to write the standard, and they're the gurus. So, there is no independent body out there. I mean it's dependent upon these people that are experts. Now, what we can say is these are experts. One of the questions that comes into my mind both about the ASME writing group and now this task group is that, when I look at the membership of those things, I see a lot of names of people that I have the impression feel like they invented this technology that are not on this list, and I'm wondering, is the ASME suffering from the fact that they haven't thrown their net wide enough in selecting the writing group to prepare this standard? DR. DROUIN: I can't speak to how ASME selected the people on their project team. DR. POWERS: That was not an issue that the technical group tried to address. DR. DROUIN: That the task group? DR. POWERS: Right. DR. DROUIN: We addressed the four things that we were directed to address by ASME and no more and no less than that. MR. PARRY: But you will notice that what the task group recommended, though, was that it be a small group, not a larger group, that pulled together Chapter 4, and I think that's a logistical thing, that the way it's been done, as Mary described, is that, you know, one group would go away and do one technical element and another one would go and do another, and it's really hard to get coordination unless you have a focused group to do it. DR. WALLIS: Who is paying for the work? DR. DROUIN: Everybody. DR. WALLIS: Who pays for the work? These aren't all volunteers that are doing all the work, or are they? DR. DROUIN: Their respective organizations pay for them. DR. WALLIS: Oh, you mean someone comes from General Electric and General Electric pays for that person's time? DR. DROUIN: Absolutely. DR. SEALE: If someone comes from NRC, then NRC pays for their time. DR. APOSTOLAKIS: The ASME's expenses, though, come from? DR. DROUIN: ASME. DR. WALLIS: ASME members are paying for this work, to some extent? MR. PARRY: Well, they're paying for their staff, I guess. DR. SEALE: Their standards effort is a self- supporting activity. That's why you pay so much for them when you get them. DR. APOSTOLAKIS: Well, we are running out of time. Is there any other thing that is relevant to the particular subject here? [No response.] DR. APOSTOLAKIS: The industry wants to make any comments? DR. DROUIN: I think Karl. MR. FLEMING: Karl Fleming. From the point of view of representation on the project team, I wanted to make three comments pertaining to the NRC review of draft 12. The first comment I wanted to make is that, after pouring through the 70 pages of detailed comments, I appreciate the effort that the staff put in to coming up with a large number of constructive comments, which I, by and large, agree with, and the resolution of those that will be guided by this project team that Mary is talking about should provide the ability for enhanced PRA standard. So, at the detail level, I really don't have much of an issue. I do strenuously disagree with the broad conclusions that the NRC reached as a result of those 70 pages of comments. I do not think that they flow from what's in there, and I think that's the reason why it appears like we do have a achievable path forward to get this thing fixed in a reasonable period of time. A second comment I wanted to make is that, with regard to the time it's taken and so forth -- and I've been with Mary, working on the project team for the last couple of years -- the one thing I kind of noticed, that up until but not including the NRC letter, I was kind of -- it was interesting for me to note that the high-level requirements which I actually suggested the introduction of after Rev. 10 as a way to try to address an industry concern with Rev. 10 that it was interpreted to be too prescriptive. And I don't necessarily agree that it was intended to be that way, but the high-level requirements were concocted or developed as a way to provide high-level requirements that were unassailable, shall be done, no negotiation whatsoever, and then the detailed requirements could be layered in in accordance with that. The high-level requirements that are in draft 12 were on the street 18 months ago, and when we introduced these, we made it clear that it was very important that we get the high-level requirements agreed upon, because everything else that needs to be done to make the detailed requirement support that flows from that, and I'm just, you know, kind of disappointed now that, 18 months later, we're still, you know, fixing the high-level requirements. I'm not arguing that they don't need to be fixed, but it's just a shame that we've been trying to write a standard for the last 18 months on the basis of high-level requirements that still need to be revised. DR. DROUIN: Karl, I agree with you, but I've just got to put something in here. The NRC has, through the project team -- and I can go pick out e-mail after e-mail where we have provided comments from just the NRC the problems with the high-level requirements. So, to say, you know, 18 months later, that this - - you know, this comment has come forward -- we've been providing comments with our concerns on those over the last 18 months. MR. FLEMING: That's interesting, because I didn't appreciate that. We spent an entire project team meeting last summer in Palo Alto where the 18-member project team went over line item by line item of those high-level requirements, and I wasn't aware, personally, that there were any comments on those until today. DR. APOSTOLAKIS: Any other comments? MR. BRADLEY: Just a quick one. In the interest of clarity and since the question was asked, I would like to point out -- and I'm not familiar with ASME, but for ANS, NRC has provided grants to ANS to support the standards development for PRA standards. DR. APOSTOLAKIS: Does it take away from the objectivity? DR. DROUIN: No. NRC does look at any grant proposal that's been submitted, and if an organization doesn't request for a grant, we can't be responsible for that. DR. APOSTOLAKIS: By the way, the ANS standard has sort of faded away? DR. DROUIN: No, they're actively working. DR. APOSTOLAKIS: Actively working. We were supposed to review something last month. Is it tied intimately to the ASME standard, so it has to wait until the ASME standard is ready? DR. DROUIN: I can't comment on -- I'm not the lead on that. DR. APOSTOLAKIS: Okay. I think we're done. DR. POWERS: Okay. I will recess us until -- for an hour. [Whereupon, at 12:40, the meeting recessed for lunch, to reconvene this same day, Thursday, October 5, 2000, at 1:40 p.m.] . AFTERNOON SESSION [1:40 p.m.] DR. POWERS: Let's come back into session. The next topic is going to be somewhat of a switch from the previous discussion of PRA to move to PTS, pressurized thermal shock, and Dr. Shack will -- DR. SHACK: Probabilistic analysis again but of a fairly exactly sort. [Laughter.] DR. POWERS: Well, in that case, Dr. Shack, maybe you could explain to me something about the bias analysis on these parameter distributions that they were calculating for this piece of work. DR. SHACK: Well, I would like to say we had a very good, full subcommittee discussion of the PTS update project, and trying to pick some pieces out of that that we should bring to the full committee, we decided it was useful to have something where we could walk through the overall probabilistic fracture mechanics calculation, so people could see how all the pieces sort of fit together, because we've been analyzing it sort of piece by piece, and then we did want to go through one particular aspect on the fracture toughness uncertainty -- or the fracture toughness distributions, where there's an interesting discussion of how the uncertainties will be treated. One approach has been a purely statistical one that was developed at Oak Ridge, and then there's an alternate approach that's being explored at the University of Maryland, and I think it's worthwhile reviewing that. One thing that did come up at the subcommittee meeting that I think is worthwhile bringing to the full committee's attention is a concern that Dr. Kress raised that's sort of similar to the problem we ran into with the spent fuel problem, that if a reactor vessel was going to fail, we would be getting a different kind of source term than the source term that we've usually been used to dealing with. That is, the core melt will occur with the -- essentially exposed to air, rather than in water. So, it will be an air environment rather than a steam environment. With a different source term, this raises into question exactly what is the appropriate kind of LERF criterion to use. The LERF criterion that we're sort of comfortably using and used to using from 1.174 is really based on a steam-driven source term, and again, PTS would have a different kind of source term, and it might well affect the kind of acceptance criteria you'd want to have for a PTS incident. So, that's an issue that we did bring up with the staff. I'm sure they haven't had a whole lot of time to address it, but it is one we think needs to be addressed before the PTS update can be completed, and with that, I'll turn over to the staff, and I see we have all sorts of staff here today. We'll have Mike Mayfield, I guess, lead off. MR. MAYFIELD: All right. I'm Mike Mayfield from Division of Engineering Technology and Research. The overall PTS project involves three different divisions in research, and the pieces you're going to hear about today, I guess, encompass some synthesis of ideas that come from the fracture mechanics world, the materials world, and some of the PRA work. We certainly have welcomed the committee's input and have appreciated the time you've been willing to invest in reviewing this project as it's gone along. We hope to continue this dialogue over the course of the next year or so, as we finish off the project. With that, unless Ed or Mark have something, I'll turn it over to Terry Dickson from Oak Ridge to talk about the FAVOR code. DR. KRESS: Before you get started, I would like to say it's nice to hear from somebody that doesn't have an accent. [Laughter.] MR. DICKSON: I never thought of myself as having an accent until -- I worked overseas for several years, back in the '80s, and I would talk to people, and they'd say where are you from? I'd say Tennessee. They'd say I thought something like that. I sort of started almost getting self-conscious about it at that time. My name is Terry Dickson, and I'm going to talk about the FAVOR code. I'd like to acknowledge two of my colleagues, Dr. Richard Bass and Dr. Paul Williams, that work with me in the heavy section steel technology program. They very much helped me put this presentation together. The presentation is sort of broken into distinct categories. The objective is to describe the evolution of an advanced computational tool for reactor pressure vessel integrity evaluations, FAVOR, and the first part of the presentation is just how FAVOR is applied in the PTS re- evaluation, and the second part is just to show how the evolving technology is being integrated into the FAVOR code for this PTS re-evaluation. The third section is just kind of about the structure of the FAVOR code, and the fourth is kind of an overall PRA methodology, and there's a fifth section that really isn't part of this presentation. This is basically the same presentation I gave in September, but it's sort of been decided since then that maybe we need maybe a little more work in this last area. So, really, this presentation will deal with the first four of these. This sort of goes, I guess, right to the heart of the matter. Application of FAVOR to the PTS re-evaluation addresses the following two questions. Here's a graph that plots or demonstrates the frequency of RPV failure in failures per reactor year plotted as a function of effective full-power years. Also, you could think of this as RTNDT, you could think of it as neutron fluence; in other words, the length of the time that the plant has been operating. And this, I might add, is the type methodology that was used in the SECY 82-465 analysis, from which the current PTS screening criteria was derived. So, the two questions here is, at what time in the operating life of the plant does the frequency of reactor pressure vessel failure exceed an acceptable value? So, we see this red line increasing, the frequency of vessel failure increasing as a function of time, and the current value is 5 times 10 to the minus 6. So, the question is, at what time in the operating life of the plant does the frequency of failure exceed a certain value, and then, the second question that we're particularly interested in here with this re-analysis is how does the integration and application of the advanced technology affect the calculated result. And this is just an attempt to show that, you know, this second curve, this blue curve, if you went back and re-did this analysis, with an improved model, which we think we have, an improved model, or a plant-specific mitigation action, that you would shift this curve in such a way that you would be able to operate the plant an additional period of time and still be in compliance with some level of failure, frequency of failure. So, in the context of this presentation, what FAVOR does it generates this type curve, and you would run FAVOR -- an execution of FAVOR would give you one point on the curve, and then you would have to -- as I talk about FAVOR -- I mean this is what we're doing. We're plotting point on this curve to see how the frequency of the failure increases as a function of the time, and specifically how these improve models. Now, one thing I will point out -- and I will probably refer back to this slide as I go through the presentation. This just shows it as a line. This just shows it as discrete values. In reality, there will be some distribution about this. In other words, you can think of this as being the mean value of the distribution that comes out. DR. APOSTOLAKIS: Was there an uncertainty analysis actually done for the red line? MR. DICKSON: Yes. DR. APOSTOLAKIS: At that time? MR. DICKSON: Well, I may ask Professor Modarres to help me out here. All the uncertainty -- as I step through the presentation, maybe I'll be able to address that question better. DR. APOSTOLAKIS: That refers to the blue line, the one you're developing now. I'm asking about the old analysis. MR. DICKSON: The old analysis did not have an uncertainty analysis. DR. APOSTOLAKIS: So, the red line is -- we don't know what it is. MR. DICKSON: Right. DR. APOSTOLAKIS: Some sort of a best estimate. MR. DICKSON: Yes. DR. APOSTOLAKIS: Would you define effective full- power years? MR. DICKSON: An effective full-power year is a calendar year at which the vessel -- at which the core was operative. DR. KRESS: If it's only part-power, you count that as a fraction of power. One of the things that bothered me about this is, if the 5 times 10 to the minus 6 were only 2 times 10 to the minus 6, which ain't a lot difference, you drop all the way down to 15 years on the red line or the blue line, too. MR. DICKSON: This really isn't numbers from an actual analysis. This is for illustrative purposes only. DR. KRESS: Right. But that's an illustration of the same point. MR. DICKSON: Yes. That was one of the outcomes of the IPTS analysis, that the shape of these curves -- as you know, there was an analysis done for each of the three domestic vendors, and the shape of that curve seemed to be slightly different for a Westinghouse versus a B&W versus a CE. DR. POWERS: Westinghouse versus CE is a little surprising, but Westinghouse versus a boiler, you wouldn't be surprised about that, would you? DR. KRESS: I don't think you've done it for boilers, have you? It's a B&W plant. DR. APOSTOLAKIS: So, if I have two years at half- power, then those would count as one effective full-power year? MR. DICKSON: Yes. Typically, I think -- I think, typically -- and somebody correct me if I'm not right, that a typical licensing period is for 32 effective full-power years, I believe, which I think sort of equates to 40 calendar years, if you figure, you know, 20-percent down time. Okay. So, the near-term schedule for the development of the FAVOR code has been recently defined, and the current schedule specifies FAVOR to be ready for the PTS re- evaluation analysis March 1st of next year. Between now and then, the models are being finalized, and the finalized models are being implemented into the FAVOR code, and scoping studies will be performed, and the idea at this time is that the Oconnee plant will be the vehicle for performing the scoping studies. The primary reason is that the thermal hydraulics folks are doing Oconnee first. So, we'll have all the input data. We'll have the PRA data in the form of initiating frequencies. We'll have the thermal hydraulics data from the thermal hydraulics branch. We'll have the flaw data from the appropriate people. So, all the data will be there to sort of start shaking down the code and seeing -- kind of seeing where the numbers come out. So, Oconnee will be the first application of all this technology. Since this presentation is about the status of the FAVOR code development, we sort of thought it might be appropriate to sort of show a little of the historical evolution of how the FAVOR code came to be. By the way, for those of you that don't know, FAVOR is an acronym, Fracture Analysis of Vessels Oak Ridge. Development of the FAVOR code was initiated in the early 1990s by combining the best attributes of the OCA and the VISA code with evolving technology. There was a series of codes -- OCA-1, OCA-2, OCA-P -- that was developed at Oak Ridge National Laboratory in the early 1980s, OCA standing for Over-Cooling Accident. There was also, in parallel, an effort that was initiated within the NRC and later was taken up PNNL. They did a code called VISA-1, VISA-2, as I said, that was done by the NRC, PNNL, in the same timeframe, and both of these codes were applied during the SECY 82-465 analyses, as well as the integrated pressurized thermal shock analyses. So, both of these codes sort of fed into -- in other words, we took the best parts of OCA, the best parts of VISA, plus lessons learned from the IPTS, integrated pressurized thermal shock, as well as a lot of lessons learned from the Yankee Rowe experience. All of these fed into the development of FAVOR. The first public release of the FAVOR code was in 1994, followed up by an improved version, the '95 version. There was a limited release in 1999, limited to this group of people that have been coming to these NRC meetings, primarily from the industry, as part of this PTS re- evaluation. So, some of the things I will be talking about were incorporated into that version, but clearly, we're -- the code is continuing to evolve, and as I said earlier, the goal right now is to have a development version fixed by March of 2001. This is just sort of a transition slide to say we'll now talk a little about the integration of evolving technology into the FAVOR code. Okay. Elements of updated technology are currently being integrated into the FAVOR computer code to re-examine the current PTS regulations, and this is just an illustration kind of all of these boxes out here on the periphery of how they are feeding in -- these are areas that clearly have been improved since the analyses that were done in the 1980s from which the current Federal regulations for PTS were derived, such as detailed neutron fluence maps, flaw characterizations, embrittlement correlations, better thermal hydraulics, better PRA methodologies. The RVID database is the reactor vessel integrity database which has been developed and is maintained by the Nuclear Regulatory Commission, which basically is kind of a repository of all the, I guess you would say, official vessel characteristics such as chemistry. If you wanted to know the chemistry of a particular weld in a particular plant, you'd go to the RVID. Extended fracture toughness databases, fracture initiation toughness, fracture mechanics, and the FAVOR code itself is one of the boxes that certainly is an improvement of technology since the 1980s. DR. WALLIS: Now, thermal hydraulics hasn't improved yet, has it? MR. DICKSON: I suppose what is intended here is - - I don't know what release of RELAP was used in 1985, but certainly, it's a later release of RELAP, and certainly, I'm sure you're well aware of the APEX experiments and so forth, that there's an attempt to validate -- DR. WALLIS: Which haven't been done yet. MR. DICKSON: I think it's supposed to happen during the fall of this year. MR. BOEHNERT: They just started, Graham. They just started testing. MR. DICKSON: So, I guess what is meant here -- we would hope that we have better thermal hydraulic analyses now than 15 years ago. So, in any case, all of these are going to feed in -- you can think of all of these sort of being input data into the process, because the FAVOR code sort of has to have a software interface here with all of these elements. So, they all feed into this updated technology PTS assessment. Hopefully, at the end of the day, we'll have a technical basis for the revision of PTS regulation. There's no way of knowing which way it's going to go at this time. I think we started into this process thinking that the potential exists for there to be a potential for relaxation of the current regulations. We'll talk a little bit more about that, but basically, we're going to put all the stuff in, turn the crank, and let the chips fall where they may. Okay. This is a little bit redundant. This is just, in words, sort of repeating what we had there in that last one. Advanced technology is integrated into FAVOR to support possible revision of the PTS regulation, flaw characterizations from the NRC research -- we'll talk a little bit more about that in a moment -- detailed fluence maps, embrittlement correlations, RVID database, fracture toughness models, surface-breaking and embedded flaws, inclusion of through-wall weld residual stresses, and a new probabilistic fracture mechanics methodology. As I said, that's slightly redundant, but probably the reason that I'm standing here now talking about this is we sort of did some analyses a couple of years ago to sort of see what -- if someone was to revisit some of the IPTS analysis and use some of the improved models, what the impact might be, and at that time, it looked like -- as I previously said, it looked like that the potential existed for a relaxation of the regulations, and the singular most important contributor to that was a significant improvement in the flaw characterizations, okay? A significant improvement since the derivations of the current PTS regulations is flaw characterization, because in those analyses, the SECY 82-465, as well as the integrated pressurized thermal shock, they assumed that all of the flaws were inner surface-breaking flaws, okay? It was known that that was a conservative assumption, but kind of in the absence of any other knowledge, that was the assumption that was made. Well, the NRC, I would say, has wisely spent their research dollars since then performing non-destructive examination as well as destructive examination of RPV material at Pacific Northwest National Laboratory to improve a technical basis for the flaw-related data used as input into these probabilistic analyses, and what has come out of this -- and it's a continuing, ongoing process, but what has come out of this is that a significantly higher number of flaws were found than were postulated in the original analyses. However, all of the flaws so far that have been detected are embedded, as opposed to inner surface-breaking. PVRUF, for those of you that don't know, is the Pressure Vessel Research User Facility, which was actually a vessel that was never put into service. It was brought to Oak Ridge, it was cut up, and the non-destructive examination as well as the destructive examination performed on it. When you take those flaw densities and apply them to a commercial pressurized water reactor, what you predict is that you will have between three and four thousand flaws in the first three-eighths thickness of the vessel. So, as I say, you have a lot more flaws, but they're sort of more benign, they're embedded, from a fracture point of view, and we'll talk a little bit more about this in a moment, but I guess the thought that I would want to leave you with with this slide is that, out of all of the little boxes feeding into this, this flaw characterization, by far, has the highest potential for impacting the answer. DR. LEITCH: When you say first three-eighth-inch thickness, that's from the inner wall? MR. DICKSON: Yeah. DR. LEITCH: Okay. Thank you. And it's not three-eighths of an inch, it's three-eighths -- MR. DICKSON: Three-eighths of the wall thickness. DR. LEITCH: Thanks. DR. POWERS: When you think about fracture mechanics on these vessels and flaws breaking the surface, you mean the surface surface or do you mean they break below the cladding? MR. DICKSON: When I say inner surface breaking, I mean inner surface breaking; they originate on the inner clads on the wetted surface. Okay. In the original analysis, SECY 82-465, as well as the integrated pressurized thermal shock, as well as the Yankee Rowe, normally what you would do is you would take the highest neutron fluence value and assume that that was acting everywhere, okay, and it was known that that was quite conservative, too. It's like let's put all the flaws on the inner surface, because we don't know anything else to do. Well, there usually wasn't detailed neutron fluence maps available at that time, so one of the things that, as I say, lessons learned, is to come up with a methodology that allows the RPV belt-line to be discretized into sub-regions, each with its own distinguishing embrittlement-related parameters, which -- therefore, this accommodates chemistries from the RVID database and detailed neutron fluence maps, because the reality is, as I'll show in some slides here in a moment, this section of the vessel may be considerably less embrittled than here. So, to assume that it all has the same embrittlement was quite conservative, and rightly so, some of the industry people were saying, but in the absence of a tool to incorporate this into the analysis, you know, you took the most conservative route. DR. WALLIS: What does this figure show? I don't understand. MR. DICKSON: Okay. I'm sorry. What this figure is showing here -- this is attempting -- think of this as the vessel rolled out, unwrapped, from zero to 360 degrees, where this is the core active region. Traditionally, the analyses have been -- when you talk about the belt-line region, you talk about, traditionally, from one foot below the core to one foot above the core, and so, these green regions here are meant to be welds, whereas the gray-like, whatever color this is, is plate material. So, what I'm saying here is that the FAVOR code has a methodology that allows you to break it down like this. DR. WALLIS: Doesn't look like a very fine grid. MR. DICKSON: Well, this is just for illustrative purposes. I mean you can break it down as fine as you -- the code leaves that up to the user. That's the discretion of the user. You break it down as fine as you want to. Well, that's not entirely true. DR. SIEBER: But that really doesn't make any difference as far as the geometry, because once a fracture starts, it will start in the most vulnerable place? MR. DICKSON: Not necessarily. We'll maybe get to that in a moment. I hate to generalize, because there's always the exceptions, but more often than not, the plate material is less embrittled than the weld material, okay? And even if you say, okay, the plate has a flaw density of one-tenth or 1/50th that of the weld material, you may still have cases where the less embrittled plate material drives the analysis, in other words contributes more to the overall risk of failing the vessel just by virtue of -- there's probably, out of 100 percent of this material, probably 99 percent of it's plate material. It has a lot less density of welds, but still it - - flaws -- but at the end of the day, you have a lot more flaws. In the old way of doing these analyses, as I said, you would take the most limiting and sort of concentrate on that. DR. UHRIG: This is a belt-line weld around the middle of the vessel right there? MR. DICKSON: Yeah, that's a circumferential weld. DR. UHRIG: All right. Then the core is above the center -- MR. DICKSON: Here's the core. DR. UHRIG: Okay. I thought it was down -- about split equally. MR. DICKSON: Well, as I say, when we talk about the belt-line region -- and maybe the next slide will help here a little bit. DR. UHRIG: Okay. MR. DICKSON: In fact, let's just move to the next slide. This is actually some real data, and the NRC contractor, by the way, that's doing the neutronics calculations to generate the neutron fluence maps -- this work is being done at Brookhaven National Laboratory, and this actually is from one of those analyses where they sent me the data, and this shows, again, from zero to 360 degrees, so if you can think back to that previous slide of the unwrapped vessel, this shows the asmuthal variation of the neutron fluence at particular axial locations. Now, this one here is at 72 inches from above the bottom of the core; in other words, kind of at mid-core. In other words, this is sort of the worst location, the worst axial location, and you can see that -- of course, this is repeating. It has a periodicity of 45 degrees. It goes down to 45 degrees, and then it's a mirror image of itself the next 45 degrees, and then it just repeats itself. So, that's 90 degrees, and it just repeats itself three more times as you come around the 365 degrees. DR. WALLIS: Why does it vary so much? MR. DICKSON: You'll have to talk to someone other than me. DR. SHACK: In the original analysis, you would assume the worst fluence, the worst chemistry, the worst thermal hydraulics, all at one location, and then you'd do a distribution of flaws and do the fracture mechanics on that? MR. DICKSON: Right. DR. KRESS: Now, you have a thermal plume coming down from the hot leg. Where is it located with respect to those high points? MR. DICKSON: At the moment, we assume an axial symmetric loading, okay? Now, to answer your question, you know, your inlets are, you know, somewhere up here, and some of the APEX experiments are directed toward this issue. In other words, have the plumes dissipated by the time you get down to the top of the core, and certainly that's been the assumption so far in most of our analyses, and it's not just out of thin air. I believe that Dr. Theofanis has some publications that states that, for commercial, domestic PWR designs, that that plume pretty much has dissipated by the time the coolant gets down to the top of the core. DR. KRESS: That was part of his remix. MR. DICKSON: Yes. We've done some work in this in the past, and I think we will try to verify this one more time. It's certainly our intent to -- we were involved in asking -- you know, in designing where the instrumentation went on the APEX experimental device for this very reason. We would like to sort of verify that one more time that that's the case. MR. MAYFIELD: Terry, could I stop you for just a second? MR. DICKSON: Sure. MR. MAYFIELD: This is Mike Mayfield from the staff. I wanted to make sure we understood the distribution of flaws Terry's talking about are fabrication- induced flaws rather than service-induced. So, we're talking about things that are in the vessel from the day it's built, as opposed to things that are induced by service. DR. SHACK: What's the rationale for ignoring the clad completely? MR. MAYFIELD: Well, we don't ignore it completely. DR. SHACK: I mean in the original calculation. MR. MAYFIELD: Even then, it was treated as -- it was taken as a conservative measure. So, it's not incorporated from a fracture mechanics standpoint but from the material toughness standpoint. It was included in terms of the differential thermal expansion you get. DR. SHACK: But you put the flaws on the inner surface of the clad. MR. MAYFIELD: Put the flaws on the inner surface, and you treated the metal like it was all faradic steel, rather than have the clad layer on it, except in the stress analysis, and there you picked up the differential thermal expansion, and I guess also, by association, the thermal analysis, because you pick up some thermal conductivity issues, but it was -- I think people didn't know exactly how to handle the duplex nature of the structure, so treated conservatively for the analyses that were done in the early '80s. MR. DICKSON: Pretty much what Mike said is still true today. Certainly, the little stainless steel clad is factored in in the calculation of the thermal response of the vessel, as well as the stress, even in the K1, the stress intensity factors for inner surface breaking flaws, but we do not check flaws that reside entirely in the clad for a cleavage fracture, because it's stainless steel, and it's much more ductile than the base material. So, to pretend that a cleavage fracture event could initiate there is just denying reality. It just isn't going to happen. Thinking back to the previous slide, this shows the level of the neutron fluences at one foot above the core and one foot -- you can see that it's decayed practically to zero by the time you get one foot above and one foot below, and this just shows the neutron fluence as a function -- it's the axial gradient. So, at the core flats, at zero, 90, 180, and 270, there is the axial variation, and there it is at other values. So, the point that these slides are -- it's just that FAVOR has the capability to handle this kind of detail in your fluence map. DR. SIEBER: In the righthand figure, where's the top of the core? MR. DICKSON: Zero is at the bottom. Zero is one foot below the bottom of the core. DR. SIEBER: I would have expected that embedded control rods would have caused that shape, as opposed to -- MR. JONES: Excuse me. This is Bill Jones from Research, NRC staff. That particular reactor has a fuel management scheme in the bottom of the core to hold the fluence down the vessel welds. So, that's why it has that maybe not characteristic shape, but that's why that particular axial looks that way. DR. SIEBER: Is that burnable poisons? MR. JONES: No, it is not. I believe it's stainless steel. DR. SIEBER: Oh, all right. DR. POWERS: If we have high burn-up fuel and get axial offset anomalies, does it distort these distributions substantially? MR. DICKSON: I'm sorry. I didn't catch the first part of that. DR. POWERS: If we use high burn-up fuel and we get axial offset anomalies, does it distort these fluences substantially? MR. DICKSON: I'll defer to the neutron people on that. MR. JONES: I'm sorry, Dr. Powers. You'll need to repeat that again. DR. POWERS: What I'm asking is, when people use very high burn-up fuels, they get a little boron absorption up high in the rods, and that causes a shift in the spectrum down to lower parts of the core. I'm wondering if it changes the fluences to the vessel enough to make a difference in your calculations. MR. JONES: Well, these calculations are pretty -- we tried to match -- the plants we did, we tried to match the way the cores would burn. For the future, what I believe ought to be done is that calculations will be done matching the way the core was burned, so the codes would be adequate to account for that spectral shift and do an adequate job of calculating what the fluence would be at the vessel wall. DR. SIEBER: I take it, just as a follow-on, that this is a sort of an average profile, as opposed to -- since axial offset occurs, even in a moderately burned core, that this is some kind of an average, as opposed to doing this in slices in time during a cycle. MR. JONES: It was done as slices in time, but certainly it's an average between those slices, yes. MR. DICKSON: If I'm not mistaken, Bill, I believe these asmuthal -- I believe they were about 3 degrees -- the increment, I believe, was like 3 degrees and the axial was like 6 inches or something like that, in that neighborhood, pretty small. I can tell you this: This is quite a bookkeeping exercise to keep up with this. A lot of data goes into the analysis. DR. KRESS: Do you digitize that before you input it? MR. DICKSON: Yeah. Of course, you know, as we know, in reality, it's a continuum, but as you often do, mathematically, you have to discretize. DR. SEALE: These neutron maps were generated using the revised DNDFB -- what is it, 6? -- cross-sections to more properly distribute the energy in the heavy steel? MR. DICKSON: I'll defer to Bill on that. MR. JONES: I believe it's 6, but the calculations only go to the inside -- we're only using fluences at the inside of the vessel wall. MR. DICKSON: Right. I guess I should have said that from the outset. It's understood that this is the magnitude of the fluences on the inside, and of course, it goes without saying, this is at a particular time in the plant life, remember, back to that first graph. You know, we're doing an analysis to plot a point as a function of EFPY. Ten years later, everything would be up, you know. Any questions, comments? DR. SHACK: What do you then do to get the distribution through the wall if you stop the fluence calculation? MR. DICKSON: We attenuate it, and we use an exponential to K constant of .24. DR. SIEBER: What's the basis of that? MR. DICKSON: There's people in this room that can speak to that better than I can. MR. LOIS: We derived in the '80s, and it came from a transport calculation for the vessel. Later on, we found that the displacement measure that some people used -- it gives approximately the same sort of grade through the vessel. But the one that is in the book, in the regulations right now, in Regulatory Guide 199, is the decay through the thickness of the vessel. MR. DICKSON: Actually, the data that Brookhaven provided actually had data through the wall, and Bill Jones has actually gone through the exercise and sort of verified that the .24 is still very much applicable, if anything slightly conservative. Is that true, Bill? MR. JONES: Yeah. MR. DICKSON: You found the .24 is still a valid number, and when it was off, it was off on the conservative side. MR. JONES: That's a true statement. MR. DICKSON: Okay. DR. WALLIS: What are the units of this .24? MR. DICKSON: Inches minus one. DR. WALLIS: Inches. You're in the dark ages. [Laughter.] DR. WALLIS: Even at MIT in the '50s, we used metric, as far as I can remember. Well, go on. MR. DICKSON: Well, divided by 25.4, I suppose. DR. KRESS: In Tennessee, we can divide. [Laughter.] MR. DICKSON: Speaking of MIT, I was at a meeting with Dr. Peter Griffith. The thing about units came up, and he said, well, I prefer to use Christian units, and I think he was talking about English units. DR. APOSTOLAKIS: I believe he's retired now. [Laughter.] MR. DICKSON: Okay. Moving along, new statistical models for enhanced plane strain static initiation and arrest fracture toughness databases have been implemented into the FAVOR code. Okay. Now, this shows K1C. This is fracture initiation toughness, plotted as function of T minus RTNDT. Now, the old ASME curve that's been around since the early 1970s was derived from a database collected by EPRI, Electric Power Research Institute, and within the last year, year-and-a-half, at Oak Ridge, we went through and found how much additional data had been generated since then -- it's been 28 years -- and found that there have been, I believe, 83 additional valid points that were valid according to certain ASTM regulations, and I'm not going to get bogged down in that detail. So, we said, okay, we'll take the enlarged database and really do a rigorous statistical analysis on it. Now, Dr. Kenny Bowman and Dr. Paul Williams did this and came up -- they fitted this with a WIBLE distribution. So, this shows your 254 points, and this actually shows the WIBLE distribution that they fitted to that. This bottom curve shows the -- it's actually the location parameter, the WIBLE location parameter, which is the lowest possible predicted K1C that you would ever predict, and this shows, you know, the 1/1,000th percentile and the 99.999 percentile, as well as the median, did the same thing for the fracture -- the crack arrest, known as K1A, and these are very important inputs into fracture analysis, into a probabilistic fracture analysis, how you statistically represent or, if you prefer, represent the uncertainty associated with the fracture data. DR. POWERS: It seems to me that the 99.999 percent line and the .001 percent line are very dependent on having selected WIBLE distribution, that if you'd selected something else, they would have greater or lesser width to them. Now, it seems to me that the WIBLE distribution has an empirical justification that applies in -- between the 90th and the 10th percentile. What justification is there to extrapolate it so far out, 99.999 percentile? MR. DICKSON: I'm not sure that I can address it. Mark, would you like to speak? DR. KIRK: Mark Kirk, NRC staff. Maybe we can just defer that to my presentation, but there are physical reasons why you would expect and can, in fact, demonstrate that fracture toughness data should have a WIBLE distribution. What Terry is showing is a result of a purely empirical analysis of the data, which happened to find that a WIBLE was the best fit to the data. Nevertheless, there's a good theoretical and physical justification for why the WIBLE should work, which I think helps to build the case that you should be using it, but you're absolutely correct, any model just picked from empiricism, out at the tails, you can have significant effects. DR. KRESS: I don't think you have intention of using the 99.999 for anything except the decision-making process. MR. DICKSON: No. DR. KRESS: It's just on there for illustration. MR. DICKSON: Right. It's on there for illustration. But certainly, the tails of the distribution can be quite important in these analyses sometimes. DR. APOSTOLAKIS: I don't see the distribution, but you must have done something else somewhere else, because here there is K versus R minus RTNDT. DR. KRESS: Is the distribution vertical? DR. APOSTOLAKIS: Where is the probability? MR. DICKSON: I actually have a little back-up slide here. I don't know if this will help. This shows the distribution at any vertical slice. DR. APOSTOLAKIS: Okay. So, R minus RTNDT is a parameter. MR. DICKSON: Yes. DR. APOSTOLAKIS: And you have a distribution for each value of that. MR. DICKSON: Yes. DR. APOSTOLAKIS: Okay. So, then you take the 99th percentile of each curve, and you plot it as a function of T minus RTNDT. MR. DICKSON: Yes. DR. KRESS: The thing about the WIBLE is it doesn't go out to infinity in both directions. It has a lower bound and an upper bound. MR. DICKSON: It's truncated on both ends. DR. POWERS: Triangled. MR. DICKSON: The WIBLE distribution -- I think it's very -- the derivation is very -- DR. KRESS: It has several parameters. You could make it flexible to fit. DR. POWERS: The triangle has the same number. DR. APOSTOLAKIS: It has three parameters, and with three parameters, you can do a lot. MR. DICKSON: The A, B, and C, the three parameters of this distribution, are functions of T minus RTNDT. So, when you say they're a function of T, that makes them very much time-dependent in the analysis, because temperature is changing through the wall. RTNDT, as well as changing as a function through the wall, it's changing as a function of when in the life of the plant you're doing this analysis. I will just say the final report on this -- there was a few other considerations other than WIBLE considered, and the report sort of discusses why the WIBLE distribution was the template that was sort of chosen to put into this, and certainly, one of them is that the WIBLE distribution, I believe, was developed especially kind of for fracture-type considerations. DR. APOSTOLAKIS: It's distribution of minimum values, is it not? DR. WALLIS: This RTNDT sets the temperature scale, doesn't it? If you slide horizontally, you can cover a big range of points. MR. DICKSON: Right. DR. WALLIS: So, how well do you know this RTNDT? MR. DICKSON: That's one of the variables that there's an awful lot of sampling going on. DR. WALLIS: That may be more important. The uncertainty you show here looks good, but if you don't know your RTNDT very well -- MR. DICKSON: The uncertainty of the RTNDT -- there's a lot of stuff going on inside of the analysis to determine that. I think Mark will talk about that. RTNDT is a function -- you know, thinking back to where we've talked all this about the discretization of the vessel, the embrittlement -- when you talk about the embrittlement, you're talking about the RTNDT, which is a function of the chemistry. DR. WALLIS: But someone who looked at these points had to decided what RTNDT was in order to plot the points? You could slide them around on that graph quite easily by having a different RTNDT. MR. DICKSON: There is actually a formula that tells you what RTNDT is as a function of chemistry and neutron fluence. DR. WALLIS: Is it a real thing? MR. DICKSON: It's a real thing, but there's a distribution associated with that. DR. POWERS: As you well know, Graham, they have been researching this embrittlement since the dawn of time. DR. WALLIS: Yeah. I'm just bringing out that there's another uncertainty; it's not just the vertical uncertainty. DR. POWERS: And that raises the legitimate question of these fitting processes. If you fit one uncertain parameter versus another uncertain parameter and you appeal to these squares, I will throw things at you, because that's just not right. Similarly, when I look at the calculations of the parameters of the WIBLE distribution that are reported in this document we've got, I am struck by -- you end up using the database to find your means and your shape parameters and shift parameters, at least when you calculate variances of an ordinary database and you use the mean of the database for the calculating of the variance, you introduce bias that you have to correct for. Don't you have to correct for bias when you do these formulae using the database to define your parameters? MR. DICKSON: Mark? You're talking about the embrittlement correlation itself. DR. KIRK: I'm not sure I completely understood that question. In fact, it would be fair to say I didn't. But let me just address the gentleman's comment over here. In Terry's model, the uncertainty -- he's just looking at part of it here. Certainly, the uncertainty in the vertical axis is considered, and this is the way it's currently been characterized. Equally, we've spent an awful lot of time -- and I suppose you could say that's the main focus of the next presentation -- in characterizing the RTNDT uncertainty. That's a major component, as well. With regards to the bias question, I'll invite Dr. Powers to ask it to me again, but I would just like to mention in passing that Terry's presenting here a purely empirical model, just to fit the data, with absolutely no appeal to any underlying physical theory. In the presentation that I'll be making, which concerns the uncertainty characterization of the K1C RTDNT uncertainty model, we do appeal to, you know, physical rationale and the underlying basis for cleavage fracture to help us get at what these distributions should be, not just argued from the data. So, I realize that's not a direct answer to your question, but I think that might help to put aside some of the questions concerning lower tails and bias and things of that nature. DR. KRESS: I'd like to point out that these are data taken from specimens, well characterized in terms of RTNDT. The chemistry is well taken. So, there's a relatively narrow uncertainty, but when you get ready to decide what RTNDT is for the vessel itself, so that you can select from the uncertainty there, it's a much different uncertainty. DR. WALLIS: That's a very helpful comment. Thank you. DR. POWERS: All except for the fact that -- yeah, they're well characterized as far as chemistry and things like that. There is still an uncertainty in the RTNDT which is non-trivial. DR. KRESS: Oh, absolutely. It's non-trivial. DR. WALLIS: How big is it? DR. APOSTOLAKIS: If you know the RTNDT, there is a spread of values for K. That's all he's telling you. DR. KRESS: I think Dana has a valid point, that how you get those variances and how they're distributed does depend on both uncertainties. DR. POWERS: Now, let's accept that we know RTNDT exactly and we look at a vertical slice, a set of your data points for a particular one, and you want to calculate the parameters of the WIBLE distribution. Okay. When you set out to do that, you need a mean, you need something like a variance, and you need something like a skew, because you've got three parameters you've got to find, so you've got to take three moments of the distribution to fit this thing. When I use the data to calculate those things, if I want to calculate the variance of a data set and I want to take the moment around the mean, right, and I calculate that mean from the data set and don't do something, I will calculate a biased estimate of the variance. It makes sense, because I've taken the data to calculate the mean, and when I calculate the skew using the mean of the data set to calculate the skew, I get a much bigger bias, even, because I'm taking the third power of it. Okay. I didn't see, in your write-up in here, how you accounted for that bias. MR. DICKSON: Which write-up are you talking about? DR. POWERS: It was a document given to us that goes through how you calculate the parameters of the distribution. MR. DICKSON: Okay. DR. KRESS: I think that came from the University of Maryland, didn't it, that particular document. DR. POWERS: So, they're really biased. They're biased clear over to the east coast. DR. KRESS: Yeah. Doesn't even have an accent. MR. DICKSON: There is a document that came from Oak Ridge. I can't answer your question. DR. SHACK: I would assume the statisticians took that into account. DR. WALLIS: There's another question, too. In a finite number of data points, you start talking about 99.999. Now, you need a certain number of data points before you can even, with any precision, talk about that sort of a number. MR. DICKSON: Point well taken. I believe that document talks about needing a minimum of 250 to do the particular analysis that they did. DR. WALLIS: And there's just about that here? MR. DICKSON: Two hundred and fifty-four. DR. WALLIS: So, just about enough to reach a .01- percent conclusion. DR. APOSTOLAKIS: What is the message you want to leave us with? MR. DICKSON: The only message here is that we consider this an improvement over what we have been doing for years. DR. APOSTOLAKIS: What were you doing before? MR. DICKSON: What we were doing for years was taking the ASME curve and saying, by definition, it is the mean minus 2 sigma, where 1 sigma was 15 percent of the mean. DR. APOSTOLAKIS: And the ASME curve was a curve like the one there? MR. DICKSON: The ASME curve actually was not a lower-bound curve. It was actually a lower-bound curve to about 88 percent of the data. DR. APOSTOLAKIS: Okay. But it was a curve like that. MR. DICKSON: Yeah. DR. APOSTOLAKIS: And now the new thing is the uncertainty. MR. DICKSON: Yes. DR. APOSTOLAKIS: Through the WIBLE distribution. MR. DICKSON: Yeah. So, we feel like this is a more rigorous statistical model than what we had. DR. APOSTOLAKIS: And why do you have this spread? Why do you have uncertainty? Dr. Kress said earlier that you had well-defined specimens. DR. KRESS: Yeah. Those are not specimens, but even those, whether or not K1C is an accurate phenomenological description of cleavage fracture is questionable. It depends on the geometry of the crack. DR. APOSTOLAKIS: So, if I take 100 of those, I will still have a spread. DR. KIRK: The thing that lights off cleavage fracture is the distribution of cleavage initiation sites or generally carbides in front of the crack, and you've got big carbides, you've got small carbides, and you've got a massive stress gradient. So, it's just a statistical process that sometimes they light off early and sometimes they light off late. DR. APOSTOLAKIS: So, this kind of an uncertainty in understanding will be part of the bigger picture later when you do the Monte Carlo study. MR. DICKSON: Yes, absolutely. DR. WALLIS: These specimens came from cutting up a pressure vessel? MR. DICKSON: No, not necessarily. Just from like pressure vessel-type steel, A508, A533. DR. WALLIS: That's not the same. It's been treated differently than a pressure vessel. MR. MAYFIELD: This is Mike Mayfield from the staff. The plate samples that were tested came from heats of plate that were thermally treated to be identical to the thermal treatments of reactor pressure vessels. They were purchased from the mills that made the plates that went into reactor pressure vessels, to the same specifications that were used, and that thermal treatments were imposed to be identical. The weld materials that have been tested were used -- were fabricated and heat treated using procedures that were as close to those used in fabricating reactor pressure vessels as was practical given that we're welding flat plate rather than cylindrical shell segments. So, there was some significant effort made to duplicate the materials that would have actually been used in fabricating reactor pressure vessels. As we have had the opportunity to cut up some pressure vessels, we have also been able to test materials from actual reactor pressure vessels that never went into service, and those materials fit in with these data quite nicely. So, there's good reason to believe that the data we have are representative of the materials that were in- service. DR. KRESS: The fluence was provided over a shorter period of time by putting them in a high flux intensity area. MR. MAYFIELD: The irradiation aspect of it from the test reactors introduces some uncertainty in the characterization, but the un-irradiated properties, the materials themselves are quite representative. DR. POWERS: Mike, have you ever found evidence of a flux dependence? MR. MAYFIELD: Yes. At the very high fluxes, we used to do test reactor irradiations where the capsules were put in core, and the flux there is high enough that we've subsequently stopped doing that and we go out to the core edge. I've forgotten the numbers, but the in-core fluxes were high enough that the theoreticians have shown us that that was the wrong thing to do. We've backed away from that, do core edge irradiations now, so we can still get some acceleration, but it's not so high as to be of concern. DR. SIEBER: Is there a dependency on the energy level of the fluence? MR. MAYFIELD: Yes, but for the range of fluences -- I'm sorry -- for the energy spectra that we typically see in power reactors, there is not such a large spectral difference for it to be an issue, and as long as we capture the neutron flux above 1 MEV, that's where the index -- it's a convenient index to use, and that's where the modeling has been done. DR. SIEBER: But through-wall, there should be a lot of attenuation, so there should be a variation in RTNDT through the wall. MR. MAYFIELD: There is absolutely a lowering of the RTNDT as you go through-wall, and that is accounted for in Terry's analyses through this attenuation parameter. So, he's attenuating the fluence as he goes through-wall and then can calculate an RTNDT, an adjusted RTNDT as a position of -- as a function of position. DR. SHACK: But you don't, for example, take into account the spectral change as you go through the wall. MR. MAYFIELD: No. DR. SIEBER: Is that important? MR. MAYFIELD: Over the ranges we're talking about, it's not a factor. DR. SIEBER: All right. Thank you. DR. KIRK: This is Mark Kirk from the staff. It might also be helpful to point out in passing that a lot of the questions that were just asked in the past few minutes regarding irradiation effects and materials effects all manifest themselves in a change in the index temperature parameter, RTNDT. They do not manifest themselves at all in a change in the vertical scatter. So, those uncertainties, material dependent differences and so on, are there and are considered, but they're taken up in a different part of the analysis. DR. SIEBER: Thank you. MR. DICKSON: Okay. This is just to maybe graphically illustrate, you know, an inner surface-breaking flaw as well as the embedded flaw, and the FAVOR code -- traditionally, the older FAVOR codes only did surface-breaking flaws, and as I said earlier, when they actually start doing destructive examination, non-destructive, destructive examination, they hadn't found any of these, but they found a ton of these. So, the FAVOR code -- and the -- actually, the mechanics that you have to do to deal with these flaws is dramatically different than the mechanics that you have to do to deal with these flaws, but the FAVOR code now will deal with either/or, you know, within the same analysis. You can have inner surface-breaking and/or embedded flaws in the same analysis. DR. WALLIS: There's an infinite variety to flaws, all kinds of flaws. MR. DICKSON: Oh, yeah. DR. WALLIS: Once you have a flaw, its shape and everything is -- it's like a snowflake, isn't it? I mean they're all different. MR. DICKSON: Yeah, sort of. DR. WALLIS: But somehow you can treat them all -- MR. DICKSON: No. Within the analysis -- I mean the things that get sampled within the analysis is the -- what we call the through-wall depth -- in other words, how deep is the flaw, how long is the flaw, where is the flaw, is this type flaw or this type flaw, where through the wall. All of those things are sampled, and the functions that they're sampled from are the characterization data that has been found in PVRUF, as well as the work that's being done here at the NRC to characterize the flaws. DR. WALLIS: Are they typically very small things? MR. DICKSON: Yes, for the most part. DR. WALLIS: What sort of size? MR. DICKSON: Let me answer it this way. I think the largest flaw that has been found in this actual effort of going and cutting this vessel material up, the largest flaw that's been found is an embedded flaw that's 17 millimeters through-wall extent. How long it was, I actually don't know. We found a whole lot more flaws than was used, not so much smaller, they were just in the wall, and from a fracture mechanics point of view, this flaw was a whole lot more benign than this flaw. DR. SIEBER: On the other hand, there is a critical flaw size where propagation occurs, which is a function of RTNDT. MR. DICKSON: Well, it's a function of everything. It's a function of the embrittlement, it's a function of the transient. DR. SIEBER: Right. MS. JACKSON: This is Debbie Jackson from the NRC staff. To date, the largest flaw that PNNL has found, like Terry said, was 17 millimeters, and it was approximately, I believe, 5 millimeters long, but they found some other flaws in other vessel material. We're doing Shoreham, River Bend, and Hope Creek, and we found some flaws, but they haven't been validated, that are a little longer than 17 millimeters. MR. DICKSON: So, if you remember that box that I showed, all the things feeding into the middle, one of them was flaw characterization, and that's a pretty general term. It's how many flaws, what characteristics, you know, how long, how deep, where in the wall, all of that gets into the analysis. DR. SIEBER: I presume that the flaws that have been found are not uniformly distributed through the wall. MR. DICKSON: Debbie can speak to that better than I can. MS. JACKSON: This is Debbie Jackson again. The majority of the flaws have been found in weld repair regions, regions of the welds that have been repaired, and we're presently doing some additional exams on the base metal. MR. DICKSON: I believe you asked are they uniformly distributed through the thickness of the wall -- DR. SIEBER: Yes. MR. DICKSON: -- and I believe the answer is approximately. MS. JACKSON: Right, where a weld repair is located, right. There are a lot of smaller flaws. MR. DICKSON: Okay. I'm going to have one slide here, just a little bit about the structure of FAVOR, which I don't know if that's real important here, but we'll talk about it anyway. When we talk about the FAVOR code, I don't know what people conjure up in their mind. This code -- the current code that we're working on consists of three separate, independent modules. The first one is -- and I'll talk a little bit more about these on subsequent slides. The first one is a load generator, and the input -- this first level, the blue boxes up here, is input data. These yellow boxes -- they're the actual executable modules. So, when you talk about FAVOR, this is what you're talking about, and this last row here is what comes out of each of the modules. So, your first one is your load generator. That's where you actually put in your thermal hydraulic boundary conditions from, typically, like this output from the RELAP code, and of course, you input your thermal elastic material properties for the clad base material, elasticity, thermal conductivity, coefficient of expansion, on and on, the RPV geometry. Now, a transient description -- typically, it will come in the form of three time histories, the pressure time history that's acting on the inner surface of the vessel, the coolant temperature time history, and the convective heat transfer coefficient. Now, each one of these, you can input 1,000 time history pairs for each of the three, and typically, I think the people from the thermal hydraulics branch told me that, for Oconnee, there's 27 transients. So, you're talking about 81,000 time history pairs. So, again, big bookkeeping exercise, as well as doing the mechanics. Out of this comes the through-wall temperature gradient as a function of location and time, stresses, and - - DR. WALLIS: I think it would be XYZT. MR. DICKSON: This is an axial symmetric. It's a one-dimensional analysis through the wall. DR. WALLIS: That's all? MR. DICKSON: Yeah. Since we are assuming that the thermal hydraulic boundary conditions are axial symmetric, you only need a one-dimensional analysis. DR. WALLIS: You don't need a vertical coordinate, too? MR. DICKSON: No. Okay. So, you run this module, and out of that comes a file that contains all of this. DR. WALLIS: If the thermal hydraulic tests showed plumes, you'd have to change this into something different. MR. DICKSON: Yeah, if it showed that they were very significant. DR. WALLIS: Now, is FAVOR able to do that? MR. DICKSON: We would have to re-do this aspect of it, not the whole thing. DR. WALLIS: You'd have to re-do that part of it. MR. DICKSON: We would have to do that part of it. It would not be trivial. Essentially, instead of doing a one-dimensional finite element analysis, we would have to do a three- dimensional finite element analysis. DR. WALLIS: In order to be cautious, you might want to do an XYT one just to see what happened if you did have a plume, even if you had reason to hope it wasn't there. MR. DICKSON: We have some publications on that. Actually -- I want to digress too far on this. In the first two versions of FAVOR that came out, the '94 and '95 versions, there actually was a option in there to include this, and in a conservative sort of way, in a bounding sort of way. So, we really did get in and do a lot of analysis, but when we went to this next version, the decision was made early on that we're not going to do that. Now, if APEX shows that this could be very significant, we may have to go back and do that. Anyway, load-generator -- MR. MAYFIELD: This is Mike Mayfield from the staff. This was something that we had some considerable debate on, and we recognize there is an element of gambling here that -- on what the APEX results will show us, but when we looked at the other analyses that had been done, we took a deep breath and said let's move forward, but at the same time perform the experiments to try and sort it out, and if we guessed wrong, then we're going to have to do some additional work, and we'll have to deal with the schedule impact, but we went into it recognizing there was an element of risk that we would have guessed wrong. DR. WALLIS: Well, computationally, doing XYZT isn't that complicated, just you've got to do so many runs? Is that what it is? MR. DICKSON: No. I mean what you're talking about -- if these plumes, which are multi-dimensional by nature -- if they are significant, you can no longer do a one-dimensional axial symmetric analysis. DR. WALLIS: That's right. MR. DICKSON: And as I'll talk about in a moment, we do finite element analyses. So, writing a finite element code -- DR. WALLIS: It's not that difficult to do. MR. DICKSON: Well, it's like everything here. The devil is in the details. Writing a three-dimensional finite element code is not a trivial -- it's not something you do in an afternoon. MR. MAYFIELD: I think the other thing we've talked about is we would have to look at some -- perhaps some simplifying approaches to the stress analysis, which would take us to a little bit different structure in this particular module. MR. DICKSON: Yeah, it might. MR. MAYFIELD: So, we could very well end up having to do some things off-line and then load in stress information, but there are other approaches that could be considered rather than the approach that Terry has been able to make use of given the one-dimensional nature of the problem. But your questions are -- we agree completely. Some of us that were involved in making that decision are waiting quite anxiously to see how close we were or weren't to being right. MR. DICKSON: Again, I'll refer back to some of Theofanis' publications saying that, for U.S. domestic commercial designs, it should not be significant. So, I guess we're putting some faith in that. Okay. Then here's the probabilistic fracture mechanics module, and the input to that is all of the flaw data, which, again, tells you the densities, number of flaws, as well as the size and location. Also coming into this PFM module is the embrittlement data. You remember that, where I showed the vessel rolled out from zero to 360 degrees. Each one of those little regions has a particular chemistry and neutron fluence, something that gives you enough information to calculate the RTNDT of each one of those regions. Okay. And also, of course, the loads from the load generator -- all this obviously is input into here. DR. WALLIS: The belt-line embrittlement isn't one-dimensional either. MR. DICKSON: No. DR. SIEBER: Well, you aren't really talking about belt-line. You're talking about the whole show, right? MR. DICKSON: We're talking about the entire belt- line region. It's two-dimensional. DR. WALLIS: How do you do that two-dimensionally and do the stress one-dimensionally? MR. DICKSON: Well, it's just the same stresses are assumed. DR. WALLIS: Okay. I guess it is independent. It's all independent. MR. DICKSON: Yes. DR. WALLIS: Yeah, I guess that's right. MR. DICKSON: The assumption is that the stress profile acting through the wall here is the same as it is anywhere else, and the temperature profile through the wall, which is a totally independent assumption from what the embrittlement is in any location. Okay. DR. WALLIS: So, embrittlement doesn't change the modulus. MR. DICKSON: No. The modulus of elasticity? No. DR. WALLIS: For the stress calculations. MR. DICKSON: Embrittlement changes the yield strength a little bit, but basically, we're doing -- DR. WALLIS: No, it doesn't change it. MR. DICKSON: No, it doesn't change the modulus, no. It changes the yield stress some, but we're doing a linear elastic analyses here. We're not doing elastic plastic analyses. DR. WALLIS: Never get that close. Never get close to that, do you? MR. DICKSON: No, there are cases where you can get into plasticity, but the assumption traditionally has been that a LEFM analysis is conservative, as opposed to where you actually consider the plasticity. DR. WALLIS: Once it begins to fail. MR. DICKSON: I think we're going to have to take a closer look at that when it comes to embedded flaws. You know, for surface-breaking flaws, I think, traditionally, it's been shown that, if you do a LEFM analysis, linear elastic fracture mechanics analysis, that that is conservative with regard to surface-breaking flaws, but for embedded flaws, I don't think we at Oak Ridge are yet convinced that that is necessarily the case. MR. MAYFIELD: Terry, this is Mike Mayfield again. For surface-breaking flaws, we've shown it to be accurate. MR. DICKSON: Yeah. MR. MAYFIELD: It correctly characterizes the phenomenology for a highly-embrittled pressure vessel at the surface, and if you have a less-embrittled pressure vessel such that you actually had to go to an elastic plastic analysis, by and large pressurized thermal shock wouldn't be an issue. So, what you're really talking about is the set of conditions that gets you to a linear elastic fracture phenomena. So, it's not just that it's conservative. It is, in fact, accurate for the surface-breaking flaws. As Terry says, for the embedded flaws, it gets a little more interesting, but by and large, if it's an elastic plastic initiation, it's not much of a challenge from pressurized thermal shock. MR. DICKSON: So, out of this PFM module comes actual distributions for the conditional probability of initiation -- in other words, the conditional probability that you initiate a flaw in cleavage fracture, the conditional probability of failure. In other words, just because you initiate a flaw does not necessarily mean it's going to propagate all the way through the wall. There's another analysis that has to be done to determine whether that flaw that initiates actually goes through and makes a hole in the side of the vessel. So, for each transient, what comes out of here is the distributions for the conditional probability of initiation, conditional probability of failure for each transient. Okay. Now, this third box over here -- this is actually the one I'm working on right now, as we speak, as far as the development aspect. The input into this, of course, is these distributions for the conditional probability of initiation, conditional probability of failure, and I keep using the word conditional. It's conditional that the transient occurred, okay, but input here is actually the distribution of the initiating frequencies, initiating frequency being, you know, how often this transient occurred. DR. APOSTOLAKIS: So, the transient does not appear for the first time there. MR. DICKSON: Yeah. The transient as a thermal hydraulic boundary condition that actually occurs is back over here. You go ahead and calculate, if this transient happens, here's the result. Here's the actual probability of the transient even happening to begin with. And I'll talk briefly in a couple of slides here about how these distributions and these distributions get together to form this final distribution, which is the frequency of RPV fracture, or in other words, the crack initiation, the frequency that you fracture the vessel and also the frequency that you fail the vessel. DR. WALLIS: These transients you start with are some sort of design basis accident? DR. SIEBER: Not necessarily. MR. DICKSON: Not necessarily. I don't feel like I'm probably the person to speak to the thermal hydraulics aspect of it. MR. CUNNINGHAM: This is Mark Cunningham from the staff. You don't start from the set of traditional design basis accidents that are in Chapter 15 and that sort of thing. You start from -- you look at information on what types of transients in the plants can cause issues of concern. DR. WALLIS: Okay. MR. CUNNINGHAM: Over-cooling, pressurization. So, it can be small-break LOCAs with operator actions, it can be a variety of things like that, but it doesn't start from the Chapter 15 analysis. DR. KRESS: It's a selection from the normal PRA of those sequences that might be important. MR. CUNNINGHAM: Yeah. Somebody when we talk about the PRA, you'll see we look at PRA information, we look at operational experience and that sort of thing, to see what's going on and what could cause these types of situations in the plant, in the vessel. DR. SHACK: Now, when you do the running crack, it's running one-dimensionally, right? You're not trying to also do a length-growth of this thing, or are you? MR. DICKSON: No. The assumption is that -- and this is consistent with experimental results that we've observed through the years at Oak Ridge, is that an initiated flaw runs long before it runs deep, propagates along the length of the vessel before it propagates through the wall. So, you could start with -- you could thermally shock this vessel right here with this flaw, and this flaw is going to want to extend this way before it goes through the wall, and also, with this flaw, the assumption is that this flaw is going to propagate in toward the inner surface, because it's propagating into a region of higher embrittlement, as well as higher stress, so it's got a higher load and a lower material resistance in this direction. So, the assumption -- you check it at the inner crack tip, you check it for initiation, you know, and if it initiates, you assume that it breaks through and becomes long. So, an initiated flaw is a long flaw, becomes a long flaw. Then the question is, now, do you, the long flaw, do you propagate through the wall? DR. POWERS: Do you get a damage accumulation in these processes? That is, if I do a little bit of insult, little bit of over-cooling, I get my cracks bigger and bigger and bigger? MR. DICKSON: No. No, there's no -- I think Mike Mayfield mentioned a moment ago, there's no service-induced crack growth here. If the flaw is predicted to crack, it's predicted to be a major -- I mean it runs long and breaks through. DR. KRESS: You don't use the time-dependent flaw distribution, is what you're saying. MR. DICKSON: Yeah. There is no time-dependent flaw distribution. DR. POWERS: I'm asking about the phenomenology here. DR. SIEBER: Well it would seem to me that you can initiate a crack through some event that arrests, and then, as the vessel runs again, you continue to embrittle it so that the next event you have, it can go even further, and I guess that's what in-service vessel inspection's all about, to try to find those situations, if you can. MR. MAYFIELD: This is Mike Mayfield from the staff. I think the notion is that if, in fact, you had a substantive over-cooling, you have to go back and inspect the vessel. So, there would be -- you would intend to go look for anything that you might have popped in. There's always, of course -- in-service inspection or non-destructive examination is not 100-percent. So, there is the potential that you could have a pop-in that you would miss, but -- it's not inconceivable, but we think that the prospect of that is not all that high. First of all there aren't that many highly-embrittled vessels out there. But I think your point is correct. DR. SIEBER: If you were to do an inspection, what is the minimum flaw size you can characterize? Could you find these 17-millimeter flaws? MR. MAYFIELD: Yes. If you know to go look for them. So, it gets to be a question of what would you ask the inspectors to do? You probably would not send them out and ask them to do a traditional Section 11 in-service inspection. So, it's a question of what you would actually -- what a licensee would actually ask their in-service inspection team to go look for. I guess I should also say that the characterization, the flaw characterizations that Terry talked about and that Debbie Jackson mentioned started out with ultrasonic examination. It's some fairly sophisticated ultrasonic examination, but that's where it starts, followed up by destructive examination. DR. SHACK: Now, presumably you have done the fracture -- the fatigue flaw growth to convince yourself that these things really don't grow by fatigue. MR. MAYFIELD: This is something that a lot of us used to make our living trying to do, and this comes up -- everybody that looks at PTS wants to go do a fatigue crack growth evaluation, and they consistently come back with the same answer, that it's a no-never mind, and the reason is the normal operating stresses are so low, even if you had a surface-breaking flaw, the cyclic component of normal operation is so low and these flaw sizes are so small, you get just no growth. MR. DICKSON: I guess, before I move on to the next slide, the main purpose of this slide -- a main purpose of this slide is that the bottom line, what comes out of -- after you run all three of these modules of FAVOR, the bottom line is a frequency, a distribution of the frequency of how often you fail the vessel, and that distribution has a mean value which you would then go plot on that curve that I showed early in the presentation, you know, let's say that we're doing this at 32EFPY. So, each time in the life of the plant has a distribution of failure associated with it, which has a mean value, which you would go back and plot on the curve I showed, and of course, there would be a distribution. DR. APOSTOLAKIS: It's not clear a priori that it will work with a mean value. I mean it depends how wide the distribution is. MR. DICKSON: Right. DR. APOSTOLAKIS: So, I understand we have a subcommittee meeting coming up, and we'll talk about how all these uncertainty calculations are being done. Is that correct? MR. DICKSON: Yes. MR. HACKETT: Professor, I think we were talking in December, most likely. This is Ed Hackett from the staff. DR. APOSTOLAKIS: I thought it was November 16th. That's not true anymore? MR. HACKETT: Terry, this is Ed Hackett. I was going to suggest, in the interest of time, because we're running a bit behind here -- I think you made a good jumping-off point to go to your slide 19, the second-to- last. Why don't we just go to that? DR. WALLIS: I have a point on number 17. You are actually looking at uncertainty in the thermal hydraulics by varying some of the parameters in that in a statistical random way or something? MR. DICKSON: Do you want me to jump to the last slide? The answer is yes, this is an attempt to capture the uncertainty associated with the thermal hydraulics. DR. WALLIS: And you're getting some good consulting on that or something from somebody? MR. DICKSON: This is an interface between the PRA people and the thermal hydraulics people. DR. WALLIS: They have a good way of estimating uncertainties in the thermal hydraulics? MR. CUNNINGHAM: In the PRA part of this, this has been one of the challenges, how do you bring that together. This will be one of the subjects we'll talk about, I guess, at the December meeting. DR. WALLIS: I think it's important to do. I just wondered if anybody knew how to do it well. MR. CUNNINGHAM: We'll see, but we think we can do it with a fair amount of comfort on our part. DR. WALLIS: Let's hear about that at some point. MR. CUNNINGHAM: Yes, in December. MR. DICKSON: Well, this is just an attempt to show that we have major categories of transient. Maybe this is a LOCA, and within that LOCA, there are several variants of that, and each one of them has a certain frequency of occurrence, a distribution of the frequency of occurrence, which we talked about. All this feeds into this load generator of FAVOR, which performs a one-dimensional, axial symmetric, finite element thermal analysis as well as a finite element stress analysis, and as I showed a moment ago, the output from that is a lot of temperatures, stresses, and stress intensity factors, and I'll try to be real brief here. I've got two more slides. This is, again, talking about this probabilistic fracture mechanics module. Again, this is redundant. The input data coming into this is all the flaw data, the embrittlement map, as well as some uncertainty, 1 sigma values, as well as the loads, and what comes out of that is an array -- this is for -- I call this PFMI. This is for initiation, as opposed to PFMF, which is conditional probability of failure. So, what comes out this PFM module is like a two- dimensional array for so many vessels and so many transients, okay? Now, I know this maybe is a little bit not clear at this point, but let's just say that each entry in this matrix is the conditional probability that that vessel initiated or failed when subjected to that particular transient, okay? So, we end up with these two arrays. Now, it would be another whole presentation to talk about the actual details of what goes into calculating that the -- the probability that that vessel fractured when subjected to that transient, and we're not going to go there in this presentation, because we don't have time. So, this is the last slide, but I think that is one of the things, when we get together for this meeting of uncertainty, that we will talk in great detail about, and this is the post-process, just trying to graphically show how we integrate the uncertainties of the transient initiating frequencies with the PFMI and PFM arrays that I just showed, which is what comes out of your probabilistic fracture mechanics analysis, to generate distributions for the frequency of RPV fracture and failure. This is an attempt to show the distribution of the initiating frequency for transient one-two-dot-dot-N that feeds in here, as well as the arrays that were calculated from the probabilistic analysis. And this is showing -- here's the bottom-line answer that comes out of this, which is the frequency of RPV fracture, and what you do to actually generate this distribution is, for each vessel -- keep in mind, we're going maybe 100,000 or a million vessels, this is a Monte Carlo process, so we sample each one of these distributions to get a frequency for each of the transients, and then we combine that with the results of the PFM analysis where the conditional probability of failure for a vessel is the summation of the products of the initiating frequency with the conditional probability of fracture for that vessel. So, what you're multiplying here is events per year, failure per event, which is failures per year. That's what you end up with. I mean this looks kind of difficult but it's really just pretty straightforward. At the end of the day, you end up -- let's say you end up with a million of these values, which you then sort and then generate this distribution, which has some mean value and some level of uncertainty about it. So, that's the bottom line. DR. WALLIS: This is all what you intend to do. This is all the specifications for FAVOR. MR. DICKSON: Yeah, this is the specifications, and basically, the first two modules are pretty complete, although there's a lot of details that are still up for grabs. This third module, I'm developing that right now. DR. UHRIG: What's the abscissa here in the lower graph? MR. DICKSON: This would be frequency of failure, the frequency of the frequency. So, it's a histogram. It's the frequency of the frequency. Did you follow that? I mean what we're talking about here is the frequency of RPV fracture, so many fractures per year, and then a histogram, by its very definition, is the relative frequency or, if you wish, the density of something. So, you can think of it as the relative frequency or the density of the frequency of fracture. Is that clear? It's a lot of words. I can see why it might not be. This is just a histogram or a distribution of the frequency of vessel failure. DR. WALLIS: What is your specification for how long it takes to run this? MR. DICKSON: Well, it takes a while. DR. WALLIS: It's not much use if it takes a year. MR. DICKSON: No, no, it doesn't take a year, but to run like 25 transients for many 100,000 vessels, on a machine that I have in my office, which is 533 megahertz -- it was the newest machine a year ago -- it's an overnight job. DR. WALLIS: That's not so bad. MR. MAYFIELD: Dr. Wallis, in comparison to the big system-level thermal hydraulics codes, this thing is a blink of the eye. It runs very quickly relative to some of the large codes. DR. WALLIS: So, the inputs you're getting from RELAP will be the bottleneck, then. MR. DICKSON: Yeah, I guess so. DR. WALLIS: So, maybe we should work on the bottleneck. [Laughter.] DR. POWERS: You began this session by saying to us that thermal hydraulics had not improved yet. Let's work on the part that's improved, and that's the fracture mechanics. MR. MAYFIELD: If we don't have any other questions for Terry, I guess I would -- we have a second presentation, looking at the materials, and I guess the question I would pose to the committee is, do you want us to try and stay within the time slot, or do you want Dr. Kirk to give you the presentation, or some variant on that? DR. POWERS: I believe you have till 20 of the hour. If he does what he professes to do, persuade me that the WIBLE distribution has a theoretical foundation, he has till midnight. [Laughter.] MR. MAYFIELD: Let me assure you, sir, there will be those of us that get the hook long before midnight. Mark, why don't you try and stay within the time slot? DR. KIRK: I'll try to keep it snappy. Okay. Well, this is just one part of many parts of Terry's overall calculation, and the goal in this effort is to characterize toughness using all available data, information, in a way that is PRA-consistent, and what I mean by that, before I get shot down by the whole committee, is a best estimate process, and that, as you'll see here, is actually quite a remarkable change from where we've been before. The process that we've gone through is, first off, to start with a data evaluation, and we asked Terry's colleagues at Oak Ridge to assemble all of the available valid -- and by valid, I mean linear elastic valid K1C and K1A data. Terry showed you that. They did a purely statistical analysis of the data, for what it's worth, but we wanted to start with the largest empirical database possible. That information was then passed to Dr. Modarres of University of Maryland and Dr. Natishan of PAI to help us establish the sources of uncertainty. They performed a root cause analysis, which I'll go through in some level of detail, and appealed to a physical basis for the process of cleavage fracture to help us to distinguish in this overall process of both RTNDT uncertainty and fracture toughness uncertainty what parts of those uncertainties are aleatory and epistemic and what's the proper way to account for them in the probabilistic fracture mechanics calculation. Professor Modarres and his students developed a mathematical model to treat the parameter and model uncertainty which is currently -- we're working the bugs out of that but is currently being coded into FAVOR. So, that's the ultimate end result of all this, is a description of initiation fracture toughness and arrest fracture toughness in FAVOR that accounts for all the uncertainties in an appropriate way. Terry showed you -- this is in your slides. Terry showed you the database before. This just points out that there was some data growth from where we were in both the crack initiation and crack arrest data, and it also points out that the bounds on the curves that we were using before in the SECY 82-465 and in the IPTS studies, which are shown in red, are considerably different than the bounds that we're using now, but of course, that all goes into the mix and you turn the crank. And as is noted on the bottom in yellow, the implications of that increase in uncertainty -- and here you should really just be looking at the vertical axis uncertainty, because that's all that is reflected on this slide -- really depends upon the transient considered. Terry has done some scoping analyses which reflect that and, I believe, were published in a pressure vessel and piping conference publication. The analysis process that Drs. Natishan and Modarres used, we've been referring to as a root cause diagram analysis, and the nice thing about this, at least from my perspective as a participant in the process, is it really -- it's a visual representation of a formula, but it can be a very complex formula, and it helps to both build consensus among the experts and it also provides a common language for discussion between the materials folks and the PRA folks. One thing that's very nice about it is it helps us to position -- everybody's got their most important variable -- the fluence folks, fluence is most important; the materials folks like myself, copper is most important. They're all, of course, very important, but they enter the process in different places, and you need to get the parameters or the boxes -- or you could think of those as distributions of variables -- and then the relationships between the variables are shown at the nodes. So, you can imagine here -- and I'll show you an actual example of one of these in a minute -- putting in copper and nickel and initial RTNDT on this side and propagating through a mathematical model and coming out with a distribution of RTNDT which then samples a K1C distribution on the other side. One big change from the old way of doing this -- by the old way, I mean in the IPTS studies and in SECY 82- 465 -- that this process represents is that here we -- as I just mentioned, we input the uncertainties in the basic variables -- copper, nickel, initial mechanical properties and so on -- and propagate those to output uncertainties, ultimately, RTNDT or somewhere way over there, probability of vessel failure. What we don't do, which is what we used to do, is to input the margins or the uncertainties from outside the analysis. That's what happened before. We had -- for example, for the embrittlement trend curve, we had a bit of uncertainty of -- a standard deviation of -- it was 28 degrees Fahrenheit on welds. So, that became one of the parameter boxes that was applied, rather than letting the analysis figure it out from copper and nickel and so on. The other thing that this provides us, which sort of goes to the end application of all of this work, is in the end, of course, we're trying to figure out a new PTS screening criteria that we can compare plants to. When we compare plants, at least in the current way of doing things, we generate a best estimate -- in this case, RTNDT -- and then we add a margin to it. As I stated now, right now, the margin was sort of decoupled from this process, whereas here, the margin is determined through the process. So, in that way, it's much more self-consistent than it was in the past. Now, this has enough details to keep us here till midnight, and we can do that if I'm allowed to make a phone call and get my son to football, but what I'd like to do is to just make a few points and then open it up to questions if you all have any. I should note that information flow on this diagram goes from right to left rather than left to right, but what we've done is we've diagrammed or mathematically modeled a process that's fully consistent with the current regulatory framework for establishing an estimate of the fracture toughness of the vessel at end-of-license fluence. So, you start somewhere back here. Now, you see, I haven't shown you everything, because the sharpie embrittlement shift, the delta-T 30 goes to a completely other view-graph which has its own set of information and fluence and through-wall attenuation and all of that. The diagram that you see here is all of -- everything that feeds into node 3 is just the process by which we estimate RTNDT un-irradiated for a particular material in the vessel, and this is the diagrammatic representation -- don't be too alarmed -- of what's actually in SECY 82-465 or Reg. Guide 1.99, Rev. 2. So, anyway, you put in all the variables, and what this treats is different levels of knowledge. Sometimes you have specific material properties for a specific weld. Other times you have to use generic data. Other times you might have only some information. But in any event, you propagate that all through, you get an estimate of RTNDT un-irradiated. You add to that an estimate of the sharpie shift. You then get an estimate of RTNDT irradiated, and then I guess I wish to make the third point second, because I'm at that node. When we get RTNDT irradiated -- and this is what came up before -- is there is a recognition here -- and I think I'll go to the next slide and then come back, and this is, again, a new feature of this analysis. There's a recognition here that RTNDT -- that the RTNDT value, which serves as an indexing parameter for placing -- here I've just shown a bounding curve, but equally, it can position a family of curves or a density distribution. There's the recognition that RTNDT sometimes does a pretty good job at predicting where the transition curve lies with relation to real data, and here we're taking the linear elastic fracture toughness as our version of reality, and sometimes it doesn't do such a good job at all, and I think that gets back to some of the questions that were over here. Now, that -- nobody should be too alarmed by that, that's, in fact, expected, since the way that RTNDT is defined in ASME NB2331 is it's designed to be a bounding estimate of the transition temperature, so it's always supposed to put the K1C curve here with what some people have called white space in between. Unfortunately, that's inconsistent with a PRA approach that is premised on having best estimates of all these parameters. So, if we're going to stay consistent with the PRA best estimate philosophy, we then need to make a bias -- make a statistical bias correction on that RTNDT value. Now, this is something that we're -- I'll just have to say we're still working out the details on, and some of the candidate bias corrections are shown up here. The mean values tend to be different, but in answer to one of the questions that was asked of how far off can RTNDT be, it can be anywhere from right on the money to 150 degrees Fahrenheit off, and that simply comes from -- in the simplest way of looking at one of these corrections, you plot the linear elastic fracture toughness data that you measure for a particular material, you position a K1C curve based on sharpie and NTD tests, which are, of course, completely independent of the fracture toughness tests, and you see how far they're apart. It can be anywhere from zero to 150 degrees Fahrenheit, and what I learned from Mohammed and his graduate students is it's probably best characterized by a uniform distribution. So, in the process, we go through the type of calculations that the licensees would have to perform to estimate an un-irradiated RTNDT. You can go to another view-graph and do the type of calculations the licensees would have to perform to get a shift, and of course, in the code, since the Monte Carlo, we do this many, many, many, many times, we go into this node, we add this to this, and then we make a bias correction by just -- this is a cumulative distribution function -- by simulating a random number from zero to one, coming in and picking off the bias correction for that run, and of course, that happens a whole host of times. You then get your estimate of RTNDT irradiated, which helps you -- which indexes you into what vertical cut you take through here, and then you get your vertical uncertainty that we were talking about before. Now, again, like I said, this is a status report of work in progress. We had a meeting at the University of Maryland yesterday, and one of the things we realized, because both Dr. Modarres, Dr. Natishan, and all the folks at Oak Ridge are still working on this, is that, as we've noted, the data says that there is an unmistakable bias in RTNDT that we need to correct for. That bias enters this calculation in two places. One is in the estimate of RTNDT irradiated that you make for a particular vessel, and that's shown, appropriately, as being corrected for here, but obviously, it's had an influence here, because you plotted all of your fracture toughness data versus RTNDT. We're still working on the details of how that should best be corrected for, but suffice it to say we need to correct for that bias in the data set in an appropriate manner, because otherwise, the data set includes a perhaps un-decipherable mix of both aleatory and epistemic uncertainties. What we'd like to do if we can get the -- if we can work out the math and -- pay no attention to the graphs, I don't really have anything to say about them, just to say that we're working on different mathematical correction procedures, but what we're aiming to get, of course, here is take the -- is a methodology to take the epistemic uncertainty in RTNDT out of this fracture toughness distribution so that we can treat it as a pure aleatory, which is where Mohammed -- which is how Mohammed and Nathan have recommended dealing with the fracture toughness distribution. The concept is to get -- and this, conceptually, should work. The details, I must admit, escape me a little bit, but the idea is to get all of the epistemic uncertainties into RTNDT, and when you look at the process, you conclude that's, indeed, where they are, and then just leave the aleatory uncertainties in the WIBLE distribution of fracture toughness, which represents the inherent material -- the inherent inhomogeneity of the material in the transition temperature regime. So, that is something that's being worked out as we speak and probably will be a good topic for the next meeting. So, so far, we've completed the statistical transition fracture toughness model, we've collected all the linear elastic data that we could lay our hands on, and did a truly empirical fit to it. I'd say we're probably about 85 to 90 percent of the way on developing our PRA uncertainty framework. We've understood the process using the root cause diagram approach, developed mathematical models of the process, and we're working on the details of the FAVOR implementation, and of course, everything's an iterative process, and in the process of getting this actually coded into FAVOR, we realized that we had treated the model uncertainty in RTNDT in the vessel estimate part but not in the toughness correlation part. So, that's something we had to go back and do. Ongoing is full implementation into FAVOR, and as I mentioned here, resolution of RTNDT bias correction function and modeling procedure, and of course, assembly of input data from the various plants that we're considering to actually run these models. Questions? DR. POWERS: I didn't get to see why my WIBLE distribution is of fundamental significance, but I guess I'll have to wait on that. DR. KIRK: I'd be happy to talk about that next time, but I don't have my slides. But I do want to point out, what we've focused on here is a lot of the empiricisms behind this, but what also stands behind all of the data that you see is really quite a good fundamental understanding of why all the variations of toughness with temperature should be the same before and after irradiation for all these product forms, for all these chemistries, and why all of the distributions should be the same, and that's sort of part of the background basis, so we can -- you know, we can brief you on that next time. DR. WALLIS: I guess what will be interesting in the end is how the results you're going to use for making decisions are sensitive to the various things you did to get those results. Now, when you've got that far, you could see how the assumptions and processes and all that influence the actual final product. DR. KIRK: Right. DR. POWERS: Are there any other questions that people want to ask? DR. LEITCH: Could you say a word about how you separate out the epistemic influences from the aleatory? DR. KIRK: Well, when you -- I'm going to go to a completely different talk, because it's color-coded better. DR. POWERS: I think, in view of our timing, maybe we should leave that for another -- alternate presentation or off-line. DR. KIRK: Okay. DR. POWERS: Okay. Well, thank you very much. I will recess this now for 15 minutes, and then we will come back and discuss our report on the nuclear plant risk studies, and we can dispense with the transcription at this point. [Whereupon, at 3:35 p.m., the meeting was concluded.]
Page Last Reviewed/Updated Tuesday, July 12, 2016
Home
About Us
Popular Documents
Stay Connected
