476th Meeting Advisory Committee on Reactor Safeguards - October 5, 2000

                       UNITED STATES OF AMERICA
                             476th MEETING
                                   Two White Flint North
                                   Room T2-B3
                                   11545 Rockville Pike
                                   Rockville, Maryland
                                   Thursday, October 5, 2000
               The committee met, pursuant to notice, at 8:30
     DANA A. POWERS, Chairman
     GEORGE APOSTOLAKIS, Vice Chairman
     .                         P R O C E E D I N G S
                                                      [8:30 a.m.]
               DR. POWERS:  The meeting will now come to order.
               This is the first day of the 476th meeting of the
     Advisory Committee on Reactor Safeguards.
               During today's meeting, the committee will
     consider the following:
               Discussion of Union of Concerned Scientists
     report, "Nuclear Plant Risk Studies:  Failing The Grade";
               NEI 00-02, "Industry PRA Peer Review Process
               Staff views on the ASME standard for PRA for
     nuclear power plant applications;
               And pressurized thermal shock technical bases
     reevaluation project.
               We will also discuss proposed ACRS reports, and we
     will discuss the topics to be raised in our meeting with the
     Commissioners tomorrow.
               Our meeting today is being conducted in accordance
     with the provisions of the Federal Advisory Committee Act. 
     Dr. John D. Larkins is the designated Federal official for
     the initial portion of the meeting.
               We have received no written comments from members
     of the public regarding today's sessions.
               A transcript of portions of the meeting is being
     kept, and it is requested that the speakers use one of the
     microphones, identify themselves, and speak with sufficient
     clarity and volume so they can be readily heard.
               I want to bring to the members' attention that Jim
     Lyons is now on-board as our Associate Director for
     Technical Support.
               Welcome aboard, Jim.
               MR. LYONS:  Thank you.  I'm glad to be here, happy
     to serve and see what we can do to make everything work as
     well as possible.
               DR. POWERS:  And we'll all do our best to try to
     really confuse him in his first few days.
               DR. SEALE:  We'll see what we can do.
               DR. POWERS:  Now I have some good news and some
     bad news.
               Let me start with the bad news.
               The bad news is Lilly Gaskins is leaving us for
     greener pastures.  She's going off -- Defense Intelligence
               MS. GASKINS:  Yes.
               DR. POWERS:  Lilly, I want to say we've very much
     appreciated having you here, and we're going to be
     disappointed, but our loss is their gain.
               DR. POWERS:  Now for the good news.
               Our own Carol Harris, Miss Carol Harris, has now
     become Ms. Carol Harris.
               DR. POWERS:  Very best wishes from the committee,
     Carol, and we'll harass you like crazy.
               MS. HARRIS:  I count on it.
               DR. POWERS:  Members do have a hand-out called
     "Items of Interest."
               I'll call to your attention that the water reactor
     safety information meeting is coming up, and I also call to
     your attention that the staff has issued their report on the
     Indian Point 2 incident.
               With that, I'll ask, are there any opening
     comments that members would like to begin?
               [No response.]
               DR. POWERS:  Seeing none, I think we can move to
     the first of our sessions, which is a discussion of the
     recent report by the Union of Concerned Scientists
     concerning nuclear plant risk studies.
               Professor Apostolakis, do you want to lead us in
     this session?
               DR. APOSTOLAKIS:  Okay.  Thank you, Mr. Chairman.
               This report was issued last August, and it is very
     critical of the NRC's risk-informed initiative, and this
     committee has been very supportive of that initiative.  So,
     naturally, we were interested in understanding better what
     the arguments of the report were.
               Mr. Lochbaum happened to be in the building last
     time we were here and very graciously agreed to come and
     chat with us in an informal way, but we felt that the more
     formal forum was needed, so we scheduled this hour-and-a-
     half today, and Mr. Lochbaum is here.
               So, how would you like to proceed?  Would you like
     to make some opening statements?  You have transparencies, I
               MR. LOCHBAUM:  I have approximately 10 or 15
     minutes of stuff, and then I'd be glad to entertain any
     questions or comments.
               DR. APOSTOLAKIS:  Why don't we go that way, then?
               MR. LOCHBAUM:  Okay.
               Roughly two months ago, we released a report
     called "Nuclear Plant Risk Studies:  Failing The Grade." 
     Almost immediately, we heard criticism that the report was
     flawed because we relied primarily on data from the
     individual plant examinations that had been done and not
     information from the more recent plant safety assessments
     that each plant owner does.
               The reason we used the IPE results is because
     those are really the only results that are publicly
     available.  So, we didn't have access to anything other than
     that information.
               We also found it curious that the NRC was among
     the ones criticizing us for using the IPE results.  Yet, in
     the site-specific work-sheets for use in the significant
     determination process, it's the IPE data the NRC uses, not
     the PSA data.
               So, you know, on one hand, we're criticized for
     using the data by the NRC that the NRC itself uses.
               So, I guess I didn't realize it was proprietary in
     that nature.
               We also heard that some folks argue that our
     evaluation, specifically the case studies that we did in our
     evaluation, were flawed because our conclusions were that
     the case studies prove that the risk assessments were bad
     because the results are so different for plants of similar
               In fact, I met, within the last week or so, with
     one critic of our report, who said that the plants are
     different, even plants at the same site are different, and
     those differences drive the risk assessments results to be
     different, so it's more surprising when the results are the
     same than when they're different.
               So, we heard that criticism, and we wanted to look
     into it a little bit.
               So, we went to the NRC's IPE database that's on
     the internet, and these following three sheets are from that
               I just printed out the overall core damage
     frequency from the IPE database and had sorted it from high
     to low, and I notice, with the exception of a few plants,
     Hatch 1 and 2, St. Lucie 1 and 2, Indian Point 2 and 3,
     Salem 1 and 2, and Beaver Valley 1 and 2, all multiple-unit
     sites that have similar reactor designs have the exact same
     core damage frequency reported for all units.
               So, if, indeed, the plants are different enough to
     drive the risk results -- assessment results different, then
     that's not done on a consistent basis.  Sometimes it is,
     sometimes it isn't, and apparently, the NRC is happy whether
     the results are the same or whether the results are
     different, and like I say, that IPE database has all the
     plant results, even for plants that are now closed.
               DR. APOSTOLAKIS:  So, are these so-called sister
     plants, or it's just a listing of the plants?
               MR. LOCHBAUM:  This is a listing of the IPE
     results that all the plants submitted.  Some of them are
     sister units and some of them are not.
               For example, Millstone -- obviously, they're at
     the same site, but they're different reactors.  I wouldn't
     expect those numbers to be the same, and they're not.
               DR. APOSTOLAKIS:  But your argument in the report
     is that sister plants report different results.
               DR. UHRIG:  Well, St. Lucie -- they have different
     cores.  One has 14-by-14 and one has 16-by-16, and there's a
     lot of other design features that are different.
               MR. LOCHBAUM:  The same thing is true with Brown's
     Ferry.  Even plants at the same site operate differently,
     whether it's the core design or the cooling water system
     design or whatever, because I don't think there's any --
     it's like snowflakes.  No two nuclear power plants are
     identical, even at the same site, yet the results are the
     same and are not the same.
               I understand the criticism.  I don't know -- based
     on the data, I can't confirm or refute it, because we have
     some in both categories.
               DR. APOSTOLAKIS:  Would you like us to raise
     questions as you go along or wait until you're done?
               MR. LOCHBAUM:  Either way, whichever is easiest
     for you.
               If it makes more sense to ask me a question about
     it as we're going --
               DR. APOSTOLAKIS:  Okay.  Let's pick up this issue.
               The issue -- I guess you are making the point that
     the PRA is such a weak technology that, when applied -- or
     immature, perhaps -- that when applied by two different
     groups to two plants that are very similar, it produces
     different results.
               MR. LOCHBAUM:  That's correct.
               DR. APOSTOLAKIS:  This is an issue, as you
     probably know, that the staff identified in its report on
     lessons learned and insights from the IPE reviews.
               So, it is really nothing new there except that you
     are taking a little more extreme position, but it's really
     the methodology that is the culprit here and not so much the
     design differences.
               The staff's conclusion was that, from the
     evidence, they could not conclude whether it was really the
     methodology or the design features, although they do say
     somewhere there, as I recall, that the differences were
     primarily driven by design and operation differences.
               So, I mean there are differences, and it's not
     just -- I mean it also depends very much on what you call
     sister plants, and my colleague, Dr. Bonaca, has more
     experience in that, and maybe you can say a few words.
               MR. BONACA:  Oftentimes, we talk about sister
     plants on different sites, and when you look at them,
     really, oftentimes they are different on the secondary site,
     because the AE was different, because they were implemented
     in different ways.
               So, the sister plant connotation is one that
     relates more to the NSS island than to the balance of plant,
     and yet, the balance of plant is critical in determining
     some of the results in the PRA.  For example, the layout of
     the auxiliary feedwater system is a fundamental element
     looking at the results.
               Now, on the same site, you have some plants that
     probably are sister plants, and maybe the case was made by
     the applicant, like Calvert Cliffs, that they're identical,
     therefore we submit only one IPE.  I will expect that
     probably was the approach taken.
               On some sites, there may significant differences
     and you have different values.
               But one point I wanted to make is that that's why
     we don't like to see a bottom line number for a PRA.  We're
     looking at uncertainty.
               If you take two teams doing a PRA for the same
     plant, you will get different results, no question about it. 
     If you get two different vendors doing a LOCA analysis for
     the same plant, the same fuel, we get different results. 
     Nobody expects to get the same value.
               What you expect to see, in fact, is a reflect of
     the uncertainty in the whole evaluation reflected in the two
     assessments, and I don't think it's surprising.
               I think it actually would be a healthy insight to
     have two different estimates of the same matrix for the
     plant, so you could understand what the subtleties are and
     what the effects are.
               MR. LOCHBAUM:  I don't disagree with that.  In
     fact, you know, the plants are different, and therefore, the
     results should be -- if the result -- different results are
     due to plant differences, that's one thing.  If the
     different results are due to different methodologies and
     both methodologies are the same, six of one, half-a-dozen of
     the other, then that's fine, too.
               What we're concerned about is the lack of controls
     over the methodology and the assumptions in inputs that
     would allow a plant owner to -- if they're contemplating a
     modification to the plant or a procedure change -- for
     example, putting kerosene in the fire water headers -- to go
     back and adjust the inputs to compensate for the actual
     increase in risk from the proposed change, make some
     methodology hand-waving to make it look -- the net effect to
     be the same or even an improvement in safety.
               We're concerned that the controls over the
     methodology and the assumptions wouldn't prevent those kind
     of abuses, whether intentional or mistaken.
               DR. APOSTOLAKIS:  I believe you are absolutely
     right on that, and this committee has been concerned about
     it, and the staff has made it clear that they are concerned
     about it.  So, I think there is no question that, in some
     IPEs, people, whenever they found an opportunity, they were
     more optimistic than the general state of knowledge would
     justify, but it seems to me that the real question -- and I
     think that's an issue that one can raise in several places
     in your report -- is not so much whether there are studies
     that are not really very good out there.
               The question should be, at least in my view, has
     the NRC used any of these studies in its decision-making
     process and was led to incorrect conclusions because the NRC
     was unable to identify the sources of the differences,
     perhaps?  I mean it all comes down to decision-making.
               I think we should make it very clear that, you
     know, the industry -- it's a private industry that can do
     studies for themselves, and if they want to use optimistic
     numbers, that's up to them, but when it comes to the agency,
     does the agency accept these analyses, and do you have any
     examples where, indeed, the agency used what, in your view,
     were inadequate PRAs to make a decision, because that's
     really the heart of the matter, the decision-making process.
               MR. LOCHBAUM:  I guess we didn't, because the
     agency is clearly moving towards risk-informed regulation,
     and the risk assessment results are going to be an input or
     a factor in that regulatory decision-making.
               We wanted to try to prevent some bumps in that
     road.  We're not saying that the road is wrong.  There are
     some problems along the way that we wanted to try to address
     in our report and get fixed before we proceeded too far down
     that road.
               So, the answer to the question is, no, we don't
     have any examples, but also, we didn't look, because we were
     trying to prevent mistakes in the future, rather than flag
     errors of the past.
               DR. APOSTOLAKIS:  This answer, I must say, is
     really surprising to me, because by reading the report, I
     didn't get the impression that you were trying to prevent
     mistakes from happening.
               I mean the report gives the impression that things
     are very bad.
               So, I must say your statement is very welcome, to
     me, at least, because the committee, of course, does not
     have a position at this point.
               So, if that was your intent, I think you're
     succeeding, but it doesn't come through by reading the
     report that you are really trying to prevent mistakes from
     happening.  I mean "Failing The Grade" is a pretty strong
     statement to put in the title.
               MR. LOCHBAUM:  I think we have some data to show
     why we think the risk assessments are bad.  We didn't look
     for any examples where those results have been used yet, but
     the agency is moving in that direction, and that's what
     troubled us.
               DR. APOSTOLAKIS:  But you do agree that this is
     really what the issue is, I mean the decision-making
     process.  I mean, you know, a private company can do
     whatever they like, if they want to kid themselves that the
     risk is 10 to the minus 20, but when they try to use it
     here, then it's a different story.
               MR. LOCHBAUM:  Well, I think the related question
     -- and I agree that that is the question, but the related
     question that Commissioner McGaffigan poses is, would this
     information or this approach lead the NRC to a different
     answer than it would using the current prescriptive
     approach, you know, because errors can be made on either
               DR. APOSTOLAKIS:  And that brings up another major
     question that I have.  I don't know whether I should raise
     it now or later.
               MR. LOCHBAUM:  One of the other criticisms we had
     was our concern about the risk assessments not accounting
     for design basis problems, and the bulk of that criticism
     was that, yes, design basis problems have been identified,
     but they haven't been risk-significant; they've been much
     ado about nothing, if I can characterize the criticism, if I
     understand it correctly, and to investigate that criticism,
     we went to a recent -- a May 2000 draft report that the
     Office of Research prepared on the design basis problems,
     and this is Figure 22 from that report that looks from 1990
     to 1997.
               The number -- the percentage of LERs with design
     basis issues that have been classified as accident sequence
     precursor events -- and while the trend is, indeed,
     downward, the important part that we think is that none of
     the years is non-zero.
               So, not all of design basis problems that have
     been reported have been able to be dismissed as non-safety-
               DR. POWERS:  I guess I'm a little bit confused. 
     Would you expect it to go to zero?  I mean I can hope it
     goes to zero, but I wouldn't expect it to.
               MR. LOCHBAUM:  That's true.  I wouldn't expect it
     to, but since the risk assessments basically assume that
     it's zero, then there's a disconnect between reality and the
     risk assessment assumptions, and that disconnect is what --
               DR. POWERS:  I don't think -- I mean it doesn't
     seem to me that the risk assessments assume this is zero. 
     They assume these things actually occur.  Most of the
     accident events result in nothing happening.
               In a typical modern PRA, there are, what -- I
     think, for 1150, the average one had three million accident
     sequences, of which nearly all of them ended just like these
     precursors ended, no consequences.
               I mean I can go through the 1150 and actually give
     a prediction on how often those things that are recorded
     there should occur.
               MR. LOCHBAUM:  I guess the point we look at is,
     for example, the stand-by liquid control system we talked
     about in our report -- the Big Rock Point plant operated for
     13 -- the last 13 years of its 39-year life with that system
     not working quite right.
               The risk assessments don't reflect --
               DR. POWERS:  Nor does the design basis analysis
     reflect -- nor does anything -- if you don't know that
     something is not right, there is no amount of analysis you
     could ever do in your entire life, by the most intelligent
     people, impossible, that will discover that if it's not
               I mean it's a non-criticism.
               MR. LOCHBAUM:  But once something becomes a
     reported event, it doesn't seem that design basis events are
     factored back into the process, like human errors.  There's
     a human error database.  There's an equipment failure
     database.  There doesn't seem to be a design deficiency
     database or -- a widget just doesn't work because it's
     designed improperly.
               If the widget doesn't work because somebody mis-
     operates it, that seems to be captured.  You can argue
     whether it's right or wrong, but at least it's captured.  If
     the widget doesn't work because it fails on start or fails
     on demand, then that seems to be in there.  But if the
     widget is designed improperly, that doesn't seem to be in
     the risk assessments, and you know, any one of those
     failures in any one of those columns can cause something
     from working properly.
               DR. SEALE:  I'm curious as to what's driving that
     curve down, then.
               MR. LOCHBAUM:  Well, I would hope one of the
     factors would be, as you find things and fix them, you have
     a work-off curve.
               MR. BONACA:  I would like to make a comment about
     this point.
               This is the trend, and that's the trend, but we
     have to recognize that we didn't look -- I mean one thing we
     found is, the more we look, the more we find, and we looked
     the most between 1995 and 1997.
               To me, it's comforting that that number of
     precursors is so low in the very period in which we looked
     so much, and there was a limit to how much we found.
               The other point I would like to make is that,
     again, for those precursors there, you know, there wasn't a
     terminicity evaluation or system failures that did not
     represent the range of operation in which the system should
               There were some conditions that, in the
     deterministic space, says the system is not functional or is
     not operable.
               So, anyway, that's a different issue, but the
     point I would like to stress here is it's -- this trend --
     it's encouraging in my mind, because we looked so much in
     '95-'97 timeframe, and we found, you know, we didn't upset
     that curve.
               MR. LOCHBAUM:  I guess you could -- statistics can
     be looked at any number of ways.  You could look at -- with
     all the things that reported in the '95 to '97 timeframe,
     the percentage would go down, because this is not absolute
     number, this is percentage, and there were so many less
     significant items found, as all those problems were flushed
     out, that the percentage would have gone down even if the
     absolute numbers stayed the same.
               MR. BONACA:  There was also a finding that, again,
     the more you look, the more you find, and there was a lot of
     looking, and so, many of these issues were to do with
     original design.
               MR. LOCHBAUM:  Seventy percent of them, according
     to this report, by the way.
               MR. BONACA:  That's right, and it seems to be that
     as you -- the plant ages and these efforts are undertaken
     and so on and so forth, and the SSFIs took place in the
     early '90s and so on and so forth, it's an encouraging
               I think we are getting to the point where,
     probably, most of this design -- original design defects are
     not there anymore.  There will be always some, and we cannot
     eliminate those.
               MR. LOCHBAUM:  That was the end of addressing the
     criticisms, so far, at least in the presentation.
               I'd like to turn now to some of the information we
     gathered as we researched the report that didn't -- wasn't
     in the final report, but we think that this information
     supports the conclusions that we drew in the report.
               This is an NRC study -- I forget the exact date --
     it's on the isolation condenser system reliability at
     boiling water reactors.
               This is a figure showing the unreliability for the
     systems from actual plant events compared to what the IPE
     results were for -- that were used for these various plants,
     and you can see, for every case, the IPE result was to the
     left of the actual plant data or actual operating
     experience, although the error bands and the uncertainty
     bands did cover all the data.
               DR. APOSTOLAKIS:  So, let's look at Dresden 2. 
     Can you show us -- I see the PRA on reliability.  Dresden 2
     is the first one.
               Where is the operating experience?
               MR. LOCHBAUM:  The last two lines are taking
     operational experience with and without credit for recovery.
               MR. BONACA:  But that's an industry mean.  I can
     tell you it was based on data, actual data.
               MR. LOCHBAUM:  Right.
               MR. BONACA:  So, I'm saying that that's a mean
     down there for the industry and doesn't represent,
     necessarily, individual --
               DR. APOSTOLAKIS:  So, you don't have the operating
     experience number for Dresden 2 in the figure.
               MR. LOCHBAUM:  Right.
               DR. APOSTOLAKIS:  I see.
               MR. LOCHBAUM:  The NRC report said not all the
     plants had enough operational data in order to do individual
     plant comparisons.
               I do have some charts that do have information in
     that regard.
               DR. POWERS:  I guess I still don't understand the
               It seems to me this is a ringing endorsement of
     what they've done.
               The data are plotted, or the number used in the
     analysis is plotted.
               In some cases, they used point values, and I have
     ugly words to say about point values, but they get used, and
     I grow more tolerant with age, I suppose, and then you show
     this industry mean with a range, which is good.
               What's wrong?
               MR. LOCHBAUM:  I didn't mean to trap anybody, but
     the next few figures, I think, will show what the problems
               This is the same approach applied for the high-
     pressure coolant injection system on boiling water reactors.
               The black closed circles with the bands are the
     operational data for each plant, with the uncertainty bands. 
     The white circles are the IPE data, without uncertainty
     bands, and you'll notice, in this case, every single one of
     the IPE results is to the left of the actual performance. 
     Most -- or some of the IPE data is not even inside the
     uncertainty bands for the actual operating experience, and
     with this unreliability curve on the axis, being to the left
     with the IPE result means that the IPEs assumed more
     reliability than what operational experience showed.
               The other thing we thought was -- we noticed as
     these curves came out was the IPEs were submitted in like
     the  '91-'92 timeframe.
               So, they would have used -- if they used anything,
     they would have used operational data from the '80s, early
     '90s, which is a little bit earlier than the data used --
     the operational data plotted here, and everybody keeps
     conceding that operating performance is getting better and
               So, you would have expected the IPE results,
     perhaps, to be closer to today's operational experience or
     perhaps even to the right of today's operational experience,
     and that wasn't the case.
               DR. APOSTOLAKIS:  I'm getting two messages from
               First of all, I really think that PRAs and the
     standards that are being developed for PRAs now should
     insist on using plant-specific data.
               This is an excellent argument here, and we'll have
     an opportunity to discuss this during this meeting with --
     when we discuss the standards, because this clearly shows
     that you have to do that.
               I mean you really have to use plant-specific data. 
     I don't know what the basis of the other curves -- the other
     estimates was, but clearly, it was not plant-specific data.
               So, that's an excellent argument for that.
               And second, again, I will come back to the point I
     made earlier.  I mean it is the NRC staff that supplies this
     figure.  It is the NRC staff that makes decisions using
     PRAs, using the integrated decision-making process.
               So, I would expect the staff to raise this issue
     if a particular licensee comes, say, like Dresden, with a
     number that is way down to the left and say, no, this is not
     acceptable, I mean you have to do something or your PRA is
     not very good.
               So, that is the right context, in my mind, of this
     figure, which I think is very enlightening.
               MR. LOCHBAUM:  I think what concerns us about this
     data -- these reports are put out by the NRC's Office of
     Research, and it seems like they go up on a shelf without
     the rest of the NRC staff relying on those, because when we
     talk about these numbers to the regions, about why Nine Mile
     Point seems to have more trouble with RCSI, they've never
     heard of this stuff.
               DR. POWERS:  You're preaching to the choir.
               DR. APOSTOLAKIS:  You are telling the truth,
     David, and this committee has expressed disappointment many
     times that a lot of the results from the former AEOD do not
     seem to be used.  I think that's an excellent point.
               MR. LOCHBAUM:  I got one.  Okay.  I'm on a roll
               I guess I don't want to -- since I'm on a roll,
     this is some more data.  This is the reactor core isolation
     cooling system for boiling water reactors.
               I think this is even more striking than the last
     chart in that seven of the 30 results or plants that are
     reporting, the uncertainty bands for the IPE data don't even
     overlap at all with the uncertainty bands for the actual
     operating experience data.
               In none of the cases do the upper end uncertainty
     bands for the IPE data match the mean for the operational
     experience data.
               So, again, there seems to be a very strong bias
     towards using IPE results that are more reliable than actual
     operating experience would show.
               Now, those are boiling water reactor examples. 
     It's totally different on the pressurized water reactor
     side, totally different.
               This is the same -- I blew it up a little bit. 
     That's why it doesn't -- it's right out of the figure, but
     this is the auxiliary feedwater system reliability study
     that the NRC issued, and you can see that it's totally
     different, because the NRC put the unreliability axis on the
     vertical axis.
               So, instead of being to the left or right, it's
     above or below.
               There is one case for design class 4 where the
     operational data -- the IPE data is actually below the
     operational experience data.  In all of the other cases,
     it's the same as the boiling water reactor data.  The IPE
     data is shown to be more reliable than the operational
     experience data.
               And I agree with you, the bias can be handled as
     long as the staff recognize that and handles it right.  What
     we're concerned about was the controls or the standards or
     how the NRC makes that -- how it factors that into its
     regulatory decision-making process.
               Once we saw the system bias or what we perceive to
     be a system bias, we wanted to try to figure out why that
     was happening, and we haven't conclusively determined why
     that's happening, but we think -- we eliminated one suspect,
     and that was that, you know, these biases being introduced
     at the component level and then just being rolled up into
     the overall system reliability number.
               So, this is a chart -- and there's about six or
     seven different charts in this NRC report.  This is
     component level for the turbine-driven feed pumps --
     turbine-driven pumps, not necessarily feed pumps.  It's the
     probability of failure on demand, and you can see there, you
     have the operational experience, and then you have what the
     individual plants reported, and it's some above, some below,
     which is what you would expect if best estimates were used
     or realistic data was used.
               So, it didn't seem that the bias was being
     introduced at the component level.  It was being introduced
     somewhere else.
               In eliminating one candidate, we didn't identify
     what it was, but what we think it is, but we can't prove, we
     think it's somehow related to this, and this figure appears
     in every one of these NRC research reports, perhaps a little
     bit easier to read.
               It's got three round things.  Region A is all
     inoperabilities that are reported by license event reports. 
     Region B is the ones that have been -- of those
     inoperabilities that have been classified as failures, and
     region C, the smaller of the round things, is the ones that
     have been -- failures that have been considered countable,
     and I think, you know, if the industry has a different
     definition for region C than the NRC does, then that could
     explain why the bias at its system level, because a
     component-level failure could be perceived as not causing
     the system to not do its function.
               If the NRC uses a different definition than the
     industry, then that could explain the two curves or the two
     -- the results being so disparate, and again, going back to
     your comment, that doesn't necessarily mean it's wrong. 
     It's a bias that has to be recognized and factored into a
     decision-making process.
               What our concern is, is while the research is
     cranking out report after report identifying and classifying
     and labeling this bias or these differences, the rest of the
     NRC staff isn't accounting for that in its decision-making
     process, and that's why, in our report, we didn't say this
     was the wrong road for the staff or the industry to be on;
     we thought that those standards, those controls needed to be
     done before anymore progress was made down this path.  I
     mean that seemed to be an important part that was not in the
     right order in our mind.
               MR. BONACA:  Probably I'm not the person to answer
     that question, but the question I would have is, has the NRC
     asked the licensees why there are these biases?  I mean,
     here, in the test, it implies there have been calculations
     done using raw data and using actual operating experience
     and using data from the IPEs, and I'm not sure there has
     been a communication back and forth to understand where this
     bias trend comes from.
               DR. POWERS:  I guess maybe this is the point to
     interject an issue that comes up here, and I'm not being
     critical of your use of the IPE, because I think I'm very
     sympathetic, in fact, using the IPE results.  That's all you
               The situation is, the NRC asked the licensees to
     conduct the IPE studies.  They didn't ask them to use PRA. 
     In many cases, the analyses that were done to support the
     IPE submissions were the first attempts to do a PRA of that
     plant, and the intentions of the NRC was not to have the
     licensee do a very rigorous, careful PRA analysis.  It was
     to familiarize themselves with the vulnerabilities of their
     plant and to gain some experience in the area, and so, they
     didn't review them in exhaustive detail.
               The question that comes up, I think that gets
     raised by all of this, is if we use a risk assessment, which
     now is rigorous and carefully reviewed as part of a
     licensing action, and the public has an interest in that
     particular licensing action, how do they routinely get
     access to these PRA analyses?
               Well, clearly, one way is to put them on the
     docket, but if you put them on the docket, you ossify them,
     and it becomes -- and that almost defeats the purpose of
     having them, and so, the question I put to you is, have you
     thought of any way to make these PRA results accessible to
     the interested parties without ossifying them?
               MR. LOCHBAUM:  We've thought about it a little
     bit, because we are on the outside trying to get at
     information, and the IPEs themselves are very large, but
     there are summary sections and tables that basically provide
     the bottom line to the results fairly quickly, and I think
     you could have that on the docket, the summary and the
     results and the basic description of the approach taken and
     a brief description of the methodology, without having all
     the component failure data, tables, and all that stuff on
     the docket.
               DR. POWERS:  The problem I see with that is that
     it doesn't take a genius to figure out how to make bottom
     lines look very, very supportive of one position or another,
     and in fact, in the regulatory actions, you very seldom use
     those bottom lines, and the regulatory actions tend to be
     insensitive to them.
               In 1.174, you use that bottom line number to
     position yourself on a horizontal axis, which is plotted in
     a log scale, so being a factor of 2 or 3 one way or another
     hardly makes a difference at all.
               It is the differentials of those results that get
     used in deciding whether they're licensing actions, and
     those differentials have to -- you now have to plunge in the
               Almost, I'm willing to give people the bottom line
     results that they come back with.  I can almost guess the
     number that they'll come back with.
               What I'm looking for is vulnerabilities in the
     system, and those vulnerabilities are discovered only by
     plunging into the details.
               MR. LOCHBAUM:  I guess the other answer to that
     question would be, if the NRC had some standards or controls
     over the methodology that plant owners did, inputs, so on,
     and had some -- so that the NRC's methodology was publicly
     available and the NRC's verification that plant owners met
     or exceeded those standards, then I'm not sure that the
     public needs -- or even I -- I wouldn't want full access to
     everybody's PSA.
               I want, and I think the rest of the public wants,
     some assurance that the NRC is making sure that people
     making decisions on risk analysis -- that those risk
     analysis is a solid foundation for making those decisions,
     and I think there's -- either make the risk assessments and
     all the attendant details available or make the NRC's role
     in verifying that that's good, make that publicly available,
     and that might be the easier of the two or the more proper
     of the two paths.
               DR. APOSTOLAKIS:  One of the fundamental points, I
     think, that needs to be made here is that the decision-
     making process that utilizes PRA is, as you know, an
     integrated process.
               As Regulatory Guide 1.174 describes it, there are
     five inputs to the process, and four of them are really
     traditional deterministic considerations, like preservation
     of the defense-in-depth philosophy, safety margins, and so
               So, in that context, it seems to me that your
     criticism in the report acquires a different flavor.  It's
     not in the report, though.  In the report, you just mention
     in passing the risk-informed initiative on page 22, giving
     it one paragraph.
               You give the impression that decisions rely -- you
     use the verb "rely" too many times -- rely on PRAs, and
     surely you're aware of the fact that decisions do not rely
     on PRAs.  I mean it's just one input, and we have figures
     with shades of gray, we have long discussions in Regulatory
     Guide 1.174 about model uncertainty, about sensitivity
     studies that sometimes drive the industry crazy.
               So, do you think that perhaps you did not present
     the actual decision-making process as accurately as you
     could have?
               MR. LOCHBAUM:  I think I understated it.
               DR. APOSTOLAKIS:  You understated what?
               MR. LOCHBAUM:  The reliance on the risk
               DR. APOSTOLAKIS:  You think you are relying too
     much on risk assessments?
               MR. LOCHBAUM:  In certain cases.
               I think the recent example is Indian Point 2, the
     Inspector General's report on how the staff handled that
     steam generator inspection, where the NRC staff thought that
     the whole thing was of low safety significance and just
     basically put it up on the shelf.
               That wasn't made by any of these five factors and
     weighing all this information.  This was based on one
     person's shooting from the hip, deciding that something
     didn't warrant further NRC resources.
               So, I think there are two many cases like -- even
     one case like that's too many, and I think that's not the
     isolated case.
               DR. APOSTOLAKIS:  So, we should be vigilant, then,
     to make sure that what we call risk-informed approach is
     actually risk-informed and not risk-based.  You're saying
     there are examples where the decision was risk-based, and
     that was inappropriate.
               MR. LOCHBAUM:  Well, it was considered -- that
     decision, technically, was considered risk-informed, because
     all factors were done, but one of them was given 99 percent
     of the weight, and the other four added up to maybe like 1
               So, that was, technically, risk-informed, but it
     really -- it was an abuse --
               DR. APOSTOLAKIS:  -- a mis-application of the
               MR. LOCHBAUM:  Right.
               DR. APOSTOLAKIS:  But a broader issue, though, is
     the following, in my mind.
               One -- and I've said that in many other contexts,
     not just in the context of your report, because many times
     we've had staff come here and give us a million reasons why
     a risk-informed approach to a particular issue is not
               People take the PRA and they scrutinize it to
     extreme detail in the absolute.
               They say, well, gee, you know, there are all these
     problems, therefore you can't use it.  It seems to me that's
     the wrong way to approach it.
               The way to approach it is to ask yourself, I have
     now a deterministic system in place.  Am I better off if I
     supplement what I'm doing there by using risk information or
               That's really the proper context to evaluate the
     PRA, and you can apply that to specific issues.
               For example, you raised the issue of design issues
     that have not been found and so on, and well, if I decide
     not to use PRA, does the existing system do a better job
     identifying those design issues, and if I use a PRA, do I
     have a better chance of finding them and evaluating them in
     the proper context, and I think you can practically pick any
     of the issues you raise, and if you place it in that context
     -- now, you probably reach a different conclusion from mine,
     but I believe that the system is better if it blends both.
               I grant you that there may be exceptions where
     people mis-use the process.  You know, it's a big agency
     handling many, many situations.  I mean we're all human. 
     But I think that's the proper context, and just to say, gee,
     PRA has all these deficiencies, therefore it should not be
     used, is really an evaluation in vacuum.
               PRA is not proposed as a methodology that will
     replace the existing system.  It will add information to it. 
     I was wondering what your thoughts were on that.
               MR. LOCHBAUM:  I agree with that.
               Again, if we thought PRA was the wrong tool to be
     using, or if it was going to replace the deterministic, the
     recommendations would be to stop and fix the standards.  It
     would be just stay where you are and stop wasting all those
     resources.  But that wasn't the conclusion we reached in the
     report.  We said fix the problem.
               I think where we see the problem is that, in the
     example being risk-informed regulation or risk-informed
     inspection, clearly the industry leaders in that approach
     have looked at plant-specific inspection results, identified
     areas where inspections are not finding problems, and have
     prudently chosen to redirect those resources into areas
     where the problems are being found, and to me, that's a
     perfect example of what you explained about deterministic
     and now factoring in risk information to be smarter, do
     things better, and we attended some of those meetings and
     thought that was fine.
               Our concern is, without the standards that the NRC
     applies, there are other plant owners who didn't spend all
     the time to really understand the subtleties of the issues
     that the leaders have done, just are going to get on that
     highway and go down the same road and might make the wrong
     decision, and the NRC, by not having established standards,
     doesn't have the proper tools or infrastructure to prevent
     those subsequent abuses.
               You know, the first -- South Texas and commercial-
     grade classification or whatever -- those guys spent an
     awful lot of time and an awful lot of effort to make sure
     they fully understand it.
               So, it's not the leaders, it's the ones that then
     jump on the highway down the road, and we're concerned that
     NRC's not policing against those.
               DR. APOSTOLAKIS:  Have you had the chance, by the
     way, since you mentioned the standards, to look at the ASME
     standard, and would you care to make a comment or two about
               MR. LOCHBAUM:  I haven't looked at the ASME
     standard.  I have looked at NEI's -- what is it -- 0002?
               DR. APOSTOLAKIS:  The industry certification
               MR. LOCHBAUM:  The peer review process.
               We talked earlier about the need for using plant-
     specific data.
               NEI's peer review process does include that, if --
     depending on what grade you're trying to get your risk
     assessment classified as.
               I forget whether 1 or 4 is good or bad, but if
     you're just trying to use it for run-of-the-mill --
               DR. APOSTOLAKIS:  Four is good.
               MR. LOCHBAUM:  Okay.
               If you just want a grade 1, you don't have to use
     plant-specific data.  So, there is recognition for things
     like that.
               The one criticism we have at this point -- it's
     preliminary review, because I only got the thing Monday --
               DR. APOSTOLAKIS:  Sure.
               MR. LOCHBAUM:  -- so I haven't -- is that those
     checklists, the things that have to be looked at, while
     they're very thorough, they seem more like inventory checks
     than they are quality checks.
               You know, the example we could cite would be --
     you could ask the captain of the Titanic if you have life-
     boats, and that answer would be yes.  If the question was,
     do you have enough life-boats for all passengers and crew,
     that's a different answer.
               So, they seem to me more questions of the first
     category than the second.
               DR. APOSTOLAKIS:  There was an example of that in
     Greece last week.
               DR. SEALE:  Two examples.
               DR. APOSTOLAKIS:  What do you think of this idea
     of grades?
               MR. LOCHBAUM:  Well, I think the whole concept of
     having -- I don't think very plant owner needs to have the
     same standard risk assessment, it depends on the
     application, and that, again, goes back to your point about
     -- you know, deterministic base is one thing.  If you opt
     for more and more risk insights, then you should have a
     stronger foundation for supporting those moves.
               So, I don't think you have to have -- it's a one-
     size-fits-all approach, and it makes sense that there should
     be varying degrees.
               Whether it's grades or -- you know, the actual
     mechanism for classifying that is -- I don't have a strong
     comment on, but I think it is good to have those tiers and
     to be used in that way.
               DR. APOSTOLAKIS:  As long as the tiers can be
     defined clearly, so everybody agrees, right?
               MR. LOCHBAUM:  The concern we have also goes back
     to the old SALP process, where they -- there were tiers in
     that approach, too, but they seemed to be somewhat
     subjective.  If the NRC thought you were a good plant owner,
     you tended to get a 1, and if they thought -- if you were in
     the regulatory dog-house, you got a 3.
               So, we're trying not to get the peer review or any
     risk assessment grade also be a reflection of how much the
     NRC likes you.
               That fondness should not be a factor in the
     overall result, whether it's a grade or anything else.
               DR. APOSTOLAKIS:  You make a statement in your
     report which I find strange, and I'm not sure that -- and
     I'm curious to see how you would react to that.
               On page 21, you say, "But it is not possible to
     properly manage risk when only reasonable instead of all
     possible measures are taken to prevent and mitigate events
     unless the probabilities and consequences are accurately
               Are you seriously proposing that all possible
     measures be taken?  I mean what does that mean?  You know,
     as you know, this is a very ill-defined concept, all
     possible.  I mean we don't do that with automobiles.  We
     don't do it with airplanes.
               Is this an exaggeration to make a point, or should
     it be taken literally?
               MR. LOCHBAUM:  It can be taken either way.  It is
     true, if you take all possible measures, then it doesn't
     really matter to know what the probability or consequences
     of any event are, because you're doing everything that's
     possible, and those risk insights wouldn't change what
     you're doing, because you're already doing everything you
               I wasn't advocating doing everything that was
               DR. APOSTOLAKIS:  I see.
               MR. LOCHBAUM:  The point I was trying to make --
     I've had several comments on this paragraph.  The comments I
     had were from people who didn't like risk assessments at
     all, and they were conceding that things could be done. 
     They thought I understated the point.
               But what I was trying to say is, if risk insights
     are then being used to draw a line between what you do and
     what you don't do, then you need to understand the
     consequences and the probabilities well enough to draw the
     line and decide not to do things that are on the wrong -- on
     one side of the line, and that was the point I was trying to
     make, and it clearly didn't come across, because I've had
     several comments on that.
               DR. APOSTOLAKIS:  I have one last question. 
     Looking back the last 25, 30 years, do you think that the
     use of risk information has made plants safer or not?
               MR. LOCHBAUM:  I think it has.  I think the IPE
     program itself identified vulnerabilities and led to several
     real changes, not paperwork changes, actual plant
     modifications or procedure changes to improve safety.
               I think the extension of that effort was the
     maintenance rule.  A lot of that research or activity led
     into the maintenance rule.
               I think the maintenance rule -- the emphasis on
     both the safety system reliability and also what were
     traditionally considered BOP systems and the increased
     attention on those has led to overall safety improvements.
               So, I'm not saying that risk insights have been a
     net loss.  There clearly have been some gains, and important
               DR. APOSTOLAKIS:  Do any of my colleagues have any
     comments or questions?
               DR. POWERS:  I'm still coming back to the access
     to information.
               One of the studies -- PRA studies that was fairly
     extensive that is publicly available was the reactor risk
     reference study.
               MR. LOCHBAUM:  Is that in NUREG-1150?
               DR. POWERS:  I think that's 1150.
               MR. LOCHBAUM:  Yes.
               DR. POWERS:  Did you consult that at all in
     preparing this document?
               MR. LOCHBAUM:  In earlier discussions over the
     last year, as this report was being researched, we
     referenced 1150 quite a bit, and NEI was the obligatory
     counterpoint on each of those arguments, and it was that
     1150's out of date, and you really need to understand the
     inputs that drove the numbers, you can't just rely on the
     end points, which is what I had been doing when I was citing
     1150.  So, we decided to go back and look at the individual
     IPEs to try to respond to that criticism.
               So, that's why we didn't use 1150 in this report,
     although we were aware of it and had used it previously.
               DR. POWERS:  Well, the upshot of that is that your
     report now harkens back more to the old WASH-1400 and Army
     studies and things like that, which are really geriatric.  I
     mean that's when the technology was really in the
     developmental stage, I would say.  So, you end up abandoning
     one study, because it's out of date, in favor of some that
     are really old.
               DR. SEALE:  A whole array of studies --
               DR. POWERS:  -- that are really old.
               MR. LOCHBAUM:  That's true.
               DR. SEALE:  And again, the quality control, if I
     may use a phrase, the legal authority for the IPEs was not
     what we would expect for any PRA that we would use today. 
     It was vulnerability identification, and we all know that
     you can do a risk assessment which can be very coarse in
     certain areas and very fine in other areas to identify
     particular vulnerabilities.
               DR. POWERS:  I think that's especially true when
     you're trying to look for vulnerabilities that you don't
     know about.
               DR. SEALE:  That's right.
               DR. POWERS:  And you say, okay, well, gee, I know
     that I'm vulnerable in this particular area, in my piping
     system, so I'll just put in some numbers there, because I
     already concede that point, I'm looking for something else,
     and so, you do strange things on your inputs there.
               DR. KRESS:  If I boil this down to a few words, it
     seems to me like your problem is that NRC doesn't seem to
     have good quality standards that it enforces in PRAs and
     that PRAs  are shortcoming in that they don't deal with
     latent design defects.
               If we had those two things fixed and the movement
     towards risk-informed regulation, I think you might be in
     favor of.
               MR. LOCHBAUM:  That's basically our conclusion. 
     You know, those standards need to be there and enforced, and
     then the move to risk-informed regulation would be -- could
     be a good thing.
               Not every plant owner is going to do that, but
     those objections would be removed.
               DR. SEALE:  Yeah, but then again, you run into the
     same problem again.
               Suppose you want to look at the difference in two
     alternatives, and those two alternatives could be very
     specifically defined.
               Now, I can do that within the context of a very
     detailed, highly formalized, high QA, overall PRA, perhaps
     even levels 1, 2, and 3, and the whole school yard, or I can
     do a modified PRA process which treats those particular
     detail differences in considerable detail, and the rest of
     it rather coarsely, and come up with a number that may not
     be valid in terms of actual risk for the overall plant but
     will tell me that the difference in risk between this
     alternative and that alternative is a factor of whatever.
               Now, under certain circumstances, if that's to
     determine how I'm going to conduct a maintenance process and
     what equipment I will have in service and what equipment I
     won't have in service, then clearly, the focused assessment
     tells me all the answer I need to know to make that
               MR. LOCHBAUM:  Well, conceivably, both of them
               DR. SEALE:  Yeah, but one of them is so long to do
     that I won't have the answer when I need to do the
     appropriate maintenance, or it's so expensive that I don't
     have the people on the floor to do the work, I have them in
     the PRA group doing the assessment.
               MR. LOCHBAUM:  I agree with that, because again
     comparing it to what we're doing today, in 50.59, you have
     to make a determination like that, if a modification or a
     procedure change or configuration affects the margin of
     safety, and you have to do some evaluation, whether it's
     PRA-based or not PRA-based.
               DR. SEALE:  Yeah.
               MR. LOCHBAUM:  So, if the approach that's
     selected, whichever of those two is done, meets or exceeds
     the decision that would have been made to the old non-PRA
     50.59, then either of those approaches should be acceptable.
               DR. SEALE:  Yes.
               MR. BONACA:  I have a question regarding those two
     plots you showed, HPCI and unreliability, and the question I
     have is, did you find these trends consistently for other
     systems, or are these the grossest examples?
               MR. LOCHBAUM:  Every volume that I received from
     the research that was sitting on my desk had this trend. 
     The only one I didn't use was the reactor protection system
     study that recently came out for B&W and Combustion
     Engineering, because I didn't see a plot in there like this. 
     I think there was some text to that effect, but there wasn't
     a plot, and I tried to illustrate with a plot.
               So, it wasn't that I only picked the ones that
     supported my argument.
               MR. BONACA:  Okay.  So, the trend is there, you
     say.  Okay.
               MR. LOCHBAUM:  I didn't look at all the
     reliability studies, but the ones that were on my desk that
     I received recently, I did, and every one of them supported
               MR. BONACA:  One thing we found -- and I'm not
     sure this is the answer to the problem -- when we reviewed
     the experience of station blackout, we found that -- the NRC
     found that the industry and the NRC were counting
     unreliability a different way, and that was because
     regulation in different forms allows different ways of
     counting, and I'm just wondering -- because I mean, this
     consistent bias seems to be -- you know, certainly is a
               MR. LOCHBAUM:  That's why I think it's some of an
     accounting system, because if it was a methodology or if it
     was a personnel problem, people just weren't getting all the
     data, then I would expect to see some plants perhaps to the
     right of the operational data, whereas there seems to be a
     fairly consistent bias for all plants, all methods.  So, I
     think there's something more generic than just --
               DR. APOSTOLAKIS:  On the good side, of course, the
     same office issued reports that showed that the frequencies
     of initiating events were, in fact, over-estimated in some
     instances by the PRAs.
               So, it would be, in fact, interesting to find out
     why these things are happening and whether it was just a
     matter of judgement or accounting or whatever.
               So, the evidence is mixed.
               But the main message that the PRAs we're using now
     should reflect this experience I think is a good one.
               MR. BONACA:  That's the most troublesome part to
     me, that there are these biases, and we don't know why.
               DR. APOSTOLAKIS:  We can investigate that.
               Any other questions from the members?  Comments?
               [No response.]
               DR. APOSTOLAKIS:  Does the NRC staff have any
     comments that they would like to make?
               MR. BARRETT:  I'd like to make a few comments.
               DR. APOSTOLAKIS:  Please identify yourself first.
               MR. BARRETT:  My name is Richard Barrett.  I'm
     Chief of the PSA -- Probabilistic Safety Assessment branch
     in the Office of Nuclear Reactor Regulation.
               I just would like to make a few comments based on
     what I heard today, and I think, to some extent, I'm
     repeating what some of the members had to say.
               I believe there's a lot in this report that we
     would say is technically correct.
               I think there are some things in this report that
     we would take issue with, technically, but I think primarily
     what we would be concerned about is the implication that
     PRAs have not been used in a judicious fashion by the staff
     in our decision-making process.
               We feel that, throughout the history of the use of
     PRA by the NRC, which goes back 25 years, we've been very
     cautious in using it, and we have used it with full
     knowledge of the weaknesses, limitations, and uncertainties
     in the methodology.
               There are some, like myself, who have felt that we
     could have been much more aggressive over time, but here we
     are now 25 years after the Rasmussen study and we're now
     moving into an arena where we are beginning to use PRA in a
     way that will challenge the methodology.
               I think that what you'll find, though, is that, in
     our decision-making process to move into risk-informing Part
     50, both option 2 and option 3, we are taking into account a
     lot of the lessons learned from our own experience and some
     of the ones that are pointed out in the UCS study, and we
     feel that we have defined -- and I would refer everyone to
     SECY 00-162, which I think is an excellent description of
     the decision-making process that we tend to use, because
     SECY 00-162, what we say is that it's the quality of the
     decision that counts, not the quality of the PRA.
               PRA -- as with Reg. Guide 1.174, PRA will be used
     in conjunction with other methodologies, with other sources
     of information, and with other consideration.  We will look
     at generic PRA results, as well as plant-specific PRA
     results.  We will look at the results of deterministic
     studies, and we will also look at considerations of defense
     in depth and safety margin.
               And having looked at all of those, the staff will
     decide what trade-offs have to be made between quality of
     the analysis that's submitted and the quality and depth,
     scope of the staff's review of individual applications.
               We know that PRAs, in the past, have had their
     weaknesses.  I think my favorite example in the UCS report
     is the discrepancies between the Wolf Creek and Calloway
     plant, because Wolf Creek and Calloway were standard plants,
     designed to very similar specification, and yet, they not
     only had different numerical results, they came up with
     different dominant contributors.
               We know that, today, those two PRAs have very
     similar results both in the numerical results and in the
     dominant contributors, and the reason for that is that there
     has been an effort over time on the part of the owners
     groups, on the part of those licensees to compare the
     results, to compare the plant designs, and to find out what
     are the reasons for these discrepancies.
               Now, you could say that, over time, these groups,
     working together, have converged on the same answer,
     possibly not the right answer.
               We believe that the opposite is true, that these
     efforts to compare the bases of these PRAs and to challenge
     each other through these peer processes actually leads to
     more correct answers.
               So, we believe that, over time, this peer review-
     type of process will give us better PRAs, PRAs of higher
               We are currently reviewing the industry's peer
     review process, and we know that, in fact, the peer review
     process does ask the question, how many life-boats you'd
     have to have on the Titanic, not just whether or not you
     have any, but we've asked the industry to document so-called
     sub-tier criteria, asked them to document them so that it's
     not only clear to the NRC staff but it's also clear to our
     other stakeholders, and we think that's important.
               I think I agree with the assertion that the UCS
     study had to be done on the basis of IPE results, because
     that's all the results that were available, and I think, as
     time goes by, we're going to see more and more information
     available on the public record.
               A couple of specifics:
               With regard to the use of --
               DR. POWERS:  Let me interrupt you on that point. 
     You're persuaded we don't have to do anything special here,
     that this is just a natural evolution, we're going to have
     publicly available data that would allow people like Gary to
     make comparisons that didn't have this question of whether
     he's comparing data that nobody would stand behind or
     something like that?
               MR. BARRETT:  I don't want to speak for the
     industry, and perhaps someone from the industry would like
     to speak, but there has been a lot of discussion on the part
     of the industry, and I think, to some extent, motivated by
     the UCS report, to make a lot more information available,
     publicly available in a scrutable way so that more accurate
     public discussion of these PRAs can be held. 
               I won't say anymore than that, but maybe the
     industry would like to say something.
               I'd like to speak to a couple of specifics.
               One is the use of the operational experience
     reports that were published by EOD, now published by
               We in NRR get those reports in draft form, we
     review them.  We are aware of the results, and we use the
     results, as applicable, in the reviews of license amendments
     and other regulatory actions.
               I'm not familiar with this particular view-graph
     about the high-pressure coolant injection systems.  I am
     surprised to see that it indicates the unreliability of this
     safety-related system is in the 30- and 40-, even as high as
     70-percent range for BWRs in this country.  I think my
     recollection of the report was that -- and the other
     operational data reports -- was that the unreliability of
     most of the systems on a train basis is in the few-percent
     range, not the few-tens-of-percent range.
               This may not be representative of the bottom line
     of these operational experience reports.
               The other thing I'd like to point out is that we
     are using -- the example of Indian Point was brought up, and
     I think that we now have a couple of examples of how risk-
     informed regulatory action has been taken in the review of
     inspections like the Indian Point case.
               I would point you to the Farley case and the ANO
     case, and in those two cases, one of which we approved a
     licensee continuing to the end of the cycle and one in which
     we did not approve it, the full risk-informed Regulatory
     Guide 1.14 process was used, and I think that, if you review
     those two SERs, you'll get an example of what happens when
     this risk-informed thinking is applied.
               So, in summary, I'd like to say we do see a lot of
     things in the UCS report that we agree with.  We think that
     PRAs have to be continuously improved.
               We also think that there is a limit to how much
     they can be improved.  We have to have a regulatory process
     that accounts for these limitations.
               We think that, in the past, we've had such a
     regulatory process.  We're committed to having such a
     regulatory process in the future.
               DR. APOSTOLAKIS:  Comments?
               MR. LOCHBAUM:  My recollection on the Farley and
     the ANO uses of Reg. Guide 1.174 is Farley was approved
     before the IP-2 accident and ANO was denied after the IP-2
     accident.  If those were flipped, I'd question very strongly
     whether the approval/denial would have been reversed.
               I think it's more a function of the accident at
     IP-2 than the technical merits of the two cases, but I could
     debate that till the cows come home.
               DR. APOSTOLAKIS:  Any other comments from the
               Dr. Parry?
               MR. PARRY:  This is Garth Parry from NRR.
               I just want to make a kind of clarification about
     the use of the IPE results in the significance determination
               There are two points to be made here.
               The first is that the IPEs were taken primarily as
     the first step and that the process then is to send that out
     to the licensees, who will review it, together with NRC
     staff and contractors, to reflect the most up-to-date
     results, but I think the more important thing about the use
     of the IPE is that the results that are being used, the
     significance determination process, are probably among the
     more robust results from the PRA in the sense that all
     that's being used is the structure of the event trees in
     terms of the systems and functions that are required to
     respond to the different initiating events.
               I think I've made this statement before in front
     of this committee that I think those results are unlikely to
     change very much.
               The IPEs differ largely in the level of detail and
     in the numerical results, which are not directly used in the
     SDP process, and remember, too, that the SDP process and the
     way that the work-sheets are used is really only a screening
               It's not the last word.  It's just intended to be
     a conservative screening tool to act as a filter.
               MR. LOCHBAUM:  I'll look into that, but this is
     the site-specific work-sheet that went out to Indian Point
     2, dated January 3rd of 2000.  The pages aren't numbered, so
     I don't know which page it is, but it says that the human
     error probability assessed in the IPE, page 3-371, is 5.62E
     to the minus 2.
               So, you know, it seems to be more than just
     structure and things like that.
               MR. PARRY:  Yeah, I'll make comments on that, too.
               In the very original SDP process, effectively,
     human error probabilities were given a choice of .1 or .01
     across the board, depending on an assessed level of stress,
     and I think we've designed it that that really isn't
     appropriate, because in fact -- take boiling water reactors
     as an example.
               Every sequence ends with initiating suppression
     pool cooling.
               If we gave that a 10 to the minus 2, then every
     transient would turn out to be -- any change that related to
     a transient would turn out to be a red, which really doesn't
     make any sense, so it defeats the object.
               So, what I think is underway is that we're trying
     to get all the results on the AGPs from the suite of IPEs
     that's out there, and on the basis of that, a conservative
     estimate for a particular function will be chosen to
     represent the screening tool.
               So, yeah, the plant-specific numbers are in there,
     but they're being used at the moment as -- they're just
     being collected.
               DR. APOSTOLAKIS:  Any other comments from the
               [No response.]
               DR. APOSTOLAKIS:  Any comments from the public?
               [No response.]
               DR. APOSTOLAKIS:  Hearing none, thank you very
     much, Mr. Lochbaum.
               MR. LOCHBAUM:  Thank you.
               DR. APOSTOLAKIS:  Back to you, Mr. Chairman.
               DR. POWERS:  Thank you, Professor Apostolakis. 
     You've given me an extra 15 minutes that, alas, I cannot
     use, so we will have to take a -- we will recess until a
     quarter after 10.
               DR. POWERS:  Let's come back into session.
               We're going to continue our discussion of PRA with
     an examination of the industry PRA peer review process
     guidelines, and again, I'll turn to Professor Apostolakis to
     provide leadership in this area.
               DR. APOSTOLAKIS:  Thank you, Mr. Chairman.
               We have at the table Mr. Fleming and Mr. Bradley,
     who will take the lead in this.
               I have a question before we start, because -- and
     it's probably a naive question, but in the Executive Summary
     of NEI 00-02, it says that one desired outcome of having a
     PRA review process is to streamline regulatory review of
     risk-informed applications.
               In other contexts, we've said that, you know, this
     would expedite reviews, will make the life of the staff
     easier, of the industry, of course.
               If that is the case, why do you need the NRC to
     approve anything?
               I mean if you have a process that you think will
     do that, won't that happen de facto?
               I mean the NRC will have to review, no matter
     what, whatever submission is made.
               Now, if you follow a process that you think is
     reasonable, then the NRC staff naturally will do this in a
     more efficient way.
               Why do we need to go through this painful process
     of getting the blessing of the NRC in advance?
               Do you anticipate that there may be a situation
     where a licensee comes in there and say, oh, my peer review
     certification process which you have approved, let's say,
     says that this PRA is grade 3, so you should accept it's
     grade 3.  I mean, clearly, that cannot be the case, because
     the staff will still have to review it.
               So, I don't know why we have to go through this
     and have the staff approve anything.
               I mean isn't it de facto, something that will
     happen de facto?
               MR. BRADLEY:  Okay.  I'll try to answer that.
               I think we will attempt to answer that in more
     detail in today's presentation.
               What we requested was a specific NRC review with
     regard to option 2 of the Part 50 regulatory reform effort,
     and maybe I'll just go ahead and put my first slide up,
     since that's on there anyway.
               We've had discussions with the staff regarding the
     use of the peer review process to facilitate review, focused
     NRC review of applications for some time now, and there have
     been continuing questions about aspects and details of the
               Submitting the process for NRC review was intended
     to give NRC the opportunity to look at the process in
     detail, ask the questions they need to, and basically try to
     achieve the comfort level we believe they need in order to
     successfully use this process for regulatory reform and
     provide focused NRC review based on their knowledge of the
     process itself, as well as what we'd have to submit in terms
     of the option 2 application.
               So, it was specific.
               You're right, it's going to be somewhat of a
     painful process.
               We've already gotten the first RAI, and I know
     what South Texas feels like now, and we'll have to kill a
     few trees to respond to that, but we believe it's a
     necessary thing that we need to go through, and it puts it
     in the public record, and it's, you know, for everyone there
     to look at.
               DR. APOSTOLAKIS:  But the question is, do you
     expect -- let's say that you finally agree on something, you
     know, you modify your process, the NRC staff is happy, and
     so on.
               Do you expect that the licensee may come to the
     staff and say, look, the supporting documentation went
     through the peer review process, which you guys have
     approved, therefore you should accept it, or is the staff
     going to review it anyway?
               MR. BRADLEY:  No, we're not asking for some type
     of carte blanche acceptance based on the fact that it's been
               We're asking for a focused -- using that result,
     to focus the review and streamline the review, not to
     obviate the review.
               DR. APOSTOLAKIS:  So, that is my question.
               MR. BRADLEY:  Right.
               DR. APOSTOLAKIS:  If that is the case and you do a
     good job, is there any need for -- okay.  The staff passes
     judgement now and you say, well, gee, we really would like
     to see that, too, and leave it at that, without asking them
     to actually bless this.
               MR. BRADLEY:  I think both pieces are necessary. 
     Given the nature of PRA, we're trying to get at this from
     all possible directions and establish a successful framework
     to get applications to go forward.
               So, it seemed appropriate to put this on the
     docket and get NRC to have a look at it.
               MR. FLEMING:  If I may add to what Biff said, I
     think another motivation for that statement that George is
     referring to is the fact that, if utilities follow the
     certification process -- and that identifies strengths and
     weaknesses in their PRA -- those strengths and weaknesses
     can be addressed in the application, and as part of the
     application process, as it's submitted to the NRC, that
     information can be presented as a way to build confidence
     that strengths and weaknesses in the PRA have been
     identified and they've been addressed for that particular
               DR. POWERS:  But the NRC staff will have to make
     that determination anyway.
               So, if you have done it already, the staff will
     breeze through it and say this is good.
               Why is there a need for the staff to bless the
     process in advance?
               I mean they will just see the results of it and
     say, gee, those guys really know what they're doing, and
     then after they do that three or four times, they will start
     saying, oh, yeah.
               I mean if they submit a grade 3, chances are we'll
     review it very quickly.
               MR. BRADLEY:  I guess our view is, if NRC is
     familiar with the process, if they've reviewed it and have
     confidence in it, that will make it that much easier.  I
     mean that might be a leap to expect them to be able to reach
     that kind of conclusion if we haven't asked them to review
     the process.
               DR. APOSTOLAKIS:  But you're not expecting that
     somebody will come in here and say this is grade 3, you have
     approved it --
               MR. BRADLEY:  That's correct.
               DR. APOSTOLAKIS:  -- therefore we need an answer
     by Monday.
               MR. BRADLEY:  That's correct.  That's not what
     we're asking for.
               We recognize -- I mean, if regulatory reform
     succeeds, we're going to get a fairly large number of
     applications in process concurrently, and we have to have
     some method to use NRC's resources in an efficient way to
     approve those, and we're all trying to achieve some
     reasonable middle ground as to how we can do that, and this
     is part of that.
               DR. SIEBER:  Is it a factor that the NEI document,
     by itself, is not a mandatory thing for utilities to use,
     and therefore, they could pick and choose whether they would
     use it at all or what parts they would use by having some
     kind of regulatory blessing that provides the incentive to
     use it all the way it stands?
               Would that be part of the reasoning?
               MR. BRADLEY:  I guess I never thought of that as
     part of the reason.
               I don't know, Karl, if you want to elaborate, but
     I guess I'm not aware of utilities that are just using
     portions of it.
               I mean we have funded through the owners groups
     the application of the process to essentially all our plants
     by the end of next year, and it's the whole thing.  It's the
     whole process.
               DR. SIEBER:  Okay.
               MR. FLEMING:  It may be a rather moot point,
     because all the owners groups have committed to completing a
     peer review process, and we're more than halfway through all
     the plants in the country applying it.
               MR. BRADLEY:  Why don't we go ahead and try to
     start here?
               I have taken the bold step of putting a fair
     amount of detail in Karl's presentation, and I guess we may
     beg the committee's indulgence to maybe let him try to get
     through as much of that as possible.
               We did want to give you a pretty good sense of how
     the peer review process can be used to support an
     application, and that's what Karl's presentation is going to
     get at.  I just wanted to set him with just a few brief
     remarks here on -- and I think we already covered some of
               We are talking about a specific application here,
     option 2 of the Part 50 regulatory reform effort.  NEI has
     developed a guideline which we've given to the staff for
     review on that whole process.
               It's an integrated decision process.  It's sort of
     a classic 1174 application.  It also uses sensitivity
     studies and other methods to -- as part of the process to
     check the overall result and the categorization.
               Therefore, the use of the PRA in that application
     is specific, and there's specific things about a PRA that
     are important for that application.
               DR. APOSTOLAKIS:  NEI 00-02 doesn't say that it's
     only for --
               MR. BRADLEY:  No, no, no.  00-02 was developed for
     broader use.
               DR. APOSTOLAKIS:  Isn't that what we're reviewing
               MR. BRADLEY:  We want to talk to you about how we
     want to use 00-02 to facilitate NRC review of option 2. 
     That's what we're going to talk about today.
               We've already briefed the committee a number of
     times on the general process of 00-02 or the peer review
     process, and wanted to move beyond that today and talk about
     specifically how we would focus the NRC review of option 2.
               DR. APOSTOLAKIS:  Well, I have some specific
     comments anyway.
               MR. BRADLEY:  Okay.  That's what we're intending
     to do.
               These are some of the slides we've used in some of
     our recent discussions with the Commission and with our own
     working group at NEI.
               We do believe that a peer review will always be
     required, that you can never reach a point where you can
     have a checklist that would give you the necessary
     information to use a PRA for an application without any
     further advanced review.
               DR. WALLIS:  What is the second bullet here?  What
     do you mean by that?
               MR. BRADLEY:  That there is a certain amount of
     engineering judgement inherent in the process.
               DR. WALLIS:  Well, I'd say there's judgement in
     how extensive it needs to be or how much evidence you need,
     but it's not judgmental.  It's based on evidence.  It's
     based on scientific reasoning.  It's not inherently
               MR. BRADLEY:  Well, there are some judgmental
               MR. FLEMING:  I think what we meant by that -- and
     maybe the choice of words could have been refined -- is that
     PRA is not a tangible, measurable type of thing, it's based
     on a state of knowledge, and of course, the state of
     knowledge --
               DR. WALLIS:  All science and engineering is,
               MR. FLEMING:  Right.
               DR. WALLIS:  That's the kind of remark I would
     have expected from UCS.
               MR. BRADLEY:  Well, I'm not UCS.
               DR. WALLIS:  I'm sorry.  You'll have to clarify
     what you mean by a statement like that.
               DR. APOSTOLAKIS:  I think the word "inherently" is
     a bit too strong.
               DR. WALLIS:  Much too strong.
               MR. BRADLEY:  The point we were trying to make
     here is that, regardless of what requirements you write into
     a standard or to a process, that you need a peer review of a
     team of experts to look at it.
               DR. WALLIS:  Well, let me try this.  If I said the
     thermodynamic analysis is judgmental in the sense that you
     have to use judgement in how far you're going to go in
     detail, that's true, but it's not inherently so.  It's
     inherently scientific.
               MR. BRADLEY:  Okay.
               DR. WALLIS:  Is that what you mean?  That's what
     you mean, isn't it?
               MR. BRADLEY:  Yeah.
               DR. WALLIS:  Okay.
               DR. SHACK:  What you mean is that you don't
     believe you can do a design to PRA standard.  Is that what
     it means?
               MR. FLEMING:  Yes, I think that's what it means,
     and I'd just amplify on it.
               The "judgmental" refers to the fact that, right
     off the bat, to select the initiating events for a PRA model
     involves many, many judgements about what's important, and
     those judgements are inherent into the process of selecting
     initiating events.
               DR. WALLIS:  Well, that's inherent in the
     thermodynamic code.  I mean how much detail are you going to
     go into?
               MR. FLEMING:  But in a thermodynamic code, at
     least I can design an experiment to go out and benchmark my
     computer code against some actual measurements.
               DR. WALLIS:  You can do that with PRA if you have
     enough time.
               DR. POWERS:  What you say is perhaps true, but
     since no one ever has enough time, I think the distinction
     has to be drawn here.
               But maybe, Karl, it would help if you gave us a
     few examples of where this judgement has to be made, because
     clearly, I could say things in a standard like, you shall
     use plant-specific data for the reliability of valves, okay. 
     I may not want to, but I could.
               MR. FLEMING:  I think where this comment was
     heading is that, no matter what you write in a book in terms
     of criteria for performing a PRA, a standard for a PRA, or a
     process for doing a review, it requires expertise to perform
     the PRA and to review the PRA, and no matter what you put in
     there, judgements will have to be made about whether the
     state-of-the-art has been applied in an appropriate way.
               So, it's just an acknowledgement of that, and I
     think it's more important than the thermodynamic example --
     I have to take issue with that -- because the quantities
     that we calculate in a PRA, core damage frequency, are not
     observable quantities.  The temperature of a mass is an
     observable quantity.
               So, we don't have the scientific ability of
     experiment to match up the theory to the same extent we had
     with thermodynamics.
               DR. APOSTOLAKIS:  I think there is a place in the
     PRA where judgement is crucial, and this is in the
     collection of data, where you have to decide whether a
     particular occurrence in a failure or not.
               You really need experienced analysts to declare
     that something is a failure or not, especially when you go
     to common cause failures, you know, whether that failure
     applies to your plant and so on, as Karl knows very well.
               So, I think this is a judgement that you don't
     really find in other more deterministic --
               DR. SEALE:  PRAs are probabilistic, but they're
     not actuarial.
               DR. APOSTOLAKIS:  They're not actuarial, and
     again, they agree that the word "inherently" perhaps was too
     strong.  I mean there is always an element of judgement. 
     So, let's go on.
               MR. BRADLEY:  I think I've already covered this
               Basically, what we're asking for with option 2 and
     with any application is the ability to focus NRC's review.
               We also -- I want to make the point that -- as
     you'll see, I think, in Karl's presentation, the peer review
     process does elucidate the areas in a PRA that need
     improvement to support an application, and we're going to
     illustrate how it does that and what some of the
     improvements that have been made in some PRAs are as a
     result of peer reviews, but we do recognize that a number of
     the existing PRAs will need improvements to support the
     option 2 application.
               We've already provided the NRC staff with the
     schedules for the upcoming peer reviews.
               They're being conducted by all four owners groups,
     and we believe the process itself, in order to understand
     the process and appreciate its value, it's not simply
     reviewing the process guidance, it's a dynamic process, and
     it's a very intensive process involving the team and the
     plant site and includes preparation prior to the review and
     meetings every day and a final interaction between the peer
     review team and the utility.
               And the whole process -- it really needs to be
     observed to be appreciated, and we've extended the
     invitation to NRC staff to observe some of these upcoming
     reviews, and I'm going to extend that same invitation to the
     members of the committee, if they're interested.  The staff
     is working now with the schedules, and you know, please let
     them know if you're interested in observing.
               We strongly believe this is a credible, good
     process, and we want as many people as possible to get out
     there and observe it firsthand.
               DR. APOSTOLAKIS:  Could you have a Heisenburg
     effect here?
               MR. BRADLEY:  Well, at some point, that's
     possible, you know, if we had 20 observers.
               There obviously are some practical limitations,
     but we certainly would work with the schedules we have and
     with the interest to facilitate as many people as we could.
               Another thing we recognize is that the process
     right now is a one-time review, although there are a few
     sites that have been through or will go through it twice,
     but in order -- there may be a need to develop a closure
     mechanism to address, one, whatever deltas may come out of
     the NRC review, if there's a belief that certain technical
     elements need to be improved or whatever, or if there are
     substantial improvements made to a plant PRA.
               There may be cases where you need to do a second -
     - some type of a streamlined peer review to help close that
     loop, and we're looking at mechanisms that might be
     available to do that.
               Also, with regard to facilitating NRC review of
     option 2, ISI is the example we use.  We're going to have 78
     units coming in to apply for risk-informed in-service
     inspection over the next two years.
               I mean that is, by far, the most successful of the
     optional type applications we've achieved yet, and in order
     to have NRC review that many applications, we developed a
     template for the review, and that template includes a
     summary of your results, as well as a discussion of the peer
     review results and what improvements you may have needed to
     make to the PRA to allow you to do the ISI application.
               DR. APOSTOLAKIS:  As you say repeatedly in the
     report, you don't give an overall grade.  You grade
     individual elements.
               MR. BRADLEY:  Right.
               DR. APOSTOLAKIS:  So, when the licensee comes for
     an ISI application, then the licensee submits the PRA and
     says, look, for the elements that are really important to
     ISI, the peer review process ended up with the highest
     grade.  For others, where the grade is lower, they are not
     really that relevant to this particular application.
               I can see that happening, but I don't see a
     licensee coming in here and saying the result of the review
     process is that, for this element, we got a low grade, and
     that's important to ISI.
               Is that correct?
               MR. BRADLEY:  That's correct.
               DR. APOSTOLAKIS:  Okay.  So, it's for the elements
     that are not relevant to the particular application where a
     lower grade would be tolerated.
               MR. BRADLEY:  That's right.
               DR. APOSTOLAKIS:  Okay.
               DR. WALLIS:  Can you explain the third bullet?
               MR. BRADLEY:  The template?
               DR. WALLIS:  That sounds to me as if you're
     telling NRC what to do.
               MR. BRADLEY:  No.
               What we want to do is work with the staff, just
     like we did on ISI, to find what elements are important to
     their review of the application and make sure that we can
     capture those in a streamlined practical way, and remember,
     NRC set out to implement option 2 with no advance review,
     which I think is an incredibly ambitious undertaking, and I
     guess industry's view is that that's really not feasible,
     that there will have to be some review.
               Obviously, the entire detailed documentation,
     models, and everything else are going to be available for
     inspection and assessment.
               DR. WALLIS:  Doesn't NRC develop its own templates
     in the form of standard review plans and things like that?
               MR. BRADLEY:  This is simply a review template. 
     This isn't an inspection template or anything else.  This is
     a template where we can agree with the staff that, if a
     licensee puts this set of information into the application,
     that will help NRC to be able to do an expeditious review. 
     That's all we're trying to achieve here.
               DR. WALLIS:  This is a question, perhaps, of
     getting the words right, but it doesn't look right, as if
     you're developing a template which looks as if you're
     telling NRC what to do.
               MR. BRADLEY:  No.
               Finally, the -- in listening to Dave Lochbaum's
     presentation, there was quite a bit of discussion of the
     need for updated risk information, and industry is aware of
     this need.
               I don't think it serves us to have to continually
     defend studies of IPE results that are 12 years old, and
     really, we've moved well beyond those for the majority of
     plants, and there are two elements I think we're looking at.
               One is, for those plants that actually would
     undertake regulatory reform, like an option 2 application,
     we're looking at developing some type of summary description
     that would go into the FSAR for that plant, because at that
     point, you really are putting the licensing basis more
     firmly into a risk-informed arena, with option 2 or option
     3, for that matter, and those plants, we believe, would need
     to develop something along those lines, and so, we're
     working with that as part of the option 2.
               Now, there's another question of general industry
     risk information and making summary information available
     for all plants, you know, in terms of updating the type of
     information that's out there for the IPEs now, so that
     there's publicly available, current risk information.
               We're not talking about docketing the models or
     anything but coming up with some reasonable high-level
     summary that we could -- probably in some kind of matrix
     form or something.
               And those are the two things we're looking at now
     as an industry to try to get updated risk information to the
               I think of the -- you know, I would agree that all
     the stakeholders would be served by having more current
               DR. APOSTOLAKIS:  So, when you say that this is to
     support the implementation of option 2, essentially you're
     talking about the classification of components --
               MR. BRADLEY:  Yeah.
               DR. APOSTOLAKIS:  -- risk 1, 2, 3, 4, which
     essentially means Fussell-Veseley and risk achievement
     worth.  Okay.
               But the first bullet also asks the ACRS to comment
     on the whole document, not just option 2.
               I mean you have briefed us before, but we never
     really reviewed it as such.
               MR. BRADLEY:  If you want to comment on the
     document, that's great, and I'm sure you will, but what
     we're really asking for here is your interest in observing
     the actual process.
               DR. APOSTOLAKIS:  Okay.  That clarifies it.
               MR. BRADLEY:  Okay.
               At this point, I'm going to turn it over to Karl,
     and he's going to go into a little more detail about how we
     would -- I guess the other thing I'd mention is we haven't
     yet sat down and developed all the details of how the option
     2 review template would work, but we have done for other
     types of submittals that we're doing now, such as tech spec
     AOT extensions, and we're going to talk about just using
     that as an example to show how we're thinking here.
               DR. SHACK:  Will the peer review results be
     publicly available in any sense?
               MR. BRADLEY:  The peer review results -- to the
     extent that you make an application for, say, an option 2 or
     ISI or whatever and you go on the docket with a summary of
     your strengths and weaknesses and how you disposition those
     to support the application, the answer is yes.
               Whether we would docket the -- you know, the
     lengthy detailed peer review report, probably less likely we
     would do that, although at this point, we are still looking
     at the level of detail of information, but I think the
     answer is generally yes, in some form.
               MR. FLEMING:  What I'd like to do in the next few
     minutes -- I apologize for the volume of material in the
     hand-out package.
               When I was warned about having too much material
     for an ACRS presentation, I told my colleagues at ComEd that
     I wasn't ambitious enough to believe that I would get
     through all my slides, even if I had a single slide, but I
     did want to kind of point out some examples of how recently,
     especially in the ComEd case, how very recently the
     certification process was used to help support a specific
     application, and the purpose of this is to try to bring out,
     I think, a better understanding of how the industry or at
     least one element of the industry plans on using this
     process and maybe clear up some possible misconceptions that
     I think may have arisen on the certification process.
               So, in this particular example, we're talking
     about the processes that apply to ComEd.
               A few things that I wanted to point out -- I don't
     want to go into the details.  I know you've had the NEI 00-
     02 report to review.  Just a few key highlights I wanted to
     bring out.
               I've been involved personally on six of these
     certifications, three on the BWR side and three on the PWR
     side, but in this particular example I'm going to go
     through, I was on the end supporting the PRA team and using
     certification findings done by others.
               But what these consist of is a team of six,
     sometimes seven people who spend about two to three person
     months total reviewing the PRA documentation in a very
     structured process.
               They do homework, quite a bit of homework before
     the actual site visit, and they probably spend a good 60
     hours in actual on-site review at the site.
               DR. APOSTOLAKIS:  Does this include a walk-down?
               MR. FLEMING:  Yes.  There is a focused walk-down
     made by a subset of the team to look at specific issues that
     have come up in the PRA.
               What's important to understand is that there's a
     very structured process.  I mean every minute of every day
     of the on-site review session is structured in terms of
     identifying assignments for different people to take the
     lead on different parts of the review, and what's also
     important to recognize is that there is a very important set
     of products that are produced in the certification team, and
     I want to make a little bit of a comment about the
     certification team itself.
               The certification teams that are being put
     together include at least one or two people who are
     recognized experts in PRA that a lot of you would probably
     have seen in the PRA community before but, importantly,
     include members of the same owners group utility PRA staffs,
     which provides some very useful features of the
     certification process.
               One of them is that the people that are
     participating on these reviews know plants, and they know
     plants of the same owners group vintage that is being
     reviewed, and they bring into the certification team
     insights from their plants in terms of the plant features
     that are important to risk, as well as the features that
     they brought from their overall PRA program.
               So, they leave on the doorstep of the
     certification process recommendations on how the PRA could
     be enhanced, and they also have a capability of going in and
     finding weaknesses, especially in the area of looking at the
     plant fidelity -- the plant model fidelity issue, and I
     think that's very important.
               You can have people that are very, very
     experienced in PRA come in and not really know the plant
     very well and not necessarily do a fruitful review.
               For each -- in the process that we come up with,
     there's a grading system, 1 through 4.  There's a lot more
     information in the report for that.
               This grading process is an important part of the
     certification process, but it's not the most important part,
     and I think its uses maybe have been somewhat overstated.
               What really is the most valuable aspect of the
     process, from my point of view, and all the people that are
     participating on these, is a detailed set of fact and
     observation forms that identify the strengths and weaknesses
     of the PRA.
               These are very, very specific issues that the team
     comes up with that are put into different classifications of
               The most important categories are the A and B
     categories, which could have a significant impact on the PRA
               Category A are recommendations for immediate
     attention and update of the PRA before it's used in
               Category B are other issues that are considered to
     be important but could be deferred until the next update.
               The C issues are issues that probably don't impact
     the baseline PSA results but could be important for specific
               Category D issues are the ones that are basically
     editorial comments that are recommendations for cleaning up
     the documentation, and a very, very important category is
     category S, where the team is identifying particular
     strengths, where this particular element of this PRA is
     recognized as an industry leader in that type of an
               DR. WALLIS:  This is like the inverse of the
     academic grade.  I mean A means bad and D means good.
               MR. FLEMING:  That's right.
               DR. WALLIS:  It's a bit unfortunate.
               DR. APOSTOLAKIS:  They keep their distance.
               MR. FLEMING:  That's right.  It probably reflects
     the fact that so many of us have been so long out of school.
               DR. WALLIS:  Don't come round saying all the
     plants got A's, therefore it's good.
               MR. FLEMING:  That's right.
               Now, the other --
               DR. LEITCH:  Before you move too far away from the
     certification team, it seems to me that there's a measure of
     subjectivity in this peer review and that that subjectivity
     is largely tied up in the team.
               Is the team always independent of the plant that
     is being evaluated?
               MR. FLEMING:  Yes.  There are requirements for
     independence, and as part of the documentation for the peer
     review is basically an affidavit, a statement by each team
     member, who identifies his independence from the PRA team.
               Now, there have been a few cases where a
     particular team member has been involved, for example, in
     the level 2 aspects of the PRA, where that person basically
     excuses himself from any of the consensus sessions, and
     that's a reality, is that if you force -- if you want to
     force too much independence, you may not have adequate
     expertise to do the review.
               DR. LEITCH:  Yeah.
               MR. FLEMING:  So, there have been a few exceptions
     like that, but there's an affidavit, a statement made in the
     documentation, a declaration of independence or a
     declaration of whatever involvement they did have, so it's
     on paper and documented, and that's part of the process.
               DR. LEITCH:  The other side of independence, as
     you point out, is having an adequate knowledge base, and you
     obviously need people that are well versed in the issues. 
     So, it's kind of a two-edged sword.
               MR. FLEMING:  That's right.
               In the formulation of the team, there's quite a
     bit of effort that goes together by the owners group
     chairman -- for example, Barry Sloan on the Westinghouse
     owners group, takes the lead on that, and Rick Hule on the
     General Electric one, and so forth -- to make sure that the
     specific team that has been put together covers all the
     expertise needed to review the elements of the PRA.
               DR. LEITCH:  Might a particular BWR, though, have
     a set of six or seven people that are totally different than
     those that are evaluating another BWR, or would there be
     some commonality between those players?
               MR. FLEMING:  They worked hard to have some common
     players on the certification team to make sure that there's
     a carry-over of consistency, and that's the main ingredient
     that's put into the process to try to improve the
     certification and certification consistency.
               An important product, though, is the third item,
     which is, having identified strengths and weakness of the
     PRA, very specific recommendations on what could be done to
     get rid of or to resolve the issue, and a very important
     part of this is that, for the most important categories of
     issues, the A/B issues on the negative side and the F's on
     the positive side, it's a requirement that there's a
     consensus of the entire six-or-seven-member team on these
     grades, because they're very important.
               We realize that we're leaving on the doorstep, in
     the case of A and B, issues that have to be addressed, and
     we want to make sure that it's not just the opinion of one
               In the consensus process and the participation of
     Bruce Logan of INPO, for example, he recognized that to be a
     very, very important part of this.
               It's not just a bunch of opinions that are rolled
     together; it's a consensus process.
               DR. KRESS:  Just for clarification, when you talk
     about each PRA element and sub-element, just exactly what do
     you mean by that?
               MR. FLEMING:  What I mean by that -- if you look
     at the NEI 00-02, the PRA is divided up into -- I can't
     remember the actual number -- about 10 or so elements.
               Initiating events would be one, and then those are
     further broken down into sub-elements, and there's a total
     of 209 of these, and these are just simply the elements
     within the elements.
               So, for an initiating event, there would be sub-
     elements for identifying and for grouping and for
     quantifying and frequency and so forth, and it just provide
     -- that checklist is simply a way to add structure to the
     process to make sure that we're looking at, you know, all
     the same things in each one of the reviews.
               It's not intended to be a comprehensive or
     complete all-inclusive list, but it's enough of a structure
     to provide some consistency for the reviews.
               DR. WALLIS:  How does this affect the uncertainty
     in the results?  It seems to me that you can keep on
     improving the structure, keep on updating, but it doesn't
     mean to say that the certainty or the confidence you have in
     the answer that's being given is necessarily increased as a
     result of all this.
               MR. FLEMING:  Well, I'm going to give you an
     example in a second that I hope to address that, but one of
     the elements that is looked at is quantification, and as far
     as the confidence in the overall results of the PRA, whether
     they make sense, that's looked at in great detail in the
     quantification element, and if there is believed to be
     technical issues, A and B issues, in particular, that could
     impact the quantification, those are brought out in the
               I'm going to walk through an --
               DR. APOSTOLAKIS:  I have questions.
               When you say 1 is IPE and 2 is risk-ranking, 1
     means identifying the dominant sequences, vulnerabilities,
     and 2 means the option 2 application?
               MR. FLEMING:  This is better explained in,
     actually, the NEI report.
               It's recognized that there's a continuum of
     quality levels for each element of the PRA, and arbitrarily,
     those were broken up into four levels, and the general
     definition is that 1 represents a PRA that just meets the
     requirements of the IPE, and it's just a anchor point,
     historical anchor point to be able to take reference to
     what's already happened in the industry.
               A 2 means that the PRA is capable to support
     application involving screening, you know, screening kind of
               We call it ranking, but what it really means is
     being able to screen SSEs into broad categories of high,
     medium, and low safety significance, where you're not really
     worrying too much about the absolute number.
               Three is the risk significance determination,
     which would be like a Reg. Guide 174 kind of an application,
     and 4 is something to capture, you know, a state-of-the-art
     level treatment of the actual element.
               DR. APOSTOLAKIS:  Well, the impression I get from
     reading the report was slightly different, that 1 was really
     identifying -- being able to identify the dominant
     sequences, 2 was importance measures, and then 3 and 4, I
     agree with you.
               MR. FLEMING:  Yeah.
               DR. APOSTOLAKIS:  This is very important, because
     if you're going to the body of the report, there are some
     questions that are raised, and I think I should raise one
               It is stated on page 9 and then on page 18 that
     you don't need to do a time-phased analysis -- is that what
     they call it? -- for 1.  I lost the page now.  In other
     words, if you have a certain window of time and the
     operators have to do something, that you don't need to do
     that if you do a 1, grade 1, I guess, and then that common
     cause failures, on page 18, are not needed.
               It says explicitly are not needed for risk
     ranking, okay?
               Now, the note is "not required for successful
     ranking or dominant contributor determination."
               Now, there was a PRA which you are extremely
     familiar with where the number 1 accident sequence was loss
     of off-site power, loss of all the diesels, and failure to
     recover power within the available time before core uncovery
     occurs.  That's the number one contributor.
               If I am not -- I mean I don't know how you lose
     the diesels there, but I'm sure common cause failure played
     a role.
               If I don't do common cause failure analysis and if
     I don't do this time-dependent human probability for
     recovery of power, I will never be able to identify this
     dominant sequence.
               So, my grade 1 and 2 will really not give a
     reasonable result, unless I go to 3.
               MR. FLEMING:  Yeah.  I don't have the document in
     front of me, but I think that may reflect a poor wording in
     the document.
               I can assure you that, if someone came and
     presented a PRA for the certification process that didn't
     model common cause failures, they could not get a grade
     higher than 1.
               DR. APOSTOLAKIS:  Well, it's very clear in the
               MR. FLEMING:  Okay.
               DR. APOSTOLAKIS:  "Not required for successful
     ranking or dominant contributor determination."
               MR. FLEMING:  Okay.  Well, yeah, I can't account
     for that.
               DR. APOSTOLAKIS:  Do you remember, perhaps, where
     the definition of the grades is given in the document?
               MR. BRADLEY:  I think if you could let us proceed,
     you'll -- what we're trying to make the point here is that
     we're not trying to hinge the review on the grades, and all
     this discussion of the grades is really a little bit
     tangential to our intent here of trying to show how we're
     going to use the process, certainly not our intent to go in
     and say I got a grade X and therefore it's okay.
               That's not how we're doing this, and obviously,
     things like common cause and time dependencies are going to
     be important for most applications that we're going to apply
     this to.
               But I think if you could possibly let Karl
     proceed, we might answer some of these questions.
               DR. APOSTOLAKIS:  Well, it seems to me that this
     is such an explicit instruction here -- it says there is a
     note, "not required for successful ranking or dominant
     contributor determination," and here is a major plant where
     the number one sequence involves these things.  I mean
     that's a little troublesome, isn't it?
               MR. FLEMING:  Well, I can assure you that it is
               In the certifications that we're doing, if you
     would come in with a model without common cause failures in
     it, it could not get a grade higher than 1, and 1 is just a
     lower bound that the system -- we don't have a zero. 
     Probably, we should get a zero in that case.
               I can't explain this particular aspect of the
     document, but the document is a problem, and I can't recall
     seeing a PRA that does not have common cause failures of
     diesels modeled.  I don't think there's any out there.
               So, I don't know whether this is really an
     operational issue or not.  The document may have some flaws
     in it.
               What I'd like to do now is to basically walk
     through a little case study of how, at the ComEd Byron and
     Braidwood stations, this certification process was used in
     an actual successful application, and if I can walk you
     through the -- in this particular case study I wanted to
     walk you through, back a few years ago, ComEd decided that
     they want to pursue a risk-informed application involving a
     14-day diesel generator AOT, and that was done in
     conjunction with a decision to upgrade their PRAs, to take
     into account advances in PRA technology and design changes
     since the original IPEs were done.
               So, they're in the process of doing a major
     upgrade to the PRA, and they also had decided to pursue a
     Reg. Guide 177-style submittal to request a 14-day extension
     on the diesel generator allowed outage time.
               So, the Westinghouse owners group certification
     was scheduled for September 1999.
               That was scheduled during the period of the PRA
     update, and it was scheduled, actually, to provide an
     opportunity to get some input while the PRA upgrade was
     being actually completed, and then what happened was that,
     on the basis of the fact and observations, strengths and
     weaknesses that were identified during the Braidwood
     certification, there was a continuing process of upgrading
     the PRA, and then, in September of 1999, there was a
     submittal to the NRC staff requesting diesel generator AOT
     extensions for both Byron and Braidwood.
               Byron is a sister plant to Braidwood, and the PRA
     models are very similar, but there are differences, as well.
               In the submittal itself, it was a Reg. Guide 177-
     style submittal.
               There was information submitted to summarize the
     updated PRA results, and there was also a representation
     that a certification process had been done to support the
     basis for the quality of the PRA supplied in the
               After that and while the NRC was in the process of
     reviewing the submittal which was made in -- actually, the
     submittal was actually made in January 2000, this year. 
     Later on, in the summer of this year, there was a followup
     certification on the Byron plant, and that was -- provided a
     special opportunity, since the Byron and Braidwood PRAs were
     being done concurrently, because of the similarities in the
               There was an opportunity for basically the same
     certification team to come back and take a look at how the
     issues identified for Braidwood had been resolved that
     applied to Byron, as well, which was essentially 98 percent
     of them, and at the same time provided sort of a
     confirmation that the strategy taken by ComEd to resolve the
     technical issues that came up had been satisfactorily
     addressed, and that was reflected by a significant
     improvement in the results of the certification process.
               DR. APOSTOLAKIS:  Now, when you say Braidwood was
     doing a PRA, what do you mean?  They were doing what you and
     I would understand as a PRA?
               MR. FLEMING:  Right.
               They were in the process of upgrading their PRA
     from soup to nuts, you know, converting the software, going
     back over the initiating events, constructing new event
     trees, success criteria, the whole aspect, and as you may
     recall, the original IPEs submitted for the ComEd plants
     were subjected to a lot of issues associated with a very
     different set of success criteria and so forth.
               So, there was just a lot of background in terms of
     lessons learned from the original IPE process that ComEd
     wanted to take advantage of, and they basically have
     completely updated all the PRA programs at all five of their
               DR. APOSTOLAKIS:  So, they were not upgrading
     their PRAs, the parts of the PRA that they felt would be
     useful to this particular application.  They were upgrading
     the PRA, period.
               MR. FLEMING:  Well, they were upgrading the -- I'm
     glad you mentioned that.
               They were upgrading the PRA for a range of
     applications that they had planned to pursue, which included
     risk-informed tech specs, included risk-informed ISI,
     included supporting their configuration risk management
     program, and others.
               DR. APOSTOLAKIS:  Okay.
               MR. FLEMING:  So, they did have a specific package
     of applications that they wanted to pursue, but the first
     one -- the first like Reg. Guide 174 application that was
     launched as a result of this upgrade was the diesel
     generator case.
               DR. APOSTOLAKIS:  So, would you say, since you are
     very familiar with the process, that they were coming close
     to having a category 2 PRA of the ASME standard?
               MR. FLEMING:  I'm going to get to the details of
     that in a -- I'll answer your question in a second, if I
     might indulge -- have your indulgence on that.
               Now, what happened was, in the course of the NRC
     review of the diesel generator tech spec submittal, they
     asked for some additional information on the results of the
     certification process, and as a result of that, ComEd
     submitted a summary of the category A and B issues, the ones
     that were given the highest priority, together with what had
     been done to address each one of the issues, and that then
     led to the final completion of the NRC review, and just
     recently, the NRC has issued the safety evaluation report
     granting the risk-informed tech spec.
               So, that's sort of a synopsis of how the process
     was used, and I want to get back into some insights came
     through the overall process.
               What's been happening here is that it sort of
     illustrates in one particular case study that a decision had
     been made to use the PRA, a certification process was
     identifying specific strengths and weaknesses of the PRA,
     and through this overall process, there were a number of
     risk-management insights that -- we can argue about
     certification processes and standards and things like that,
     but the bottom line is that the risk-management process was
     working and working very well.
               The two big issues that had been identified in the
     Byron and Braidwood PRA -- one involved vulnerability due to
     internal flooding scenarios in the auxiliary building, where
     there was a possibility of floods that would take out the
     safety-related service water pumps located in the basement
     of the aux building, and loss of service water, of course,
     was a very serious event at this plant, Westinghouse plant,
     which would lead to a reactor coolant pump seal LOCA
               The other issue was that there was a very large
     contribution due to reactor coolant pump seal LOCAs, and the
     first insight was that, actually through Westinghouse's --
     I'm sorry -- through ComEd's participation on the
     Westinghouse owners group certification process, they became
     aware of different strategies that were being used by
     different Westinghouse plants to reduce the risk of reactor
     coolant pump seal LOCAs.
               And they actually decided to implement one of
     these changes, which has to do with providing a way to use
     the fire water system to provide an alternate component
     cooling pathway for the charging pumps so that, in the event
     that you would lose service water and you had the charging
     pumps available, you could maintain a path of seal
               It turns out that a very large number of the
     Westinghouse plants have been using, you know, techniques
     like this to reduce the probability of the conditions for
     the pump seal LOCA sequence.
               So, the actual -- you know, the participation in
     the certification process actually led to this insight, led
     to a decision to change the plant design and improve the
     risk with respect to this aspect.
               The second one was that plant modifications were
     made to address the internal flooding issue, which was the
     dominant contributor in the PRA reviewed by the
     certification team, and plant modifications were identified
     to also reduce this risk contributor.
               The other thing that was discovered through this
     process was that, in going through the evaluations required
     by Reg. Guide 177 and Reg. Guide 174, the risk metrics that
     we were using to evaluate the acceptability of the 14-day
     allowed outage time turned out to be not affected by either
     the flooding risk or the modifications that were put in
     place, and what was a little bit difficult about this
     application was that ComEd was in the midst of managing the
     risk of flooding, managing the risk of reactor coolant pump
     seal LOCA during the course of making this submittal to the
               Another risk-management insight that we found was
     that, when we tried to calculate the incremental risk
     metrics that Reg. Guide 177 calls for for evaluating tech
     specs, we discovered that just a straight application of
     those risk metrics led to problems meeting the acceptance
               And that led to insights back into the PRA to
     determine insights from the configuration risk management
     program on what compensatory measures needed to be taken
     while you're taking a diesel generator out of service to be
     able to justify risk acceptance criteria being met, and as a
     result of this process, it was determined that the risk
     acceptability or the risk insights that bear on the question
     of acceptability from this overall process actually was
     dictated by how the plant configuration was managed during
     the 14-day diesel generator outage time, and these insights
     were actually reflected in the license amendment request and
     in the NRC safety evaluation report.
               Now, how did the -- just want to talk a little bit
     -- how did the certification impact all of this, and this
     happens to be a roll-up of the grades that were obtained in
     the original Braidwood IPE or PRA review process, and as
     noted in NEI 00-02, there's grades given at the sub-element
     and element level but not on the overall PRA.
               This is a rack-up of what grades were given by the
     team for each of the elements of the PRA.
               The parentheses (c) means that the grade level 3
     was provided under specific conditions that specific issues
     that came up in the PRA were identified, and those issues
     are identified in the specific fact and observation sheets
     that are sort of tallied here in this table.
               So, the overall flavor of the certification review
     process was that they either got 3's or condition 3's but
     the conditions were conditioned on very, very specific
     issues that the certification team took issue with that
     didn't think were quite adequate for supporting the
               MR. BRADLEY:  So, that would be conditional on
     those being resolved.
               MR. FLEMING:  Yeah.
               So, what this suggests here is that, you know, it
     wasn't so much the grades themselves that were important,
     was the specific things that had to be done to be able to
     support the risk-informed application.
               DR. APOSTOLAKIS:  Let me understand the initiating
     events, the first row.
               MR. FLEMING:  Right.
               DR. APOSTOLAKIS:  You have A, B, C, D, S are the
     possible grades.
               MR. FLEMING:  No, the grades are 1, 2, 3, 4 for
     initiating events, and what's listed in the rest of the
     table are basically a frequency distribution of the number
     of fact and observation issues that came up for initiating
               So, a total of nine comments were made or
     technical comments were made for initiating events by the
     whole group, and they are distributed according to priority.
               You know, there was one A issue, two B issues, and
     four C issues and so forth, and each one of these is
     documented in the form of here's the technical issue, here's
     what we think ought to be done to resolve it, and so forth.
               DR. APOSTOLAKIS:  And what you call a grade was
     derived from those how?
               MR. FLEMING:  The grade was derived by looking at
     all the sub-elements, the grades for the sub-elements, which
     I haven't showed you here --
               DR. APOSTOLAKIS:  For initiating events.
               MR. FLEMING:  -- for initiating events, the
     specific fact and observations -- in other words, the
     technical issues that were identified for that, and then
     there was -- those were weighed against the overall criteria
     for grades 1, 2, and 3.
               So, what you don't see here -- there's a big
     detailed checklist for initiating events that has grades for
     maybe 25 or 30 sub-elements for initiating events,
     identification, grouping, support system initiators, and so
     forth, and what this table simply shows is that the
     certification team gave an overall grade for initiating
     events of a conditional 3, meaning that, if these three
     items in column A and B, if those issues were resolved --
               DR. APOSTOLAKIS:  If they are resolved.
               MR. FLEMING:  If they are resolved, they would
     qualify for 3.
               They're effectively a 2, with the path to get to a
     3 by meeting these particular issues.
               DR. APOSTOLAKIS:  There is a 3 and a C.  Three
     means it can be used for risk-informed applications?
               MR. FLEMING:  Under conditions.
               DR. APOSTOLAKIS:  Under these conditions.
               MR. FLEMING:  Under these conditions.
               DR. APOSTOLAKIS:  And C means desirable for
     applications?  Why is the parentheses in a C?
               MR. FLEMING:  I'm sorry, that's a different C. 
     The C in the grade column simply means that there is a
     condition on meeting the grade, whereas C in the other
     column means it's categories -- those are different C's. 
     I'm sorry to confuse you.
               DR. APOSTOLAKIS:  So, the grade, then, coming back
     to your second slide or so, is 3 refers to risk-informed --
               MR. FLEMING:  That's right.
               DR. APOSTOLAKIS:  -- could be used for risk-
     informed applications --
               MR. FLEMING:  Yeah.
               DR. APOSTOLAKIS:  -- according to 1.174.
               MR. FLEMING:  And the C means that you don't get
     the grade 3 unless you meet specific -- if you address
     specific issues, and I'm going to give you what those issues
     are in a second.
               MR. BONACA:  The A is significant.  There is no
     modeling of the ABG design feature, but there is a 3 without
     a condition.
               DR. APOSTOLAKIS:  System analysis?
               MR. BONACA:  System analysis, for example.
               MR. FLEMING:  Well, in the case of systems
     analysis, the team did not feel that the issues in this case
     were significant enough to affect the grade of 3, but keep
     in mind that the utility is still left with A and B issues,
     and that's one of the points I want to get here.  They're
     not just going to stop because they get a 3.  They've also
     got to resolve their category A and B issues.
               MR. BONACA:  I think the issue -- the second issue
     in the next table -- it was resolved.
               MR. FLEMING:  Yeah, right.
               DR. APOSTOLAKIS:  Why can't the industry do this
     for every single unit and have a grade 3 PRA so we will not
     have to argue about 1 and 2?
               MR. FLEMING:  I think the industry wants to get
     there, but they want to get there along an optimal path of
     allocating resources.  They want to be able to see where
     they are, measure where they are right now, see what kind of
     applications they want to do this year, next year, and the
     year after that, and they want to advance towards quality in
     the most cost-effective strategy.  I think that's what they
     want to do.
               DR. APOSTOLAKIS:  That's my fundamental problem. 
     Can one identify the dominant contributors without having at
     least a grade 3 PRA?  That's my problem, because there is an
     allowance for that.
               You know, in category 1, all you're looking for is
     vulnerabilities.  Can you really do that without having a
     level 3 PRA?  That's my problem.
               I agree with you that they want to follow the
     optimal path and get some return for their investment on the
     way, but it seems to me this is the baseline PRA that we
     should have, and this process is very good.
               MR. FLEMING:  What I believe is, in this scheme,
     the category 2, 3, and 4 all have to be able to identify the
     dominant sequences.  Category 1 is simply a historical
               DR. APOSTOLAKIS:  Okay.
               MR. FLEMING:  Okay?
               DR. APOSTOLAKIS:  But again, you dismissed the
     document earlier, but we have to go by the document, and if
     the document says that, for category 2, I don't have to look
     at the time calculations, like, you know, recovering AC
     power and then I know that the PRA you managed came up with
     a number one sequence that involved that, I'm having a
               MR. FLEMING:  I think what the time comment
     referred to in the document is the level of detail in the
     calculation of these time-sensitive sequences.
               In other words, in category 2, you could roll up
     your frequency of loss of off-site power and probability of
     non-recovery in a very simplistic time-independent model,
     whereas for the category 3 and 4, you'd have to basically be
     able to delineate how much was happening in the first hour
     and second hour and third hour.
               I mean it's a question of level of detail and
     simplicity.  It's not things that are missing.
               DR. APOSTOLAKIS:  But it seems to me that -- you
     know, later on, we're going to discuss, also, the ASME
               A lot of the disagreement comes from the fact or
     from the apparent claim that you can do certain things by
     doing a limited number of things for a PRA.
               I mean if you do what you're describing here, I
     think a lot of the controversy would go away, but that's not
     what the documents say.
               MR. FLEMING:  The document is the document.  The
     document is part of the process, and that's one of the
     things I wanted to try to get off in this presentation, is
     to walk you through in a complete soup-to-nuts application
     to show you how this is actually being used.
               The document is one part of it.  It may be an
     imperfect document.
               DR. APOSTOLAKIS:  My problem is not what you're
     describing, because you see you are ending up with a grade 3
               The question is, will there be other licensees who
     will be happy with a 2 there, instead of a 3, and they would
     demand risk-informed decisions from the staff?
               MR. BRADLEY:  It depends on what the risk-informed
     decisions are and how you're using the PRA to support those. 
     It's conceivable there could be certain risk-informed
               We're doing it today with a number of things that
     we're doing.  You can't distill it down to a black-and-white
               With regard to 00-02, which you have in front of
     you, try to look at that in the context of the letter with
     which we submitted that to NRC, and we are asking for a
     review with respect to a specific application and also with
     regard to the ability to focus NRC's review using the facts
     and observations, not to obviate their review in a specific
               We're trying to get this down to some pragmatic
     thing we can do to get option 2 implemented here, and I
     think a lot of the questions you're asking have to do more
     with the general approach of 00-02 and specifically the four
     categories, which were developed a long time ago, and you
     know, we've been through this on the ASME standard ad
     nauseam with trying to define what fits into what category,
     and we're really trying to just sort of get away from that
     here with regard to how we would use this in option 2.  It's
     a specific application.
               DR. APOSTOLAKIS:  I understand that, and the
     question is, then, is the methodology required for
     identifying the dominant sequences different from the
     methodology required to classify systems, structures, and
     components, and if so, why?
               Do you require a simpler methodology to place SSCs
     in those four groups?
               MR. BRADLEY:  It depends on the details of the
     categorization process.
               That's why we're asking for these things to be
     reviewed in concert.
               In depends on how -- where you draw the line, what
     sensitivity studies you use, and all kinds of other aspects,
     and how this feeds into the integrated decision process of
     the option 2 application.
               That's why you have to look at this in context
     with the categorization.
               DR. APOSTOLAKIS:  And that's the problem.  Can I
     use a methodology that will miss a major accident sequence
     and yet will give me satisfactory results?
               MR. FLEMING:  Let me see if I can address that.
               First of all, the only of these four grades that
     have any practical significance is grades 2, 3, and 4. 
     Grade number 1 is basically for historical reference
     purposes, and I don't think anyone in the industry would be
     satisfied with a grade level 1 anything in their PRA.
               They're getting some grade level 1's at elements
     and sub-element levels, and they're fixing those and getting
     up at least to grade level 2, but for grade level 2, 3, and
     4, it is necessary to be able to identify the dominant
     sequences, and for grade level 2, whatever else you have to
     have in order to be able to do risk screening, and there are
     utilities that are happy to use a grade level 2 PRA to be
     able to identify that something is not important, to be able
     to say that this set of motor-operated valves is definitely
     not important.
               You don't have to have a lot of detailed PRA
     information, necessarily, to be able to do that.
               If you have enough -- a minimum threshold or
     critical mass, if you will, to call this thing a PRA, you
     need to be able to identify the dominant sequences and at
     least be able to put components, SSCs, into broad categories
     of safety significance.
               DR. POWERS:  I get the impression most people
     understand that.
               The question is that you do need some details PRA
     results to do that categorization.
               MR. FLEMING:  Yes.
               DR. POWERS:  The question is which detailed ones?
               Professor Apostolakis has found a contradiction
     that he brings to your attention here in discussing level 1. 
     You don't want to discuss level 1, because it's meaningless
     now, but the same contradictions are potentially available
     to us in levels 2, 3, and 4, aren't they?
               MR. FLEMING:  I guess I don't really appreciate
     what the contradiction is.
               DR. APOSTOLAKIS:  I followed what the document
     said, and it specifically says, for 1 and 2, in fact -- I
     think it includes 2, and it's a very important point, so
     bear with me for a second.  I'll find it.
               DR. POWERS:  While you're looking, I will comment
     you're succeeding well on not getting through your view-
               DR. APOSTOLAKIS:  So, on page B-9, designated AS-
     13, time-phased evaluation is included for sequences with
     significant time-dependent failure modes; for example,
     batteries for station blackout, BWR RCPC LOCA, and
     significant recoveries."
               And then it says, for PSA grades 1 and 2, you
     don't need to do this, and I'm telling you, the number one
     sequence in the PRA you managed a number of years ago was
               MR. FLEMING:  I don't think that's what's
               DR. APOSTOLAKIS:  There may be an error.
               MR. FLEMING:  There's a level of detail.  It's not
     that you don't have to include it.  I think you have to
     include it for the lower grades.
               It's a question of whether you have to include it
     using a time-dependent model or a simplified model.  I think
     that's what's intended for that.
               DR. APOSTOLAKIS:  Okay.
               MR. FLEMING:  That's the way we are using it.
               DR. POWERS:  You pose a challenge to understanding
     your document, then, because a blank no longer means a
     blank, it means kind of a blank.
               MR. BRADLEY:  I think maybe you're answering the
     question you asked earlier about why we submitted this for
     NRC review.
               We'll go through this thing in detail and
     specifically ferret out any issue that's going to impact
     option 2.
               DR. APOSTOLAKIS:  I'm willing to accept Karl's
     point that maybe there are some mistakes here, but the
     intent was not to do that.
               Now, the fundamental question in my mind is, is
     there a difference in the methodology that identifies the
     dominant sequences from the one that identifies the
     significance of SSCs?
               MR. FLEMING:  No.
               DR. APOSTOLAKIS:  There shouldn't be.
               MR. FLEMING:  I don't believe there is.
               DR. APOSTOLAKIS:  You might argue that I can do a
     cruder analysis for the classification, because I will be
     very conservative in placing things in boxes.
               MR. FLEMING:  That's what's intended.
               DR. APOSTOLAKIS:  But in principle, there
     shouldn't be a difference.
               MR. FLEMING:  No, there isn't.
               DR. APOSTOLAKIS:  I agree with you.
               Dr. Bonaca.
               MR. BONACA:  I just had a question.
               If you put back the previous slide and you take
     off the C's, you will have a number of areas where you call
     it a 2 right now.
               MR. FLEMING:  That's right.
               MR. BONACA:  And it will be mostly 2's.
               MR. FLEMING:  Right.
               MR. BONACA:  And here it seems to me that the
     effort required to go from a 2 to a 3 is really a minor
     effort, seems to be, almost.
               MR. FLEMING:  Excuse me?  It's a minor effort?
               MR. BONACA:  Yeah, it seems to be.  I mean there
     are a few issues -- granted, these are only the A's, but
     there are, you know, a number of B's.
               The question I'm having is that I could say, well,
     this PRA was already almost a 3, had just a minor number of
     issues that made it a 2, and I'm trying to understand what
     is the range of quality in a grade 2, for example.
               MR. FLEMING:  Well, first of all, let me clarify
     something that I didn't want to mislead you on.  The effort
     it took ComEd to go from the three C's to 3 for the aspects
     of the PRA that were important for the diesel generator AOT
     was a major PRA update.
               MR. BONACA:  Okay.
               MR. FLEMING:  It wasn't just going in and doing a
     few things.
               In fact, that's what I wanted to -- let me see if
     I can get through this key slide, because that's really the
     one I was trying to get to.
               In the way in which this particular certification
     was used in this particular application, the grades
     themselves were not used directly, and what I mean by that
     is that ComEd didn't come in and say, hey, we got grade
     level 3's or, you know, we got grade level 3's subject to
     these conditions and, therefore, you know, we're a grade
     level 3.
               The grades were in the process and they're an
     important part of the process to try to ensure consistency,
     but you know, the way the process was used is that they
     found specific technical issues that stood in the way of
     getting a grade level 3, and they figured out a resolution
     strategy to get rid of those, most of which involved updates
     to the PRA.
               So, there was a substantial improvement in the
     quality of the PRA driven by the need to get approval for a
     particular application.
               The second thing that ComEd does is that all the
     issues identified -- A, B, C, and D -- the S's are only
     retained to not lose things that were successful -- are put
     into an action tracking system, and it's ComEd's commitment
     to address all these issues, you know, in some priority, but
     they're going to try to roll in the schedule for addressing
     the issues in areas that are significant for the given
     applications, because they don't have unlimited resources to
     do this.
               And the final point I wanted to make here is that,
     you know, the submittal was made in January 2000 for the 14-
     day AOT for four reactor units at two stations.
               The safety evaluation report was granted in
     September of this year, and in that process, NRC was given
     sufficient information to review this submittal and address
     quality concerns by information that was presented in the
     licensing submittal itself, which included summary
     information of the PRA.
               An RAI process extracted the A and B issues and
     what ComEd was doing about them, and that happened about a
     month before the SER was issued, and the process was
               DR. APOSTOLAKIS:  Is it possible for me to get
     copies of the submittal and the SER?  I really would like to
     read them.  Are these public documents?  I would like to
     have those.  I would appreciate that.
               Karl, the point is -- I mean you are arguing very
     forcefully about how good this was.  I'm with on that.  I
     agree with you.  I think what you're presenting is very
               The problem I'm having is -- and maybe it's a
     misunderstanding on my part -- is the lower grades.  I'm
     under the impression that both the ASME standard and the NEI
     peer review process allow a licensee to petition for
     something by using only limited parts of PRA, without having
     a level 3 PRA somewhere else, just to support that
     application and then demanding that the NRC staff not look
     at other things, because you know, some guide says there is
     a blank there.  That's my problem.
               MR. BRADLEY:  It's not the purview of the
     licensees to demand anything of the NRC staff.
               DR. APOSTOLAKIS:  I'm sorry?
               MR. BRADLEY:  The NRC staff can certainly look at
     any aspect they want in reviewing any application.
               DR. APOSTOLAKIS:  Yeah.
               MR. BRADLEY:  We would never demand that they not
     look at --
               DR. APOSTOLAKIS:  If they have blessed something,
     you know, either the ASME standard or this, then you can
     come back and argue very forcefully that, gee, you know, you
     guys are going beyond the rules of the game.
               Why do we need category 1 in the ASME?  Why do we
     need category 1 here?  And 2.  Why don't we all agree that 3
     makes sense?
               Let's do it and use pieces of it as appropriate in
     applications, which is what you're doing now with the
     extension of the AOTs for diesels.
               I think this is great.  You told them what is
     required to come up to standards of level 3 or grade 3, they
     did it, now they're going to use it in a number of
               That's beautiful.
               MR. FLEMING:  Well, let me give you an example. 
     In the particular example that I gave you here, what ComEd
     needed to do is to identify the issues that stood in the way
     for grade level 3 for those portions of the PRA that were
     important to the diesel generator AOT submittal, and that
     turns out to be a rather narrow range of sequences that
     involve extended maintenance on the diesel generator that
     don't involve issues of LOCA and ECCS and switch over to
     recirculation and things like that.
               DR. APOSTOLAKIS:  So, they went to level 4 for
               MR. FLEMING:  No.  What I'm trying to say is that
     they only had to make the case that the A and B issues that
     had been identified had been resolved to the extent needed
     for that application.
               Now, the next application, there's another set
     that are going to become important.
               I think eventually -- I think that eventually
     we'll get there, George, but --
               DR. APOSTOLAKIS:  That's a good point that you're
               MR. FLEMING:  I think eventually we'll get there,
     but I think one of the reasons why the industry wanted to do
     this certification process is that, you know, if I start
     with South Texas -- the South Texas experience, South Texas
     had gone down a pathway, they had invested a lot in their
     PRA, they paid the NRC for a detailed nuts-and-bolts review,
     much more than the IPE submittal, and they went down the
     particular path that was successful for them, and they're
     industry leaders in that process.
               One of the things that the industry wanted to do
     in the certification process is to say let's see what we've
     got now, let's benchmark what's out there right now and
     clarify what current applications the utility could do now
     and delineate which ones he has to defer until he invests
     the resources necessary to bring the PRA up, as opposed to
     going to a situation where the industry has to go off and
     spend millions and millions of dollars to get everything up
     to grade level 3 and now start applications.
               DR. APOSTOLAKIS:  I think that's a very reasonable
               MR. FLEMING:  So, it's a question of allocating
               DR. APOSTOLAKIS:  What you just said I cannot find
     written anywhere, and I agree with what you said.  I think
     that, if I look at this table you just had there, you know,
     with the ABC's and so on --
               MR. FLEMING:  Yeah.
               DR. APOSTOLAKIS:  Would you put it back on?
               MR. FLEMING:  Sure.
               By the way, I wasn't originally planning on even
     presenting this.
               DR. APOSTOLAKIS:  If everyone wants to do this, it
     seems to me that's great, and then individual pieces, you
     know, for particular applications, can afford to wait until
     they fix the A's and B's.  That's good, but that's not
     what's in the document.
               So, let's go on.
               MR. FLEMING:  I think part of that is that the
     document doesn't really describe the application process.
               DR. APOSTOLAKIS:  Can you accelerate your -- be
     more efficient?
               MR. FLEMING:  I'm just going to come to the
     conclusion right here.
               MR. BRADLEY:  Coming from you, that's an
     interesting request.
               MR. FLEMING:  I just wanted to summarize.
               From this particular example, I just wanted to,
     you know, get across a few points, that for those of us who
     participate both on the reviewing side and the receiving
     review comment side, the most important results of this peer
     review process is, first of all, the delineation of specific
     strengths and weaknesses of existing PRAs, and a clear road
     map that results from that on what exactly does the PRA team
     have to do to bring his particular PRA up to the level
     needed for a given application.
               Over time, as the certification process continues,
     because of the participation of owners group utility
     representatives from different plants and the information
     that they carry back to their PRA programs, this will
     eventually -- and it already has increased the level of
     consistency across the PRAs, and I think where we were a few
     years ago -- or at the IPE stage, we had -- most of the
     variabilities in PRA results were driven by assumptions,
     judgements, scope, and things that had nothing to do with
     the plant.
               I think that we're going down the right path
     towards getting more consistent application, and the main
     thing that's contributing to that is the make-up of the
     teams from the owners group plant PRAs.
               The grades were good in the sense that they
     provided an element of consistency from certification to
     certification.  I don't want to discount that, but we're not
     using the grades in the sense of trying to abuse them by
     saying, hey, we got a grade level 3, leave us alone, don't
     bother reviewing our PRA.  That's not what we're saying.
               We're saying we went through the process, we
     identified the strengths and weaknesses, here's what they
     were, tell the world what they were and what you did about
     them, and I think that's the most valuable part of the whole
               So, that's the summary of my presentation.
               MR. BONACA:  Do you have any idea when this
     Braidwood PRA will be a 3?
               MR. FLEMING:  What happened was that, in the Byron
     -- the Byron PRA was reviewed approximately a year later,
     and all of these C's but one were eliminated, and we're in
     the process right now of trying to figure out what it takes
     to get that up there.
               So, I think the Byron and Braidwood PRAs are at
     grade level 3 right now.
               MR. BONACA:  Okay.  Thank you.
               MR. FLEMING:  Or if they're not, there may be one
     or two specific issues that need to be resolved.
               This was a year ago, and today, we're much further
     than that, and that's another key point, is that the
     certification process by itself does not produce quality,
     the standard by itself doesn't produce quality, but what
     does lead to quality is exercising the PRA in actual
     applications, trying to make decisions from the PRA, and as
     a technical analyst, we ask ourselves the question, how does
     the technical conclusion derive from the analysis that I
     did, does it logically flow to be able to support a
     decision, and it's through exercising the PRA in specific
     decisions that lead to quality.
               DR. LEITCH:  You mentioned earlier that, within
     the next year, some 78 or 79 ISI applications might be
     received.  It seems to me, just on a -- thinking about it
     for a few minutes, that perhaps all of those PRA elements
     would be in some way tied up with the ISI program.
               Might I then imply that, by that time, those 78
     units would all have grade 3 PRAs, or am I getting something
     mixed up there?
               MR. BRADLEY:  I don't think ISI necessarily would
     exercise all those elements.
               As risk-informed applications go, it has a fairly
     limited scope of PRA elements.
               DR. LEITCH:  Which ones would you think would be
               MR. BRADLEY:  You're getting a little bit beyond
     my expertise here.
               DR. LEITCH:  Okay.
               MR. FLEMING:  I think they are involved, but I
     think one of the things that reduces the -- I don't know --
     the anxiety, if you will, about the PRA quality issue and
     the risk-informed ISI process is that the individual
     decisions are done on sort of a weld-by-weld basis.
               There may be thousands of welds that you're
     processing this evaluation, and when you start looking in
     detail about how much the risk associated with a pipe
     rupture at a weld is going to change because I have it in or
     outside the inspection program, you come to get an
     appreciation that the changes in risk that are at stake here
     are very, very -- they tend to be very, very small changes,
     because whether you have something in or outside the
     inspection program doesn't mean that the probability of
     failure goes from high to zero.
               There's a very indirect effect of doing an
     inspection and whether the pipe's going to rupture in the
     first place, because you can't inspect for all the damage
     mechanisms that could occur in the pipe, for example, and
     the localized effects of one weld failing, either as an
     initiating event or a consequence of initiating event -- you
     tend to come up with very, very small numbers.
               I don't know if you've looked at the risk-informed
     ISI thing, but the delta risks that we're calculating are
     very, very small, which gives us -- you know, we're so far
     away from the decision criteria that, you know, we don't
     have a lot of anxiety, but if you do a risk-informed tech
     spec, you take the diesel out for 14 days, you can see some
     very, very real potential effects on the PRA.
               So, now, we get much more anxious about how well
     we're calculating diesel general failure rates and common
     cause failures and these time-dependent issues that were
     brought up.
               There can be big swings in the results.
               DR. LEITCH:  Let me just ask my question another
     way, then.
               Regardless of what the grades are, then is it
     reasonable to conclude that these 78 or 79 units would have
     this peer review process completed in a year, when they
     submit these?  I mean are we that far along?
               MR. BRADLEY:  The 78 units is over the next two
     years, and given the schedule we have for peer review, I
     think it is reasonable to conclude that they will have been
     through that at that time.
               It's possible that some of the plants that submit
     will not have completed their peer reviews yet, in which
     case they can -- there are other ways to get -- you know,
     NRC is reviewing these applications, and there are other
     ways to identify the PRA information if you don't have the
     peer review results.
               I'm sure there are probably a handful of sites
     that will be ahead of that curve.
               DR. SIEBER:  It would seem to me that PRAs don't
     model welds.
               What's important is an importance measure for the
     system or some portion of the system that would reflect a
     chance of rupture, which you could do with a level 2 PRA.
               MR. FLEMING:  Right.
               DR. SIEBER:  Okay.  So, the demand on the PRA
     quality and content is not as high as it would be for other
     kinds of requests.
               MR. FLEMING:  Having been involved to some extent
     in the risk-informed ISI arena, in the risk-informed
     application, one of the steps in the process is to -- having
     recognized that you don't have the welds in the PRA, you
     don't have the identity of the welds in the PRA, is to
     exercise the PRA models that you do have so that you can
     simulate what the effects of a postulated weld failure would
               So, you end up getting to the end point, and
     eventually, what you end up doing is developing the
     capability to do risk significance on welds.
               DR. SIEBER:  Right.
               DR. POWERS:  I'm going to have to cut this
     interesting discussion off.  I thank you very much.  Thank
     you for the view-graphs, because I think they do merit study
     beyond the lecture.
               MR. BRADLEY:  I'd appreciate it if you would look
     at those, because we put a lot of effort into putting those
     together, and we didn't get through all of them today.
               DR. POWERS:  We'll continue with the Professor
     Apostolakis show into the staff views on ASME standard for
     PRA for nuclear power plant applications.
               DR. APOSTOLAKIS:  Okay.
               We have reviewed the ASME standard.  What we have
     not had the chance to do is to review the staff's comments
     on the standard.
               So, today, Ms. Drouin and Dr. Parry are here to
     enlighten us on that.
               DR. DROUIN:  Okay.
               Mary Drouin from the Office of Research, and with
     me is Gareth Parry from the Office of Reactor Regulation.
               We're going to go through today to talk about the
     recent activities that have happened since the issuance of
     Revision 12 of the ASME PRA standard.
               Back in June, on the 14th, ASME issued what they
     called Rev. 12 of the standard for probabilistic risk
     assessment for nuclear power plant applications.  This was
     the second time for public review and comment.
               The NRC spent quite a bit of time during the
     public review and comment and went through Rev. 12 in quite
     detail, and we provided substantial comments that were a
     combined effort between the two offices, and we provided
     those in a letter to ASME on August the 14th.
               In doing our review of the ASME standard, there
     was SECY-162, which provided a lot of the guidance that we
     used in coming up with our comments, using Attachment 1, and
     we also went back and looked at our comments that we had
     made on Rev. 10 to see if we still had some of those
     concerns, if they were still valid, and looked at that in
     terms of Rev. 12.
               In our letter to ASME that was submitted on August
     the 14th -- and these four bullets are lifted verbatim from
     the letter.  I did not try and paraphrase them or anything. 
     There were four points that the staff concluded:
               One, that Rev. 12 was not a standard that
     addresses PRA quality.  It's difficult to use in determining
     where there are weaknesses and strengths in the PRA results
     and, therefore, will have limited use in the decision-making
               It only provides limited assistance to the staff
     in performing a more focused review of the licensee PRA
     submittals, and the last conclusion, it provides minimal
     assistance in making more efficient use of NRC resources,
     and those were the four conclusions, even though there was
     backed up with the letter, I think, about 70 pages of
     comments of why the staff came to those conclusions.
               What I am going to do at this point -- because we
     don't have time to go through all 70 pages of comments but
     try and give you a general feeling at a high level from each
     of the chapters where our major concerns and comments were.
               Starting with Chapter 1, the biggest thing that
     you see in Rev. 12, in Chapter 1, was the definition of the
     categories, and our main concerns there is that, when you
     look at the single categories and you look at applications,
     there's no single application that fits under each category.
               So, from that aspect, we felt that the categories
     were not very useful or very helpful, and the categories
     also were being defined more from an application process,
     and since you don't have a single application that fits
     under any category, we felt that was the wrong way to
     approach defining the categories.
               When you went to Chapter 2 and looked at the term
               DR. SHACK:  Presumably, you would have the same
     objection to the grades and the peer review process.
               DR. DROUIN:  In terms of defining those, yes.
               DR. SHACK:  Yes.
               DR. DROUIN:  Yes.
               DR. SHACK:  And I think everybody sort of agrees
     you probably can't do that, that we really, really shouldn't
     be focusing on -- you know, that's somebody's dream of how
     it can be done.
               I mean they can propose, but you dispose.
               DR. DROUIN:  Yes.
               DR. DROUIN:  That's one way of saying it, yes.
               Jump in any time.
               DR. DROUIN:  In Chapter 2, the definitions, when
     we looked at these, I think the words here really captured
     our feelings that many were inaccurate, many were not
     written for the context in which they were used, and many of
     them just simply unnecessary, we didn't see why there was a
     definition there proposed.
               Maybe there were already well-known definitions
     for these and it wasn't necessary to come up with one.
               Chapter 3, the risk assessment application
     process, we had several concerns in this chapter, but the
     biggest ones is the way it was written is that, one, it
     doesn't provide any requirements in there, and then, also,
     because of the way it was written, it sort of exclude any
     minimum requirements, so that when you get to Chapter 4,
     which was the technical content, Chapter 3 almost came in
     and said you don't have to meet anything in Chapter 4,
     because it always allowed you to do supplementary analysis
     that were equally acceptable.  So, in essence, you ended up
     without a standard because of the way Chapter 3 was phrased.
               MR. PARRY:  Also, I think, in that chapter, there
     is somewhat of -- the logic isn't quite right in the sense
     that either you meet the standard or you don't meet the
     standard, but you present reasons to the decision-making
     panel why you didn't, why that doesn't matter, and I think,
     instead, the documentation seems to suggest you do something
     else and you get around the standard and say you've met it.
               So, it was a little -- the logic was a little
               DR. POWERS:  The committee certainly commented on
     precisely that unusual feature of the standard.  You cannot
     get an N-stamp, but you do get an N-stamp if you do
     something that's undescribed.
               DR. DROUIN:  Okay.
               Section 4, which some might say is the heart of
     the standard, because it gets into the technical content,
     and in our 70 pages of comments, probably at least three-
     fourths of our comments were on this particular chapter, and
     in summation, where we had problems was a lack of
     completeness, in many places just a lack of accuracy.
               We felt that it was inaccurate in terms of some of
     the technical requirements.
               The logic, the organization, and the structure,
     the supporting requirements against the high-level
     requirements, we saw lots of problems in those area, and
     this was probably the main one where it led back to our
     conclusions that we had in our cover letter, were the
     problems associated with Chapter 4.
               DR. APOSTOLAKIS:  Now, the long table you have in
     your comments on data analysis, comparing Rev. 10 to Rev.
     12, is under section 4, right?
               DR. DROUIN:  Is under section 4, lack of
               DR. APOSTOLAKIS:  That was really a very good
               DR. DROUIN:  Thank you.
               DR. APOSTOLAKIS:  And in light of what Mr.
     Lochbaum said this morning, it acquires even greater
     significance, because I notice there was an effort in Rev.
     12 to get away as much as possible from using plant-specific
     data, and you point that out in several places --
               DR. SEALE:  Yes.
               DR. APOSTOLAKIS:  -- and you know, that's the
     complaint from UCS, that the numbers that are being used for
     plants are generic, non-conservative, and so on.
               DR. DROUIN:  I think in our --
               DR. APOSTOLAKIS:  But that was really a good
               DR. DROUIN:  In our Executive Summary, we gave, I
     thought, two good examples of things that were in Rev. 10
     that were not in Rev. 12 that got into that -- appropriate
     plant-specific estimate of equipment unreliability shall be
               DR. APOSTOLAKIS:  Yeah, yeah.
               DR. DROUIN:  That was missing in Rev. 12.
               DR. APOSTOLAKIS:  There's a broader issue here,
     though, and I mean I realized it when Karl Fleming was
     making his presentation.
               A standard is not a procedures guide.
               DR. DROUIN:  We agree.
               DR. APOSTOLAKIS:  So, a standard cannot tell you
     which method to use, I suppose.
               DR. DROUIN:  Right.  We agree with that.
               DR. APOSTOLAKIS:  So, your first statement that
     this standard does not address PRA quality -- you didn't
     really mean that it had to tell you had to do certain
               DR. DROUIN:  No.  It was getting into these
     problems here, because of these problems.
               DR. APOSTOLAKIS:  But how to do it becomes a
     factor -- is Fleming still here? -- becomes a factor when,
     for example, in their peer review process, the peer
     reviewers say this is an A or B or C.
               In other words, you're relying now on the peer
     reviewers to know the methods and see whether the
     appropriate method was used for a particular requirement.
               Is that correct for both ASME and the peer review
               DR. DROUIN:  Yes.
               DR. APOSTOLAKIS:  And everyone is happy with that.
               DR. DROUIN:  Yes.
               MR. PARRY:  That gets a comment on Chapter 6 that
     you'll see in a minute.
               DR. APOSTOLAKIS:  Okay.
               DR. DROUIN:  I mean, from the very beginning,
     ASME, with NRC, support that the peer review was an
     essential ingredient of the standard, because we're never
     going to be able to get prescriptive in the standard.
               DR. APOSTOLAKIS:  What was an essential
               DR. DROUIN:  A peer review.  That's why a peer
     review was part of the standard.
               DR. APOSTOLAKIS:  There's something that bothers
     me about the peer review.  Is this an appropriate time to
     raise it?
               DR. DROUIN:  Well, I'm going to get to it in two
     more bullets.
               We didn't have a whole lot to say on Chapter 5. 
     We felt that that was a strength of the standard.
               DR. APOSTOLAKIS:  You didn't have a whole lot bad
     to say.
               DR. DROUIN:  That's right.  The comments we
     provided on Chapter 5 were more editorial, but we felt this
     was one strength in the standard.
               In Chapter 6, we had several comments.  The most
     significant was the one I put here, where we felt that the
     focus was not on the need for reviewers to make value
     judgements on the appropriateness of the assumptions and
     approximations and an assessment of the impact of their
               This was -- should be an essential part of the
     peer review, and we didn't see that coming out when you read
     Chapter 6 of the standard.
               Do you want to elaborate on that, Gareth?
               MR. PARRY:  Yeah.
               Really, if you read Chapter 6, it almost sounds
     like it's a QA check of the calculations, rather than an
     assessment of how well the assumptions have been justified
     and how well the thing has been modeled, basically.  It had
     the wrong focus, I think.
               DR. APOSTOLAKIS:  The thing that struck me as odd
     in the peer review process, certification process, was that
     the criteria for selecting the peers were formal.
               In other words, does a guy have a Bachelor's
     degree or doesn't, and if he doesn't, does he have so many
     years of experience.
               DR. DROUIN:  That's not in the ASME.  That's in
     the NEI 02 document.
               DR. APOSTOLAKIS:  What are the criteria for
     choosing the peers here?  Is it experience again?
               DR. DROUIN:  It's experience, and in Rev. 10, it
     read more like the certification, and we were -- ASME was
     heavily criticized for that.
               So, we tried to -- now, I'm speaking more as one
     of the ASME members -- to approach it differently, and it
     got into, you know, of course, independence, which we agreed
     with, but we tried to move away from saying number of years,
     because a lot of people can have a long number of years, but
     it doesn't necessarily make them an expert.
               So, it read more, be knowledgeable, the
     requirements, have demonstrated experience, have collective
     knowledge of the plant design.
               That was at the general requirements, and then it
     went on to be more specific about what it meant by that and
     not come in and say, you know, you have to have five years. 
     You could have somebody who could have two years who could
     be an outstanding person.
               So, it tried to get more into explaining what we
     meant by the word "expertise."
               DR. APOSTOLAKIS:  Okay.
               DR. DROUIN:  Whether it accomplished it well
     enough could be argued.
               I'm sure, as you're aware, the NRC letter came
     out, a lot of other public comments, but the NRC was
     probably -- their letter was probably the catalyst for some
     very recent activities, and that's what I'm going to speak
               ASME did appoint this task group to look at Rev.
     12, to provide advice back to ASME, and participating in
     this effort, the staff did come in and propose a set of
     principles and objectives of the standard, and these were
     through different phone calls, came to a consensus on these
     principles and objectives between NRC and industry, and this
     is what was used by the ASME task group.
               This task group then met on September the 19th and
     20th, and immediately thereafter, this task group did brief
     the NRC peer -- the NRC PRA steering committee and NEI's
     risk-informed regulation working group and ASME on September
     the 21st.
               So, these next slides -- this task group did issue
     a report on their finding.
               I've tried not to paraphrase any of it but lift
     the words directly from the report.
               I wasn't going to go over these, but I did put
     them in the hand-out.
               These were -- there's two pages of them, and
     that's pretty clear, actually.
               I'm impressed.
               But they were high-level objectives and
     principles, starting off -- if we just look at the first one
               DR. WALLIS:  Are these written after the work was
     done or before?
               DR. DROUIN:  This was written before the task
     group met.
               DR. WALLIS:  So, they should have done all these
               I mean if they've produced a good standard, it
     would have met all these requirements?
               MR. PARRY:  That would have been the conclusion,
               DR. WALLIS:  So, these were the objectives before
     they started out, and somehow they went astray?
               MR. PARRY:  These are the objectives before the
     task group started.
               DR. DROUIN:  After Rev. 12.
               DR. WALLIS:  That's what surprised me, is it would
     seem to me this would have been written right at the
     beginning as the objectives, specifications, and standards -
               DR. DROUIN:  That probably would have helped.
               DR. WALLIS:  -- and you wouldn't have had 12 revs
     that didn't meet the objectives.
               MR. PARRY:  I think it's taken time to develop.
               DR. WALLIS:  But isn't this the design process?  I
     mean they were just learning the design process?
               DR. DROUIN:  Fair comment.
               DR. WALLIS:  Somebody's just learning the design
               DR. APOSTOLAKIS:  Mary, in connection with this,
     is the basic position of the staff that the quality of the
     PRA -- there is a minimum standard for a quality of a PRA
     that doesn't belong to any category.
               You start talking about categories when you talk
     about applications, as opposed to having categories that
     have different quality requirements for different
     applications, and it seems to me that the staff wants to
     have a good baseline PRA --
               DR. DROUIN:  Yes.
               DR. APOSTOLAKIS:  -- where the quality is not
               DR. DROUIN:  That's correct.
               DR. APOSTOLAKIS:  And then, if you want to apply
     it to risk-informed ISI, then you do more or less, right? 
     You take the appropriate pieces of the PRA that apply and
     you say, for this category, this is what I need.
               DR. DROUIN:  Let's get to the word "minimum."  I
     don't know that you need to have a minimum.  I think what
     you -- I would tend to use the word "benchmark."  You want
     to have a set, whether that set is the minimum, but where do
     you line up to the left and right in terms of your
     weaknesses and strengths of that.
               DR. APOSTOLAKIS:  Good PRA practice, let's say.
               DR. DROUIN:  Yes.
               DR. APOSTOLAKIS:  Present good PRA practice.
               DR. DROUIN:  You know, what are your good current
     PRA practices?
               DR. APOSTOLAKIS:  So, you're not going to tie that
     to the application.
               DR. DROUIN:  That's right.
               DR. APOSTOLAKIS:  And that's what the ASME
     standard does right now.
               It says, for different categories, the quality can
     be different.
               DR. DROUIN:  Did Rev. 12 do that?
               DR. APOSTOLAKIS:  I thought so.  For category 1,
     here are the requirements; for category 2, here are the
     requirements, different requirements.
               DR. DROUIN:  I think it attempted to do it.  I
     don't think it was successful.
               DR. APOSTOLAKIS:  No, I'm talking about the
     approach, and this was also different from what Mr. Fleming
               What he said was that everybody's striving to go
     to grade 3, but then, for different applications, maybe, you
     know, if you have a comment A that is irrelevant to this
     application, you don't take care of it for this application,
     but you are trying to get there.
               I think this is an important point, because it's
     really at the root of the disagreement, I think.  It's one
     thing to try to define quality according to application and
     quite another to have a PRA of certain quality and then,
     depending on the application, I may do more or less or use
     pieces of the PRA.
               DR. DROUIN:  Okay.  I'd like to get to that. 
     There was a finding on that by the task group, and I'm not
     trying to put you off, but I'd like to answer it when we get
     to that point.
               DR. APOSTOLAKIS:  One last point.
               On 3, to facilitate the use of the standard for a
     wide range of applications, categories can be defined, it
     seems to me that, instead of trying to define categories, if
     you give a set of examples, you avoid a lot of the debate
     you're having right now.
               DR. DROUIN:  I think that's the same point.
               DR. APOSTOLAKIS:  Okay.
               DR. DROUIN:  I'm going to skip the next slide,
     which is just the rest of the principles and objectives,
     quickly show you who was on the task group.
               DR. APOSTOLAKIS:  That's interesting that Mr.
     Fleming is not there.
               DR. DROUIN:  I cannot say why who was on this
     side.  Industry proposed their people, NRC proposed their
               MR. PARRY:  I think Mr. Fleming was not available
     that week.
               DR. DROUIN:  I don't know.  This is who industry
               DR. APOSTOLAKIS:  Who's the chairman of this
               DR. DROUIN:  There was not a chairman.  There was
     what we call a facilitator.
               DR. APOSTOLAKIS:  Who is that?
               DR. DROUIN:  Syd Bernson was the facilitator.
               DR. APOSTOLAKIS:  Okay.
               DR. DROUIN:  In the task report that was issued by
     the task group -- and again, I lifted these words verbatim
     from the report -- this is what was stated.  Let me rephrase
     that a little bit.
               This is what was given to the task group by ASME
     as the charge for what the task group was to look at, and of
     course, they're restated in the report as what the charge
     was, and the task group was asked to evaluate the principles
     and objectives that we were given and provide conclusions
     and recommendations on the following.
               The first one was, you know, is it possible and/or
     appropriate for the standard to meet each objective, to what
     extent does draft 12 of the standard meet each objective,
     identify the critical technical issues associated with as
     many technical elements as possible, and propose resolution
     for the issues identified in 3 above and provide examples of
     changes that could be made affecting the structure and
     organization of the technical elements.
               So, those were the four specific things that the
     task group was directed to do during the two-day period, in
     looking at Rev. 12.
               When you just look at the charge there, just at a
     very high level, again, as stated, I did not rewrite
     anything by the task group.
               The general conclusions they came to is that, when
     you looked at the principles and objectives, they felt that
     the standard was appropriate and it was possible to meet all
     those objectives and standards.
               DR. WALLIS:  This is rather strange in light of
     your -- your conclusions are very strong about what the
     standard doesn't do, and yet, you end up here saying that
     they can now be modified to essentially meet all the things
     you didn't meet before.
               It looked to me as if it would need drastic
     surgery, not just be modified.
               DR. DROUIN:  I don't mean to -- from a personal
     opinion, when you look at the third one, where it says it
     should and can be modified, to imply that that's a trivial
     process to get there.
               DR. WALLIS:  No, it's a big modification you're
     asking for.
               DR. DROUIN:  It depends on --
               DR. WALLIS:  Your criticisms really implied that
     it hasn't taken the main thrust, the main thrust was wrong,
     not the details.
               Modification, to me, means details, but you're
     essentially attacking the main thrust of the standard in
     your critique.
               That would seem to me that they have to really
     revise their approach, not just modify it.
               DR. DROUIN:  I don't think the approach was
     revised, but the --
               MR. PARRY:  I think it really was the structure,
     certainly of Chapter 4.  It's not logically structured, and
     I think what we felt was that the lower-level requirements
     were addressing the issues we'd like to address in a
     standard.  It's just that they were not in a format that
     would make the standard itself a quality document and that
     could be easily used.
               So, you could call that major surgery.  It's sort
     of shifting things around.
               DR. WALLIS:  So, it's a reorganization of the
               MR. PARRY:  And some rewriting of the objectives
     and high-level requirements.
               DR. DROUIN:  The next slides get into the details
     of it, but I agree, it wasn't a trivial thing to do.
               In going through the task report, I will say I did
     take a little bit of literary license, because I wanted to
     match up the recommendation to each of the observations.
               When you read the task report, I think it came out
     with 12 detailed observations, and then you went to another
     chapter, and for each of the observations, there was a
               So, for the sake of -- to try to put it on as few
     slides as possible, in many cases the recommendation just
     rewords the observation, so I didn't exactly quote on the
     recommendation always.
               But when you go to the detailed observations, the
     current objective statements for the technical elements do
     not always provide a clear description of the overall
     objective for each element.
               When you looked at Rev. 12, what the task group
     was getting into is that, at the beginning of each technical
     element, there was a set of bullets that got into the
               MR. PARRY:  We're talking specifically about
     Chapter 4, which is the technical requirements, for all
     these detailed observations.
               DR. DROUIN:  Thank you.  Good point.
               When you looked at these bullets that were the
     objectives for that particular element you never could find
     a clear statement in there of the objective that was unique
     and specific to that element, and we thought that that was -
     - the task group felt that that was something that was
     important and that was missing, because then to go there to
     the high-level requirements, you didn't see the tie in the
     relationship, and they just weren't always consistent.
               So, the recommendation from the task group was,
     you know, go fix it, essentially, provide these objectives,
     and try and make them clear.
               Then, when you went from these objective
     statements to the next part in Chapter 4, with the high-
     level requirements, the task group also felt they were not
     logically related, and they should be logically related. 
     So, of course, that was the recommendation that came out.
               When you go into the support requirements, the
     task group felt that the supporting requirements should
     fully implement the high-level requirements, and what they
     mean by that is there seemed to be an interpretation by some
     members of the task group that, when you read the high-level
     requirement, that there were supporting requirements that
     were missing.
               So, you could meet the supporting requirements and
     you didn't necessarily meet the high-level requirements, and
     the task group felt that that should be the other way
     around, that if you've met the supporting requirements, then
     you should, by definition, meet the high-level requirement.
               So, that was one recommendation that came out, and
     also, the supporting requirements should be your minimum
               The next one that came out in terms of the
     supporting requirements was that, when you went from
     technical element to technical element and disregarding
     whether the logic was appropriate or the organization but
     just looking at the supporting requirements themselves, the
     task group did feel that it went, for the most part, to the
     right level of detail.
               The two exceptions that the task group came to was
     the data section was incomplete, but the quantification
     section tended to be too detailed.
               So, the recommendation there -- this one went hand
     in hand with the above bullet.
               So, when you looked at the recommendation, it was
     written as one across those two observations, wanting to pay
     particular attention to data and quantification as you went
     through and looked at the supporting requirements.
               The next one was getting into particular issues or
     topics that can have a major influence on your results but
     also where there's not usually a consensus on how to
     approach it, and the task group felt that those should, as
     best as possible, all of them -- you should be as complete
     as you can be, and those should be addressed in the
     standard.  Some examples there were BWR ATWS, the
     consequential steam generator tube rupture, dual unit
               An example of one that is in the standard would be
     RCP seal LOCA.
               The recommendation that came out from the task
     group is that, in addressing it, we did not feel that you
     needed to give an accepted methodology but to come in, and
     it should be part of the standard to require what approach
     you used, document what assumptions were done, and what was
     the significance of it.
               MR. PARRY:  It's not that we expect people not to
     deal with BWR ATWS, but it was specific issues related to
     that, to the timing issues and the interrelation of various
     operator actions that need to be addressed.
               DR. DROUIN:  The next one got into the clarity of
     the supporting requirements need to be improved.
               We had quite a few recommendations, I just pulled
     out the main ones.
               We saw a lot of places to the extent necessary to
     support category X application.  We felt that was
     inappropriate and should be replaced and explained what you
               The word "may" we felt was inappropriate, because
     it's totally permissive, so you don't know what they're
     going to do, and the term "consider" also was another place
     that brought a lot of ambiguity to the process.
               Now, getting to the categories, getting back to
     your comment, George -- now, in hindsight, I wish I had
     copied some other things from the task group report, because
     this was an area that the task group did spend a lot of time
               The conclusion coming to the task group was that
     the current definitions of the categories were not clear and
     not adequate enough to help formulate the supporting
     requirements, and it went further to say that the
     specification applications, since they may span categories,
     therefore, categories cannot be defined by applications.
               And I think this is a very important point, is
     that when you look at Rev. 12 and you go into Chapter 1 and
     you look at the criteria that are used to differentiate the
     application -- I mean the categories -- they were more
     application-driven, but since you don't have a single
     application that goes across a -- sorry -- since you don't
     have an application that stays within one category, then it
     doesn't make sense to use that as your criteria to
               The task group then went the step further and
     spent a lot of time and have proposed criteria to be used to
     differentiate the categories and then have also come up with
     a set of words to define each of the categories.
               The three -- they came up with three criteria --
     please help my memory here.
               The first one got into the scope and level of
     detail of your PRA, the second one dealt with how much
     plant-specific information should be included in the PRA,
     and then the third one was the level of realism you should
     be bringing into the PRA, so that when you go from -- and
     Rev. 12 only has three categories, doesn't have four.
               So, as you go from category 1 to category 3,
     you're going to -- if you look at the first category, which
     is scope and level of detail, you're going to increase your
     scope and level of your detail of your PRA as you go from
     category 1 to category 3.
               When you look at your degree of plant-specific
     information, again as you go from category 1 to category 3,
     you're going to increase the amount of plant-specific
     information you bring in, and the same thing on the degree
     of realism as you go from category 1 to category 3, you're
     going to increase the level of realism.
               MR. PARRY:  Another way of saying "the degree of
     realism," I think, is a reduction in the conservatism as you
     go from one end to the other in terms of modeling.
               DR. POWERS:  When you say the scope varies as you
     go from one category to the next, is there a category in
     which it is not necessary to consider common cause failure
     or not necessary to consider time-dependencies?
               DR. DROUIN:  No.
               DR. APOSTOLAKIS:  I think the issue of categories
     really can be bypassed completely.
               I mean if we come back to the basic idea that you
     should include in the argument the things that matter to the
     decision and things that don't matter can be left out, I
     think if you provide, as the committee has recommended, a
     number of examples where decisions were actually made and
     elaborate on those, you know, what insights we gained, what
     was important from the PRA, what was not, I think that
     should be sufficient, in my view, and that way, you avoid
     debates, again, as to whether category 2 makes sense, has it
     been defined correctly, and so on.
               I think if we see examples -- and maybe down the
     line, after we have sufficient experience, we will be able
     to define categories, but I really don't see the value of
     trying to define categories.
               It's really case by case.  Karl gave us a few
     examples earlier.  You have a number of examples in your
     earlier --
               DR. DROUIN:  You're not going to get any argument
     from us on this point.
               DR. APOSTOLAKIS:  Yeah, but I mean --
               DR. POWERS:  Maybe I'll throw up an argument if
     you're not going to get an argument from them.
               DR. APOSTOLAKIS:  Okay.
               DR. POWERS:  I mean it seems to me that we're
     going to have people who would like to know what is the
     minimum that I can do and still use my PRA for some
     applications that look attractive to me, and rather than
     having to plow through all these requirements and make some
     judgement on which ones are applicable for some minimalist
     activity, they'd like somebody to tell them.
               DR. APOSTOLAKIS:  My point is that you will have
     to plow through.  There is no way you won't.  And all you're
     doing now by trying to formalize it is create this debate.
               DR. SHACK:  I would look at the categories and the
     grades -- I think you should divorce them completely from
     applications, because I think that's a decision you make on
     a case-by-case basis.
               I think it's convenient to have categories or
     grades as a shorthand description for how complete and how
     much detail this particular PRA has gone into this element.
               DR. DROUIN:  And that's what task group did.
               DR. SHACK:  And I think that's a useful purpose to
     have categories and grades, because PRAs, element by
     element, will differ in those things.  You could divorce it
     completely from deciding what application -- trying to
     determine a priori what application --
               DR. APOSTOLAKIS:  But that was Dana's argument.
               DR. SHACK:  Well, I don't like Dana's argument.  I
     want categories for a different reason.
               DR. APOSTOLAKIS:  Let me put it a different way. 
     I think defining categories right now is premature, and
     we've seen the debate, we've seen the agony.  Why don't we
     look at a number of --
               DR. SHACK:  It's a useful way to describe the
     level of detail and completeness of a PRA, at least to the
     utility, so he knows what he needs to -- the most useful
     thing out of all of this is to identify weak spots in
     existing PRAs, so the guy can go off and do something about
               DR. APOSTOLAKIS:  But if, per chance, I have four
     or five analyses and studies like the ones that Fleming
     presented, don't I get that feeling?
               I mean he had nice tables, he told you that for
     risk-informed ISI there is a comment A, but it's irrelevant
     to this, so, you know, we're not going to take care of it. 
     If I look at a number of those --
               DR. SHACK:  It isn't up to Karl to define how good
     it has to be for risk -- you know, he can suggest that maybe
     this is good enough, but these people decide how good it is.
               DR. APOSTOLAKIS:  Sure.
               DR. SHACK:  And I really think you should get away
     from using those grades as applications and think of them
     more as --
               DR. APOSTOLAKIS:  I fully agree with that, but
     what I'm saying is that it's a risk-benefit calculation
     here.  Attempting to define categories will create more
     headaches than the benefit you get from them, at least right
               Was it option 2 where you have Appendix B with a
     number of examples where, you know, in some cases, you
     needed this kind of thing from the PRA.  We commented on it
     in our letter last time.
               I thought that was great.
               So, let's build up that information base first and
     then worry about the categories.
               The categories have been a problem with the ASME
     standard; the grades have been a problem with the PRA
     certification process.
               The way Karl presented it, though, makes sense to
               DR. DROUIN:  What the task group did was to define
     the categories without any -- totally divorce it from
               DR. APOSTOLAKIS:  And quality.  You can't define
     that a priori.
               DR. DROUIN:  You have quality in all categories.
               DR. APOSTOLAKIS:  Yeah, right.
               DR. DROUIN:  So, that's what I'm saying.  It got
     into the scope and level of detail, the amount of plant-
     specific information, and the amount of conservatism or
               We felt that those were the three criteria that
     you can use from a PRA perspective to define it.
               Now, I said I wasn't going to present an argument
     for the categories, but I will pick up on Dr. Powers'
     argument, because I have been -- without mentioning names,
     several utilities who have explained to me why they want at
     least the category 1, and that is to have a minimum, because
     when we sat down and we went through with the task group and
     came up with -- I just gave you the criteria.
               We actually then came up with a definition for
     each of the categories, and we're going through the
     elements, but our guideline that we were using for category
     2 in determining what should be the scope and level of
     detail, what should be the amount of plant-specific
     information, what should be the degree of realism, it was
     what is the current good practice.
               So, as we went from element to element, you know,
     as the eight people -- we all put in our views, what do we
     think is the current good practice, and then we said, okay,
     now, stepping aside from the current good practice for
     category 1, what's the minimum that we think is acceptable,
     so that if you don't meet that, you just don't have a PRA
     that's of anything.
               DR. APOSTOLAKIS:  But the minimum for that
     application or the minimum for a PRA?
               DR. DROUIN:  Minimum for a PRA.
               DR. APOSTOLAKIS:  PRA.  Oh, then I'm with you.
               DR. DROUIN:  Minimum for a PRA.
               DR. APOSTOLAKIS:  I'm with you.  And I think
     that's where Fleming was going.
               DR. DROUIN:  And that's -- you know, if you're
     going to argue categories, I think that is a good argument,
     if you want to know what the minimum is.
               DR. APOSTOLAKIS:  My basic objection is I don't
     want everybody to get the impression that, boy, I don't have
     any PRA now and the NRC is telling me, if I do A, B, C, I
     can have a category 1 application, and I don't think that
     will ever work.
               You have to have a baseline PRA, and then a piece
     of that may be appropriate only for category 1.
               DR. DROUIN:  And that's been our criticism of Rev.
     12 from the beginning.  They've been trying to do it on an
     application basis, which is not appropriate.
               Okay.  Moving on.
               MR. PARRY:  In terms of section 6 on the peer
     review, the major thing, I think, was to try to, again,
     emphasize that what the peer review team needs to do is make
     a value judgement about essentially the quality of the
     analysis, first of all see that it's met the requirements,
     to see that it, indeed, does meet all the requirements of a
     PRA, and then to provide an assessment of how well that's
     been done, over and above that.
               In terms of the application process, section 3,
     what we felt about that was that, in terms of describing an
     application process, it was too short, and really, to define
     that in terms of a standard, I think you'd have to make it
     much, much more detailed.
               The alternative, I think, is to have a chapter
     that defines how you would use this standard in a decision-
     making process or in an application process, what role the
     standard has in that process, and allow that process to be
     described in another document, such as -- one of the ideas
     that we threw out was an update of the PSA application, for
               We made one additional comment -- editorial
     comment, really, is that additional references in the
     document would be useful.
               It currently has very few references, and again,
     the reference, though, would not be to acceptable methods
     but be to documents that were used to explain why the
     requirements were necessary, because again, I want to get
     away from defining acceptable methods in the standard.
               Finally, the definitions -- well, we've already
     mentioned that they're pretty poor and they need a lot of
               I think they haven't been given much attention. 
     Everybody's just blown over that chapter.
               DR. DROUIN:  The last part in the task report by
     the task group was what future actions that would be
     undertaken by the task group.  Some recommendations were
     also made in that area.
               Most of the recommendations had to be where the
     task group would provide support on the previous
     recommendations.  The task group undertook to write
     objective statements for each element, modify the high-level
     requirements, to go and identify where we thought there were
     missing technical topics, not to then go through and write
     the requirements for that, the project team, but just to
     identify to the project team what the topics were that were
     missing, define the categories.
               They did that at the high level, and right now,
     the task group is going through and writing for each of the
     technical elements, and then to identify suggested
               The project team is also recommending that -- and
     that all pertains to Chapter 4, and while the task group is
     doing that, there's no reason why they shouldn't initiate
     the review and resolution of the public comments on the
     remaining chapters, and so, the recommendation was to move
     forward with that.
               The last recommendation from the task group is
     they felt a small group should be organized to come through
     -- and we used the word "organize and edit," but again, it's
     not quite a simple as that probably implies, to go back and
     fix that according to the principles and objectives.
               There was several reasons why the task group felt
     the small group from the task group ought to be formed, was
     because, one, to approach it in your holistic manner so that
     you were looking at all the elements together, so you could
     deal with the consistency and make sure you had the right
     organization and logic, instead of piecemealing it out.
               When I say piecemeal, you know, have one group go
     off and do one element and another group another element. 
     That's part of the problem, so to keep it to the small group
     that did it all together, and because the task group had
     undertaken to do the objective statements and modify the
     high-level requirements, that consistency could be carried
     on and just to re-clean up this part and then turn it back
     over to the project team to go through the public comments
     then and to resolve the public comments.
               And that's -- I hope I've characterized correctly
     what came out of the task group.
               DR. POWERS:  Maybe I missed it.  Did they speak to
     this issue of the supplemental analyses?
               DR. DROUIN:  I'm trying to remember.  Indirectly.
               MR. PARRY:  I don't think specifically, but
     certainly we did in the NRC comments, anyway.  So, that's a
     public comment that's going to have to be dealt with.
               I guess, in a way, it is, by virtue of the fact
     that it really is -- the supplemental analyses are really to
     do with how you make decisions, and I think we were
     recommending that the standard be somewhat divorced from the
     decision-making process in this document but that Chapter 3
     should make it clear how the standard would be used in such
     a process.
               DR. POWERS:  Somebody put in the discussion of
     supplemental analyses in Rev. 12 for a purpose.  They didn't
     succeed in whatever that purpose is, but I don't understand
     what that purpose was, nor do I understand how the technical
     group addressed that.
               DR. DROUIN:  As I said, I don't think we
     explicitly addressed it.
               To talk about Rev. 12, I think what happened is
     you have to look at the history in terms of some evolution
     between Rev. 10 and Rev. 12, and in Rev. 10, the same words
     were in there, except there were a few others that said you
     were then outside the scope of the standard.
               When Rev. 12 came in, those words got dropped, but
     they didn't get dropped with the intention of meaning that
     you were still in the standard.
               I wish I could remember some of the discussion,
     but it had to do with the way ASME writes a standard.
               So, the intent was never, I don't think, to say
     that you were still -- that if you went off and did some
     supplementary analysis, that you had met the requirements,
     for example, of Chapter 4.
               DR. LEITCH:  Mary, maybe I have a semantic
     problem, but I'm a little confused with what you're calling
     categories and grades.
               That is, ASME has set out to write a standard, and
     in terms of the previous NEI discussion on 00-02, are they
     trying to write a standard for a PRA that would be grade 1,
     2, 3, or 4?
               DR. DROUIN:  ASME elected to adopt the word
     "category" versus the word "grade."
               DR. LEITCH:  Okay.  So, you're using those terms
     kind of interchangeably.
               DR. DROUIN:  Yes.  We just thought the word
     carried meaning to it that was not truly should have been
               DR. LEITCH:  Okay.
               So, then, using the term "category," then, you're
     aiming at category 1?
               In other words, this is what you call the
     benchmark, a minimum standard to be a PRA.
               DR. DROUIN:  Okay.  The task group feels that
     category 1 should be your minimum
               DR. LEITCH:  Uh-huh.
               DR. DROUIN:  I also think that was also the intent
     in ASME.
               DR. LEITCH:  Okay.
               DR. DROUIN:  Has the NRC had an opportunity to
     comment on a number of previous revisions, or is this the
     first opportunity?
               DR. DROUIN:  In terms of ASME?
               DR. LEITCH:  Yeah.
               DR. DROUIN:  Yes.  There has only been two that
     have gone out publicly for review.
               The other ones were just -- every time we make a
     little change, as someone who's on the project team, it's
     probably misleading to the public to say there have been 12
               There's only been, really, two revisions.
               DR. LEITCH:  Okay.  Because I'd be a little
     discouraged if I saw 71 pages of comments on the 12th
               DR. DROUIN:  And what happens is, you know,
     Revision 1 of the ASME maybe only had two chapters in it.
               DR. LEITCH:  Okay.
               DR. DROUIN:  So, every time we put out a new one
     internally to ourselves, just to keep track of where we
     were, we kept calling those a revision when it wasn't truly
     a revision to the whole standard.
               So, the first revision that came out was Revision
     10, and then, when the ASME team got together, you know, we
     tweaked some things and we tweaked some more things, and
     then we came out with the next revision, which we call 12. 
     That's an internal counting.
               DR. LEITCH:  Do you have any knowledge about the
     schedule from here?  When might Revision 13 hit the streets?
               DR. DROUIN:  They're looking to try and do it
     within six months, and there has been a proposed schedule,
     but that is still under refinement.  That's why I didn't
     want to get into details, because it's still proposed, but I
     think that the goal they're looking to is to have it ready
     for balloting within six months.
               DR. LEITCH:  So, our role here is just
     information, discussion?  What's the ACRS doing with this
     presentation today?
               DR. DROUIN:  I don't know.  We were asked to come
     and present.
               DR. LEITCH:  Okay.
               DR. APOSTOLAKIS:  Are you requesting a letter?
               DR. DROUIN:  Are we requesting a letter?  I don't
     think so.
               DR. APOSTOLAKIS:  Okay.
               Would it be beneficial to structure the peer
     review process in the standard along the lines of Karl's
     presentation, forgetting for a moment that there is a NEI
     00-02, because you may disagree with what's in there, but
     the idea of identifying elements that need to be done right
     away or others, you know, are good to do but you can wait,
     so that you can eventually reach category 2 -- I thought
     that was a good idea, and maybe the peer review process in
     the standard can follow something like that, instead of
     saying, you know, just make sure that it's okay.
               MR. PARRY:  I think that's also true of -- if you
     are familiar with the IAEA review guidance, they do a
     similar thing.
               They categorize their comments in things that you
     need to do straight away, you can leave till later.
               DR. APOSTOLAKIS:  I think that's a good idea, and
     I don't see why the standard cannot adopt something like
     that, because it's also more specific that way, and the
     comments, of course, will refer to what's in the standard,
     not what's in the NEI document.  That's why I'm saying
     divorce the two.
               DR. DROUIN:  Yes.
               DR. APOSTOLAKIS:  But the table that Karl showed
     was very good, because if you do all these things, then you
     will have a category X.
               Any other comments from the members?
               DR. WALLIS:  I'm trying to get an overview of
     what's going on here, and maybe I don't know enough history,
     but the value to having an ASME standard is that ASME is an
     independent body and it gives some authority.  It's not
     being biased by NRC habits and so on, and therefore, it has
     some sort of authority out there of representing the public
     or some other group.     
               Now, the impression I get is that it's being
     wagged completely by NRC.
               So, if NRC is influencing the specifications and
     objectives of the standard, it's no longer an ASME standard;
     it's more a kind of NRC standard.  It's like an SRP or
               DR. DROUIN:  My question is what gave you that
               DR. WALLIS:  Just the way things have been
     described and in the recent documents I've read.
               DR. DROUIN:  We were asked to give a presentation
     on the activities.
               I tried to, as best as possible, quote directly
     from the task group, and the task group had four NRC people
     on it and it had five industry people on it, and the
     recommendations and observations that I presented here -- I
     mean I happen to work with NRC, but these were observations
     that unilaterally and unanimously were derived by those nine
               MR. PARRY:  There's one thing that could be
     confusing in terms of one of these bullets here.  It says
     the staff proposed a set of principles and objectives. 
     Okay.  That was really -- to be honest, that was a
     negotiated set of principles and objectives.
               Somebody had to start it, and I can't remember
     which -- whether it was the industry side or ourselves that
     started it, but it was negotiated over.
               DR. WALLIS:  ASME could come back and say that
     we're an independent body, we know what we're doing, why
     should we respond to all these staff things.
               MR. PARRY:  It's not just the staff.  It's also
     industry.  And you could argue, in fact, that the industry
     had a major influence in changing from Rev. 10 to Rev. 12.
               DR. DROUIN:  But everything I've presented here is
     the ASME task group.
               DR. POWERS:  Well, I think, in fact, Graham, that
     what I've come to learn -- I, like you, naively assumed that
     there was some body of people called the exalted ASME that
     produced this boiler and pressure vessel code that was the
     fount of all wisdom, and they were independent and
     unassailable, and but that turns out not to be the case,
     that in fact, any time they write these standards, they
     solicit volunteers, and in this case, they solicited
     volunteers from the staff and the industry to write the
     standard, and they're the gurus.
               So, there is no independent body out there.  I
     mean it's dependent upon these people that are experts.
               Now, what we can say is these are experts.
               One of the questions that comes into my mind both
     about the ASME writing group and now this task group is
     that, when I look at the membership of those things, I see a
     lot of names of people that I have the impression feel like
     they invented this technology that are not on this list, and
     I'm wondering, is the ASME suffering from the fact that they
     haven't thrown their net wide enough in selecting the
     writing group to prepare this standard?
               DR. DROUIN:  I can't speak to how ASME selected
     the people on their project team.
               DR. POWERS:  That was not an issue that the
     technical group tried to address.
               DR. DROUIN:  That the task group?
               DR. POWERS:  Right.
               DR. DROUIN:  We addressed the four things that we
     were directed to address by ASME and no more and no less
     than that.
               MR. PARRY:  But you will notice that what the task
     group recommended, though, was that it be a small group, not
     a larger group, that pulled together Chapter 4, and I think
     that's a logistical thing, that the way it's been done, as
     Mary described, is that, you know, one group would go away
     and do one technical element and another one would go and do
     another, and it's really hard to get coordination unless you
     have a focused group to do it.
               DR. WALLIS:  Who is paying for the work?
               DR. DROUIN:  Everybody.
               DR. WALLIS:  Who pays for the work?  These aren't
     all volunteers that are doing all the work, or are they?
               DR. DROUIN:  Their respective organizations pay
     for them.
               DR. WALLIS:  Oh, you mean someone comes from
     General Electric and General Electric pays for that person's
               DR. DROUIN:  Absolutely.
               DR. SEALE:  If someone comes from NRC, then NRC
     pays for their time.
               DR. APOSTOLAKIS:  The ASME's expenses, though,
     come from?
               DR. DROUIN:  ASME.
               DR. WALLIS:  ASME members are paying for this
     work, to some extent?
               MR. PARRY:  Well, they're paying for their staff,
     I guess.
               DR. SEALE:  Their standards effort is a self-
     supporting activity.  That's why you pay so much for them
     when you get them.
               DR. APOSTOLAKIS:  Well, we are running out of
     time.  Is there any other thing that is relevant to the
     particular subject here?
               [No response.]
               DR. APOSTOLAKIS:  The industry wants to make any
               DR. DROUIN:  I think Karl.
               MR. FLEMING:  Karl Fleming.
               From the point of view of representation on the
     project team, I wanted to make three comments pertaining to
     the NRC review of draft 12.
               The first comment I wanted to make is that, after
     pouring through the 70 pages of detailed comments, I
     appreciate the effort that the staff put in to coming up
     with a large number of constructive comments, which I, by
     and large, agree with, and the resolution of those that will
     be guided by this project team that Mary is talking about
     should provide the ability for enhanced PRA standard.
               So, at the detail level, I really don't have much
     of an issue.
               I do strenuously disagree with the broad
     conclusions that the NRC reached as a result of those 70
     pages of comments.
               I do not think that they flow from what's in
     there, and I think that's the reason why it appears like we
     do have a achievable path forward to get this thing fixed in
     a reasonable period of time.
               A second comment I wanted to make is that, with
     regard to the time it's taken and so forth -- and I've been
     with Mary, working on the project team for the last couple
     of years -- the one thing I kind of noticed, that up until
     but not including the NRC letter, I was kind of -- it was
     interesting for me to note that the high-level requirements
     which I actually suggested the introduction of after Rev. 10
     as a way to try to address an industry concern with Rev. 10
     that it was interpreted to be too prescriptive.
               And I don't necessarily agree that it was intended
     to be that way, but the high-level requirements were
     concocted or developed as a way to provide high-level
     requirements that were unassailable, shall be done, no
     negotiation whatsoever, and then the detailed requirements
     could be layered in in accordance with that.
               The high-level requirements that are in draft 12
     were on the street 18 months ago, and when we introduced
     these, we made it clear that it was very important that we
     get the high-level requirements agreed upon, because
     everything else that needs to be done to make the detailed
     requirement support that flows from that, and I'm just, you
     know, kind of disappointed now that, 18 months later, we're
     still, you know, fixing the high-level requirements.
               I'm not arguing that they don't need to be fixed,
     but it's just a shame that we've been trying to write a
     standard for the last 18 months on the basis of high-level
     requirements that still need to be revised.
               DR. DROUIN:  Karl, I agree with you, but I've just
     got to put something in here.
               The NRC has, through the project team -- and I can
     go pick out e-mail after e-mail where we have provided
     comments from just the NRC the problems with the high-level
               So, to say, you know, 18 months later, that this -
     - you know, this comment has come forward -- we've been
     providing comments with our concerns on those over the last
     18 months.
               MR. FLEMING:  That's interesting, because I didn't
     appreciate that.
               We spent an entire project team meeting last
     summer in Palo Alto where the 18-member project team went
     over line item by line item of those high-level
     requirements, and I wasn't aware, personally, that there
     were any comments on those until today.
               DR. APOSTOLAKIS:  Any other comments?
               MR. BRADLEY:  Just a quick one.
               In the interest of clarity and since the question
     was asked, I would like to point out -- and I'm not familiar
     with ASME, but for ANS, NRC has provided grants to ANS to
     support the standards development for PRA standards.
               DR. APOSTOLAKIS:  Does it take away from the
               DR. DROUIN:  No.  NRC does look at any grant
     proposal that's been submitted, and if an organization
     doesn't request for a grant, we can't be responsible for
               DR. APOSTOLAKIS:  By the way, the ANS standard has
     sort of faded away?
               DR. DROUIN:  No, they're actively working.
               DR. APOSTOLAKIS:  Actively working.  We were
     supposed to review something last month.  Is it tied
     intimately to the ASME standard, so it has to wait until the
     ASME standard is ready?
               DR. DROUIN:  I can't comment on -- I'm not the
     lead on that.
               DR. APOSTOLAKIS:  Okay.
               I think we're done.
               DR. POWERS:  Okay.
               I will recess us until -- for an hour.
               [Whereupon, at 12:40, the meeting recessed for
     lunch, to reconvene this same day, Thursday, October 5,
     2000, at 1:40 p.m.]
     .                           AFTERNOON SESSION
                                                      [1:40 p.m.]
               DR. POWERS:  Let's come back into session.
               The next topic is going to be somewhat of a switch
     from the previous discussion of PRA to move to PTS,
     pressurized thermal shock, and Dr. Shack will --
               DR. SHACK:  Probabilistic analysis again but of a
     fairly exactly sort.
               DR. POWERS:  Well, in that case, Dr. Shack, maybe
     you could explain to me something about the bias analysis on
     these parameter distributions that they were calculating for
     this piece of work.
               DR. SHACK:  Well, I would like to say we had a
     very good, full subcommittee discussion of the PTS update
     project, and trying to pick some pieces out of that that we
     should bring to the full committee, we decided it was useful
     to have something where we could walk through the overall
     probabilistic fracture mechanics calculation, so people
     could see how all the pieces sort of fit together, because
     we've been analyzing it sort of piece by piece, and then we
     did want to go through one particular aspect on the fracture
     toughness uncertainty -- or the fracture toughness
     distributions, where there's an interesting discussion of
     how the uncertainties will be treated.
               One approach has been a purely statistical one
     that was developed at Oak Ridge, and then there's an
     alternate approach that's being explored at the University
     of Maryland, and I think it's worthwhile reviewing that.
               One thing that did come up at the subcommittee
     meeting that I think is worthwhile bringing to the full
     committee's attention is a concern that Dr. Kress raised
     that's sort of similar to the problem we ran into with the
     spent fuel problem, that if a reactor vessel was going to
     fail, we would be getting a different kind of source term
     than the source term that we've usually been used to dealing
     with.  That is, the core melt will occur with the --
     essentially exposed to air, rather than in water.
               So, it will be an air environment rather than a
     steam environment.
               With a different source term, this raises into
     question exactly what is the appropriate kind of LERF
     criterion to use.
               The LERF criterion that we're sort of comfortably
     using and used to using from 1.174 is really based on a
     steam-driven source term, and again, PTS would have a
     different kind of source term, and it might well affect the
     kind of acceptance criteria you'd want to have for a PTS
     incident.  So, that's an issue that we did bring up with the
               I'm sure they haven't had a whole lot of time to
     address it, but it is one we think needs to be addressed
     before the PTS update can be completed, and with that, I'll
     turn over to the staff, and I see we have all sorts of staff
     here today.
               We'll have Mike Mayfield, I guess, lead off.
               MR. MAYFIELD:  All right.
               I'm Mike Mayfield from Division of Engineering
     Technology and Research.
               The overall PTS project involves three different
     divisions in research, and the pieces you're going to hear
     about today, I guess, encompass some synthesis of ideas that
     come from the fracture mechanics world, the materials world,
     and some of the PRA work.
               We certainly have welcomed the committee's input
     and have appreciated the time you've been willing to invest
     in reviewing this project as it's gone along.
               We hope to continue this dialogue over the course
     of the next year or so, as we finish off the project.
               With that, unless Ed or Mark have something, I'll
     turn it over to Terry Dickson from Oak Ridge to talk about
     the FAVOR code.
               DR. KRESS:  Before you get started, I would like
     to say it's nice to hear from somebody that doesn't have an
               MR. DICKSON:  I never thought of myself as having
     an accent until -- I worked overseas for several years, back
     in the '80s, and I would talk to people, and they'd say
     where are you from?  I'd say Tennessee.  They'd say I
     thought something like that.
               I sort of started almost getting self-conscious
     about it at that time.
               My name is Terry Dickson, and I'm going to talk
     about the FAVOR code.
               I'd like to acknowledge two of my colleagues, Dr.
     Richard Bass and Dr. Paul Williams, that work with me in the
     heavy section steel technology program.  They very much
     helped me put this presentation together.
               The presentation is sort of broken into distinct
               The objective is to describe the evolution of an
     advanced computational tool for reactor pressure vessel
     integrity evaluations, FAVOR, and the first part of the
     presentation is just how FAVOR is applied in the PTS re-
     evaluation, and the second part is just to show how the
     evolving technology is being integrated into the FAVOR code
     for this PTS re-evaluation.
               The third section is just kind of about the
     structure of the FAVOR code, and the fourth is kind of an
     overall PRA methodology, and there's a fifth section that
     really isn't part of this presentation.
               This is basically the same presentation I gave in
     September, but it's sort of been decided since then that
     maybe we need maybe a little more work in this last area. 
     So, really, this presentation will deal with the first four
     of these.
               This sort of goes, I guess, right to the heart of
     the matter.
               Application of FAVOR to the PTS re-evaluation
     addresses the following two questions.
               Here's a graph that plots or demonstrates the
     frequency of RPV failure in failures per reactor year
     plotted as a function of effective full-power years.  Also,
     you could think of this as RTNDT, you could think of it as
     neutron fluence; in other words, the length of the time that
     the plant has been operating.
               And this, I might add, is the type methodology
     that was used in the SECY 82-465 analysis, from which the
     current PTS screening criteria was derived.
               So, the two questions here is, at what time in the
     operating life of the plant does the frequency of reactor
     pressure vessel failure exceed an acceptable value?
               So, we see this red line increasing, the frequency
     of vessel failure increasing as a function of time, and the
     current value is 5 times 10 to the minus 6.
               So, the question is, at what time in the operating
     life of the plant does the frequency of failure exceed a
     certain value, and then, the second question that we're
     particularly interested in here with this re-analysis is how
     does the integration and application of the advanced
     technology affect the calculated result.
               And this is just an attempt to show that, you
     know, this second curve, this blue curve, if you went back
     and re-did this analysis, with an improved model, which we
     think we have, an improved model, or a plant-specific
     mitigation action, that you would shift this curve in such a
     way that you would be able to operate the plant an
     additional period of time and still be in compliance with
     some level of failure, frequency of failure.
               So, in the context of this presentation, what
     FAVOR does it generates this type curve, and you would run
     FAVOR -- an execution of FAVOR would give you one point on
     the curve, and then you would have to -- as I talk about
     FAVOR -- I mean this is what we're doing.
               We're plotting point on this curve to see how the
     frequency of the failure increases as a function of the
     time, and specifically how these improve models.
               Now, one thing I will point out -- and I will
     probably refer back to this slide as I go through the
               This just shows it as a line.  This just shows it
     as discrete values.
               In reality, there will be some distribution about
               In other words, you can think of this as being the
     mean value of the distribution that comes out.
               DR. APOSTOLAKIS:  Was there an uncertainty
     analysis actually done for the red line?
               MR. DICKSON:  Yes.
               DR. APOSTOLAKIS:  At that time?
               MR. DICKSON:  Well, I may ask Professor Modarres
     to help me out here.
               All the uncertainty -- as I step through the
     presentation, maybe I'll be able to address that question
               DR. APOSTOLAKIS:  That refers to the blue line,
     the one you're developing now.
               I'm asking about the old analysis.
               MR. DICKSON:  The old analysis did not have an
     uncertainty analysis.
               DR. APOSTOLAKIS:  So, the red line is -- we don't
     know what it is.
               MR. DICKSON:  Right.
               DR. APOSTOLAKIS:  Some sort of a best estimate.
               MR. DICKSON:  Yes.
               DR. APOSTOLAKIS:  Would you define effective full-
     power years?
               MR. DICKSON:  An effective full-power year is a
     calendar year at which the vessel -- at which the core was
               DR. KRESS:  If it's only part-power, you count
     that as a fraction of power.
               One of the things that bothered me about this is,
     if the 5 times 10 to the minus 6 were only 2 times 10 to the
     minus 6, which ain't a lot difference, you drop all the way
     down to 15 years on the red line or the blue line, too.
               MR. DICKSON:  This really isn't numbers from an
     actual analysis.  This is for illustrative purposes only.
               DR. KRESS:  Right.  But that's an illustration of
     the same point.
               MR. DICKSON:  Yes.
               That was one of the outcomes of the IPTS analysis,
     that the shape of these curves -- as you know, there was an
     analysis done for each of the three domestic vendors, and
     the shape of that curve seemed to be slightly different for
     a Westinghouse versus a B&W versus a CE.
               DR. POWERS:  Westinghouse versus CE is a little
     surprising, but Westinghouse versus a boiler, you wouldn't
     be surprised about that, would you?
               DR. KRESS:  I don't think you've done it for
     boilers, have you?  It's a B&W plant.
               DR. APOSTOLAKIS:  So, if I have two years at half-
     power, then those would count as one effective full-power
               MR. DICKSON:  Yes.
               Typically, I think -- I think, typically -- and
     somebody correct me if I'm not right, that a typical
     licensing period is for 32 effective full-power years, I
     believe, which I think sort of equates to 40 calendar years,
     if you figure, you know, 20-percent down time.
               So, the near-term schedule for the development of
     the FAVOR code has been recently defined, and the current
     schedule specifies FAVOR to be ready for the PTS re-
     evaluation analysis March 1st of next year.
               Between now and then, the models are being
     finalized, and the finalized models are being implemented
     into the FAVOR code, and scoping studies will be performed,
     and the idea at this time is that the Oconnee plant will be
     the vehicle for performing the scoping studies.
               The primary reason is that the thermal hydraulics
     folks are doing Oconnee first.  So, we'll have all the input
               We'll have the PRA data in the form of initiating
               We'll have the thermal hydraulics data from the
     thermal hydraulics branch.
               We'll have the flaw data from the appropriate
               So, all the data will be there to sort of start
     shaking down the code and seeing -- kind of seeing where the
     numbers come out.
               So, Oconnee will be the first application of all
     this technology.
               Since this presentation is about the status of the
     FAVOR code development, we sort of thought it might be
     appropriate to sort of show a little of the historical
     evolution of how the FAVOR code came to be.
               By the way, for those of you that don't know,
     FAVOR is an acronym, Fracture Analysis of Vessels Oak Ridge.
               Development of the FAVOR code was initiated in the
     early 1990s by combining the best attributes of the OCA and
     the VISA code with evolving technology.
               There was a series of codes -- OCA-1, OCA-2, OCA-P
     -- that was developed at Oak Ridge National Laboratory in
     the early 1980s, OCA standing for Over-Cooling Accident.
               There was also, in parallel, an effort that was
     initiated within the NRC and later was taken up PNNL.  They
     did a code called VISA-1, VISA-2, as I said, that was done
     by the NRC, PNNL, in the same timeframe, and both of these
     codes were applied during the SECY 82-465 analyses, as well
     as the integrated pressurized thermal shock analyses.
               So, both of these codes sort of fed into -- in
     other words, we took the best parts of OCA, the best parts
     of VISA, plus lessons learned from the IPTS, integrated
     pressurized thermal shock, as well as a lot of lessons
     learned from the Yankee Rowe experience.  All of these fed
     into the development of FAVOR.
               The first public release of the FAVOR code was in
     1994, followed up by an improved version, the '95 version. 
     There was a limited release in 1999, limited to this group
     of people that have been coming to these NRC meetings,
     primarily from the industry, as part of this PTS re-
     evaluation.  So, some of the things I will be talking about
     were incorporated into that version, but clearly, we're --
     the code is continuing to evolve, and as I said earlier, the
     goal right now is to have a development version fixed by
     March of 2001.
               This is just sort of a transition slide to say
     we'll now talk a little about the integration of evolving
     technology into the FAVOR code.
               Elements of updated technology are currently being
     integrated into the FAVOR computer code to re-examine the
     current PTS regulations, and this is just an illustration
     kind of all of these boxes out here on the periphery of how
     they are feeding in -- these are areas that clearly have
     been improved since the analyses that were done in the 1980s
     from which the current Federal regulations for PTS were
     derived, such as detailed neutron fluence maps, flaw
     characterizations, embrittlement correlations, better
     thermal hydraulics, better PRA methodologies.
               The RVID database is the reactor vessel integrity
     database which has been developed and is maintained by the
     Nuclear Regulatory Commission, which basically is kind of a
     repository of all the, I guess you would say, official
     vessel characteristics such as chemistry.  If you wanted to
     know the chemistry of a particular weld in a particular
     plant, you'd go to the RVID.
               Extended fracture toughness databases, fracture
     initiation toughness, fracture mechanics, and the FAVOR code
     itself is one of the boxes that certainly is an improvement
     of technology since the 1980s.
               DR. WALLIS:  Now, thermal hydraulics hasn't
     improved yet, has it?
               MR. DICKSON:  I suppose what is intended here is -
     - I don't know what release of RELAP was used in 1985, but
     certainly, it's a later release of RELAP, and certainly, I'm
     sure you're well aware of the APEX experiments and so forth,
     that there's an attempt to validate --
               DR. WALLIS:  Which haven't been done yet.
               MR. DICKSON:  I think it's supposed to happen
     during the fall of this year.
               MR. BOEHNERT:  They just started, Graham.  They
     just started testing.
               MR. DICKSON:  So, I guess what is meant here -- we
     would hope that we have better thermal hydraulic analyses
     now than 15 years ago.
               So, in any case, all of these are going to feed in
     -- you can think of all of these sort of being input data
     into the process, because the FAVOR code sort of has to have
     a software interface here with all of these elements.
               So, they all feed into this updated technology PTS
               Hopefully, at the end of the day, we'll have a
     technical basis for the revision of PTS regulation.  There's
     no way of knowing which way it's going to go at this time. 
     I think we started into this process thinking that the
     potential exists for there to be a potential for relaxation
     of the current regulations.
               We'll talk a little bit more about that, but
     basically, we're going to put all the stuff in, turn the
     crank, and let the chips fall where they may.
               This is a little bit redundant.
               This is just, in words, sort of repeating what we
     had there in that last one.  Advanced technology is
     integrated into FAVOR to support possible revision of the
     PTS regulation, flaw characterizations from the NRC research
     -- we'll talk a little bit more about that in a moment --
     detailed fluence maps, embrittlement correlations, RVID
     database, fracture toughness models, surface-breaking and
     embedded flaws, inclusion of through-wall weld residual
     stresses, and a new probabilistic fracture mechanics
               As I said, that's slightly redundant, but probably
     the reason that I'm standing here now talking about this is
     we sort of did some analyses a couple of years ago to sort
     of see what -- if someone was to revisit some of the IPTS
     analysis and use some of the improved models, what the
     impact might be, and at that time, it looked like -- as I
     previously said, it looked like that the potential existed
     for a relaxation of the regulations, and the singular most
     important contributor to that was a significant improvement
     in the flaw characterizations, okay?
               A significant improvement since the derivations of
     the current PTS regulations is flaw characterization,
     because in those analyses, the SECY 82-465, as well as the
     integrated pressurized thermal shock, they assumed that all
     of the flaws were inner surface-breaking flaws, okay?
               It was known that that was a conservative
     assumption, but kind of in the absence of any other
     knowledge, that was the assumption that was made.
               Well, the NRC, I would say, has wisely spent their
     research dollars since then performing non-destructive
     examination as well as destructive examination of RPV
     material at Pacific Northwest National Laboratory to improve
     a technical basis for the flaw-related data used as input
     into these probabilistic analyses, and what has come out of
     this -- and it's a continuing, ongoing process, but what has
     come out of this is that a significantly higher number of
     flaws were found than were postulated in the original
     analyses.  However, all of the flaws so far that have been
     detected are embedded, as opposed to inner surface-breaking.
               PVRUF, for those of you that don't know, is the
     Pressure Vessel Research User Facility, which was actually a
     vessel that was never put into service.
               It was brought to Oak Ridge, it was cut up, and
     the non-destructive examination as well as the destructive
     examination performed on it.
               When you take those flaw densities and apply them
     to a commercial pressurized water reactor, what you predict
     is that you will have between three and four thousand flaws
     in the first three-eighths thickness of the vessel.
               So, as I say, you have a lot more flaws, but
     they're sort of more benign, they're embedded, from a
     fracture point of view, and we'll talk a little bit more
     about this in a moment, but I guess the thought that I would
     want to leave you with with this slide is that, out of all
     of the little boxes feeding into this, this flaw
     characterization, by far, has the highest potential for
     impacting the answer.
               DR. LEITCH:  When you say first three-eighth-inch
     thickness, that's from the inner wall?
               MR. DICKSON:  Yeah.
               DR. LEITCH:  Okay.  Thank you.  And it's not
     three-eighths of an inch, it's three-eighths --
               MR. DICKSON:  Three-eighths of the wall thickness.
               DR. LEITCH:  Thanks.
               DR. POWERS:  When you think about fracture
     mechanics on these vessels and flaws breaking the surface,
     you mean the surface surface or do you mean they break below
     the cladding?
               MR. DICKSON:  When I say inner surface breaking, I
     mean inner surface breaking; they originate on the inner
     clads on the wetted surface.
               In the original analysis, SECY 82-465, as well as
     the integrated pressurized thermal shock, as well as the
     Yankee Rowe, normally what you would do is you would take
     the highest neutron fluence value and assume that that was
     acting everywhere, okay, and it was known that that was
     quite conservative, too.
               It's like let's put all the flaws on the inner
     surface, because we don't know anything else to do.
               Well, there usually wasn't detailed neutron
     fluence maps available at that time, so one of the things
     that, as I say, lessons learned, is to come up with a
     methodology that allows the RPV belt-line to be discretized
     into sub-regions, each with its own distinguishing
     embrittlement-related parameters, which -- therefore, this
     accommodates chemistries from the RVID database and detailed
     neutron fluence maps, because the reality is, as I'll show
     in some slides here in a moment, this section of the vessel
     may be considerably less embrittled than here.
               So, to assume that it all has the same
     embrittlement was quite conservative, and rightly so, some
     of the industry people were saying, but in the absence of a
     tool to incorporate this into the analysis, you know, you
     took the most conservative route.
               DR. WALLIS:  What does this figure show?  I don't
               MR. DICKSON:  Okay.  I'm sorry.  What this figure
     is showing here -- this is attempting -- think of this as
     the vessel rolled out, unwrapped, from zero to 360 degrees,
     where this is the core active region.
               Traditionally, the analyses have been -- when you
     talk about the belt-line region, you talk about,
     traditionally, from one foot below the core to one foot
     above the core, and so, these green regions here are meant
     to be welds, whereas the gray-like, whatever color this is,
     is plate material.
               So, what I'm saying here is that the FAVOR code
     has a methodology that allows you to break it down like
               DR. WALLIS:  Doesn't look like a very fine grid.
               MR. DICKSON:  Well, this is just for illustrative
               I mean you can break it down as fine as you -- the
     code leaves that up to the user.  That's the discretion of
     the user.
               You break it down as fine as you want to.
               Well, that's not entirely true.
               DR. SIEBER:  But that really doesn't make any
     difference as far as the geometry, because once a fracture
     starts, it will start in the most vulnerable place?
               MR. DICKSON:  Not necessarily.  We'll maybe get to
     that in a moment.
               I hate to generalize, because there's always the
     exceptions, but more often than not, the plate material is
     less embrittled than the weld material, okay?  And even if
     you say, okay, the plate has a flaw density of one-tenth or
     1/50th that of the weld material, you may still have cases
     where the less embrittled plate material drives the
     analysis, in other words contributes more to the overall
     risk of failing the vessel just by virtue of -- there's
     probably, out of 100 percent of this material, probably 99
     percent of it's plate material.
               It has a lot less density of welds, but still it -
     - flaws -- but at the end of the day, you have a lot more
               In the old way of doing these analyses, as I said,
     you would take the most limiting and sort of concentrate on
               DR. UHRIG:  This is a belt-line weld around the
     middle of the vessel right there?
               MR. DICKSON:  Yeah, that's a circumferential weld.
               DR. UHRIG:  All right.  Then the core is above the
     center --
               MR. DICKSON:  Here's the core.
               DR. UHRIG:  Okay.  I thought it was down -- about
     split equally.
               MR. DICKSON:  Well, as I say, when we talk about
     the belt-line region -- and maybe the next slide will help
     here a little bit.
               DR. UHRIG:  Okay.
               MR. DICKSON:  In fact, let's just move to the next
               This is actually some real data, and the NRC
     contractor, by the way, that's doing the neutronics
     calculations to generate the neutron fluence maps -- this
     work is being done at Brookhaven National Laboratory, and
     this actually is from one of those analyses where they sent
     me the data, and this shows, again, from zero to 360
     degrees, so if you can think back to that previous slide of
     the unwrapped vessel, this shows the asmuthal variation of
     the neutron fluence at particular axial locations.
               Now, this one here is at 72 inches from above the
     bottom of the core; in other words, kind of at mid-core.  In
     other words, this is sort of the worst location, the worst
     axial location, and you can see that -- of course, this is
               It has a periodicity of 45 degrees.  It goes down
     to 45 degrees, and then it's a mirror image of itself the
     next 45 degrees, and then it just repeats itself.  So,
     that's 90 degrees, and it just repeats itself three more
     times as you come around the 365 degrees.
               DR. WALLIS:  Why does it vary so much?
               MR. DICKSON:  You'll have to talk to someone other
     than me.
               DR. SHACK:  In the original analysis, you would
     assume the worst fluence, the worst chemistry, the worst
     thermal hydraulics, all at one location, and then you'd do a
     distribution of flaws and do the fracture mechanics on that?
               MR. DICKSON:  Right.
               DR. KRESS:  Now, you have a thermal plume coming
     down from the hot leg.
               Where is it located with respect to those high
               MR. DICKSON:  At the moment, we assume an axial
     symmetric loading, okay?
               Now, to answer your question, you know, your
     inlets are, you know, somewhere up here, and some of the
     APEX experiments are directed toward this issue.
               In other words, have the plumes dissipated by the
     time you get down to the top of the core, and certainly
     that's been the assumption so far in most of our analyses,
     and it's not just out of thin air.
               I believe that Dr. Theofanis has some publications
     that states that, for commercial, domestic PWR designs, that
     that plume pretty much has dissipated by the time the
     coolant gets down to the top of the core.
               DR. KRESS:  That was part of his remix.
               MR. DICKSON:  Yes.
               We've done some work in this in the past, and I
     think we will try to verify this one more time.  It's
     certainly our intent to -- we were involved in asking -- you
     know, in designing where the instrumentation went on the
     APEX experimental device for this very reason.
               We would like to sort of verify that one more time
     that that's the case.
               MR. MAYFIELD:  Terry, could I stop you for just a
               MR. DICKSON:  Sure.
               MR. MAYFIELD:  This is Mike Mayfield from the
               I wanted to make sure we understood the
     distribution of flaws Terry's talking about are fabrication-
     induced flaws rather than service-induced.
               So, we're talking about things that are in the
     vessel from the day it's built, as opposed to things that
     are induced by service.
               DR. SHACK:  What's the rationale for ignoring the
     clad completely?
               MR. MAYFIELD:  Well, we don't ignore it
               DR. SHACK:  I mean in the original calculation.
               MR. MAYFIELD:  Even then, it was treated as -- it
     was taken as a conservative measure.  So, it's not
     incorporated from a fracture mechanics standpoint but from
     the material toughness standpoint.
               It was included in terms of the differential
     thermal expansion you get.
               DR. SHACK:  But you put the flaws on the inner
     surface of the clad.
               MR. MAYFIELD:  Put the flaws on the inner surface,
     and you treated the metal like it was all faradic steel,
     rather than have the clad layer on it, except in the stress
     analysis, and there you picked up the differential thermal
     expansion, and I guess also, by association, the thermal
     analysis, because you pick up some thermal conductivity
     issues, but it was -- I think people didn't know exactly how
     to handle the duplex nature of the structure, so treated
     conservatively for the analyses that were done in the early
               MR. DICKSON:  Pretty much what Mike said is still
     true today.
               Certainly, the little stainless steel clad is
     factored in in the calculation of the thermal response of
     the vessel, as well as the stress, even in the K1, the
     stress intensity factors for inner surface breaking flaws,
     but we do not check flaws that reside entirely in the clad
     for a cleavage fracture, because it's stainless steel, and
     it's much more ductile than the base material.
               So, to pretend that a cleavage fracture event
     could initiate there is just denying reality.  It just isn't
     going to happen.
               Thinking back to the previous slide, this shows
     the level of the neutron fluences at one foot above the core
     and one foot -- you can see that it's decayed practically to
     zero by the time you get one foot above and one foot below,
     and this just shows the neutron fluence as a function --
     it's the axial gradient.  So, at the core flats, at zero,
     90, 180, and 270, there is the axial variation, and there it
     is at other values.
               So, the point that these slides are -- it's just
     that FAVOR has the capability to handle this kind of detail
     in your fluence map.
               DR. SIEBER:  In the righthand figure, where's the
     top of the core?
               MR. DICKSON:  Zero is at the bottom.  Zero is one
     foot below the bottom of the core.
               DR. SIEBER:  I would have expected that embedded
     control rods would have caused that shape, as opposed to --
               MR. JONES:  Excuse me.  This is Bill Jones from
     Research, NRC staff.
               That particular reactor has a fuel management
     scheme in the bottom of the core to hold the fluence down
     the vessel welds.
               So, that's why it has that maybe not
     characteristic shape, but that's why that particular axial
     looks that way.
               DR. SIEBER:  Is that burnable poisons?
               MR. JONES:  No, it is not.  I believe it's
     stainless steel.
               DR. SIEBER:  Oh, all right.
               DR. POWERS:  If we have high burn-up fuel and get
     axial offset anomalies, does it distort these distributions
               MR. DICKSON:  I'm sorry.  I didn't catch the first
     part of that.
               DR. POWERS:  If we use high burn-up fuel and we
     get axial offset anomalies, does it distort these fluences
               MR. DICKSON:  I'll defer to the neutron people on
               MR. JONES:  I'm sorry, Dr. Powers.  You'll need to
     repeat that again.
               DR. POWERS:  What I'm asking is, when people use
     very high burn-up fuels, they get a little boron absorption
     up high in the rods, and that causes a shift in the spectrum
     down to lower parts of the core.  I'm wondering if it
     changes the fluences to the vessel enough to make a
     difference in your calculations.
               MR. JONES:  Well, these calculations are pretty --
     we tried to match -- the plants we did, we tried to match
     the way the cores would burn.
               For the future, what I believe ought to be done is
     that calculations will be done matching the way the core was
     burned, so the codes would be adequate to account for that
     spectral shift and do an adequate job of calculating what
     the fluence would be at the vessel wall.
               DR. SIEBER:  I take it, just as a follow-on, that
     this is a sort of an average profile, as opposed to -- since
     axial offset occurs, even in a moderately burned core, that
     this is some kind of an average, as opposed to doing this in
     slices in time during a cycle.
               MR. JONES:  It was done as slices in time, but
     certainly it's an average between those slices, yes.
               MR. DICKSON:  If I'm not mistaken, Bill, I believe
     these asmuthal -- I believe they were about 3 degrees -- the
     increment, I believe, was like 3 degrees and the axial was
     like 6 inches or something like that, in that neighborhood,
     pretty small.
               I can tell you this:  This is quite a bookkeeping
     exercise to keep up with this.  A lot of data goes into the
               DR. KRESS:  Do you digitize that before you input
               MR. DICKSON:  Yeah.
               Of course, you know, as we know, in reality, it's
     a continuum, but as you often do, mathematically, you have
     to discretize.
               DR. SEALE:  These neutron maps were generated
     using the revised DNDFB -- what is it, 6? -- cross-sections
     to more properly distribute the energy in the heavy steel?
               MR. DICKSON:  I'll defer to Bill on that.
               MR. JONES:  I believe it's 6, but the calculations
     only go to the inside -- we're only using fluences at the
     inside of the vessel wall.
               MR. DICKSON:  Right.  I guess I should have said
     that from the outset.
               It's understood that this is the magnitude of the
     fluences on the inside, and of course, it goes without
     saying, this is at a particular time in the plant life,
     remember, back to that first graph.
               You know, we're doing an analysis to plot a point
     as a function of EFPY.  Ten years later, everything would be
     up, you know.
               Any questions, comments?
               DR. SHACK:  What do you then do to get the
     distribution through the wall if you stop the fluence
               MR. DICKSON:  We attenuate it, and we use an
     exponential to K constant of .24.
               DR. SIEBER:  What's the basis of that?
               MR. DICKSON:  There's people in this room that can
     speak to that better than I can.
               MR. LOIS:  We derived in the '80s, and it came
     from a transport calculation for the vessel.
               Later on, we found that the displacement measure
     that some people used -- it gives approximately the same
     sort of grade through the vessel.
               But the one that is in the book, in the
     regulations right now, in Regulatory Guide 199, is the decay
     through the thickness of the vessel.
               MR. DICKSON:  Actually, the data that Brookhaven
     provided actually had data through the wall, and Bill Jones
     has actually gone through the exercise and sort of verified
     that the .24 is still very much applicable, if anything
     slightly conservative.
               Is that true, Bill?
               MR. JONES:  Yeah.
               MR. DICKSON:  You found the .24 is still a valid
     number, and when it was off, it was off on the conservative
               MR. JONES:  That's a true statement.
               MR. DICKSON:  Okay.
               DR. WALLIS:  What are the units of this .24?
               MR. DICKSON:  Inches minus one.
               DR. WALLIS:  Inches.  You're in the dark ages.
               DR. WALLIS:  Even at MIT in the '50s, we used
     metric, as far as I can remember.
               Well, go on.
               MR. DICKSON:  Well, divided by 25.4, I suppose.
               DR. KRESS:  In Tennessee, we can divide.
               MR. DICKSON:  Speaking of MIT, I was at a meeting
     with Dr. Peter Griffith.  The thing about units came up, and
     he said, well, I prefer to use Christian units, and I think
     he was talking about English units.
               DR. APOSTOLAKIS:  I believe he's retired now.
               MR. DICKSON:  Okay.
               Moving along, new statistical models for enhanced
     plane strain static initiation and arrest fracture toughness
     databases have been implemented into the FAVOR code.
               Now, this shows K1C.  This is fracture initiation
     toughness, plotted as function of T minus RTNDT.
               Now, the old ASME curve that's been around since
     the early 1970s was derived from a database collected by
     EPRI, Electric Power Research Institute, and within the last
     year, year-and-a-half, at Oak Ridge, we went through and
     found how much additional data had been generated since then
     -- it's been 28 years -- and found that there have been, I
     believe, 83 additional valid points that were valid
     according to certain ASTM regulations, and I'm not going to
     get bogged down in that detail.
               So, we said, okay, we'll take the enlarged
     database and really do a rigorous statistical analysis on
               Now, Dr. Kenny Bowman and Dr. Paul Williams did
     this and came up -- they fitted this with a WIBLE
     distribution.  So, this shows your 254 points, and this
     actually shows the WIBLE distribution that they fitted to
               This bottom curve shows the -- it's actually the
     location parameter, the WIBLE location parameter, which is
     the lowest possible predicted K1C that you would ever
     predict, and this shows, you know, the 1/1,000th percentile
     and the 99.999 percentile, as well as the median, did the
     same thing for the fracture -- the crack arrest, known as
     K1A, and these are very important inputs into fracture
     analysis, into a probabilistic fracture analysis, how you
     statistically represent or, if you prefer, represent the
     uncertainty associated with the fracture data.
               DR. POWERS:  It seems to me that the 99.999
     percent line and the .001 percent line are very dependent on
     having selected WIBLE distribution, that if you'd selected
     something else, they would have greater or lesser width to
               Now, it seems to me that the WIBLE distribution
     has an empirical justification that applies in -- between
     the 90th and the 10th percentile.
               What justification is there to extrapolate it so
     far out, 99.999 percentile?
               MR. DICKSON:  I'm not sure that I can address it.
               Mark, would you like to speak?
               DR. KIRK:  Mark Kirk, NRC staff.
               Maybe we can just defer that to my presentation,
     but there are physical reasons why you would expect and can,
     in fact, demonstrate that fracture toughness data should
     have a WIBLE distribution.
               What Terry is showing is a result of a purely
     empirical analysis of the data, which happened to find that
     a WIBLE was the best fit to the data.
               Nevertheless, there's a good theoretical and
     physical justification for why the WIBLE should work, which
     I think helps to build the case that you should be using it,
     but you're absolutely correct, any model just picked from
     empiricism, out at the tails, you can have significant
               DR. KRESS:  I don't think you have intention of
     using the 99.999 for anything except the decision-making
               MR. DICKSON:  No.
               DR. KRESS:  It's just on there for illustration.
               MR. DICKSON:  Right.  It's on there for
               But certainly, the tails of the distribution can
     be quite important in these analyses sometimes.
               DR. APOSTOLAKIS:  I don't see the distribution,
     but you must have done something else somewhere else,
     because here there is K versus R minus RTNDT.
               DR. KRESS:  Is the distribution vertical?
               DR. APOSTOLAKIS:  Where is the probability?
               MR. DICKSON:  I actually have a little back-up
     slide here.
               I don't know if this will help.
               This shows the distribution at any vertical slice.
               DR. APOSTOLAKIS:  Okay.  So, R minus RTNDT is a
               MR. DICKSON:  Yes.
               DR. APOSTOLAKIS:  And you have a distribution for
     each value of that.
               MR. DICKSON:  Yes.
               DR. APOSTOLAKIS:  Okay.  So, then you take the
     99th percentile of each curve, and you plot it as a function
     of T minus RTNDT.
               MR. DICKSON:  Yes.
               DR. KRESS:  The thing about the WIBLE is it
     doesn't go out to infinity in both directions.  It has a
     lower bound and an upper bound.
               MR. DICKSON:  It's truncated on both ends.
               DR. POWERS:  Triangled.
               MR. DICKSON:  The WIBLE distribution -- I think
     it's very -- the derivation is very --
               DR. KRESS:  It has several parameters.  You could
     make it flexible to fit.
               DR. POWERS:  The triangle has the same number.
               DR. APOSTOLAKIS:  It has three parameters, and
     with three parameters, you can do a lot.
               MR. DICKSON:  The A, B, and C, the three
     parameters of this distribution, are functions of T minus
     RTNDT.  So, when you say they're a function of T, that makes
     them very much time-dependent in the analysis, because
     temperature is changing through the wall.
               RTNDT, as well as changing as a function through
     the wall, it's changing as a function of when in the life of
     the plant you're doing this analysis.
               I will just say the final report on this -- there
     was a few other considerations other than WIBLE considered,
     and the report sort of discusses why the WIBLE distribution
     was the template that was sort of chosen to put into this,
     and certainly, one of them is that the WIBLE distribution, I
     believe, was developed especially kind of for fracture-type
               DR. APOSTOLAKIS:  It's distribution of minimum
     values, is it not?
               DR. WALLIS:  This RTNDT sets the temperature
     scale, doesn't it?  If you slide horizontally, you can cover
     a big range of points.
               MR. DICKSON:  Right.
               DR. WALLIS:  So, how well do you know this RTNDT?
               MR. DICKSON:  That's one of the variables that
     there's an awful lot of sampling going on.
               DR. WALLIS:  That may be more important.  The
     uncertainty you show here looks good, but if you don't know
     your RTNDT very well --
               MR. DICKSON:  The uncertainty of the RTNDT --
     there's a lot of stuff going on inside of the analysis to
     determine that.
               I think Mark will talk about that.
               RTNDT is a function -- you know, thinking back to
     where we've talked all this about the discretization of the
     vessel, the embrittlement -- when you talk about the
     embrittlement, you're talking about the RTNDT, which is a
     function of the chemistry.
               DR. WALLIS:  But someone who looked at these
     points had to decided what RTNDT was in order to plot the
     points?  You could slide them around on that graph quite
     easily by having a different RTNDT.
               MR. DICKSON:  There is actually a formula that
     tells you what RTNDT is as a function of chemistry and
     neutron fluence.
               DR. WALLIS:  Is it a real thing?
               MR. DICKSON:  It's a real thing, but there's a
     distribution associated with that.
               DR. POWERS:  As you well know, Graham, they have
     been researching this embrittlement since the dawn of time.
               DR. WALLIS:  Yeah.  I'm just bringing out that
     there's another uncertainty; it's not just the vertical
               DR. POWERS:  And that raises the legitimate
     question of these fitting processes.
               If you fit one uncertain parameter versus another
     uncertain parameter and you appeal to these squares, I will
     throw things at you, because that's just not right.
               Similarly, when I look at the calculations of the
     parameters of the WIBLE distribution that are reported in
     this document we've got, I am struck by -- you end up using
     the database to find your means and your shape parameters
     and shift parameters, at least when you calculate variances
     of an ordinary database and you use the mean of the database
     for the calculating of the variance, you introduce bias that
     you have to correct for.
               Don't you have to correct for bias when you do
     these formulae using the database to define your parameters?
               MR. DICKSON:  Mark?
               You're talking about the embrittlement correlation
               DR. KIRK:  I'm not sure I completely understood
     that question.  In fact, it would be fair to say I didn't.
               But let me just address the gentleman's comment
     over here.
               In Terry's model, the uncertainty -- he's just
     looking at part of it here.
               Certainly, the uncertainty in the vertical axis is
     considered, and this is the way it's currently been
               Equally, we've spent an awful lot of time -- and I
     suppose you could say that's the main focus of the next
     presentation -- in characterizing the RTNDT uncertainty. 
     That's a major component, as well.
               With regards to the bias question, I'll invite Dr.
     Powers to ask it to me again, but I would just like to
     mention in passing that Terry's presenting here a purely
     empirical model, just to fit the data, with absolutely no
     appeal to any underlying physical theory.
               In the presentation that I'll be making, which
     concerns the uncertainty characterization of the K1C RTDNT
     uncertainty model, we do appeal to, you know, physical
     rationale and the underlying basis for cleavage fracture to
     help us get at what these distributions should be, not just
     argued from the data.
               So, I realize that's not a direct answer to your
     question, but I think that might help to put aside some of
     the questions concerning lower tails and bias and things of
     that nature.
               DR. KRESS:  I'd like to point out that these are
     data taken from specimens, well characterized in terms of
     RTNDT.  The chemistry is well taken.
               So, there's a relatively narrow uncertainty, but
     when you get ready to decide what RTNDT is for the vessel
     itself, so that you can select from the uncertainty there,
     it's a much different uncertainty.
               DR. WALLIS:  That's a very helpful comment.  Thank
               DR. POWERS:  All except for the fact that -- yeah,
     they're well characterized as far as chemistry and things
     like that.
               There is still an uncertainty in the RTNDT which
     is non-trivial.
               DR. KRESS:  Oh, absolutely.  It's non-trivial.
               DR. WALLIS:  How big is it?
               DR. APOSTOLAKIS:  If you know the RTNDT, there is
     a spread of values for K.  That's all he's telling you.
               DR. KRESS:  I think Dana has a valid point, that
     how you get those variances and how they're distributed does
     depend on both uncertainties.
               DR. POWERS:  Now, let's accept that we know RTNDT
     exactly and we look at a vertical slice, a set of your data
     points for a particular one, and you want to calculate the
     parameters of the WIBLE distribution.  Okay.
               When you set out to do that, you need a mean, you
     need something like a variance, and you need something like
     a skew, because you've got three parameters you've got to
     find, so you've got to take three moments of the
     distribution to fit this thing.
               When I use the data to calculate those things, if
     I want to calculate the variance of a data set and I want to
     take the moment around the mean, right, and I calculate that
     mean from the data set and don't do something, I will
     calculate a biased estimate of the variance.
               It makes sense, because I've taken the data to
     calculate the mean, and when I calculate the skew using the
     mean of the data set to calculate the skew, I get a much
     bigger bias, even, because I'm taking the third power of it.
               I didn't see, in your write-up in here, how you
     accounted for that bias.
               MR. DICKSON:  Which write-up are you talking
               DR. POWERS:  It was a document given to us that
     goes through how you calculate the parameters of the
               MR. DICKSON:  Okay.
               DR. KRESS:  I think that came from the University
     of Maryland, didn't it, that particular document.
               DR. POWERS:  So, they're really biased.  They're
     biased clear over to the east coast.
               DR. KRESS:  Yeah.  Doesn't even have an accent.
               MR. DICKSON:  There is a document that came from
     Oak Ridge.  I can't answer your question.
               DR. SHACK:  I would assume the statisticians took
     that into account.
               DR. WALLIS:  There's another question, too.  In a
     finite number of data points, you start talking about
               Now, you need a certain number of data points
     before you can even, with any precision, talk about that
     sort of a number.
               MR. DICKSON:  Point well taken.
               I believe that document talks about needing a
     minimum of 250 to do the particular analysis that they did.
               DR. WALLIS:  And there's just about that here?
               MR. DICKSON:  Two hundred and fifty-four.
               DR. WALLIS:  So, just about enough to reach a .01-
     percent conclusion.
               DR. APOSTOLAKIS:  What is the message you want to
     leave us with?
               MR. DICKSON:  The only message here is that we
     consider this an improvement over what we have been doing
     for years.
               DR. APOSTOLAKIS:  What were you doing before?
               MR. DICKSON:  What we were doing for years was
     taking the ASME curve and saying, by definition, it is the
     mean minus 2 sigma, where 1 sigma was 15 percent of the
               DR. APOSTOLAKIS:  And the ASME curve was a curve
     like the one there?
               MR. DICKSON:  The ASME curve actually was not a
     lower-bound curve.  It was actually a lower-bound curve to
     about 88 percent of the data.
               DR. APOSTOLAKIS:  Okay.  But it was a curve like
               MR. DICKSON:  Yeah.
               DR. APOSTOLAKIS:  And now the new thing is the
               MR. DICKSON:  Yes.
               DR. APOSTOLAKIS:  Through the WIBLE distribution.
               MR. DICKSON:  Yeah.  So, we feel like this is a
     more rigorous statistical model than what we had.
               DR. APOSTOLAKIS:  And why do you have this spread? 
     Why do you have uncertainty?  Dr. Kress said earlier that
     you had well-defined specimens.
               DR. KRESS:  Yeah.  Those are not specimens, but
     even those, whether or not K1C is an accurate
     phenomenological description of cleavage fracture is
     questionable.  It depends on the geometry of the crack.
               DR. APOSTOLAKIS:  So, if I take 100 of those, I
     will still have a spread.
               DR. KIRK:  The thing that lights off cleavage
     fracture is the distribution of cleavage initiation sites or
     generally carbides in front of the crack, and you've got big
     carbides, you've got small carbides, and you've got a
     massive stress gradient.
               So, it's just a statistical process that sometimes
     they light off early and sometimes they light off late.
               DR. APOSTOLAKIS:  So, this kind of an uncertainty
     in understanding will be part of the bigger picture later
     when you do the Monte Carlo study.
               MR. DICKSON:  Yes, absolutely.
               DR. WALLIS:  These specimens came from cutting up
     a pressure vessel?
               MR. DICKSON:  No, not necessarily.  Just from like
     pressure vessel-type steel, A508, A533.
               DR. WALLIS:  That's not the same.  It's been
     treated differently than a pressure vessel.
               MR. MAYFIELD:  This is Mike Mayfield from the
               The plate samples that were tested came from heats
     of plate that were thermally treated to be identical to the
     thermal treatments of reactor pressure vessels.  They were
     purchased from the mills that made the plates that went into
     reactor pressure vessels, to the same specifications that
     were used, and that thermal treatments were imposed to be
               The weld materials that have been tested were used
     -- were fabricated and heat treated using procedures that
     were as close to those used in fabricating reactor pressure
     vessels as was practical given that we're welding flat plate
     rather than cylindrical shell segments.
               So, there was some significant effort made to
     duplicate the materials that would have actually been used
     in fabricating reactor pressure vessels.
               As we have had the opportunity to cut up some
     pressure vessels, we have also been able to test materials
     from actual reactor pressure vessels that never went into
     service, and those materials fit in with these data quite
               So, there's good reason to believe that the data
     we have are representative of the materials that were in-
               DR. KRESS:  The fluence was provided over a
     shorter period of time by putting them in a high flux
     intensity area.
               MR. MAYFIELD:  The irradiation aspect of it from
     the test reactors introduces some uncertainty in the
     characterization, but the un-irradiated properties, the
     materials themselves are quite representative.
               DR. POWERS:  Mike, have you ever found evidence of
     a flux dependence?
               MR. MAYFIELD:  Yes.  At the very high fluxes, we
     used to do test reactor irradiations where the capsules were
     put in core, and the flux there is high enough that we've
     subsequently stopped doing that and we go out to the core
               I've forgotten the numbers, but the in-core fluxes
     were high enough that the theoreticians have shown us that
     that was the wrong thing to do.
               We've backed away from that, do core edge
     irradiations now, so we can still get some acceleration, but
     it's not so high as to be of concern.
               DR. SIEBER:  Is there a dependency on the energy
     level of the fluence?
               MR. MAYFIELD:  Yes, but for the range of fluences
     -- I'm sorry -- for the energy spectra that we typically see
     in power reactors, there is not such a large spectral
     difference for it to be an issue, and as long as we capture
     the neutron flux above 1 MEV, that's where the index -- it's
     a convenient index to use, and that's where the modeling has
     been done.
               DR. SIEBER:  But through-wall, there should be a
     lot of attenuation, so there should be a variation in RTNDT
     through the wall.
               MR. MAYFIELD:  There is absolutely a lowering of
     the RTNDT as you go through-wall, and that is accounted for
     in Terry's analyses through this attenuation parameter.  So,
     he's attenuating the fluence as he goes through-wall and
     then can calculate an RTNDT, an adjusted RTNDT as a position
     of -- as a function of position.
               DR. SHACK:  But you don't, for example, take into
     account the spectral change as you go through the wall.
               MR. MAYFIELD:  No.
               DR. SIEBER:  Is that important?
               MR. MAYFIELD:  Over the ranges we're talking
     about, it's not a factor.
               DR. SIEBER:  All right.  Thank you.
               DR. KIRK:  This is Mark Kirk from the staff.
               It might also be helpful to point out in passing
     that a lot of the questions that were just asked in the past
     few minutes regarding irradiation effects and materials
     effects all manifest themselves in a change in the index
     temperature parameter, RTNDT.
               They do not manifest themselves at all in a change
     in the vertical scatter.
               So, those uncertainties, material dependent
     differences and so on, are there and are considered, but
     they're taken up in a different part of the analysis.
               DR. SIEBER:  Thank you.
               MR. DICKSON:  Okay.
               This is just to maybe graphically illustrate, you
     know, an inner surface-breaking flaw as well as the embedded
     flaw, and the FAVOR code -- traditionally, the older FAVOR
     codes only did surface-breaking flaws, and as I said
     earlier, when they actually start doing destructive
     examination, non-destructive, destructive examination, they
     hadn't found any of these, but they found a ton of these.
               So, the FAVOR code -- and the -- actually, the
     mechanics that you have to do to deal with these flaws is
     dramatically different than the mechanics that you have to
     do to deal with these flaws, but the FAVOR code now will
     deal with either/or, you know, within the same analysis. 
     You can have inner surface-breaking and/or embedded flaws in
     the same analysis.
               DR. WALLIS:  There's an infinite variety to flaws,
     all kinds of flaws.
               MR. DICKSON:  Oh, yeah.
               DR. WALLIS:  Once you have a flaw, its shape and
     everything is -- it's like a snowflake, isn't it?  I mean
     they're all different.
               MR. DICKSON:  Yeah, sort of.
               DR. WALLIS:  But somehow you can treat them all --
               MR. DICKSON:  No.
               Within the analysis -- I mean the things that get
     sampled within the analysis is the -- what we call the
     through-wall depth -- in other words, how deep is the flaw,
     how long is the flaw, where is the flaw, is this type flaw
     or this type flaw, where through the wall.
               All of those things are sampled, and the functions
     that they're sampled from are the characterization data that
     has been found in PVRUF, as well as the work that's being
     done here at the NRC to characterize the flaws.
               DR. WALLIS:  Are they typically very small things?
               MR. DICKSON:  Yes, for the most part.
               DR. WALLIS:  What sort of size?
               MR. DICKSON:  Let me answer it this way.
               I think the largest flaw that has been found in
     this actual effort of going and cutting this vessel material
     up, the largest flaw that's been found is an embedded flaw
     that's 17 millimeters through-wall extent.  How long it was,
     I actually don't know.
               We found a whole lot more flaws than was used, not
     so much smaller, they were just in the wall, and from a
     fracture mechanics point of view, this flaw was a whole lot
     more benign than this flaw.
               DR. SIEBER:  On the other hand, there is a
     critical flaw size where propagation occurs, which is a
     function of RTNDT.
               MR. DICKSON:  Well, it's a function of everything. 
     It's a function of the embrittlement, it's a function of the
               DR. SIEBER:  Right.
               MS. JACKSON:  This is Debbie Jackson from the NRC
               To date, the largest flaw that PNNL has found,
     like Terry said, was 17 millimeters, and it was
     approximately, I believe, 5 millimeters long, but they found
     some other flaws in other vessel material.
               We're doing Shoreham, River Bend, and Hope Creek,
     and we found some flaws, but they haven't been validated,
     that are a little longer than 17 millimeters.
               MR. DICKSON:  So, if you remember that box that I
     showed, all the things feeding into the middle, one of them
     was flaw characterization, and that's a pretty general term. 
     It's how many flaws, what characteristics, you know, how
     long, how deep, where in the wall, all of that gets into the
               DR. SIEBER:  I presume that the flaws that have
     been found are not uniformly distributed through the wall.
               MR. DICKSON:  Debbie can speak to that better than
     I can.
               MS. JACKSON:  This is Debbie Jackson again.
               The majority of the flaws have been found in weld
     repair regions, regions of the welds that have been
     repaired, and we're presently doing some additional exams on
     the base metal.
               MR. DICKSON:  I believe you asked are they
     uniformly distributed through the thickness of the wall --
               DR. SIEBER:  Yes.
               MR. DICKSON:  -- and I believe the answer is
               MS. JACKSON:  Right, where a weld repair is
     located, right.
               There are a lot of smaller flaws.
               MR. DICKSON:  Okay.
               I'm going to have one slide here, just a little
     bit about the structure of FAVOR, which I don't know if
     that's real important here, but we'll talk about it anyway.
               When we talk about the FAVOR code, I don't know
     what people conjure up in their mind.
               This code -- the current code that we're working
     on consists of three separate, independent modules.
               The first one is -- and I'll talk a little bit
     more about these on subsequent slides.  The first one is a
     load generator, and the input -- this first level, the blue
     boxes up here, is input data.  These yellow boxes -- they're
     the actual executable modules.  So, when you talk about
     FAVOR, this is what you're talking about, and this last row
     here is what comes out of each of the modules.
               So, your first one is your load generator.  That's
     where you actually put in your thermal hydraulic boundary
     conditions from, typically, like this output from the RELAP
     code, and of course, you input your thermal elastic material
     properties for the clad base material, elasticity, thermal
     conductivity, coefficient of expansion, on and on, the RPV
               Now, a transient description -- typically, it will
     come in the form of three time histories, the pressure time
     history that's acting on the inner surface of the vessel,
     the coolant temperature time history, and the convective
     heat transfer coefficient.
               Now, each one of these, you can input 1,000 time
     history pairs for each of the three, and typically, I think
     the people from the thermal hydraulics branch told me that,
     for Oconnee, there's 27 transients.  So, you're talking
     about 81,000 time history pairs.
               So, again, big bookkeeping exercise, as well as
     doing the mechanics.
               Out of this comes the through-wall temperature
     gradient as a function of location and time, stresses, and -
               DR. WALLIS:  I think it would be XYZT.
               MR. DICKSON:  This is an axial symmetric.  It's a
     one-dimensional analysis through the wall.
               DR. WALLIS:  That's all?
               MR. DICKSON:  Yeah.  Since we are assuming that
     the thermal hydraulic boundary conditions are axial
     symmetric, you only need a one-dimensional analysis.
               DR. WALLIS:  You don't need a vertical coordinate,
               MR. DICKSON:  No.
               So, you run this module, and out of that comes a
     file that contains all of this.
               DR. WALLIS:  If the thermal hydraulic tests showed
     plumes, you'd have to change this into something different.
               MR. DICKSON:  Yeah, if it showed that they were
     very significant.
               DR. WALLIS:  Now, is FAVOR able to do that?
               MR. DICKSON:  We would have to re-do this aspect
     of it, not the whole thing.
               DR. WALLIS:  You'd have to re-do that part of it.
               MR. DICKSON:  We would have to do that part of it. 
     It would not be trivial.
               Essentially, instead of doing a one-dimensional
     finite element analysis, we would have to do a three-
     dimensional finite element analysis.
               DR. WALLIS:  In order to be cautious, you might
     want to do an XYT one just to see what happened if you did
     have a plume, even if you had reason to hope it wasn't
               MR. DICKSON:  We have some publications on that. 
     Actually -- I want to digress too far on this.
               In the first two versions of FAVOR that came out,
     the '94 and '95 versions, there actually was a option in
     there to include this, and in a conservative sort of way, in
     a bounding sort of way.
               So, we really did get in and do a lot of analysis,
     but when we went to this next version, the decision was made
     early on that we're not going to do that.
               Now, if APEX shows that this could be very
     significant, we may have to go back and do that.
               Anyway, load-generator --
               MR. MAYFIELD:  This is Mike Mayfield from the
               This was something that we had some considerable
     debate on, and we recognize there is an element of gambling
     here that -- on what the APEX results will show us, but when
     we looked at the other analyses that had been done, we took
     a deep breath and said let's move forward, but at the same
     time perform the experiments to try and sort it out, and if
     we guessed wrong, then we're going to have to do some
     additional work, and we'll have to deal with the schedule
     impact, but we went into it recognizing there was an element
     of risk that we would have guessed wrong.
               DR. WALLIS:  Well, computationally, doing XYZT
     isn't that complicated, just you've got to do so many runs? 
     Is that what it is?
               MR. DICKSON:  No.  I mean what you're talking
     about -- if these plumes, which are multi-dimensional by
     nature -- if they are significant, you can no longer do a
     one-dimensional axial symmetric analysis.
               DR. WALLIS:  That's right.
               MR. DICKSON:  And as I'll talk about in a moment,
     we do finite element analyses.
               So, writing a finite element code --
               DR. WALLIS:  It's not that difficult to do.
               MR. DICKSON:  Well, it's like everything here. 
     The devil is in the details.  Writing a three-dimensional
     finite element code is not a trivial -- it's not something
     you do in an afternoon.
               MR. MAYFIELD:  I think the other thing we've
     talked about is we would have to look at some -- perhaps
     some simplifying approaches to the stress analysis, which
     would take us to a little bit different structure in this
     particular module.
               MR. DICKSON:  Yeah, it might.
               MR. MAYFIELD:  So, we could very well end up
     having to do some things off-line and then load in stress
     information, but there are other approaches that could be
     considered rather than the approach that Terry has been able
     to make use of given the one-dimensional nature of the
               But your questions are -- we agree completely. 
     Some of us that were involved in making that decision are
     waiting quite anxiously to see how close we were or weren't
     to being right.
               MR. DICKSON:  Again, I'll refer back to some of
     Theofanis' publications saying that, for U.S. domestic
     commercial designs, it should not be significant.  So, I
     guess we're putting some faith in that.
               Then here's the probabilistic fracture mechanics
     module, and the input to that is all of the flaw data,
     which, again, tells you the densities, number of flaws, as
     well as the size and location.
               Also coming into this PFM module is the
     embrittlement data.  You remember that, where I showed the
     vessel rolled out from zero to 360 degrees.  Each one of
     those little regions has a particular chemistry and neutron
     fluence, something that gives you enough information to
     calculate the RTNDT of each one of those regions.
               And also, of course, the loads from the load
     generator -- all this obviously is input into here.
               DR. WALLIS:  The belt-line embrittlement isn't
     one-dimensional either.
               MR. DICKSON:  No.
               DR. SIEBER:  Well, you aren't really talking about
     belt-line.  You're talking about the whole show, right?
               MR. DICKSON:  We're talking about the entire belt-
     line region.  It's two-dimensional.
               DR. WALLIS:  How do you do that two-dimensionally
     and do the stress one-dimensionally?
               MR. DICKSON:  Well, it's just the same stresses
     are assumed.
               DR. WALLIS:  Okay.  I guess it is independent. 
     It's all independent.
               MR. DICKSON:  Yes.
               DR. WALLIS:  Yeah, I guess that's right.
               MR. DICKSON:  The assumption is that the stress
     profile acting through the wall here is the same as it is
     anywhere else, and the temperature profile through the wall,
     which is a totally independent assumption from what the
     embrittlement is in any location.
               DR. WALLIS:  So, embrittlement doesn't change the
               MR. DICKSON:  No.  The modulus of elasticity?  No.
               DR. WALLIS:  For the stress calculations.
               MR. DICKSON:  Embrittlement changes the yield
     strength a little bit, but basically, we're doing --
               DR. WALLIS:  No, it doesn't change it.
               MR. DICKSON:  No, it doesn't change the modulus,
     no.  It changes the yield stress some, but we're doing a
     linear elastic analyses here.  We're not doing elastic
     plastic analyses.
               DR. WALLIS:  Never get that close.  Never get
     close to that, do you?
               MR. DICKSON:  No, there are cases where you can
     get into plasticity, but the assumption traditionally has
     been that a LEFM analysis is conservative, as opposed to
     where you actually consider the plasticity.
               DR. WALLIS:  Once it begins to fail.
               MR. DICKSON:  I think we're going to have to take
     a closer look at that when it comes to embedded flaws.  You
     know, for surface-breaking flaws, I think, traditionally,
     it's been shown that, if you do a LEFM analysis, linear
     elastic fracture mechanics analysis, that that is
     conservative with regard to surface-breaking flaws, but for
     embedded flaws, I don't think we at Oak Ridge are yet
     convinced that that is necessarily the case.
               MR. MAYFIELD:  Terry, this is Mike Mayfield again.
               For surface-breaking flaws, we've shown it to be
               MR. DICKSON:  Yeah.
               MR. MAYFIELD:  It correctly characterizes the
     phenomenology for a highly-embrittled pressure vessel at the
     surface, and if you have a less-embrittled pressure vessel
     such that you actually had to go to an elastic plastic
     analysis, by and large pressurized thermal shock wouldn't be
     an issue.
               So, what you're really talking about is the set of
     conditions that gets you to a linear elastic fracture
               So, it's not just that it's conservative.  It is,
     in fact, accurate for the surface-breaking flaws.
               As Terry says, for the embedded flaws, it gets a
     little more interesting, but by and large, if it's an
     elastic plastic initiation, it's not much of a challenge
     from pressurized thermal shock.
               MR. DICKSON:  So, out of this PFM module comes
     actual distributions for the conditional probability of
     initiation -- in other words, the conditional probability
     that you initiate a flaw in cleavage fracture, the
     conditional probability of failure.
               In other words, just because you initiate a flaw
     does not necessarily mean it's going to propagate all the
     way through the wall.
               There's another analysis that has to be done to
     determine whether that flaw that initiates actually goes
     through and makes a hole in the side of the vessel.
               So, for each transient, what comes out of here is
     the distributions for the conditional probability of
     initiation, conditional probability of failure for each
               Now, this third box over here -- this is actually
     the one I'm working on right now, as we speak, as far as the
     development aspect.
               The input into this, of course, is these
     distributions for the conditional probability of initiation,
     conditional probability of failure, and I keep using the
     word conditional.
               It's conditional that the transient occurred,
     okay, but input here is actually the distribution of the
     initiating frequencies, initiating frequency being, you
     know, how often this transient occurred.
               DR. APOSTOLAKIS:  So, the transient does not
     appear for the first time there.
               MR. DICKSON:  Yeah.  The transient as a thermal
     hydraulic boundary condition that actually occurs is back
     over here.
               You go ahead and calculate, if this transient
     happens, here's the result.
               Here's the actual probability of the transient
     even happening to begin with.
               And I'll talk briefly in a couple of slides here
     about how these distributions and these distributions get
     together to form this final distribution, which is the
     frequency of RPV fracture, or in other words, the crack
     initiation, the frequency that you fracture the vessel and
     also the frequency that you fail the vessel.
               DR. WALLIS:  These transients you start with are
     some sort of design basis accident?
               DR. SIEBER:  Not necessarily.
               MR. DICKSON:  Not necessarily.  I don't feel like
     I'm probably the person to speak to the thermal hydraulics
     aspect of it.
               MR. CUNNINGHAM:  This is Mark Cunningham from the
               You don't start from the set of traditional design
     basis accidents that are in Chapter 15 and that sort of
     thing.  You start from -- you look at information on what
     types of transients in the plants can cause issues of
               DR. WALLIS:  Okay.
               MR. CUNNINGHAM:  Over-cooling, pressurization. 
     So, it can be small-break LOCAs with operator actions, it
     can be a variety of things like that, but it doesn't start
     from the Chapter 15 analysis.
               DR. KRESS:  It's a selection from the normal PRA
     of those sequences that might be important.
               MR. CUNNINGHAM:  Yeah.  Somebody when we talk
     about the PRA, you'll see we look at PRA information, we
     look at operational experience and that sort of thing, to
     see what's going on and what could cause these types of
     situations in the plant, in the vessel.
               DR. SHACK:  Now, when you do the running crack,
     it's running one-dimensionally, right?  You're not trying to
     also do a length-growth of this thing, or are you?
               MR. DICKSON:  No.
               The assumption is that -- and this is consistent
     with experimental results that we've observed through the
     years at Oak Ridge, is that an initiated flaw runs long
     before it runs deep, propagates along the length of the
     vessel before it propagates through the wall.
               So, you could start with -- you could thermally
     shock this vessel right here with this flaw, and this flaw
     is going to want to extend this way before it goes through
     the wall, and also, with this flaw, the assumption is that
     this flaw is going to propagate in toward the inner surface,
     because it's propagating into a region of higher
     embrittlement, as well as higher stress, so it's got a
     higher load and a lower material resistance in this
               So, the assumption -- you check it at the inner
     crack tip, you check it for initiation, you know, and if it
     initiates, you assume that it breaks through and becomes
               So, an initiated flaw is a long flaw, becomes a
     long flaw.
               Then the question is, now, do you, the long flaw,
     do you propagate through the wall?
               DR. POWERS:  Do you get a damage accumulation in
     these processes?  That is, if I do a little bit of insult,
     little bit of over-cooling, I get my cracks bigger and
     bigger and bigger?
               MR. DICKSON:  No.  No, there's no -- I think Mike
     Mayfield mentioned a moment ago, there's no service-induced
     crack growth here.
               If the flaw is predicted to crack, it's predicted
     to be a major -- I mean it runs long and breaks through.
               DR. KRESS:  You don't use the time-dependent flaw
     distribution, is what you're saying.
               MR. DICKSON:  Yeah.  There is no time-dependent
     flaw distribution.
               DR. POWERS:  I'm asking about the phenomenology
               DR. SIEBER:  Well it would seem to me that you can
     initiate a crack through some event that arrests, and then,
     as the vessel runs again, you continue to embrittle it so
     that the next event you have, it can go even further, and I
     guess that's what in-service vessel inspection's all about,
     to try to find those situations, if you can.
               MR. MAYFIELD:  This is Mike Mayfield from the
               I think the notion is that if, in fact, you had a
     substantive over-cooling, you have to go back and inspect
     the vessel.
               So, there would be -- you would intend to go look
     for anything that you might have popped in.  There's always,
     of course -- in-service inspection or non-destructive
     examination is not 100-percent.
               So, there is the potential that you could have a
     pop-in that you would miss, but -- it's not inconceivable,
     but we think that the prospect of that is not all that high. 
     First of all there aren't that many highly-embrittled
     vessels out there.
               But I think your point is correct.
               DR. SIEBER:  If you were to do an inspection, what
     is the minimum flaw size you can characterize?  Could you
     find these 17-millimeter flaws?
               MR. MAYFIELD:  Yes.  If you know to go look for
               So, it gets to be a question of what would you ask
     the inspectors to do?  You probably would not send them out
     and ask them to do a traditional Section 11 in-service
               So, it's a question of what you would actually --
     what a licensee would actually ask their in-service
     inspection team to go look for.
               I guess I should also say that the
     characterization, the flaw characterizations that Terry
     talked about and that Debbie Jackson mentioned started out
     with ultrasonic examination.
               It's some fairly sophisticated ultrasonic
     examination, but that's where it starts, followed up by
     destructive examination.
               DR. SHACK:  Now, presumably you have done the
     fracture -- the fatigue flaw growth to convince yourself
     that these things really don't grow by fatigue.
               MR. MAYFIELD:  This is something that a lot of us
     used to make our living trying to do, and this comes up --
     everybody that looks at PTS wants to go do a fatigue crack
     growth evaluation, and they consistently come back with the
     same answer, that it's a no-never mind, and the reason is
     the normal operating stresses are so low, even if you had a
     surface-breaking flaw, the cyclic component of normal
     operation is so low and these flaw sizes are so small, you
     get just no growth.
               MR. DICKSON:  I guess, before I move on to the
     next slide, the main purpose of this slide -- a main purpose
     of this slide is that the bottom line, what comes out of --
     after you run all three of these modules of FAVOR, the
     bottom line is a frequency, a distribution of the frequency
     of how often you fail the vessel, and that distribution has
     a mean value which you would then go plot on that curve that
     I showed early in the presentation, you know, let's say that
     we're doing this at 32EFPY.
               So, each time in the life of the plant has a
     distribution of failure associated with it, which has a mean
     value, which you would go back and plot on the curve I
     showed, and of course, there would be a distribution.
               DR. APOSTOLAKIS:  It's not clear a priori that it
     will work with a mean value.  I mean it depends how wide the
     distribution is.
               MR. DICKSON:  Right.
               DR. APOSTOLAKIS:  So, I understand we have a
     subcommittee meeting coming up, and we'll talk about how all
     these uncertainty calculations are being done.  Is that
               MR. DICKSON:  Yes.
               MR. HACKETT:  Professor, I think we were talking
     in December, most likely.
               This is Ed Hackett from the staff.
               DR. APOSTOLAKIS:  I thought it was November 16th. 
     That's not true anymore?
               MR. HACKETT:  Terry, this is Ed Hackett.  I was
     going to suggest, in the interest of time, because we're
     running a bit behind here -- I think you made a good
     jumping-off point to go to your slide 19, the second-to-
     last.  Why don't we just go to that?
               DR. WALLIS:  I have a point on number 17.  You are
     actually looking at uncertainty in the thermal hydraulics by
     varying some of the parameters in that in a statistical
     random way or something?
               MR. DICKSON:  Do you want me to jump to the last
               The answer is yes, this is an attempt to capture
     the uncertainty associated with the thermal hydraulics.
               DR. WALLIS:  And you're getting some good
     consulting on that or something from somebody?
               MR. DICKSON:  This is an interface between the PRA
     people and the thermal hydraulics people.
               DR. WALLIS:  They have a good way of estimating
     uncertainties in the thermal hydraulics?
               MR. CUNNINGHAM:  In the PRA part of this, this has
     been one of the challenges, how do you bring that together. 
     This will be one of the subjects we'll talk about, I guess,
     at the December meeting.
               DR. WALLIS:  I think it's important to do.  I just
     wondered if anybody knew how to do it well.
               MR. CUNNINGHAM:  We'll see, but we think we can do
     it with a fair amount of comfort on our part.
               DR. WALLIS:  Let's hear about that at some point.
               MR. CUNNINGHAM:  Yes, in December.
               MR. DICKSON:  Well, this is just an attempt to
     show that we have major categories of transient.
               Maybe this is a LOCA, and within that LOCA, there
     are several variants of that, and each one of them has a
     certain frequency of occurrence, a distribution of the
     frequency of occurrence, which we talked about.
               All this feeds into this load generator of FAVOR,
     which performs a one-dimensional, axial symmetric, finite
     element thermal analysis as well as a finite element stress
     analysis, and as I showed a moment ago, the output from that
     is a lot of temperatures, stresses, and stress intensity
     factors, and I'll try to be real brief here.  I've got two
     more slides.
               This is, again, talking about this probabilistic
     fracture mechanics module.
               Again, this is redundant.  The input data coming
     into this is all the flaw data, the embrittlement map, as
     well as some uncertainty, 1 sigma values, as well as the
     loads, and what comes out of that is an array -- this is for
     -- I call this PFMI.  This is for initiation, as opposed to
     PFMF, which is conditional probability of failure.
               So, what comes out this PFM module is like a two-
     dimensional array for so many vessels and so many
     transients, okay?
               Now, I know this maybe is a little bit not clear
     at this point, but let's just say that each entry in this
     matrix is the conditional probability that that vessel
     initiated or failed when subjected to that particular
     transient, okay?
               So, we end up with these two arrays.
               Now, it would be another whole presentation to
     talk about the actual details of what goes into calculating
     that the -- the probability that that vessel fractured when
     subjected to that transient, and we're not going to go there
     in this presentation, because we don't have time.
               So, this is the last slide, but I think that is
     one of the things, when we get together for this meeting of
     uncertainty, that we will talk in great detail about, and
     this is the post-process, just trying to graphically show
     how we integrate the uncertainties of the transient
     initiating frequencies with the PFMI and PFM arrays that I
     just showed, which is what comes out of your probabilistic
     fracture mechanics analysis, to generate distributions for
     the frequency of RPV fracture and failure.
               This is an attempt to show the distribution of the
     initiating frequency for transient one-two-dot-dot-N that
     feeds in here, as well as the arrays that were calculated
     from the probabilistic analysis.
               And this is showing -- here's the bottom-line
     answer that comes out of this, which is the frequency of RPV
     fracture, and what you do to actually generate this
     distribution is, for each vessel -- keep in mind, we're
     going maybe 100,000 or a million vessels, this is a Monte
     Carlo process, so we sample each one of these distributions
     to get a frequency for each of the transients, and then we
     combine that with the results of the PFM analysis where the
     conditional probability of failure for a vessel is the
     summation of the products of the initiating frequency with
     the conditional probability of fracture for that vessel.
               So, what you're multiplying here is events per
     year, failure per event, which is failures per year.  That's
     what you end up with.
               I mean this looks kind of difficult but it's
     really just pretty straightforward.
               At the end of the day, you end up -- let's say you
     end up with a million of these values, which you then sort
     and then generate this distribution, which has some mean
     value and some level of uncertainty about it.
               So, that's the bottom line.
               DR. WALLIS:  This is all what you intend to do. 
     This is all the specifications for FAVOR.
               MR. DICKSON:  Yeah, this is the specifications,
     and basically, the first two modules are pretty complete,
     although there's a lot of details that are still up for
     grabs.  This third module, I'm developing that right now.
               DR. UHRIG:  What's the abscissa here in the lower
               MR. DICKSON:  This would be frequency of failure,
     the frequency of the frequency.  So, it's a histogram.
               It's the frequency of the frequency.  Did you
     follow that?
               I mean what we're talking about here is the
     frequency of RPV fracture, so many fractures per year, and
     then a histogram, by its very definition, is the relative
     frequency or, if you wish, the density of something.  So,
     you can think of it as the relative frequency or the density
     of the frequency of fracture.
               Is that clear?  It's a lot of words.  I can see
     why it might not be.
               This is just a histogram or a distribution of the
     frequency of vessel failure.
               DR. WALLIS:  What is your specification for how
     long it takes to run this?
               MR. DICKSON:  Well, it takes a while.
               DR. WALLIS:  It's not much use if it takes a year.
               MR. DICKSON:  No, no, it doesn't take a year, but
     to run like 25 transients for many 100,000 vessels, on a
     machine that I have in my office, which is 533 megahertz --
     it was the newest machine a year ago -- it's an overnight
               DR. WALLIS:  That's not so bad.
               MR. MAYFIELD:  Dr. Wallis, in comparison to the
     big system-level thermal hydraulics codes, this thing is a
     blink of the eye.
               It runs very quickly relative to some of the large
               DR. WALLIS:  So, the inputs you're getting from
     RELAP will be the bottleneck, then.
               MR. DICKSON:  Yeah, I guess so.
               DR. WALLIS:  So, maybe we should work on the
               DR. POWERS:  You began this session by saying to
     us that thermal hydraulics had not improved yet.  Let's work
     on the part that's improved, and that's the fracture
               MR. MAYFIELD:  If we don't have any other
     questions for Terry, I guess I would -- we have a second
     presentation, looking at the materials, and I guess the
     question I would pose to the committee is, do you want us to
     try and stay within the time slot, or do you want Dr. Kirk
     to give you the presentation, or some variant on that?
               DR. POWERS:  I believe you have till 20 of the
     hour.  If he does what he professes to do, persuade me that
     the WIBLE distribution has a theoretical foundation, he has
     till midnight.
               MR. MAYFIELD:  Let me assure you, sir, there will
     be those of us that get the hook long before midnight.
               Mark, why don't you try and stay within the time
               DR. KIRK:  I'll try to keep it snappy.
               Well, this is just one part of many parts of
     Terry's overall calculation, and the goal in this effort is
     to characterize toughness using all available data,
     information, in a way that is PRA-consistent, and what I
     mean by that, before I get shot down by the whole committee,
     is a best estimate process, and that, as you'll see here, is
     actually quite a remarkable change from where we've been
               The process that we've gone through is, first off,
     to start with a data evaluation, and we asked Terry's
     colleagues at Oak Ridge to assemble all of the available
     valid -- and by valid, I mean linear elastic valid K1C and
     K1A data.  Terry showed you that.
               They did a purely statistical analysis of the
     data, for what it's worth, but we wanted to start with the
     largest empirical database possible.
               That information was then passed to Dr. Modarres
     of University of Maryland and Dr. Natishan of PAI to help us
     establish the sources of uncertainty.
               They performed a root cause analysis, which I'll
     go through in some level of detail, and appealed to a
     physical basis for the process of cleavage fracture to help
     us to distinguish in this overall process of both RTNDT
     uncertainty and fracture toughness uncertainty what parts of
     those uncertainties are aleatory and epistemic and what's
     the proper way to account for them in the probabilistic
     fracture mechanics calculation.
               Professor Modarres and his students developed a
     mathematical model to treat the parameter and model
     uncertainty which is currently -- we're working the bugs out
     of that but is currently being coded into FAVOR.
               So, that's the ultimate end result of all this, is
     a description of initiation fracture toughness and arrest
     fracture toughness in FAVOR that accounts for all the
     uncertainties in an appropriate way.
               Terry showed you -- this is in your slides.  Terry
     showed you the database before.
               This just points out that there was some data
     growth from where we were in both the crack initiation and
     crack arrest data, and it also points out that the bounds on
     the curves that we were using before in the SECY 82-465 and
     in the IPTS studies, which are shown in red, are
     considerably different than the bounds that we're using now,
     but of course, that all goes into the mix and you turn the
               And as is noted on the bottom in yellow, the
     implications of that increase in uncertainty -- and here you
     should really just be looking at the vertical axis
     uncertainty, because that's all that is reflected on this
     slide -- really depends upon the transient considered.
               Terry has done some scoping analyses which reflect
     that and, I believe, were published in a pressure vessel and
     piping conference publication.
               The analysis process that Drs. Natishan and
     Modarres used, we've been referring to as a root cause
     diagram analysis, and the nice thing about this, at least
     from my perspective as a participant in the process, is it
     really -- it's a visual representation of a formula, but it
     can be a very complex formula, and it helps to both build
     consensus among the experts and it also provides a common
     language for discussion between the materials folks and the
     PRA folks.
               One thing that's very nice about it is it helps us
     to position -- everybody's got their most important variable
     -- the fluence folks, fluence is most important; the
     materials folks like myself, copper is most important.
               They're all, of course, very important, but they
     enter the process in different places, and you need to get
     the parameters or the boxes -- or you could think of those
     as distributions of variables -- and then the relationships
     between the variables are shown at the nodes.
               So, you can imagine here -- and I'll show you an
     actual example of one of these in a minute -- putting in
     copper and nickel and initial RTNDT on this side and
     propagating through a mathematical model and coming out with
     a distribution of RTNDT which then samples a K1C
     distribution on the other side.
               One big change from the old way of doing this --
     by the old way, I mean in the IPTS studies and in SECY 82-
     465 -- that this process represents is that here we -- as I
     just mentioned, we input the uncertainties in the basic
     variables -- copper, nickel, initial mechanical properties
     and so on -- and propagate those to output uncertainties,
     ultimately, RTNDT or somewhere way over there, probability
     of vessel failure.
               What we don't do, which is what we used to do, is
     to input the margins or the uncertainties from outside the
               That's what happened before.  We had -- for
     example, for the embrittlement trend curve, we had a bit of
     uncertainty of -- a standard deviation of -- it was 28
     degrees Fahrenheit on welds.
               So, that became one of the parameter boxes that
     was applied, rather than letting the analysis figure it out
     from copper and nickel and so on.
               The other thing that this provides us, which sort
     of goes to the end application of all of this work, is in
     the end, of course, we're trying to figure out a new PTS
     screening criteria that we can compare plants to.
               When we compare plants, at least in the current
     way of doing things, we generate a best estimate -- in this
     case, RTNDT -- and then we add a margin to it.
               As I stated now, right now, the margin was sort of
     decoupled from this process, whereas here, the margin is
     determined through the process.
               So, in that way, it's much more self-consistent
     than it was in the past.
               Now, this has enough details to keep us here till
     midnight, and we can do that if I'm allowed to make a phone
     call and get my son to football, but what I'd like to do is
     to just make a few points and then open it up to questions
     if you all have any.
               I should note that information flow on this
     diagram goes from right to left rather than left to right,
     but what we've done is we've diagrammed or mathematically
     modeled a process that's fully consistent with the current
     regulatory framework for establishing an estimate of the
     fracture toughness of the vessel at end-of-license fluence.
               So, you start somewhere back here.
               Now, you see, I haven't shown you everything,
     because the sharpie embrittlement shift, the delta-T 30 goes
     to a completely other view-graph which has its own set of
     information and fluence and through-wall attenuation and all
     of that.
               The diagram that you see here is all of --
     everything that feeds into node 3 is just the process by
     which we estimate RTNDT un-irradiated for a particular
     material in the vessel, and this is the diagrammatic
     representation -- don't be too alarmed -- of what's actually
     in SECY 82-465 or Reg. Guide 1.99, Rev. 2.
               So, anyway, you put in all the variables, and what
     this treats is different levels of knowledge.
               Sometimes you have specific material properties
     for a specific weld.  Other times you have to use generic
     data.  Other times you might have only some information. 
     But in any event, you propagate that all through, you get an
     estimate of RTNDT un-irradiated.
               You add to that an estimate of the sharpie shift. 
     You then get an estimate of RTNDT irradiated, and then I
     guess I wish to make the third point second, because I'm at
     that node.
               When we get RTNDT irradiated -- and this is what
     came up before -- is there is a recognition here -- and I
     think I'll go to the next slide and then come back, and this
     is, again, a new feature of this analysis.
               There's a recognition here that RTNDT -- that the
     RTNDT value, which serves as an indexing parameter for
     placing -- here I've just shown a bounding curve, but
     equally, it can position a family of curves or a density
               There's the recognition that RTNDT sometimes does
     a pretty good job at predicting where the transition curve
     lies with relation to real data, and here we're taking the
     linear elastic fracture toughness as our version of reality,
     and sometimes it doesn't do such a good job at all, and I
     think that gets back to some of the questions that were over
               Now, that -- nobody should be too alarmed by that,
     that's, in fact, expected, since the way that RTNDT is
     defined in ASME NB2331 is it's designed to be a bounding
     estimate of the transition temperature, so it's always
     supposed to put the K1C curve here with what some people
     have called white space in between.
               Unfortunately, that's inconsistent with a PRA
     approach that is premised on having best estimates of all
     these parameters.
               So, if we're going to stay consistent with the PRA
     best estimate philosophy, we then need to make a bias --
     make a statistical bias correction on that RTNDT value.
               Now, this is something that we're -- I'll just
     have to say we're still working out the details on, and some
     of the candidate bias corrections are shown up here.
               The mean values tend to be different, but in
     answer to one of the questions that was asked of how far off
     can RTNDT be, it can be anywhere from right on the money to
     150 degrees Fahrenheit off, and that simply comes from -- in
     the simplest way of looking at one of these corrections, you
     plot the linear elastic fracture toughness data that you
     measure for a particular material, you position a K1C curve
     based on sharpie and NTD tests, which are, of course,
     completely independent of the fracture toughness tests, and
     you see how far they're apart.
               It can be anywhere from zero to 150 degrees
     Fahrenheit, and what I learned from Mohammed and his
     graduate students is it's probably best characterized by a
     uniform distribution.
               So, in the process, we go through the type of
     calculations that the licensees would have to perform to
     estimate an un-irradiated RTNDT.
               You can go to another view-graph and do the type
     of calculations the licensees would have to perform to get a
     shift, and of course, in the code, since the Monte Carlo, we
     do this many, many, many, many times, we go into this node,
     we add this to this, and then we make a bias correction by
     just -- this is a cumulative distribution function -- by
     simulating a random number from zero to one, coming in and
     picking off the bias correction for that run, and of course,
     that happens a whole host of times.
               You then get your estimate of RTNDT irradiated,
     which helps you -- which indexes you into what vertical cut
     you take through here, and then you get your vertical
     uncertainty that we were talking about before.
               Now, again, like I said, this is a status report
     of work in progress.
               We had a meeting at the University of Maryland
     yesterday, and one of the things we realized, because both
     Dr. Modarres, Dr. Natishan, and all the folks at Oak Ridge
     are still working on this, is that, as we've noted, the data
     says that there is an unmistakable bias in RTNDT that we
     need to correct for.
               That bias enters this calculation in two places.
               One is in the estimate of RTNDT irradiated that
     you make for a particular vessel, and that's shown,
     appropriately, as being corrected for here, but obviously,
     it's had an influence here, because you plotted all of your
     fracture toughness data versus RTNDT.
               We're still working on the details of how that
     should best be corrected for, but suffice it to say we need
     to correct for that bias in the data set in an appropriate
     manner, because otherwise, the data set includes a perhaps
     un-decipherable mix of both aleatory and epistemic
               What we'd like to do if we can get the -- if we
     can work out the math and -- pay no attention to the graphs,
     I don't really have anything to say about them, just to say
     that we're working on different mathematical correction
     procedures, but what we're aiming to get, of course, here is
     take the -- is a methodology to take the epistemic
     uncertainty in RTNDT out of this fracture toughness
     distribution so that we can treat it as a pure aleatory,
     which is where Mohammed -- which is how Mohammed and Nathan
     have recommended dealing with the fracture toughness
               The concept is to get -- and this, conceptually,
     should work.  The details, I must admit, escape me a little
     bit, but the idea is to get all of the epistemic
     uncertainties into RTNDT, and when you look at the process,
     you conclude that's, indeed, where they are, and then just
     leave the aleatory uncertainties in the WIBLE distribution
     of fracture toughness, which represents the inherent
     material -- the inherent inhomogeneity of the material in
     the transition temperature regime.
               So, that is something that's being worked out as
     we speak and probably will be a good topic for the next
               So, so far, we've completed the statistical
     transition fracture toughness model, we've collected all the
     linear elastic data that we could lay our hands on, and did
     a truly empirical fit to it.
               I'd say we're probably about 85 to 90 percent of
     the way on developing our PRA uncertainty framework.  We've
     understood the process using the root cause diagram
     approach, developed mathematical models of the process, and
     we're working on the details of the FAVOR implementation,
     and of course, everything's an iterative process, and in the
     process of getting this actually coded into FAVOR, we
     realized that we had treated the model uncertainty in RTNDT
     in the vessel estimate part but not in the toughness
     correlation part.  So, that's something we had to go back
     and do.
               Ongoing is full implementation into FAVOR, and as
     I mentioned here, resolution of RTNDT bias correction
     function and modeling procedure, and of course, assembly of
     input data from the various plants that we're considering to
     actually run these models.
               DR. POWERS:  I didn't get to see why my WIBLE
     distribution is of fundamental significance, but I guess
     I'll have to wait on that.
               DR. KIRK:  I'd be happy to talk about that next
     time, but I don't have my slides.
               But I do want to point out, what we've focused on
     here is a lot of the empiricisms behind this, but what also
     stands behind all of the data that you see is really quite a
     good fundamental understanding of why all the variations of
     toughness with temperature should be the same before and
     after irradiation for all these product forms, for all these
     chemistries, and why all of the distributions should be the
     same, and that's sort of part of the background basis, so we
     can -- you know, we can brief you on that next time.
               DR. WALLIS:  I guess what will be interesting in
     the end is how the results you're going to use for making
     decisions are sensitive to the various things you did to get
     those results.
               Now, when you've got that far, you could see how
     the assumptions and processes and all that influence the
     actual final product.
               DR. KIRK:  Right.
               DR. POWERS:  Are there any other questions that
     people want to ask?
               DR. LEITCH:  Could you say a word about how you
     separate out the epistemic influences from the aleatory?
               DR. KIRK:  Well, when you -- I'm going to go to a
     completely different talk, because it's color-coded better.
               DR. POWERS:  I think, in view of our timing, maybe
     we should leave that for another -- alternate presentation
     or off-line.
               DR. KIRK:  Okay.
               DR. POWERS:  Okay.
               Well, thank you very much.
               I will recess this now for 15 minutes, and then we
     will come back and discuss our report on the nuclear plant
     risk studies, and we can dispense with the transcription at
     this point.
               [Whereupon, at 3:35 p.m., the meeting was

Page Last Reviewed/Updated Tuesday, July 12, 2016