Reliability and Probabilistic Risk Assessment - June 28, 2000

                       UNITED STATES OF AMERICA
                                                                 Nuclear Regulatory Commission
                                   Room T-2B3
                                                                 Two White Flint North
                                   11545 Rockville Pike
                                   Rockville, Maryland
                                                                 Wednesday, June 28, 2000
               The committee met, pursuant to notice, at 8:30
               HOMAS KRESS, ACRS Member
               JOHN D. SIEBER, ACRS Member
               MARIO V. BONACA, ACRS Member
               ROBERT E. UHRIG, ACR Member
               WILLIAM J. SHACK, ACRS Member
               NOEL F. DUDLEY, ACRS Staff
               HOWARD J. LARSON, ACRS/ACNW Staff
               JOHN T. LARKINS, Executive Director, ACRS
               MICHAEL T. MARKLEY, ACRS Staff
               GERRY EISENBERG, ASME Project Team
               SID BERNSEN, ASME Project Team
               KARL FLEMING, ASME Project Team
               RON SIMARD, Chair, ASME Project Team
               RICK HILL, GE Nuclear Energy
               BARRY SLOANE, Westinghouse
               RAY SCHNEIDER, Westinghouse
               MARY DROUIN, NRC, Office of Research
               IAN WALL, Consultant, EPRI
               BOB BUDNITZ, ASME Project Team (via speakerphone)
               BRUCE MROWCA, Baltimore Gas & Electric.                            C O N T E N T S
     ATTACHMENT                                              PAGE
     PROPOSED SCHEDULE                                         4 
     INTRODUCTORY STATEMENT                                    4 
     INTRODUCTION                                              6 
     IN RESPONSE TO PUBLIC COMMENTS                            9 
     APPLICATION CHARACTERISTICS                              21 
     .                         P R O C E E D I N G S
                                                      [8:30 a.m.]
               CHAIRMAN APOSTOLAKIS:  The meeting will now come
     to order.  This is the first day of the meeting of the ACRS
     Subcommittee on Reliability and Probabilistic Risk
     Assessment.  I am George Apostolakis, Chairman of the
               ACRS members in attendance are Mario Bonaca,
     Thomas Kress, William Shack, Jack Sieber and Robert Uhrig.
               The purpose of this meeting is to discuss the
     proposed final ASME standard for probably risk assessment
     for nuclear power plant applications.
               Tomorrow the subcommittee will discuss the status
     of risk-informed revisions to 10 CFR Part 50, including
     proposed revision to 10 CFR 50.44 concerning combustible gas
     control systems, issues in the Nuclear Energy Institute
     letter dated January 19, 2000, Option 3, and public comments
     related to the advance notice of proposed rulemaking on 10
     CFR 50.69 and Appendix T, Option 2.
               The subcommittee will gather information, analyze
     relevant issues and facts, and formulate proposed positions
     and actions, as appropriate, for deliberation by the full
               Michael T. Markley is the cognizant ACRS staff
     engineer for this meeting.
               The rules for participation in today's meeting
     have been announced as part of the notice of this meeting
     previously published in the Federal Register on May 16,
               A transcript of the meeting is being kept and will
     be made available, as stated in the Federal Register notice. 
     It is requested that speakers first identify themselves and
     speak with sufficient clarity and volume so that they can be
     readily heard.
               We have received no written comments or requests
     for time to make oral statements from members of the public
     regarding today's meeting.  However, Mr. Robert Christie of
     Performance Technology, Incorporated, has requested time to
     make a presentation during tomorrow's session concerning
     proposed revision to 10 CFR 50.44.
               We will now proceed with the meeting and I call
     upon Mr. Gerry Eisenberg of ASME to begin.
               MR. EISENBERG:  Thank you.  I am Gerry Eisenberg,
     Director of Nuclear Codes and Standards at ASME, and I want
     to thank the subcommittee for this opportunity to brief the
     committee as well as to receive direct and early feedback on
     our proposed ASME PRA standard.  I think this feedback is a
     very important part of our process.
               With me at the table here, all the way to my
     right, is Karl Fleming, a member of our Project Team and
     Standards Committee; Sidney Bernsen, who is the Chairman of
     our Standards Committee; and Ron Simard, who is Chairman of
     our Project Team.  Also, supporting Project Team members
     Rick Hill, Barry Sloane, Ray Schneider and Ian Wall.
               With that, I would like to turn it over to Dr.
               MR. BERNSEN:  Good morning.  My name is Sid
     Bernsen.  As Gerry said, I am Chair of the Committee on
     Nuclear Risk Management, the Standards Committee that is
     responsible for approving the standard and maintaining it.
               We have a few visuals and they are also in a
     handout.  It was prepared for both the workshop that we held
     yesterday and for this meeting today.  I don't intend to
     cover in detail all of the slides, but they are for your
               The first, just to review where we have been and
     we are finally -- we are happy we are finally back here
     again to talk to you.  We are using the ASME redesign
     process which involves using a project team of experts to
     develop the document, publish it for early public and
     comment, and then it will be approved by our committee,
     which is a balanced committee without any dominance in any
     sector, and the work is overseen by the Board of Codes &
     Standards.  And we intend for the standard to be recognized
     as an American National Standard, we are going to submit it
     to ANSI for approval.
               The current status, historically, as you know, we
     issued draft 10 for comment in the spring of '99.  We
     received 49 responses and well over 2,000 general and
     specific comments and suggestions.  This project team has
     worked intensively to address the comments.  I am not aware
     of any effort in standards that involved as much as time
     investment on the part of the people.  The NRC, the industry
     have all participated heavily in this thing.  Project team
     members have worked extremely hard to address the comments.
               Our draft 12, which is the one in your handouts,
     was issued for comment June 14th, and with it is included a
     white paper that summarizes where we have been and where we
     have come to.
               Just briefly again, the scope and purpose of the
     standard, it covers a Level I PRA analysis of internal
     events at power, excluding fire, and a limited Level II,
     sufficient for LERF evaluation.  It is developed to support
     risk-informed applications, including, of course, those
     within the ASME Codes & Standards framework, the inservice
     inspection, inservice testing, and others underway.  And it
     is developed to support the use of existing PRAs, which, as
     we get into our discussion, is something to keep in mind.
               It provides a process for determining the ability
     of a PRA to support an application and it provides options
     for augmenting the PRA either by adding to it or by
     supplementary analysis to handle those cases where the PRA
     has weaknesses and deficiencies.
               Projected schedule, perhaps a bit optimistic, but
     we are going to work toward it.  August 14th, the comment
     period ends.  The project team will work to disposition the
     comments and we hope by early October it will go to the
     committee for approval, and that particular package will
     include responses to the substantive comments.  We will
     probably go for a parallel public review at that time, the
     formal public review.
               Then the votes from the committee are due back in
     a month.  The team will work to resolve the comments.  And
     if we are successful, the whole package can go to ASME Board
     of Codes & Standards for their concurrence before the end of
     the year.  And the ANSI process may take a month or so more.
               The purpose of this review, and, as you know, we
     held a workshop yesterday where we introduced this to a
     number of members of the industry and public, is primarily
     we want to make sure that we have resolved your specific
     comments, your meaning in the case, obviously, ACRS sent a
     lot of comments through the staff.  And we have tried to
     address them, but we really need the feedback from you on
     how well we have done, the acceptability of other changes we
     have made in response to other comments.
               Recommendations for the future.  This is a living
     document.  We are probably going to have to defer a number
     of the comments and recommendations for future
     consideration, so long as we come up with a standard now
     that is adequately comprehensive and usable.  And we hope
     that the comments will be supported with a basis,
     justification and proposed wording.
               The only other thing I would like to mention is we
     do have a number of representatives of the project team here
     today.  They are participating as individual experts.  Their
     comments don't necessarily represent the position of the
     committee or ASME.  Obviously, we haven't formally approved
     the standard, and, therefore, we don't have an ASME position
     on the standard, but I think you will hear from people who
     are quite knowledgeable and, in a few cases, we may even
     expose some still areas that need resolution, where there
     are some differences of opinion and approach.
               And we certainly welcome your interest, which we
     know has been continuing, and the input that you have
     provided to us.  So, with that, I am going to turn the
     meeting over to Ron Simard, who will discuss in more detail
     the comments and what we have come to.  Thank you.
               MR. SIMARD:  Good morning.  I am Ron Simard.  I
     would like to acknowledge two more Project Team members who
     have joined us since Sid made his introduction, Frank Rahn
     and Mary Drouin.
               Gerry, I would like to skip right to the slide
     that summarizes the comments that we got on Rev. 10, because
     what I would like to do is set the stage for Karl Fleming's
     presentation and the more detailed discussion that I expect
     we will get into about the approach we have taken in Rev.
               But let me try to help you understand what was
     behind our rationale for the approach in Rev. 12.  As Sid
     said, we got a substantial number of written comments at the
     end of the comment period on Rev. 12, and I am holding this
     up.  This is a two-sided copy.  And in addition to the
     comments that you see here that were submitted in writing,
     we had discussions at a public workshop held shortly after
     Rev. 10 was released, and at a number of key industry
     meetings throughout the year.  And what you see on this
     viewgraph is my attempt to summarize what were the very
     strong and clear messages that came through in all these
     various discussions.
               There was a very strong sense that Rev. 10, was it
     was written, was too prescriptive, and it didn't allow the
     flexibility needed to apply it to a variety of risk-informed
     applications.  One thing that we heard throughout the year
     was that somebody had counted the number of "shall"
     statements that were in the standard, and I am not going to
     propagate that number by repeating it here, but there was
     perceived to be a large number of "shalls."
               Now, there were a number of concerns with that,
     and I think one concern that really bothered us the most is
     that they said the large number of shalls made it very
     difficult to use with the process that we had laid out in
     this standard for our risk-informed application.  And the
     related remark in the second bullet here is that we needed
     to do more to allow users to distinguish among the grades of
     application, given that there is, you know, a pretty broad
     spectrum of applications that require different levels of
     PRA capability.  And again, another related comment in the
     third bullet is that the applications that we are trying to
     support today are applications that involve the wide mix of
     PRAs.  I think you all are familiar with the variety of PRAs
     that are out there today.
               And finally, there is a considerable amount of
     work that has gone on in parallel with us developing this
     standard to assess the quality of PRAs, and that is through
     the industry certification process, which I understand you
     are going to talk about tomorrow.  But as we were working to
     develop our standard, the guidelines for that process were
     being developed, and visits were being carried out.  I am
     not quite sure where we stand today, but I have heard that
     by the time -- well, certainly, by the time this standard is
     out, most of the plants today will have had one of these
               So, again, a very strong comment came through in
     the written comments, in the workshop, in the discussions
     throughout the year, that these visits were providing a lot
     of good insights.  And they also represented a significant
     commitment of resources, and we needed, where possible, to
     acknowledge that and allow a user to make use of any
     insights from a previous peer review in the way that we
     structured the peer review requirements in our standard.
               And finally, although it is not on there, there
     were also a number of comments that were favorable with
     respect to Rev. 10.  A number of commenters felt that,
     despite the various comments that I just said about the lack
     of flexibility and difficulty in applying the requirements
     in Rev. 10 to the process, that, in fact, there was some
     good stuff in there.  There were some very -- some
     characterizations of a PRA that really made sense and were
     worth maintaining.
               So, this is what we have tried to do in our
     approach to Rev. 12.  I won't get into too much detail in
     the interest of time, knowing that Karl is going to cover
     the approach that we have taken to recognizing different
     categories of application and restructuring.
               But I would like to point out a couple of other
     differences that you will notice between Rev. 12 and Rev.
     10.  There is a fair amount of restructuring.  For example,
     we had what we would proposed as a mandatory appendix to
     Rev. 10 of the standard, that had a database to be used for
     generic data.  And it was decided by the Committee on
     Nuclear Risk Management that the standard is not the right
     vehicle for that, but they have taken that on for
     consideration in the future, whether or not it would be
     appropriate for them to issue a separate standard on that.
               Another thing that we have tried to do is we have
     tried to emphasize that, really, the heart of this standard
     is the process we have laid out for using the standard in a
     risk-informed application.  So, cosmetically, we have moved
     that process from the bank of the standard now to the very
     first thing that you see once you have read the definitions. 
     And second, we have tried to make that standard more usable.
               The other thing that we have tried to do, again,
     responding to those comments that we talked about earlier,
     is we have tried to link the requirements for the various
     aspects of a PRA in our standard to corresponding criteria
     in that industry certification process where we could make
     the linkage.  So, where we could see that one of the
     requirements in our standard was equivalent to a criterion
     that was being used in the cert process, we explicitly
     recognized that.
               When Karl -- if Karl goes into the viewgraphs he
     has got, for example, of one of the tables of requirements
     in Rev. 12, you will see in the leftmost column, there is a
     unique identifier for each requirement.  And where we can
     identify a corresponding criterion in the cert process,
     there is also -- that number is there.
               The other thing that you will notice is that where
     we have retained a Rev. 10 requirement, we have also put in
     the number of the subsection where that requirement appeared
     in Rev. 10.  Only for this review and only to assist you as
     you compare what you are looking at today with what was in
     Rev. 10.  Those numbers will come out when it is published.
               The only thing that I might do -- Gerry, would you
     put up the last viewgraph, the flow chart, please?
               This is something that Karl is not going into in
     detail, and that we wouldn't expect to be -- I want to make
     sure that we hit it now before we get into the way we have
     structured the requirements.  I want to emphasize the
     importance of the process again.
               This slide summarizes some of the main points of
     the process as we have laid it out in Rev. 12.  We
     emphasize, for example, that the process is -- that the
     requirements in the standard, for example, apply only to a
     PRA that is going to be used in this process.  So, the
     requirements in Section 4 apply only to a PRA that is going
     to be used in Section 3.  They are not meant to describe a
     PRA that is going to be used outside that context.
               MR. EISENBERG:  You might point out that it is the
     last one.  They are looking for it.
               MR. SIMARD:  In case you are having the trouble
     finding the slide that is up there now, it should be the
     last slide that is in the handout with my name on the front. 
     You got it?
               The other thing is that in the second bullet, we
     have added a statement to say that we -- the process
     intended to be used with a PRA, that has had a peer review,
     that meets the requirements of Section 6 of the standard.
               A third point that I think came up again
     yesterday, we had some useful feedback yesterday, I think
     maybe we need to emphasize this a little bit more, is that
     in the process we go through the various aspects of the PRA
     requirement by requirement, as opposed to saying the entire
     PRA has this level of capability.  In certification
     language, we don't say this is a Grade 2 PRA or a Grade 3
               And finally, it is only those aspects of a PRA
     that you need for the application that you are considering
     that would have to meet the capability level that we lay out
     in our standard.
               DR. BONACA:  I would like to just make a comment
     for the record, because we discussed this yesterday at the
     workshop.  I still have an issue or a concern with the
     presumption that there is in Box A, that one can say this is
     my problem and this, all I need to do is to develop this
     primitive model and that is good enough, because, as I
     mentioned yesterday, I have seen it hundreds of times and
     that we use PRAs for so many years.
               The PRAs always surprised us with findings about
     dependencies that we did not understand when we were trying
     to address a problem.  PRAs always surprised the
     specialists, they surprised the electrical engineers or the
     mechanical engineers about things that they had not
     imagined, and most of them were in the description of the
     support systems.
               And I am saying that I don't think it is a major
     issue, however, I feel that the standard right now, it
     doesn't provide any warning to this kind of issue, at least
     in the forward where the distinction is being made in the
     process.  There have to be some forewarning that says that
     changes proposed to be addressed with a Category I type of
     capability should be very limited.  I mean there is a
     message somewhere here, but it is not very well
     communicated.  And this point of the importance of the
     dependencies that cannot be intuitively understood up front
     has to be presented.  That is a judgment I have.  And I
     present yesterday, Karl.  And, you know, anybody who has
     used extensively PRA always gets these kind of findings and
               CHAIRMAN APOSTOLAKIS:  So, how would you change
     Box A?
               DR. BONACA:  I would not, maybe not change Box A,
     but in the text where you have a description, in fact, of
     how the steps are being done, there has to be a very clear
     warning that there is always a risk in limiting your
     projection of a model that you may miss something there.
               MR. SIMARD:  Thank you.  That is useful comment.
               DR. BONACA:  And I can verbalize it and put it
     down in writing and send it to you as a comment, I think.
               CHAIRMAN APOSTOLAKIS:  Yes.  I think I would
     appreciate it.
               DR. KRESS:  Would it be useful, Mario, to say --
     say, you have identified your issue as a Category 1 type PRA
     need, to use that Category I PRA in an iterative fashion to
     verify that, sure enough, it was a Category I?  Or is that
     lifting yourself up by your bootstraps too much?
               DR. BONACA:  Well, I guess what I am trying to say
     here is that if I had the Category I that I tailored to
     address my issue, and then I did the same evaluation with a
     Category III, for some changes, Category III will tell me
     something different than Category I.
               DR. KRESS:  Tell you something too much different
     than Category I.
               CHAIRMAN APOSTOLAKIS:  It is really Box C that you
     comment is addressed --
               DR. KRESS:  Yes, determining.
               CHAIRMAN APOSTOLAKIS:  It determines the category. 
     That is where the warning should be.
               DR. BONACA:  It maybe ought to go there.  Yeah.
               CHAIRMAN APOSTOLAKIS:  Determine the category of
               DR. BONACA:  Okay.
               CHAIRMAN APOSTOLAKIS:  That is I, II, or III,
     Roman I, II or III.
               DR. BONACA:  Okay.
               CHAIRMAN APOSTOLAKIS:  Mario is questioning
     whether Category I is always sufficient, even when you think
     it is.
               DR. BONACA:  Or that if you upfront can make a
               DR. SHACK:  I think it is really Box A he is
     talking about, it is always Box 2, that you have somehow
     identified the problem and you have limited it already
               CHAIRMAN APOSTOLAKIS:  Then how about Box 5?  That
     is where you determine the category.  2 and 5 are related, I
               DR. BONACA:  No, this is in the choice of the
     specific requirements, the set of requirements they are
     going through.  The first assessment up there is how large
     -- how well is my model supposed to be in order to address
     this specific question.
               CHAIRMAN APOSTOLAKIS:  I think we are going to
     have a discussion of the categories when Karl gets up there,
     the appropriate slides.  So let's say that we note the
               I think Mr. Bernsen wanted to say something.
               MR. BERNSEN:  I was just going to say that perhaps
     we do have something, I think it is in the quantification
     area, where we say, when you are all done, you have got to
     review this for reasonableness.  And it may be that it would
     be better to consider as an option, when you get done doing
     the application, look back and see that you have had a
     reasonable --
               DR. BONACA:  But if you look at the
     quantification, I mean you have statements like, you know,
     for Category I, you may want to check that the truncation
     total does not exceed the CDF from the rest.  I mean you may
     want to do that.
               MR. BERNSEN:  That type of thing, right.
               DR. BONACA:  This is so loose that, you know,
     there is not really a verification that you are making.  You
     know, if I go through those requirements on the
     quantification, they don't give you any --
               MR. BERNSEN:  What I am saying, a similar thing at
     the end of the application, when you have done it and you
     have your results, then you need to sit back and look at it
     and say, what have I done, is it reasonable?
               DR. BONACA:  I just, the last testimony, my main
     concern is that if there is a presumption in this, and
     people in good faith may think, and probably they are
     thinking today, that they have very limited model and they
     can do the world with it, because there is sufficient
     description  But the fact is I can tell -- I mean anybody
     who uses the PRA, how many times the PRA provides surprises
     to the deterministic people, because it provides
     dependencies that they don't understand upfront, so.
               MR. SIMARD:  Well, I think at this point what I
     will do is, I think at this point I will end and let Karl
     start walking us through the way the requirements are
     structured in more detail.  And then that will help to give
     us specific examples before us that we can talk about.
               CHAIRMAN APOSTOLAKIS:  Good idea.
               MR. SIMARD:  I will just note one thing, if we
     have any comments in particular about the Level II LERF
     analysis, all the sections, the nine elements of the PRA
     that we describe in our standard were assigned to various
     members of the team with one team member as the lead.  In
     the case of the Level II LERF analysis, the team lead was
     Ray Schneider, who, unfortunately, has a conflict and will
     have to leave here around 10:00.  So, if we have additional
     comments beyond that, I think other team members can help,
     but it might be good, if there is anything really
     substantive, to try to involve Ray if we can.
               CHAIRMAN APOSTOLAKIS:  Perhaps after Mr. Fleming
     gives us an overall view of the methodology, we can jump
     into LERF and make sure that the comments are covered.  Now,
     who is this Mr. Fleming?
               MR. FLEMING:  My name is Karl Fleming and I am --
               CHAIRMAN APOSTOLAKIS:  The committee is not
     familiar with you.
               MR. FLEMING:  I am a member of the Project Team
     working on the standard.
               CHAIRMAN APOSTOLAKIS:  Okay.
               MR. FLEMING:  I would like to begin my
     presentation with a few comments on Mario's concern, because
     I think it is a valid concern.  But there are a couple of
     comments I want to make that could perhaps mitigate the
     impact of your comment.
               On one, I think one part of your comment
     indicates, and I agree with this wholeheartedly, there is a
     critical mass for a PRA, that before we can even put the PRA
     label on something called a PRA, it has to meet some minimum
     qualifications.  And it is certainly our intention that the
     Category I requirements capture that, and if there are some
     specific problems or limitations with our requirements that
     don't get us to that critical mass, we certainly are anxious
     to get that feedback.
               But another reflection I want to make is Ron
     indicated there has been, you know, more than half of the
     plant PSAs have been subjected to this industry
     certification peer review process.  I have participated on
     about 10 of them myself.  And I don't think there is a
     Category I -- I mean I doubt, I haven't seen all of them,
     but, based on my evidence, I would doubt if there is a full
     Category I PRA out there.
               I think every PRA out there has many elements that
     would classify as Category II and some Category III.  And I
     think the concept of the block diagram that we have shown
     earlier is to try to clarify that a given PRA may have an
     outstanding accident sequence model for transients and
     LOCAs, but may be very weak for ATWS or very weak for
     station blackout.  So, there may be specific areas of the
     PSA that are Category I or maybe not even Category I, but
     other aspects of their PRA and systems and data treatment
     that may be very good.
               So the block diagram is meant to clarify that, for
     some applications, what -- the current PSA, with its
     weakness and strengths, could be adequate for a given
     application, and to advance the concept that perhaps one can
     use the PSA today and incrementally, you know, build on its
     capabilities without having to invest huge resources to
     bring the whole PRA up to some level before they can begin
     to apply it.
               CHAIRMAN APOSTOLAKIS:  Speaking of resources,
     Karl, you are very experienced with these things, given that
     most units have an IPE now, what do you think the cost would
     be, roughly, for the utilities to upgrade those to a good
     Level II PRA and then a good Level III PRA?  What are these
     huge resources we are talking about all the time?
               Is it $10 million or half a million dollars?
               MR. FLEMING:  I would say that if there is an
     example of a PSA that went to the minimum, might be
     requirements and not much further, and did not update it and
     so forth and needed to do risk-informed applications, I
     would say that the typical cost upgrade, if they just sort
     of purchased the services from a consultant company, may be
     one million dollars to update the Level I PSA, and perhaps
     half a million for the Level II.
               CHAIRMAN APOSTOLAKIS:  So with a million and a
     half, they would have a very good LEVEL II PRA?
               MR. FLEMING:  Right.
               CHAIRMAN APOSTOLAKIS:  And the would not need to
     agonize over Category I, II, III, and all these things?
               MR. FLEMING:  Right.
               CHAIRMAN APOSTOLAKIS:  So I'm a little puzzled
     here.  Where are these limitations in resources and so on? 
     It seems to me a million and a half, considering the
     benefits that the utilities will have from the PRAs, is
               And yet we hear all the time that there are
     limited resources, that we have to develop standards that
     recognize that you don't need a good PRA for all
     applications, and debate it.
               You know, we spend a million and a half debating
     when you need a PRA, instead of spending it doing a good
               Now, Mr. Sieber, I think, has something to say.
               MR. SIEBER:  Well, my comment is that in the
     context of budgeting for a nuclear power plant, a million
     dollars is something.  And it takes at least two people,
     full-time, to keep the PRA up, and that adds to your
     employment list.
               And so it's not inconsequential.
               CHAIRMAN APOSTOLAKIS:  It is no inconsequential,
     but I think we're spending that much money arguing about
     quality and arguing about -- instead of just doing it.  Of
     course, this has nothing to do with the ASME standard which
     is facing reality, of course.
               But I was just wondering why we have all these
     things.  But anyway, you answered my question.
               MR. FLEMING:  I think that whatever the resources
     are and whoever wants to decide to allocate those resources,
     it's also a legitimate consideration to optimally allocate
     those resources so that you're adding the resources in the
     parts of the PSA that you need to apply today, so don't
     necessarily have to go out and put a bit chunk of resources
     in at once.
               CHAIRMAN APOSTOLAKIS:  It seems to me that the
     Revised Oversight Process has sent a clear message that this
     Agency is serious about risk-informing the regulations.
               MR. FLEMING:  Right.
               CHAIRMAN APOSTOLAKIS:  So whether the utilities
     want to spend a million dollars now, or drag their feet and
     spend it three years from now, I think it's coming.
               MR. FLEMING:  Right.
               CHAIRMAN APOSTOLAKIS:  And the Staff was
     authorized recently to request risk information from the
     licensees, even if they choose not to submit risk
     information.  So now you tell me what those signs are.
               MR. SIEBER:  Right.
               DR. BONACA:  I would like to add just one more
     thing, Mr. Chairman, which is --
               CHAIRMAN APOSTOLAKIS:  George, for you.
               DR. BONACA:  I totally agree with your statement
     that I don't know of any PRA out there that is just a
     Category I.  But also, I would like to remake the statement
     that I made yesterday that if there was one Category I PRA,
     it would be a dog.  I mean, I would be really something that
     you would not want to use for anything.
               And then we have a standard, however, that would
     allow for a PRA to be that poor, because it doesn't say here
     that only some aspects should be Category I, and others
     shouldn't be.
               CHAIRMAN APOSTOLAKIS:  If you remember yesterday,
     we discussed something that I believe the gentleman from
     ASME agreed that if you think in terms of Regulatory Guide
     1.174, and you remember the almost white lower left-hand
     side corner, then as you move towards the boundary, it
     becomes darker and darker.    
               Category II, I believe we said, really would apply
     to the nearly-white area.  As you moved to the darker areas,
     then you enter Category III, which I think is a very good
     description of the categories.       I mean, that really
     cleared it up for me.
               And the other important thing, Mario, which is
     relevant to your question, is that there is no room for
     Category I there.  In the context of 1.174, I don't see a
     Category I playing a role.
               DR. BONACA:  You're right.  I'm not saying that. 
     I'm only saying that when I look at the standard, I have
     always had an expectation that standards are the standards,
     which is, you know, I have something I can look up to, and I
     know I can do something with that standard.
               And so, I'm a little bit troubled by -- and I
     recognize, totally, the point that you are making, about
     only looking at certain attributes.
               But, taken in a vacuum, you could think about, you
     know, these are Category I and what could I do with that? 
     And the issue -- and the answer is, you can do much with it,
     which is a very poor --
               CHAIRMAN APOSTOLAKIS:  Karl, you remember many
     years ago, in order to streamline the PRA, there was a Phase
     I where people used rough point estimates, looking at other
     similar PRAs, PRAs for similar plants.
               And they came up with a list of dominant sequences
     before they started doing a more detailed analysis.
               Now, as I recall, that list was pretty good.  I
     mean, a detailed analysis did not really upset the order
     that you got.
               Would you say that that kind of a crude ranking
     would be a Category I application?
               MR. FLEMING:  Well, I think that it would.  I
     think the main difference might be some of the documentation
               Those limited-scope, Phase I PRAs that you're
     talking about, their primary purpose was to optimize the
     resources for the full PRA.
               CHAIRMAN APOSTOLAKIS:  Right.
               MR. FLEMING:  And I don't recall many important
     decisions being made.
               CHAIRMAN APOSTOLAKIS:  No, no.
               MR. FLEMING:  On the basis of that.
               CHAIRMAN APOSTOLAKIS:  I agree.
               MR. FLEMING:  It was a way to risk-inform the PRA
               CHAIRMAN APOSTOLAKIS:  But the results, though,
     were fairly robust.
               MR. FLEMING:  Yes, if experienced people are doing
     the PRA, they are capable of coming up with dominant
     sequences very quickly, with maybe ten percent of the
     resources of the PRA.
               DR. KRESS:  George, with respect to your
     categorization, linked to the white to dark, the problem I
     have with that is that white-to-dark space has -- is in a
     plane at which you have to have the absolute value of the
     CDF already, and the absolute value of the LERF.
               That means you have to have a Category III to
     determine those numbers, before you even enter into that
               CHAIRMAN APOSTOLAKIS:  If you want to be a purist,
     that's correct.
               DR. KRESS:  Yes, well, I am a purist.
               CHAIRMAN APOSTOLAKIS:  There are situations, I
     think, where you have an idea that you are really way down
               Some of the newer plants are highly redundant,
     they produce numbers like ten to the minus six.  Now, you
     might say that if you don't have a complete PRA, that number
     could be as high as ten to the minus five.
               But you're still --
               DR. KRESS:  You still could estimate.
               CHAIRMAN APOSTOLAKIS:   -- in that region, so I
     mean, at least trying to tie it to the decisionmaking
     process, helps, I think.  But, again, you can never draw a
     line and say Category II to the left and III to the right.
               We have a request?
               MR. SCHNEIDER:  Yes, Ray Schneider, Westinghouse.
               One of the issues with the Category I is that if
     you view that some PSAs will have Category I elements where
     they were intended to be conservative in the modeling, tried
     to give a higher estimate of CDF, which tried to give a
     higher estimate of LERF, and in those cases, you can make
     decisions within certain regions, as long as you're making
     the decision based on a well-focused assessment, and you
     understand what the limitations of the PSA is, and that you
     understand the uncertainty bounds are with respect to the
               And so if you are on the high end, you can make
     reasonable assumptions, so that while we're not -- the
     standard isn't purporting to say you should have -- anyone
     should have a Category I PSA, but Category I PSAs could have
     Category I elements within them that -- where certain
     assessments can be made, and made quite effectively and
     quite robustly.
               DR. BONACA:  I understand.
               MR. RAHN:  Mr. Chairman?
               CHAIRMAN APOSTOLAKIS:  Yes.
               MR. RAHN:  Frank Rahn, a member of the Project
               I know it's hard to believe, but there are
     potentially some ramifications that are non-regulatory in
     nature, where we don't even need to, for instance, consider
     a Category II, but where a Category I may be well sufficient
     to make a decision.
               Again, the purpose of a PRA is a guide to your
     thinking, and there are applications, as example, making
     insurance decisions, which are based on some insights in PRA
     where we've used this, things like trip meters which may be
     economic decisions.
               So the ASME is not only serving, if you will, the
     regulatory applications, but a whole spectrum of other
     applications where a Category I application may be
               MR. SIMARD:  Karl, are you done?
               CHAIRMAN APOSTOLAKIS:  No, we are discussing
     Categories without -- I suggest going directly to the
               MR. EISENBERG:  Show me which one.
               MR. FLEMING:  The next one, actually.
               CHAIRMAN APOSTOLAKIS:  Either the second or the
               MR. FLEMING:  In the effort that we went through
     to prepare this draft, we were attempting to meet several
     objectives, one of which was to retain the technical
     resources that had been set forth in Draft 10, and also to
     try to match up the requirements to the certification
               We spent quite a bit of time in the last six
     months, working on the definition of the application
     categories, because the detailed supporting requirements are
     all specified in terms of three different application
               We came up with three categories that match the
     top three categories of the industry's peer review and
     certification process.
               We go in there, recognizing that a given PRA will
     have to be examined for their capabilities with respect to
     the details of the PRA.  Individual elements and individual
     parts of the PRA within an element may fall into different
     categories, and with that recognition, we'd like to be able
     to provide a set of tools for the utility to use, so that
     they can find the appropriate applications to support the
               We might move to the next slide, please.  A little
     bit on the definition:  I think George's descriptions were
     provided some good insights.
               The Category I applications, we define in terms of
     decisions that are normally made based on deterministic
     analyses.  And if you had a PRA, you could supplement those
     deterministic insights with PRA insights.    
               But these are applications that refer to actions
     that the utility has to do anyway, with or without a PRA,
     and with the availability of the PRA resources, can provide
     additional insights.
               Category II was intended to line up with risk-
     informed applications, the minimum applications that might
     be required to support a risk-informed application in which
     you need a balanced set of PRA insights and deterministic
               Category III applications get up into the area
     where in Reg 1.174, you need to increase management
     attention, where the decision more heavily hinges on the
     validity and absolute values of the PSA.
               DR. KRESS:  Let me ask another question:  I like
     to think in terms of uncertainty.  And it seems to me like
     you could link each category to the degree to which you need
     to know the uncertainty.
               For example, Category I looks to me like you need
     to know the uncertainty, because the application is of such
     a nature that you cover it otherwise with the deterministic
               Category II, you probably need to know something
     about the uncertainty, but you can probably do it with a
     sensitivity-type analysis.
               MR. FLEMING:  Right.
               DR. KRESS:  Category III looks to me like it needs
     a full uncertainty analysis.  Is that a good way to look at
               MR. FLEMING:  Yes.  There are several different
     attributes of the PRA that we have looked at across these
     application categories.
               And uncertainty is one of those.  In fact, we'll
     go on to Slide Number 4 where we identified the
     differentiation across these categories with respect to the
     expectations for uncertainties.    
               In Category I, there certainly is a need to
     appreciate the sources of uncertainty and the general
     concepts of uncertainty that are behind the PSA results.
               In the Category II, there is an expectation that
     you can understand uncertainties well enough to be able to
     identify your CDF and LERF estimates with mean values.
               That means you have to think adequately through
     your uncertainties to be able to say that the point
     estimates you're calculating are reasonable estimates of the
     mean value.
               And then finally, in Category III, a full
     quantification of the epistemic and alliatory uncertainties
     is expected, which is consistent with Reg Guide 1.174
               DR. KRESS:  I should have looked ahead to see your
               MR. FLEMING:  Yes.  So that is one of the
     dimensions.  Another dimension is the extent to which the
     decisions may impact the licensing basis with respect to
     safety-related systems, structures, and components.    
               And as Frank Rahn mentioned, there may be
     applications in which the utility might want to make some
     changes to the balance of plant to reduce the -- to improve
     the reliability of the plant, in which case, it does not
     have to apply to the NRC for these types of decisions, and
     may have somewhat less requirements to document the PSA so
     that a regulatory body can participate in the peer review
               CHAIRMAN APOSTOLAKIS:  Now, again, this is
     something that came up yesterday.  When we discuss these
     things, I think it's important to always bear in mind what
     the purpose of this is in the standard.
               In other words, I don't think anyone will come to
     the NRC and say, well, this is a Category II issue, and
     that's why I did it this way; don't ask me any questions.
               The staff will say, well, excuse me, but here are
     100 questions.  So that's not the intended use.
               The intended use is before they come here, to
     think about the issues.  What would be required?  So there
     is a contribution to the general, I would say, elevation of
     the state of the art to a certain level.
               So the licensee will know in advance, what kinds
     of things are really expected of the PRA.  So when they come
     here, they will be prepared.
               So in that sense, I'm fairly comfortable with
     this, because it recognizes, you know,  reality.
               I mean, we can argue about the words and put in
     1.174 references and so on, but I -- but if the intent was,
     I mean, to have somebody come and say, gee, the standard
     says Category II, and you are asking questions about
     Category III, well, excuse me, then I'm against it.
               But the Staff will always be free to ask the
     questions that they feel are appropriate to ask.
               So, that's fine with me.  If the licensee wants to
     think that it's Category I and come here and be surprised,
     well, that's one more surprise for Mario here.
               PRAs surprise people in a lot of ways.  So, I'm
     happy with the -- I mean, not the details, but the whole
               DR. BONACA:  I didn't not express an opposition to
     the way that the standard is being -- I believe, however,
     that there is need for -- I think, in the text, you know,
     the presentation, I think, is clearer than the text.
               There is a need to translate some of this into the
     text, so there is a clearer understanding of the limitations
     of PRA Category I, and, therefore, you don't stray from this
               CHAIRMAN APOSTOLAKIS:  In the context of what I
     just said, of helping the licensee understand what is
     happening here, so that he won't be surprised before the
     Staff, that would be a very valuable thing to do.
               DR. BONACA:  Yes.
               MR. FLEMING:  The other thing that came out in our
     presentation yesterday, and I think we got some feedback
     that we could improve our presentation of this in the text.
               And that is that there is also an expectation that
     in terms of the scope of coverage of these requirements, in
     terms of the dominant and risk-significant accident
               And in this slide we bring out the expectation
     that for Category I applications, we have a set of
     requirements, and we expect those requirements would capture
     the critical mass issues before we could put the label of
     PRA on the product.
               But we impose the requirements on the treatment of
     the dominant sequences.  And so, for example, there may be
     some requirements that have to be applied to the dominant
     sequences that are not important for the non-dominant parts
     of the accident sequences.
               When we go into Categories II and III, we have to
     extend the application of these supporting requirements to
     all the risk-significant sequences.  And if we go up into
     this area of increased management scrutiny, we may have to
     go beyond the risk-significant sequences to some of the even
     less important sequences, to the extent that that may impact
     the decision.
               So, that's another characteristic of these
     requirements, and one of the feedback discussions we had
     yesterday is that we probably need to work on a definition
     of what we're talking about when we use these terms,
     dominant, and risk-significance.
               We did not include those in the actual definitions
     section, and I think we got some feedback that we would be
     well advised to add that.
               If we can skip the next slide, so, working sort of
     from a top-down fashion, we, of course, then have the
     elements, the nine elements of the PSA, and these are the
     same nine elements that we used in Draft 10.
               And they are very typical of what you would see in
     the breakdown:  Initiating Events, Sequence Development,
     Systems Analysis, Data Analysis, and so forth.
               There are nine of these that cover the scope for
     internal events, including internal flooding, but not
     including internal fires.
               If you look at these attributes, the attributes
     call out the concepts of dominant versus risk-significant
     accident sequences, and the other concept that's clearly
     differentiated across these three columns is that
     conservatism is tolerated, if you will, more completely --
     more freely in the Category I applications, whereas it's not
     really tolerated in the risk-significant arena for the
     Category II and III applications.
               And anytime that we permit or provide the
     opportunity to meet requirements with conservative
     assumptions, we have the caveat that the conservatisms do
     not distort the ability to make risk screening applications
     that you would need in a Category I.
               The basic Category I type of applications are
     applications in which you just want to make course screening
     of elements of your PSA into very course risk categories, so
     that conservatism would be permitted, only to the extent
     that it does not distort that kind of application.
               So that's -- these attributes provide the logic
     for how we tried to come up with a differentiation, when
     appropriate, for the supporting requirements for each of the
               CHAIRMAN APOSTOLAKIS:  Under Data Analysis, it
     says realistic quantification of mean values.
               MR. FLEMING:  Right.
               CHAIRMAN APOSTOLAKIS:  Many PRA type analysis -- a
     lot of people take a point estimate, and they say, well,
     this is a mean value.
               That's not what you mean here.  You have to alert
     people to the fact that the mean is not the same as
     somebody's best estimate.
               MR. FLEMING:  That's right.  The concept for data
     and quantification is that point estimates, which could be
     conservative estimates, as long as they don't distort the
     risk profile, are accepted for Category I.
               Mean point estimates are expected for Category II,
     and that means that you have to carry through your
     uncertainty analysis to a sufficient extent to be able to
     show, demonstrate that you have mean values.
               CHAIRMAN APOSTOLAKIS:  That's stated somewhere? 
     It should be clarified.
               MR. FLEMING:  We certainly intended it to be.
               CHAIRMAN APOSTOLAKIS:  Yes.  I don't remember
     seeing that.
               DR. KRESS:  That's the mean value of only the
     alliatory uncertainty?
               MR. FLEMING:  Alliatory and --
               CHAIRMAN APOSTOLAKIS:  Epistemic uncertainties, as
     well.  I think it is really epistemic.
               DR. KRESS:  Yes.  That's only the --
               CHAIRMAN APOSTOLAKIS:  The failure rate is
               DR. KRESS:  Yes.
               MR. FLEMING:  Whatever epistemic uncertainties
     that are included in the model.
               CHAIRMAN APOSTOLAKIS:  Yes, the human error rates.
               MR. FLEMING:  Yes, the human error rates, and --
               CHAIRMAN APOSTOLAKIS:  So that's a key point, and
     I think that maybe we can look for a place to make sure --
               DR. KRESS:  Both of those things need
               MR. FLEMING:  For example, that would require some
     kind of uncertainty analysis be done at the data level, but
     not necessarily propagated all the way through to CDF and
               CHAIRMAN APOSTOLAKIS:  But when you propagate mean
     values, in some instances, as you know, the variance plays a
               MR. FLEMING:  Right.
               CHAIRMAN APOSTOLAKIS:  I think people will find it
     easier to just do a Monte Carlo simulation.  That is at
     least numerical, you know.  Just do it.
               MR. FLEMING:  And that may be, in fact, the case.
               CHAIRMAN APOSTOLAKIS:  I wonder whether we can,
     before Karl moves on to discussing requirements, maybe we
     can start with the LERF, the very last one, Level II
     analysis, and make sure we cover it before the expert
     leaves?  How about that?  Is that okay?
               MR. FLEMING:  Sure.
               CHAIRMAN APOSTOLAKIS:  Unless there is something -
               MR. FLEMING:  Sure.
               CHAIRMAN APOSTOLAKIS:  Do you have any viewgraphs
     on this subject?
               MR. SCHNEIDER:  No.  I was just basically going to
     take questions from the Committee, but I would want to put
     something in overview in terms of what was done with the
     LERF section.
               The intent was not to be a full Level II PSA, but
     to look at the LERF surrogate that the NRC's been using for
     regulatory review.  And the three categories in --
               The words probably don't specifically state the
     way it was structured, but the three categories, Category I,
     was generally intended to be the conservative estimate of
     LERF, using bounding assumptions, where bounding assumptions
     would be -- would provide acceptable results in sufficient
               As you move it to the Categories, you will get
     increased resolution and increased precision.  You include
     more information, more phenomena, and more information on
     the -- more details on the quantification.
               So as you go from Category I, II, and III, what
     you should be getting is a more refined prediction of LERF,
     generally moving down.
               The expectation is that Category I estimates
     should not under-predict LERF.
               Okay, I guess with that as an overview, I'll take
               DR. KRESS:  Well, I had a couple of questions,
     mostly -- I thought that was a fairly good section of the
     standards, but I had some questions that I think are mostly
     just of a clarification nature.
               On page 126 of my version of the document, in the
     Category III applications, you say in the bottom box there,
     you say you include a requirement that the effects of in-
     vessel melt retention ought to be included.
               And I wonder why you felt it necessary to actually
     spell that out.  Is that at all applicable to any operating
     plants we have?
               MR. SCHNEIDER:  There are several C-plants that
     have the ability to be bottom-flooded, and some of them have
     credited a certain proportion in their detail Level IIs,
     more or less a certain proportion of the events wouldn't
     necessarily go to failure, because they could flood all the
     way up to the nozzles.
               So, there's a -- we have integral lower heads, so,
     as a result of the design differences, at least for the
     plants that I'm familiar with, it is a consideration that
     has shown up in PSAs, and could result in certain events
     that would have gone into failure/not going to failure.
               DR. KRESS:  My understanding is that they all go
     to failure if you include the uncertainties, and the size of
     the vessels and the power levels are such that none of them
     really can take much credit for in-vessel retention.
               I would rethink whether or not I wanted to have
     that called out, specifically, there.  But maybe I'm wrong
               MR. SCHNEIDER:  I believe that most of them can,
     and the analyses depend on -- there is, I guess, the
     probabilistic assessment that a certain fraction of them,
     under -- I guess there were two issues.
               One was a delayed injection into the RCS, coupled
     with a flooding of the external would give you a high
     probability of recovery.
               And that wouldn't be recovery in another high-
     pressure event or another event that progressed slightly
     differently.  That's why it says level of precision that's
               What's happening is that you're reducing your LERF
               DR. KRESS:  My feelings are that the uncertainties
     are so large in that that it probably is not useful.
               MR. SCHNEIDER:  Understood.
               DR. KRESS:  Likewise, on page 128, in defining a
     large early release it says that the analyst may consider
     mitigating factors, such as played out and deposition of
     fission products released from the fuel and the release
     pathway characteristics.
               I certainly again with that but unfortunately
     nowhere in the standards do you mention any standards for
     fission product release modeling that I can see at all
     because you are dealing mostly with LERF, which doesn't
     really involve the modelling of fission product release, but
     if you are going to take credit for mitigating issues, then
     you have to know something about the timing of the release. 
     You have to know something about the species.  You have to
     know something about the aerosol characteristics which
     depend on those things, so when you say they may take credit
     for it, you don't go the next step and say but if you do you
     will have to meet certain standards in your fission product
     release model, so this is more just a comment than a
               MR. SCHNEIDER:  Good point.
               DR. KRESS:  And I guess I had one other.  This is
     a clarification question.
               On Table 4.49 on the dominant contributors to be
     considered in LERF, I was a little bit interested in why
     under hydrogen combustion for example you included Mark 1s
     and Mark 2s, which I thought were inerted and why you didn't
     include large dries because in combination with other loads
     hydrogen combustion could be the straw that puts you over
     the brink so I was just wondering why the check marks, how
     the check marks came about in that table?
               MR. SCHNEIDER:  For the large dries, the Level II
     analyses and the experiments that have been followed up,
     that have been used to support this is that the DCH and
     hydrogen combustion really aren't concurrent.  They do occur
     displaced in time and while if you add the two together
     would put you above the brink, they probability that they
     will be there as a dominant contributor hasn't shown out to
     be in practice, in experiments.
               I think that is why we didn't put the check box
     there for that and why we kept it but we did keep it for the
     DCH induced failure with a certain probability, but when you
     start adding the DCD in hydrogen the probability would be a
     lot lower.
               Also, a lot of the analyses that are actually
     being done often when they do the DCH add to it, consider
     the hydrogen combustion in conjunction with the DCH as well,
     so this is really for the hydrogen combustion independent of
     the high pressure melt ejection.
               DR. KRESS:  Okay, and they detect the Mark 1s and
     Mark 2?
               MR. SCHNEIDER:  I'll turn that over to Rick Hill.
               MR. HILL:  This is Rick Hill, GE, and a member of
     the project team.
               Hydrogen combustion is listed for Mark 1, Mark 2
     even though they are inerted plants.  They are oxygen
     controlled plants and there are scenarios where you could
     de-inert of have oxygen in the containment and we feel that
     that is a question of Level II modeling that should take
     place even though obviously the risks are very low.
               DR. KRESS:  Okay, it wasn't screened out on low
               MR. HILL:  Right.
               DR. KRESS:  Okay.  A similar question on this
     table.  Why did you feel like you could exclude steam
     explosions from consideration in large dries and ice
               MR. SCHNEIDER:  It goes pretty much back to what
     the existing Level IIs tend to show is that the steam
     explosion phenomenon is, once officially uncertain and low
     probability, that for a LERF assessment it just was over-
     dominated by all the other processes.
               The main issues in terms of releases to the public
     pragmatically are where you have the loss of containment
     isolations above ground typically and it would be loss of
     containment isolation, the IS LOCA and the steam generator
     tube rupture.
               To a much lower extent you have the probabilistic
     potential that you can fail containment due to the high
               The steam explosions typically occur in the lower
     portions of the cavity.  You would have to fail the
     containment in a way that would affect the above-ground
     releases and it was just felt to be a much lower probability
     event that would be more than covered by the others as long
     as you are not doing a detailed Level II.
               DR. KRESS:  So you are relying on the risk
     insights --
               MR. SCHNEIDER:  From the Level IIs that were
     done --
               DR. KRESS:  -- from the Level IIs that were done
     by the IPEs.
               I suspect that that might be a risky thing to rely
     on for this.  I am not sure I would want to exclude steam
     explosions, at least I don't think the explosion itself is
     going to damage the containment.
               We are dealing with containment here --
               MR. SCHNEIDER:  Right.
               DR. KRESS:  -- but I think there is a high
     probability it can add pressure to an already pressurized
     containment and might ought to be considered for looking at.
               MR. SCHNEIDER:  Well, we have looked at that issue
     and that is not the driver.
               You can vaporize a lot of the water but the
     robustness of the containments are such that you are not
     going to, pragmatically you are not going to have enough
     water in the containment to take that to a containment
     failure condition, but we could look at that and reconsider
     and check the numbers out.
               DR. KRESS:  Okay.  Well, that is the extent of the
     questions I had.  Do you have some?
               CHAIRMAN APOSTOLAKIS:  I have one or two, but 
     maybe it is because of my ignorance of the subject.
               I have always been mystified by the definition of
     large early release, so I was looking for a definition.
               So on page 8 it says that large early release is
     the rapid, unscrubbed release of airborne fission products
     from the containment to the environment occurring before the
     effective implementation of offsite emergency response and
     protective actions.
               Then on page 128 it says you define LERF
     consistent with the definition given on page 8, Section 2,
     but then it goes on and elaborates a little bit on early --
     which means, early refers to a timeframe -- prior to
     effective evacuation of the inhabitants of the exclusionary
     area boundary.
               My question is why are we avoiding giving a time,
     a rough time?  I have heard in the past before three hours,
     but I don't know.  Is it before any effective implementation
     of offsite emergency response?  I wonder if that is a
     scientific definition -- early.
               What if the emergency response measures fail and
     they are delayed?  Well, then early release is anything that
     is released before that?  There has to be some time --
               MR. SCHNEIDER:  For most of the transients if you
     look at what contributes to LERF, it mostly isn't an issue. 
     It comes because like if you have loss of containment
     isolation it is core damage events that occur and have an
     early core damage failure.
               CHAIRMAN APOSTOLAKIS:  Early?
               MR. SCHNEIDER:  Yes, and so you are generally
     talking the first several hours, so when you initially get
     to this, you are dealing with 4 to 8 and then that depends
     on how quickly they can get the information out to the
     public, how much population they have around the site.
               For example, in Arizona, it's not going to be that
     bad.  They know everyone's phone number, but in other areas
     it may take longer for evacuation so to but a rule on the
     time was -- we didn't feel comfortable doing an exact,
     precise time, but the issues that you have to consider are
     about how rapidly is the staff going to be able to recognize
     they are undergoing a core damage state, how quickly can the
     information get out, and when do they expect the releases to
     be felt given the event?  For example, steam generator
     ruptures may occur very late in time.
               What this does is gives them the flexibility to
     say not all steam generator tube ruptures have to be
     considered large early if you can keep the core covered for
     12 to 15 to 18 hours, but if you have a steam generator tube
     rupture that rapidly progresses to failure with an open
     MSSV, then that would be an early release, so --
               CHAIRMAN APOSTOLAKIS:  I guess my question is why
     is the condition of early or late, why does it depend on the
     evacuation and not on some physical characteristics of the
               MR. SCHNEIDER:  The QHO was the original --
               CHAIRMAN APOSTOLAKIS:  The unscrubbed --
               MR. SCHNEIDER:  Because you go back to the
     original definition of what was trying to be accomplished
     maybe five to eight years ago when you had the Qualitative
     Health Objective, and that was basically to limit the number
     of fatalities, to put it into a certain level -- to make it
     consistent with the rest of the industry, and what they did
     is they made a surrogate and the surrogate was LERF.   
               So you have taken away now a lot of the features
     that went into what the QHO was but the QHO included
     evacuation, sheltering and all of those features so the LERF
     retained some of that flexibility without the clear
     definition of how it affects the population and so you need
     to define something and if you put a defining time for
     certain plants that may not be an appropriate timeframe, so
     we're allowing them the flexibility to adjudge their
     emergency planning procedures that match up against the
     various events and then determine whether they would class a
     specific steam generator tube rupture as a large, early or a
               Otherwise you may end up in situations where you
     have later releases that because of some issue associated
     with the transient that may have prevented them from
     alerting the public might really be classified as a large
     but if we put -- an early, but if we put a short timeframe
     involved that would just automatically throw it out and if
     by the same token if you put a long timeframe you probably
     are including too many events, especially for the plants
     with low population area.
               So we did allow some flexibility.  It mainly will
     affect issues like steam generator tube rupture and some of
     the high pressure melt events.  They have to justify how
     they are binning it.
               DR. KRESS:  I think when NRC, and I may be
     interpreting them incorrectly, but when they went to the
     LERF what their intention was to do was to more or less
     separate outside issues from design issues and do it in such
     a way that the LERF would cover essentially most of the
               CHAIRMAN APOSTOLAKIS:  Right.
               DR. KRESS:  And now we seem to be going away from
     that and going back and saying now we have to -- if you are
     going to do a LERF that is site specific, you have got to
     have Level III PRA, which we are not dealing with in here at
     all.  We have no standards for Level III.  We don't
     discussion fission product standards and I think it is a
     mistake to in this particular standard to go wawy from NRC's
     intended use where the LERF that we have is related to plant
     accident issues, like George says.
               I think you do need some sort of tighter
     definition of large and early release that relates to
     actually the timing of the accident that would be site-
     independent, frankly, and that is a problem I had with it
               CHAIRMAN APOSTOLAKIS:  The definition seems to
     depend on site characteristics in emergency --
               DR. KRESS:  Yes, but those fall into Level III
     categories and you have no standards for Level III, so I
     think you have a bit of a problem with that.
               CHAIRMAN APOSTOLAKIS:  Please.
               MR. FLEMING:  There was also some industry
     perspective on the definition of LERF that we put into the
     EPRI PSA Applications Guide, and we offered a definition in
     the PSA Applications Guide which is consistent with this
     definition but it was a little bit different.
               The philosophy from the industry perspective was
     to expand the range of risk informed applications to be able
     to consider some of the containment systems that might be
     involved in the applications, and we came up with a
     definition of LERF in the EPRI PSA Applications Guide which
     was based on the philosophy of capturing all the risk of
     early health effects and we used the definition that was
     based on the assumption that Seabrook, the Seabrook Level
     III PSA, which had a vast inventory of Level III analyses,
     and also the Staff had pretty much concluded that Seabrook
     had one of the more limiting sites with respect to the
     emergency plan, we came up with a definition in the
     Applications Guide, which I believe was earlier was
     something like within four hours of vessel breach, which for
     Seabrook was the time it took to clear out the EPZ based on
     their site specific emergency plan, and that was part of the
     definition for quite awhile.
               We dropped the hour definition in recognition of
     the fact that some plants may be able to clear out their EPZ
     in two hours or one hour and if they have site-specific
     analyses to be able to tighten up their definition of LERF
     and not use the conservative definition for Seabrook they
     would have that option.
               But the philosophy was to provide a surrogate for
     a Level III PRA that would expand the range of applications
     beyond what CDF could look at without dragging in all the
     issues that we have difficulty -- rebed cooling and basemat
     melt-through -- and just take a subset of the Level II
     issues into the risk-informed arena.
               CHAIRMAN APOSTOLAKIS:  I appreciate the effort but
     the problem I see with that is that somebody may declare
     their plant as capable of evacuating within 2 hours
     without -- and then that is buried somewhere there and that
     may be significantly uncertain.
               DR. KRESS:  You say how do you know that, whether
     you did Level III, how good is your Level III.
               CHAIRMAN APOSTOLAKIS:  And to base a quantity that
     plays such an important role in decisionmaking on these
     kinds of assumptions makes me a little uneasy.
               I would rather have a definition that depends on
     the design, as Tom said, and the accident characteristics,
     at least to have some bounds and give maybe some flexibility
     because perhaps Karl's point is an important one that you
     can't really ignore the fact that they may have very good
     evacuation plans, but limit the impact of that.  Perhaps
     that would be a better way of doing it, because what if
     someone says we can do it in an hour, and that is a sentence
     there somewhere there in a three volume PRA and, you know,
     the whole calculation of LERF depends on that --
               DR. KRESS:  Depends on that --
               CHAIRMAN APOSTOLAKIS:  -- and it would be very
     hard to touch it.
               I would feel better if there were some
     recognition, some acknowledgement that these issues are
     important because I fully appreciate the arguments you made.
               Now there is also a page 128 user definition of
     LERF that captures the contributions to the risk of early
     health effects, but it seems to me that that has been stated
     several times.  It is just a matter of editorial cleaning
     up, I think.
               I think a lot of the discussion in 128 on the
     right-hand column is very repetitive.  That's your business.
               MR. SCHNEIDER:  Okay.  I will take that into
               CHAIRMAN APOSTOLAKIS:  Maybe we should start using
     fuzzy sets, you know -- so dead set against them, but now I
     see those definitions.
               CHAIRMAN APOSTOLAKIS:  Are you guys willing to
     develop a standard for fuzzy PRA?  Say no.
               Okay. Are we done with LERF?  Well, back to Mr.
               MR. FLEMING:  Thank you.
               CHAIRMAN APOSTOLAKIS:  I really hate to work for
     an hour and a half without a break.
               DR. KRESS:  Yes, me too.
               CHAIRMAN APOSTOLAKIS:  Is our Federal employee
     objecting to taking a break now?  Okay.  We will take a
     break now for -- oh, well.  How do we define a break without
     using a clock?  What about 15 minutes.
               CHAIRMAN APOSTOLAKIS:  Okay.  Back to session. 
               MR. FLEMING:  Karl Fleming from the project team.
               Before I return to Section 4, I wanted to make a
     comment.  The cost estimates I provided earlier were for
     time and materials and not a fixed price contract.
               MR. FLEMING:  Getting back to Section 4, one of
     the comments that we wrestled with from Draft 10 was that
     somebody counted up 900 and some odd requirements that had
     the work "shall" and we were trying to avoid a frankly silly
     exercise where we sit down and negotiate how many "shalls"
     could be sent to "shoulds" or "mays" or whatever and it
     didn't seem to be a very useful exercise, so what we decided
     to do as part of our effort for Rev. 12 was to back up to,
     say, 20,000 feet and from the point of view of people who
     are competent to perform peer reviews and people who have
     lots of experience in PSAs is to boil down these
     requirements into a set of irreducible high level
     requirements that point to basic attributes of a PRA that we
     are all aware of.
               These would be attributes such as the completeness
     of the PRA, treatment of dependencies, the degree of realism
     in the assumptions and the success criteria, the degree of
     fidelity with the plant and the PRA model, and how well it
     reflects the as-built, as-operated, and design change plant
     and so forth and go across each of the nine elements of the
     PRA and come up with high level requirements phrased in
     "shall" language that everybody would agree have to be
     present and form the critical mass of what is needed for the
     product that we are going to put the PRA label on, whether
     it is Category I, II or III.
               One of the tasks that we laid out here, and I will
     walk through some examples of those in a few minutes for
     accident sequences, is to capture the essence of the
     requirements in these high level requirements that typically
     are a number in the range of maybe four or six high level
     requirements for each of the nine elements, and we used this
     as a starting point for organizing and defining the detailed
     supporting requirements.
               Many of these high level requirements are actually
     in Draft 10 but they may be difficult to find because it was
     presented in sort of a textual format and we wanted to bring
     them out and make them very clear and explicit in this
               The concept is that each of these high level
     requirements would apply to all three application
     categories, but the extent and the context in which you
     would apply them would be different depending on the
     characteristics that I mentioned in the earlier
               With this kind of a concept what I would like to
     do is actually walk through some of the high level
     requirements for accident sequences.x
               DR. BONACA:  So you are in Chapter 4?
               MR. FLEMING:  Yes, we are in Chapter 4.
               CHAIRMAN APOSTOLAKIS:  I was wondering whether the
     members had any comments on the definitions and the risk
     assessment application process that are Chapters 2 and 3.
               DR. BONACA:  The Definitions section, you mean?
               CHAIRMAN APOSTOLAKIS:  Yes.  I just got a comment
     from Mr. Barton, who could not be here today.  On page 10,
     unavailability is defined as follows -- the fraction of time
     that a test or maintenance activity disables a system or
     component, also the average unreliability of a system or
     component over a defined period of time.
               His comment is the word "unreliability" is not
     defined, so there should be a definition of unreliability as
               That brings me to another comments, which is a
     favorite of mine.  This definition I recognize is one that
     the industry has been using for a long time, the fraction of
     time that a test or maintenance activity disables a system.
               It is not consistent with the definition in
     reliability theory, which is that the component or system is
     unavailable due to any reason at Time T, and this has been
     an issue before in other contexts.
               What was the last time we had an appendix with a
     definition and I didn't like it there either?  The
     maintenance rule.
               It seems to me that if one decides to go with this
     definition of unavailability then one would have to have in
     the expressions for the probability of the thing not
     responding --
               MR. BUDNITZ:  [By Telephone]  This is Bob
     Budnitz --
               CHAIRMAN APOSTOLAKIS:  Okay.  We know who you are.
               MR. BUDNITZ:  Oh.  I know who you are too.
               CHAIRMAN APOSTOLAKIS:  Can you see us?
               MR. BUDNITZ:  I cannot see you.  I am only on a
     phone.  Are you more gorgeous than usual?
               DR. KRESS:  Yes.  The answer is yes.
               CHAIRMAN APOSTOLAKIS:  Okay, we can hear you very
     well, Bob.
               MR. BUDNITZ:  Look, I have to be out of here at a
     quarter after, which is just over an hour from now.
               CHAIRMAN APOSTOLAKIS:  Okay, don't worry. We will
     be done by then.
               DR. KRESS:  Do you have some comments you want to
     make, Bob?
               MR. BUDNITZ:  You mean upfront?
               DR. KRESS:  Yes.
               MR. BUDNITZ:  Where are you in the agenda?
               CHAIRMAN APOSTOLAKIS:  We are talking about
     definitions.  We finished LERF.
               I will give you a few minutes to catch up, okay?
               MR. BUDNITZ:  Yes.  I thought I was on because
     when you come to expert judgment I am the one.
               CHAIRMAN APOSTOLAKIS:  We will make sure we do
     this before you have to go.
               MR. BUDNITZ:  Okay.
               MR. SIMARD:  Expert judgment as well as any
     questions about initiating events -- Bob and Steve in that
               CHAIRMAN APOSTOLAKIS:  Initiating events is coming
               So as I was saying, if we adopt this definition,
     which the industry seems to be comfortable with, then there
     has to be an extra term, probably of failure on demand.
               I am not sure that we have that in the
               Now if you go to standard mathematical books on
     reliability, unavailability includes that so I don't know
     what the resolution should be.
               At some point we have to make sure we have one
               I think the industry refers to the latter, the
     probability of failure on demand is unreliability, which
     again conflicts with the mathematical definition which says
     it is the probability of not performing in a period of time,
     so I don't know.
               Do the members have any suggestions?  Should we
     try to change the way the industry uses these terms?
               DR. BONACA:  Well, for me, not including other
     reasons why a system or component is unavailable, it just
     doesn't make any sense.
               CHAIRMAN APOSTOLAKIS:  They may include it in the
     calculations.  I don't know.
               DR. BONACA:  I understand that.
               CHAIRMAN APOSTOLAKIS:  The definitions should
     include it, in my view --
               DR. BONACA:  -- should include it.
               CHAIRMAN APOSTOLAKIS:  Okay, so we will probably
     make a comment to that effect and the committee will have to
               Now speaking of definitions, I also have a
     question, but maybe we can wait until -- the human error. 
     The definition of latent human error, do you want to do it
     now or when we talk about human errors?
               MR. FLEMING:  As you wish.
               CHAIRMAN APOSTOLAKIS:  Well, it says here on page
     8 a human error typically by mispositioning or
     miscalibrating a component that if not detected or corrected
     predisposes the affected component to failure when demanded.
               This is a very limited definition of the latent
     error, and I would recommend that you use Jim Reason's
     definition of latent conditions and latent, which is any
     human action before the actual active error takes place and
     not just mispositioning or miscalibrating.
               Any other comments from the members on the
               DR. BONACA:  I would like to provide one.
               On page 5 on the definition of accident class
     there is a use of the word "severe" accidents, and I believe
     that is a little bit of a narrow connotation there, somewhat
     confusing.  I would certainly prefer to see a grouping of
     accidents that by severe accidents we indicate very specific
     ones but you include the category of transients that are not
     necessarily ending up in a severe accident.
               Now on the issue of accident consequences, here is
     it more of a -- you know, in regulatory space is meant only
     doses.  I mean is meant only radiological release, and I am
     not sure that you may not want to look at that definition
     there if it creates an unintended conflict, a confusion.
               This I am just raising as a question and would let
     the ASME decide what is the proper approach.
               At the bottom of page 5, "available time"
     specifically talks about time from which an indication is
     given that human action is needed to where the action was
     performed to "avert" -- first of all, the word "avert" --
     but core damage I think again is a very narrow definition
               I don't think it is intended only that sense of
     available time.  I think there are actions that prevent
     other events, not only core damage and again it is very
     narrow to focus on core damage -- maybe to where the action
     was to be performed to achieve success, whatever that means. 
     You could let it be in the analysis.
               The definition on page 6 on containment analysis
     needs work.  There is some editorial problem there -- no,
     that's okay -- definition of extended events on page 6,
     again there is always this pointing may lead to core damage
     or larger releases, but really you are looking at extended
     events in a broader sense -- and again that reference to
     core damage early release I don't think that is necessary in
     the context of the definition.
               CHAIRMAN APOSTOLAKIS:  Well, for extended events
     though that is really what you worry about.  Isn't it?  I
     mean if you have an earthquake or --
               DR. BONACA:  Yes.  I am talking about there
     extended events is initiating event originating outside -- I
     mean you may conclude in the analysis they will lead you to
     that.  You are still defining certain external events.
               For example, I could have conceivably a typical
     external event for a PRA that you always analyze that in
     that particular plant will not lead to CDF or LERF.
               It would still be --
               CHAIRMAN APOSTOLAKIS:  But the concern is that it
     might.  That is why you analyze it.
               DR. BONACA:  Yes, but if you look at the
     definition -- may lead.
               CHAIRMAN APOSTOLAKIS:  May lead to the part of
     external events?
               MR. FLEMING:  Internal events.  It's in the scope.
               DR. BONACA:  Page 7 on the harsh environment,
     there is a reference to appropriate for design basis or
     beyond design basis accidents.
               I would rather a definition that does not include
     -- not narrow that much.  Again, a environment  -- as a
     result of the postulated accident condition.
               I mean, there are some others one, and I don't
     want to spend any more time.  I will provide them.
               CHAIRMAN APOSTOLAKIS:  Sure.  We'll have an
               DR. BONACA:  Again, the word, unavailability on
     page 10, mirrors the comments we had.
               MR. BERNSEN:  Let me just ask one question.  We
     would like, wherever possible, to use existing definitions,
     definitions that have been published, if they're at all
     consistent with our intent.  So that if you have some
     alternative definitions that have been published, if you
     could cite the reference or whatever for them, that would be
     very helpful.
               CHAIRMAN APOSTOLAKIS:  Sure.  We will probably
     have appendix to our letter with the detailed comments,
     maybe line-by-line.  I don't know.
               One last comment on the definitions which may
     involve Bob Budnitz.  On page 6, there is a definition of
     expert solicitation.
               First of all, I would suggest that you change it
     to expert opinion elicitation.  It's not the elicitation
     that's expert; it's the opinion.
               And second, it says a formal highly structured and
     documented process.  Now, if you go to the actual section on
     expert opinion, there is allowance for less than highly
     structured processes.
               So, it seems to me that it's overly restrictive to
     define it as a highly-structured process.  Above, when you
     use the technical integrator, the technical integrator, then
     the process is not necessarily highly structured.
               It's highly structured when you go to the full
     treatment that the technical facilitator, integrator,
               And I think that in Section 6, you make -- I'm
     sorry, 4.6, you make that point well.  So it seems to me
     this definition here should delete -- maybe you can say a
     structured formal approach, rather than highly structured.
               MR. BUDNITZ:  Which definition are you looking at?
               CHAIRMAN APOSTOLAKIS:  Expert elicitation on page
     6.  Okay?
               MR. BUDNITZ:  Yes, you can just take the "highly"
     out of there.  I understand that point.  It's a good point.
               CHAIRMAN APOSTOLAKIS:  Or maybe completely highly
     structured and say a formal and documented process, and you
     differentiate in 4.6, regarding the various levels.
               MR. BUDNITZ:  It's got to be structured, George.
               CHAIRMAN APOSTOLAKIS:  Okay.  Can you have a
     formal process that's not structured?
               DR. KRESS:  Yes.
               CHAIRMAN APOSTOLAKIS:  Okay.  My expert in English
     tells me yes.  I will not question it.  I have questioned it
     in the past and have regretted it.
               MR. BERNSEN:  I would observe that this is a
     formal process.
               CHAIRMAN APOSTOLAKIS:  But it's not structured.
               CHAIRMAN APOSTOLAKIS:  Thank you very much, Mr.
               DR. KRESS:  I had a couple of items on the
               CHAIRMAN APOSTOLAKIS:  Sure.
               DR. KRESS:  Most of mine were covered by Mario,
     but on page 6, the definition of core damage frequency, I
     wonder why the shied away from the usual connotation that's
     per year instead of per unit time, although, you could
     define it anyway you want to, but it's usually in the use of
     CDF and LERF, it's always per year, per reactor year.
               CHAIRMAN APOSTOLAKIS:  Actually, in the text
     somewhere they say that it's not per reactor year; that's
     it's per calendar year.  That was a question I wanted to
     ask, why-- because you're considering all modes of
     operation, so even if the reactor --
               But if the reactor is in cold shutdown, do you
     really care?  I mean, the definition is somewhere, and let
     me see if I can find it.
               You're saying it in the text, but --
               DR. KRESS:  If Dana were here, he's say, yes, I
               CHAIRMAN APOSTOLAKIS:  Yes, I what?
               DR. KRESS:  If Dana were here, he's say, yes, I
     care, to your question.
               CHAIRMAN APOSTOLAKIS:  I think there is an
     inconsistency between the definition and the text.  Karl?
               MR. FLEMING:  Yes, with respect to the -- I
     believe that in the technical requirements for quantifying
     initiating event frequencies, for example, you'll see the
     need for expressing units in terms of calendar year.   
               That's just to clarify that the alternative might
     be to calculate it per reactor operating year, and then
     you're going to be coming up with units that may be
     inconsistent with the criteria, you know, all the safety
     goals and core damage objectives, and so forth, really are
     calendar year.
               There has actually been some confusion out there
     in the industry about what calculations should be performed.
               DR. KRESS:  Okay, the other question I had was on
     common-cause failure.  I thought that defining it in terms
     of a short period is a good idea, but it leave me wanting a
     little bit more in terms of what is meant by short.
               It has something to do with whether the two
     failures are close enough in time that they actually impact
     the sequence somehow.
               And so somebody needs to add a little more of a
     definition of "short" in there that I thought it could be
     expanded on.
               DR. BONACA:  I think it's a very good comment. 
     For example, you may have oftentimes a  -- mode failure is
     caused by a replacement, say, of a component with a
     different material that will lead to the failure later on.
               Many of them are latent, and then may develop
     themselves in a long time.
               DR. KRESS:  You to have a certain probability
     that's going to impact the sequence.
               DR. BONACA:  That's right.  So that's a good
     requirement to clarify that.
               CHAIRMAN APOSTOLAKIS:  Karl, do you plan to spend
     any time on initiating events?  I would like to finish
     initiating events and expert judgments, so that Bob can be
     off the line.
               MR. FLEMING:  Fine.  If you have questions, I did
     not prepare.
               CHAIRMAN APOSTOLAKIS:  Okay, let's finish first.
               DR. BONACA:  I just have one more comment,
     unfortunately, on -- a question, actually.  That's why I'm
     raising it, on the definition on page 9, under PRA Upgrade.
               It says the incorporation into the PRA models of a
     new methodology that has not been previously peer-reviewed. 
     I assume that if I incorporate it into my model, a new
     methodology, whether or not it was peer-reviewed, it would
     be an upgrade of my PRA.
               MR. FLEMING:  Yes.
               DR. BONACA:  Unless I misunderstand what you
               DR. KRESS:  It doesn't matter whether it's peer-
     reviewed or not.
               DR. BONACA:  That's right.
               DR. KRESS:  It's still an upgrade.
               DR. BONACA:  May I'm missing something.
               MR. BERNSEN:  We could let Rick answer that, but I
     think the intent here is that this is a definition that's
     unique to the standard, and, in particular, to the peer
     review section where we're talking about what changes in the
     PRA need to have a peer review.
               So it's kind of -- it's unique to the standard,
     and that's why the differentiation.  Is that right, Rick?
               MR. HILL:  Well, actually, I don't think I'd have
     a problem with taking out the, "that has not been previously
               Yes, it is unique to the standard, but the context
     of what an upgrade is, is a change in methodology, rather
     than just a change in time phasing like data or something
     like that.
               I also think that this has not previously been
     peer-reviewed, might skew the definition to somebody
     thinking, well, this particular methodology has been
     reviewed someplace else, so, therefore, it's acceptable
     here, without thinking about the application of that
               MR. WALL:  Mr. Bonaca, I'd like to draw your
     attention -- sorry, this is E.M. Wall, a team member.
               Mr. Bonaca, I'd like to draw your attention to
     page 136, configuration control, Section 5, and Subsection
               We used these two definitions to distinguish an
     upgrade from maintenance.  For a maintenance, we kind of
     even have kind of an internal review.  It's for very minor
               An upgrade is a major thing which will require
     some incremental peer review, pursuant to Section 6.
               DR. BONACA:   I understand, but still, I mean, I
     may decide to upgrade by adding seismic or fire, okay?  And
     I'm going to use a PRA methodology which has been previously
     reviewed.  I'm asking somebody to put it in.
               That's a major upgrade of the PRA.  And so I would
     call it an upgrade, irrespective of whether or not that
     methodology has been peer-reviewed.
               MR. SIMARD:  We'll look at that.  It sounds like
     we ought to delete that phrase and just end the sentence
     after new methodology.
               DR. BONACA:  I don't want to belabor it, I'm just
     pointing out that it is something to look at.  Thank you.
               MR. SIMARD:  Okay.
               CHAIRMAN APOSTOLAKIS:  Are we ready to move on to
     initiating events?  Since Carl Doesn't have any -- maybe you
     can put up the viewgraph you have which is the -- no, it's
     on page 21, where it talks about --
               MR. BUDNITZ:  George?
               CHAIRMAN APOSTOLAKIS:  Yes?
               MR. BUDNITZ:  In the sequence of the text expert
     opinion. comes first.
               CHAIRMAN APOSTOLAKIS:  Really?
               MR. EISENBERG:  No, it doesn't.
               MR. BUDNITZ:  Doesn't it?
               CHAIRMAN APOSTOLAKIS:  It's on --
               MR. BUDNITZ:  It's 4.6.  I apologize.
               CHAIRMAN APOSTOLAKIS:  It's page 135.
               MR. BUDNITZ:  Of course.
               CHAIRMAN APOSTOLAKIS:  Are there any comments on
     initiating events from the members?
               [No response.]
               MR. BUDNITZ:  Well, that was easy.
               DR. SHACK:  My one comment is sort of really just
     that it does apply that there is an awful lot of detail that
     was in the Draft 10 that disappeared from the Draft 12.  I
     mean, that's common to the whole thing, and it's this
     philosophy, perhaps --
               I mean, typical ASME standards are fairly
     prescriptive, and they provide a lot of detail.  You know,
     I've heard references here that the philosophy here is to
     provide sort of a high level guidance to the peer review
     panel who are assumed to be knowledgeable.
               And you've omitted a great deal of detail that is
     in other guidance documents like NUREG 1602, which was
     another attempt to sort of set up guidance for PRA, or the
     Draft 10 version.
               And haven't you really lost something here in
     omitting these details?
               MR. BUDNITZ:  Well, about initiating events --
     this is Bob Budnitz from 3,000 miles away -- about
     initiating events -- and I believe that this was something
     that was true all the way through --
               Remember -- and it's very important for you to
     understand that the whole standard is telling the analyst or
     the analyst team what to do and not how to do it.  What to
     do, and not how to do it.
               Now, in the course of reviewing Rev 10, there was
     some stuff in there that told them how to do it, and I, like
     the others, took that out.  Actually, I didn't have to take
     a lot of it out; it was taken out in the intermediate thing
     you never saw called Rev 11.
               When a subset of our group took Rev 10, they made
     the major, major changes of going to three columns of
     requirements, and integrating the NRC certification
     requirements with what had been there before to make a
     larger list and straightening things out, a whole lot of
     detail was taken out that was of the character of how to do
               And that was true here, too, however, I don't
     think that, unless you find one -- and I'd be, of course,
     eager to know -- I don't think there was any what-to-do's
     that we lost in the course of taking out a lot of that how-
               So although it comes up here in initiating events,
     it's really a question of philosophy for the whole thing. 
     It just happens to come up here, first, I suppose.
               CHAIRMAN APOSTOLAKIS:  Karl?
               MR. FLEMING:  Yes, to amplify on what Bob just
     said, if you go back to Rev 10, on the Section 3, I guess it
     was, that had the detailed requirements, the entire content
     of all the requirements for initiating events was on page.
               All the other material that you found in Rev 10 on
     initiating events and other issues like that, was back in
     the Appendix, which was in the form of guidance and things
     like that.
               So, actually, if you look at the detailed
     requirements we have in initiating events in Rev 12, there
     is actually more here.  There is more because we have
     integrated in additional requirements that were in the
     certification process that were not in Draft 10.
               MR. BUDNITZ:  But either that one -- it was really
     one that had pages, Karl, you're right.
               Even there, there was some stuff that was how to
     do it, that I then went through and took out, in the spirit
     of what we were trying to do with the whole thing.
               You see, if there are five different ways to
     accomplish a certain thing, we made a decision up front that
     it was erroneous for us to prescribe one of them.
               Now, by the way, if they're all equipment -- and
     no one had ever done a PRA in this area -- it might be
     useful to prescribe and have everybody do it the same way.
               But, in fact, we've got 100-odd PRAs out there,
     that all did it different ways.  And you don't want anyone
     of them to say, gee, you did it incorrectly, because you
     didn't do it the way we told them how to do it.
               So, if you're reacting to that, I believe your
     observation is completely correct, and we did it on purpose.
               CHAIRMAN APOSTOLAKIS:  There is some
     inconsistency.  I agree with you, Bob, that this is a
     broader issue than just initiating events.
               MR. BUDNITZ:  Oh, of course.
               CHAIRMAN APOSTOLAKIS:  And I was planning to bring
     it up when we discuss human reliability analysis, and expert
               In other words, in some instances, you give more
     detailed guidance in the form of references, and in others,
     you don't.
               MR. BUDNITZ:  Well, I  --
               CHAIRMAN APOSTOLAKIS:  It's a matter of being
               MR. BUDNITZ:  Well, without arguing the case, it
     is -- if you can point out places where we can give
     references that provide a good example of how one goes about
     it, why, those are very valuable.
               CHAIRMAN APOSTOLAKIS:  I mean, I realize that this
     particular standard is not really a procedures guide.  It
     doesn't really give you methods.
               MR. BUDNITZ:  Quite the opposite.
               CHAIRMAN APOSTOLAKIS:  You stayed away from it,
     and, in fact, one of the criticisms, as you told us earlier,
     was that you were too prescriptive in Rev 10.
               So, the least we can do then is, when we discuss
     HRA, and expert judgment, is to make sure that there we
     eliminate the more specific advice that is given, which is
     inconsistent with the other chapters.
               Anything else in initiators?
               DR. KRESS:  I had one, George.
               CHAIRMAN APOSTOLAKIS:  Sure.
               DR. KRESS:  On page 33, Table 4.4-1(d), under Item
     1(e)-D14, we talk about that the frequencies need to be
     weighted by the fraction of time the plant is at power.  I
     think that needs to be made a little more clear that the
     weighting goes in the denominator instead of the numerator. 
     It may be clear to everybody else.
               MR. BUDNITZ:  By the way, this is exactly the
     place where the adjustment is made to the difference between
     a reactor year and a calendar year.
               DR. KRESS:  That's right.
               MR. BUDNITZ:  That's exactly the point that we
     spoke about this about five minutes ago?
               DR. KRESS:  That's it.
               MR. BUDNITZ:  This is the only place it's done. 
     This is the only place where frequency comes in, in quite
     this way, right?
               DR. KRESS:  That's right, and that's why I thought
     it need to be made a little clearer as to what you're doing
               MR. BUDNITZ:  Well, explain -- no sweat. What
     wording would you --
               DR. KRESS:  Well --
               MR. BUDNITZ:  You're going to tell them to do the
     arithmetic right, or something?
               DR. KRESS:  Well, that's basically that's it.
               CHAIRMAN APOSTOLAKIS:  We do that in the
               DR. KRESS:  If it's clear to everybody, okay.
               MR. BUDNITZ:  If it isn't clear --
               DR. KRESS:  It was clear enough to me, but I
     wasn't sure it would be clear to everybody.
               I have another sort of comment on page 25 where
     we're introduced to key safety functions, which I thought
     was --
               MR. BUDNITZ:  Which requirement?
               DR. KRESS:  This is high level requirements for
     initiating event analyses on page 25 of my version.  At the
     footnote, we're introduced to key safety functions.
               I like that.  I liked the list that they have
     there, but the problem I have with it is, I hated to see
     that relegated only to a footnote.  I wish there was a
     section in there talking about key safety functions and the
     role they play here.     
               In fact, you see this footnote showing up with
     multiple tables, all along through here.   I thought that if
     you could take care of it up front, and not have to repeat
     it every time, it might help the readability a little bit.
               But somehow I thought this was too-important a
     concept just to relegate to a footnote, and that was the
     only comment there.
               CHAIRMAN APOSTOLAKIS:  Can you give me an example
     of a safety function that is not a key safety function?  If
     I look the way you define them, it seems to me that you
     covered everything, reactivity control, core heat removal,
     reactor coolant inventory control, reactor coolant heat
     removal and containment bypass integrity.
               MR. BUDNITZ:  Oh, I suppose you'll find -- I
     imagine that if I told you that the Center for Disease
     Control in Atlanta concentrated on key diseases, I could
     probably come up with some minor diseases they don't
     concentrate on.  I bet there are some.
               CHAIRMAN APOSTOLAKIS:  No, I'm not sure.
               MR. BUDNITZ:  Maybe not.  Control --
               CHAIRMAN APOSTOLAKIS:  What you have listed here -
               MR. BUDNITZ:  Inventory, that's pretty much the
     whole thing.
               CHAIRMAN APOSTOLAKIS:  It is pretty high level,
     and it sounds like it's all-inclusive, so they are really
     safety functions, so there isn't such a thing as a key
     safety function.
               I mean, the moment you talk about core heat
     removal, reactor coolant inventory control and reactor
     coolant heat removal --
               DR. KRESS:  Where would you put flooding the
     cavity in that?
               CHAIRMAN APOSTOLAKIS:  Is that a safety function?
               DR. KRESS:  I consider it one.  Where would you
     put operation of the sprays?
               CHAIRMAN APOSTOLAKIS:  I would say if you deleted
     the word, bypass, and you said containment integrity, then
     all these things are included there.
               DR. KRESS:  But they didn't.  They had containment
     bypass integrity.
               CHAIRMAN APOSTOLAKIS:  Just because of the word,
     bypass, we define a new class of safety function?
               MR. BUDNITZ:  No, no, these aren't those.  These
     are functional initiating event categories or categories
     that affect these things.
               DR. KRESS:  Oh, you're talking about initiating
               CHAIRMAN APOSTOLAKIS:  I think the word, key, is
     redundant here.  I mean, you really have to try very hard to
     find something that doesn't belong there.
               MR. BUDNITZ:  Well, going once, going twice, it's
     out.  It also has the phrase, are the minimum set -- well,
     these include, at a minimum, X, Y, and Z, so you're right,
     there is the freedom to throw something else in there; do
     you see it?
               DR. KRESS:  You're right.
               CHAIRMAN APOSTOLAKIS:  Anyway, we don't want to
     make a big deal out of it.
               MR. BUDNITZ:  Are you taking notes about these
     things, because I'm not.
               MR. SIMARD:  Yes, we're taking notes and we're
     also going to get a copy of the transcript.
               CHAIRMAN APOSTOLAKIS:  There is a transcript.
               MR. BUDNITZ:  Enough said; it's done, okay?
               CHAIRMAN APOSTOLAKIS:  Anything else on
               [No response.]
               CHAIRMAN APOSTOLAKIS:  From the members?
               [No response.]
               CHAIRMAN APOSTOLAKIS:  One quick question on page
     26 under Transients in the bottom box:  And this is now a
     requirement that applies to all three categories.
               MR. BUDNITZ:  Which requirement are you in?
               CHAIRMAN APOSTOLAKIS:  This is Table 4.4-1(a),
     page 26, 331, 331-B, transients, loss of offsite power and
     manual shutdowns.  Are these the only transients we're
     looking at?
               MR. SIMARD:  Well, we do say that the following
     list is not intended to be all-inclusive.
               CHAIRMAN APOSTOLAKIS:  I mean, automatic shutdowns
     for some reason are not a transient?
               DR. KRESS:  They generally are categorized as
               CHAIRMAN APOSTOLAKIS:  Why do we distinguish?  Why
     the word, manual?  Should it be just shutdowns?
               MR. SIMARD:  Absolutely.
               CHAIRMAN APOSTOLAKIS:  Bob?
               MR. BUDNITZ:  I can't remember why that's there.
               CHAIRMAN APOSTOLAKIS:  Well, maybe you guys can
     think about it.
               DR. SHACK:  That's really one of the things that
     really got stripped down compared to Version 10.  There was
     a much longer list and much more detailed thing in 10 than
     there is in 12.
               MR. BUDNITZ:  Yes.  It should say manual and
     automatic, just to flesh it out.
               CHAIRMAN APOSTOLAKIS:  Yes.
               MR. BUDNITZ:  I don't argue that for a moment.
               CHAIRMAN APOSTOLAKIS:  Especially since these
     things are now part of the performance indicators, right?
               MR. BUDNITZ:  That isn't relevant to us.  We're
     doing a PRA.
               CHAIRMAN APOSTOLAKIS:  To support risk-informed
               MR. BUDNITZ:  In part.
               CHAIRMAN APOSTOLAKIS:  Why am I arguing with you,
     Bob.  Maybe now we can go back to Mr. Fleming.  You plan to
     talk about accident sequence analysis?
               MR. FLEMING:  Yes.
               CHAIRMAN APOSTOLAKIS:  Good.  I don't know, but
     how do we handle this expert judgment?  Should we do it now
     so that Bob can --
               MR. FLEMING:  It's your call.
               CHAIRMAN APOSTOLAKIS:  Okay, let's do expert
     judgment.  That is on page 155, as I recall.  One of the
     comments of the Committee was time -- was too detailed and
     focused on one approach.
               MR. BUDNITZ:  Well, it's fair to say that the
     original version 18 months ago was ten times as long.
               CHAIRMAN APOSTOLAKIS:  Yes.  The first comment,
     Bob, is --
               MR. BUDNITZ:  With a whole lot of detail that's
     just gone, including a long appendix that's gone.
               CHAIRMAN APOSTOLAKIS:  Yes.  Unlike other chapters
     -- we're coming back to the earlier comment about
     consistency and so on.
               You are giving two references here.  Other
     chapters, or most of them stay away from providing
               MR. BUDNITZ:  But you notice that they're
     permissive, may be used to meet the requirements in the --
               CHAIRMAN APOSTOLAKIS:  But you know, the moment
     you say "may be" in a standard, I mean, that's --
               MR. BUDNITZ:  No, "may" is a crucial word that is
     used in standards to indicate a permissive that is not
               CHAIRMAN APOSTOLAKIS:  Right, but then --
               MR. BUDNITZ:  Other approaches may also be used.
               CHAIRMAN APOSTOLAKIS:  But then the question is
     why these two?  For example, the second one would seem to me
     to apply to high level waste repositories.  It's from the
     NMSS Branch of the NRC, and it says in the title, in the
     high level radioactive waste program.
               Why does this belong in a reactor standards?
               MR. BUDNITZ:  Because it's a method.
               CHAIRMAN APOSTOLAKIS:  It's a method?  I thought
     they just reviewed existing methods?
               MR. BUDNITZ:  Well --
               CHAIRMAN APOSTOLAKIS:  The first reference is a
     method, but the second one really was a review to state the
     Branch position and I'm not sure that this helps anybody
               And then the big question you're going to get is,
     why are you ignoring NUREG 1150?  If you are going to put a
     Branch technical position on high level radioactive waste
     repositories, you are -- you are not citing the major study
     that involved expert judgments sponsored by the Nuclear
     Regulatory Commission.
               MR. BUDNITZ:  George, you might know the answer to
               CHAIRMAN APOSTOLAKIS:  I know.
               MR. BUDNITZ:  That was written by Apostolakis and
     Budnitz and a bunch of other people.  If you don't know
     that, George and I were the authors of 6372.
               The answer, George, is that in 6372, after a lot
     of thinking, we rejected some of the methodology used in
               CHAIRMAN APOSTOLAKIS:  But we never really look at
     what the staff did in NMSS, and I'm pretty sure if we
     reviewed that, we would have some comments as well.
               My point is that the moment you start putting
     references, you get these questions.  You know, why didn't
     you include this guide?  Why didn't you include that guide? 
     Why do you have this fellow?
               I would say it would be probably best to not have
     any references at all.  Now, that severely limits the
     ability of the user to really do something, but it would be
     consistent with the rest --
               Or, just take out the second reference, which I
     think is irrelevant here, and put two or three more.  I'm
     sure you're going to get this comment about NUREG 1150.
               I mean, they went through a major exercise there. 
     They spent a lot of the Agency's money, and now we are not
     even citing them.
               MR. BUDNITZ:  On 1150, I'm prepared to write a
     rebuttal if anybody says that, and I assume that you will
     review and tell me I was right about it.
               You remember what they didn't do that was right.
               CHAIRMAN APOSTOLAKIS:  But since you are allowing,
     Bob, a graded approach to the use of expert judgment --
               MR. BUDNITZ:  Sure.
               CHAIRMAN APOSTOLAKIS:  Surely there is a role for
     1150 somewhere there?  I mean, in Category II issue, for
     example, I mean, we're even allowing the technical
     integrator to do it internally to the company, you know,
     without even going to outside experts.
               In that sense, there must be a role for 1150
     somewhere.  I mean, I am on your side when you say that in
     the full treatment, the first reference we have here goes
     beyond 1150.  And now it just occurred to me, can we comment
     on things that we have co-authored?
               MR. BUDNITZ:  Well, I am on the phone with you so
     we can do what we want.  You just stated on the record that
     you and I co-authored that, so everybody understands.
               CHAIRMAN APOSTOLAKIS:  There is Mr. Markley here
     who has some views.
               MR. MARKLEY:  Well, George, to the extent that you
     can, you should avoid discussing your own work.
               CHAIRMAN APOSTOLAKIS:  What I am doing here, Mike,
     is I am staying away from the technical content.
               MR. MARKLEY:  You can state the facts.
               CHAIRMAN APOSTOLAKIS:  I am just stating that
     there are other references that I think belong.
               DR. KRESS:  Right.
               MR. MARKLEY:  There is nothing wrong with that.
               DR. KRESS:  You can provide clarifications, and to
     the extent that you are not supporting the reference, you
     can actually add to the discussion.
               CHAIRMAN APOSTOLAKIS:  Sure.  And I think it is
     evidence from our exchange with Bob that we are not really
     getting into the details.  My point is that since you
     decided to cite the reference, and I think that is
     appropriate here, because it is not easy to find these
     things, it seems to me you have to cite a few more for
     completeness.  And especially since your write-up, the text,
     does allow for different approaches that involve different
     levels of sophistication, if you will.  That is all the
     comment I have to make here.
               DR. KRESS:  But when you start adding more
     references you always have the completeness problem.
               CHAIRMAN APOSTOLAKIS:  Well, yeah, but I mean
     there are two or three major, like 1150, I mean, for
     heaven's sake, it introduced the formal use of expert
     judgment to the nuclear safety business.  There were lots of
     little papers here and there, some of them mine, but 1150
     really pulled the whole thing together.
               DR. KRESS:  But there is a whole science out there
     on expert elicitation.
               CHAIRMAN APOSTOLAKIS:  Sure.
               DR. KRESS:  With books and texts, and where do you
               CHAIRMAN APOSTOLAKIS:  But what I am saying is we
     should limit ourselves to things that have been used in the
     nuclear business, especially the ones -- I don't know that
     the industry has supported any major studies in this area,
     but the NRC certainly has.  There was another one later on
     Level III.
               DR. KRESS:  Level III.
               CHAIRMAN APOSTOLAKIS:  Which was in collaboration
     with the European Union.
               MR. BERNSEN:  George, we will certain consider the
               One of the other observations I would make is that
     we have talked to Bob about the fact that there was a lot of
     valuable material in the earlier drafts that shouldn't be
     lost.  And I believe he is committed to write a paper which
     might be suitable for reference in this issue or some
     subsequent issue of our standard.  We felt that a technical
     paper would be more useful for that type of information
     perhaps than a standard.
               CHAIRMAN APOSTOLAKIS:  That's fine.  But, again,
     looking at the standard alone, since there are no references
     in other places.
               MR. BERNSEN:  Understood.  And we will certainly
     consider that.
               CHAIRMAN APOSTOLAKIS:  Jack?
               MR. SIEBER:  I was just thinking that that is a
     good idea to write a supplementary paper, because I think
     that then becomes the tutorial for the application of the
     standard, and without it, I think there is something
               MR. BUDNITZ:  Well, on the other end, I said I
     would do that, but it hasn't been done yet, and it certainly
     isn't going to get into this edition of the standard, you
     know, because, obviously, you know how long these things
               George, by the way, so here is George -- George is
     talking about this, and without about arguing about
     conflicts, George, offline, let's have a conversation about
     what other references might be appropriate here, and I will
     give it some thought.
               CHAIRMAN APOSTOLAKIS:  Sure.
               MR. BUDNITZ:  And, by the way, just to broaden
     this, I can call up and have discussions with two or three
     other members of our team, there were seven authors there,
     like Peter Morris and so on, and see if I can pull together
     an improved little list.
               CHAIRMAN APOSTOLAKIS:  Yeah, I am not talking
     about, you know, 35 references.  I am talking about --
               MR. BUDNITZ:  Certainly.  You are talking about
     two or three more.
               CHAIRMAN APOSTOLAKIS:  Two or three key, major
     references, you know, that included nuclear related issues.
               One or two more comments.  I think, Bob, you
     undertook a very difficult task here trying to give guidance
     as to when to use, you know, the facilitator approach or the
     technical integrator, and there is a series of four bullets
     on page 135.  I would suggest, I mean I understood what you
     meant here, but, you know, I have spent three years with you
     working on this, earlier on the standard says that examples
     will be used to clarify things.  In fact, in Section 4.4 on
     requirements, there is a series of examples.  I would
     suggest that on this Section 4.6, you give a few examples of
     what you mean by certain things.
               For example, are there any Level I issues that
     would require a TFI treatment, or would the technical
     integrator treatment would be good enough?  The one that
     comes to mind from 1150 is the coolant pump seal LOCA, where
     there is model uncertainty.  Would that be a good example? 
     And maybe that one can be handled by the utility itself,
     since you allow them to do that, by a technical integrator. 
     Then, as you move on to Level II, I suspect for some of
     these issues, one would have to do a more rigorous expert
     judgment elicitation process, you know, so people will get a
     better idea.
               I am afraid that this is not clear now, unless you
     really have read some of the citations.  And especially when
     you say, on page 135, 4.6.3, "The PRA analysis team may
     elect to resolve a technical issue using their own expert
     judgment or the judgment of others within their
     organization."  Now, if I were a utility person, I would say
     this is great, we can resolve all the issues internally. 
     Maybe we will call up one or two consultants to make sure we
     are not doing anything really bad, and then I would not read
     the rest.
               Why should I worry about uncertainties are large
     and significant judgments of outside technical experts are
     useful?  I mean since you allow me to resolve technical
     issues within my organization, I would probably do that.
               MR. BUDNITZ:  Yes, but read the next sentence.  I
     will read the sentence you read, "The PRA analysis team may
     elect to resolve a technical issue using their own expert
     judgment or --"  Right.  But the next sentence, "The PRA
     analysis team shall use outside experts when the needed
     expertise on the commission is not available inside."
               CHAIRMAN APOSTOLAKIS:  Well, I understand that,
     and that again --
               MR. BUDNITZ:  And then there is a "should" which
     is sort of in between.  It says maybe you have the experts,
     there is a "should" which is in between.  You got it, but
     there are other reasons why you want to go outside.
               CHAIRMAN APOSTOLAKIS:  But, again, I mean then it
     comes down to deciding whether I have the expertise or not,
     which, of course, --
               MR. BUDNITZ:  Yes, but that is always discussion
     that is left up to the analyst team.  Nobody but the analyst
     team could ever make that call.  I think that is intrinsic
     to this game.
               CHAIRMAN APOSTOLAKIS:  But all I am saying is --
               MR. BUDNITZ:  Do you agree with that?
               CHAIRMAN APOSTOLAKIS:  A few -- no, I agree.  But
     a few examples, I mean not 10, but two or three.
               MR. BUDNITZ:  I can cite some examples.
               CHAIRMAN APOSTOLAKIS:  Of issues.
               MR. BUDNITZ:  The way I can cite it is I can cite
     two or three reports which cover issues.
               CHAIRMAN APOSTOLAKIS:  Yeah.
               MR. BUDNITZ:  In other words, the analyst who is
     trying to figure out what the hell is what, could go to that
     coolant pump seal example, or they could go to, for example,
     full elicitation at Yucca Mountain for seismic hazard or
     something, to see the whole big, gory thing.
               CHAIRMAN APOSTOLAKIS:  Well, this is actually a
     good example, or 1150.  I mean in Level II analysis, there
     are all sorts of issues that require expert judgment, right.
               MR. BUDNITZ:  Right.
               CHAIRMAN APOSTOLAKIS:  Although I don't know if
     you would limit yourself to LERF, or whether there is the
     same number of issues.  But, certainly, if you do the
     traditional Level II, with the release of --
               DR. KRESS:  There are a lot less issues if you do
     it to LERF.
               CHAIRMAN APOSTOLAKIS:  A lot less.  But the
     question that would come to my mind would be, if I were a
     utility executive, why can't I go to NUREG-1150?  They did
     all this analysis, maybe I can take their results, use my
     expertise in my facility, maybe hire a consultant, and maybe
     adopt those results to my plant.  So I don't have to go
     through this expert elicitation process and all that.  I
     mean do you allow that reality here, which I suspect a lot
     of people would find very attractive?
               MR. BUDNITZ:  Well, of course.
               CHAIRMAN APOSTOLAKIS:  Because the idea of going
     through a NUREG-1150, it is just out of the question for a
     private company to do.  I mean it is okay for a federal
     agency that wants to gain insight and so on.
               MR. BUDNITZ:  By the way, of course, it is not
     only allowed, it is explicitly -- it is expected, I suppose. 
     But they do have to get by their peer reviewers.
               CHAIRMAN APOSTOLAKIS:  That's right.  And all I am
     saying is by giving two or three specific examples, like I
     just did, I think you will make this section much easier for
     people to understand and implement.
               Also, you don't emphasize enough this community
     perspective, which, for a private company, may not be
     relevant.  Remember when we were doing this other thing, a
     very important concern for a federal agency that is looking
     at broader issues is what is the community of experts' view
     or a spectrum of views on a particular issue?  Because this
     is a federal agency, they have to regulate 103 units.  But
     if I am one utility with one or two plants, I probably don't
     care about the community of experts, do I?  I mean I really
     worry about what applies to my facility.  Although, of
     course, there I can have the community's views.  So, I would
     suggest that this become clearer.
               MR. BUDNITZ:  Well, it is right there, the last
     paragraph of 4.6.4, probably this whole thing is only, you
     know, half, two-thirds of a page.  Read it.
               "The utility shall be responsible for aggregating
     the judgments so as to develop the composite distribution of
     the informed technical community."
               CHAIRMAN APOSTOLAKIS:  Yeah, but what I am saying
     is that this is not sufficient to bring up the issue of the
     community view.  Maybe you can emphasize it a little more. 
     I mean every word here, every sentence is loaded with
               MR. BUDNITZ:  You and I know that this was 100
     pages turned into three-quarters of a page.
               CHAIRMAN APOSTOLAKIS:  I know.  I know.  So, maybe
     by using a few examples, you can make it a little clearer
     and that is all I have.  And maybe thinking again about the
     issue of references, either eliminate all of them or add two
     or three more.
               MR. BUDNITZ:  I see your point.
               CHAIRMAN APOSTOLAKIS:  Yeah.  Are there any -- I
     guess the issue of expert opinion elicitation does not arise
     when you do a Category I.  I mean it really has to be
     Category III, right
               MR. BUDNITZ:  Well, I mean if you are just having
     a couple of experts in, which is not only allowed, it is
     probably the most common thing.
               CHAIRMAN APOSTOLAKIS:  A Category II perhaps.  But
     Category I, which is --
               MR. BUDNITZ:  I mean, you know, you have a couple
     of experts in, you still have got to follow this, you just
     do it in a certain way.  Right?
               CHAIRMAN APOSTOLAKIS:  That's right.  I think it
     is so short that it probably will not be of great use to
     people, but I don't expect individual utilities to really
     resort to expert judgment elicitation to a large degree
     anyway.  I mean this is more like a federal kind of
               Any other members have any comments on this
     particular issue?
               [No response.]
               CHAIRMAN APOSTOLAKIS:  Well, I guess we are done
     with Bob.  Bob, do you have any comments?
               MR. BUDNITZ:  Yes.  I am not sure, and my
     colleagues are sitting around the table there, how much more
     here -- well, you know, we cut this way down on purpose
     because there didn't seem any middle ground between
     something that was real short and the whole big banana,
     which didn't make sense, it was out of context.  That thing
     that was in the first thing was out of context, it was as
     long as the rest of the standard practically.
               CHAIRMAN APOSTOLAKIS:  Yeah.
               MR. BUDNITZ:  I suppose, you know, 25 percent more
     doesn't place it too much out of -- you know, doesn't screw
     up the balance
               CHAIRMAN APOSTOLAKIS:  Yeah.
               MR. BUDNITZ:  And I will see what I can do.  Maybe
     it is only just a sentence here and there.
               CHAIRMAN APOSTOLAKIS:  Okay.  Have you thought
     about eliminating the whole section, or is that out of the
               MR. BUDNITZ:  Of course we did.  I didn't think
     eliminating it made sense because without some guidance, you
     leave it wide open.
               CHAIRMAN APOSTOLAKIS:  Okay.
               MR. BUDNITZ:  I mean you do want to say things
     like look at the last one.  You do want to tell them who is
     responsible.  You do want to tell them that they have go to
     identify the issue and that they shall go outside.  You do
     want to tell them they shall go outside when they don't have
     the needed expertise, I think, don't you?
               CHAIRMAN APOSTOLAKIS:  Yeah.  Mario.
               DR. BONACA:  Just I want to make sure before Bob
     goes, I want to pick up again on something we talked about
               CHAIRMAN APOSTOLAKIS:  Okay.  Are we done with
     expert judgment?  I think we are done.
               MR. BERNSEN:  Just let me make an observation with
     regard to this.  I mean our primary purpose here is we are
     responding to comments received.
               MR. BUDNITZ:  Yes.
               MR. BERNSEN:  And in that context, I don't recall
     any comment that said, take it out.  I do think we had
     comments that said it seemed to be over-weighted in terms of
     the total approach in the standard.  And yet, obviously, you
     want to recognize that this is a part of the process and
     must be considered.  The user is obligated to address, and
     the peer review team has the opportunity to evaluate.  So, I
     don't think we should take it out.
               And, obviously, your comments on the examples and
     detail and references are quite appropriate.  But we did not
     get any suggestion that said delete it
               CHAIRMAN APOSTOLAKIS:  And you are not getting one
     now either.  I am not suggesting that.
               MR. BERNSEN:  Right.
               CHAIRMAN APOSTOLAKIS:  I just asked a question
     whether you have considered it.
               I think this kind of exercise is really foreign to
     most utility practitioners.  I don't think they will go
     through this thing, unless somebody is about to shut down
     their facility and there is a major seismic issue, and then
     it is a different story.  But in their routine application
     of PRA, I doubt it very much.  But I agree with you, there
     should be some guidance.
               So let's go on to the other issue that Dr. Bonaca
               DR. BONACA:  Yeah, I just had -- I wanted to pick
     up on the issue, Bob, of what you described before, that is,
     from Rev. 10 to Rev. 12, you really took out how to do it.
               MR. BUDNITZ:  Well, that is not fair.  And the
     others in the room can elaborate.  Sprinkled throughout all
     of this text are how to do it, you know, at different
     levels.  Because sometimes you couldn't describe what to do
     without telling them how.  You know, there has to be a
     flavor of that or it can be sterile in some places.
               But where there were five ways to do something for
     sure, it was erroneous to tell them how to do one of those,
     you see.
               DR. BONACA:  Yeah.  And I don't disagree with the
     approach you have taken.
               MR. BUDNITZ:  Do people around the table agree
     with me, my team members there?
               DR. BONACA:  Well, let me finish my question.
               MR. BUDNITZ:  Yeah.
               DR. BONACA:  At least I don't disagree that you
     should do that.  I am only considering some cases not always
     to do one thing are equal, I mean some of them leave behind
     some problems.  And, you know, I have heard time and time
     again from PRA practitioners, you know, a discussion about
     the method, because if you follow a certain method, then you
     have to do something else later on to back up some possible
     shortcoming in the approach.  And I think the original
     version we reviewed, Rev. 10 had some of those elements
     inside that
               Now, I am not saying Rev. 12 doesn't have it,
     because I haven't performed that kind of evaluation.  I only
     see that, you know, I am just concerned that -- it seems to
     me, okay that a lot of the burden now has been placed on the
     back of the peer review process, that is supposed to make
     sure that all this possible, you know, pitfalls in the kind
     of approach you use are being dealt with.  Am I correct or
     not?  I don't know.  That is a question.
               MR. BUDNITZ:  Does anybody else want to try to
     answer that, too?  I don't know where the burden is.  The
     burden is always on the analyst to do it right, and for the
     peer reviewers to check, isn't it?
               DR. BONACA:  Well, I mean --
               MR. BUDNITZ:  It is just like a running a reactor,
     the burden is on the reactor operator to run the reactor. 
     The NRC can't.
               DR. BONACA:  Let me explain to you why I have got
     a problem, okay.
               MR. BUDNITZ:  No, I understand.
               DR. BONACA:  No, no, no.  No, let me just finish. 
     I have got a problem because you keep saying that this
     standard applies to -- as a standard is going to be used by
     the power plants and there is a full process here that, in
     fact, describes, you know, the utility use of this.  And, in
     fact, the peer review process is also very focused on
     utility use.  Most of these utility people did not perform
     the PRAs.  PRAs were performed by, in a lot of cases, by
     specialists who were not participated in the peer review
     process, who are not parties to this.  And so those kind of
     capabilities are not being applied in this review process. 
     That is why I am raising it.  That is the only reason.
               I would have no problem if all this is going to be
     administered by, you know, top, experienced industry
     practitioners.  This is going to be applied individual
     utilities, typically, with one or two PRA people, that is
     the whole staff they got, plus a lot of other people in
     their expert panel, and that is why I am raising these
               And there may be an excellent answer.  I just said
     it seems to me that there has been a significant shift, and
     maybe it is not significant, but some shift of
     responsibility to the peer review process.
               CHAIRMAN APOSTOLAKIS:  Well, I think the issue you
     are raising, Mario, is really much bigger, and it comes back
     to the earlier discussion we had on whether this is a
     procedures guide or a high level guidance document.  It
     seems to me it is inevitable that this will happen, what you
     just described, because they have to stay away from actually
     prescribing methods.  In fact, even the previous version,
     Rev. 10, that we saw was criticized as being overly
               And I have a comment, for example, on the common
     cause failure part here that lists five methods, and without
     any qualification, and I think one of them should not really
     be there at all.  So, you know, unless you are an expert in
     that field, unless you are Karl Fleming, you will not really
     know which method to use.
               DR. BONACA:  And I want to point out, --
               CHAIRMAN APOSTOLAKIS:  I would rather eliminate
     all five.
               DR. BONACA:  -- George, in fact, I didn't disagree
     that this could be done.  And I am not not supportive of
     this.  I am only making a statement that, in my judgment, I
     see some shifting of the role to the review process, or the
     peer process, which, in fact, you have.  And maybe we will
     talk about the peer process, review process later.
               CHAIRMAN APOSTOLAKIS:  We will.  But I think you
     see that in many, many applications like Option 2 and so on. 
     And it is inevitable.  When you really don't trust the
     numbers or the models to guide you to a decision, you have
     to rely on the judgment of people.  And this is what is
     happening here, too.  It may be in a different context, but
     it is the same thing.
               DR. BONACA:  Yes.
               MR. BUDNITZ:  Mario, let me talk about how this
     standard differs from most standards that ASME puts out. 
     And I will tell you that I just had this exact same comment. 
     I am chairing the group that is writing, under the American
     Nuclear Society, the seismic external hazards piece.  And we
     had a meeting of our oversight committee just last week and
     it came up, the exact same question.
               One of our committee members objected strongly to
     what we had written because it wasn't prescriptive enough. 
     And I will just read you an example, but I am going to read
     it from initiating events.  If you have initiating events,
     the first high level requirement says, I will read it to
     you, "The initiating event analysis shall provide a
     reasonably complete and appropriately group treatment of
     initiating event categories."  Okay.
               Now, let's focus on the word "reasonably."  Shall
     provide a reasonably complete treatment of the stuff. 
     Right.  Now, and the same thing came up in the seismic PRA,
     but it is generic throughout here.  The person said, what do
     you mean by reasonably complete?  I mean, you know, when you
     are telling people how to design, how to do a calculation of
     the stress on a piece of metal, you don't talk about
     reasonably complete, you give them a method.
               And we had this discussion, and, of course,
     everybody understands in PRA that if we ever said the word
     "all" or "complete," it is death.  There is no "all" and
     there is no "complete" in PRA, because you have to screen
     things out that are unimportant compared to other things. 
     So, you have to use the word "reasonably."  And then you
     come -- what is reasonably complete?  Well, the analyst has
     to make that call, and the peer reviewers have to agree with
     it.  You are stuck with that, there is no way around that,
     Mario and the rest of you, in my view.  And I hope everybody
     in the room will nod at what I am saying.
               It is that sort of judgment that in the end --
     that is why this standard, this whole area is different than
     most ASME standards or ANS standards that one writes in
     other areas.
               DR. BONACA:  And, Bob, I totally agree with you, I
     understand it.  I am only --
               MR. BUDNITZ:  So it is dilemma that we are in from
     the start.
               DR. BONACA:  Yeah.  But this sets certain parts in
     my mind about the peer review.
               MR. BUDNITZ:  Of course.
               DR. BONACA:  I know that when I was still -- and I
     knew we had the peer reviews of a number of PRAs, and I am
     going back in my memory to see what the qualification of the
     people were.  Just because -- and probably they were
     adequate.  I am only saying that I didn't even think about
     the peer review when I was reviewing Rev. 10, and maybe that
     was my problem, but now I am thinking about it more
               CHAIRMAN APOSTOLAKIS:  Okay.  Any other comments?
     I'm sorry.
               MR. BERNSEN:  Just one.  I did look at the changes
     from 10 to 12 fairly thoroughly in the initiating event
     area.  And substantially, most of 10 is there.  A lot of the
     narrative stuff may have been deleted, but the what
     requirements by and large remain.  In the initiating event
     area, there wasn't that much change.
               MR. BUDNITZ:  Wait, wait, wait.  There was no
     requirement in Rev. 10 that isn't here.
               MR. BERNSEN:  What I mean is, and Bob did the same
     thing independently, so --
               MR. BUDNITZ:  There is absolutely no requirement
     that is missing.
               MR. BERNSEN:  Right.  Okay.  I guess I stand
     corrected.  That is true.  And so, I don't think there is a
     major change, in fact, from -- going from 10 to 12 for this
     particular element.
               DR. BONACA:  No, I am only mentioning that in some
     cases there were examples.  Remember, you can use this
     approach or you can use that approach.  And if you use this,
     you should be doing also this and this and that.  There was
     that part.
               MR. BERNSEN:  That was kind of -- it was in 10 as
               DR. SHACK:  It sort of comes back to George's
     question.  You know, if you look at (a)(4) in 12 versus the
     list in of 10, there really is a great deal of
     difference in the list.  You know, you get the transients
     and the LOCAs in (a)(4) and you have a much more descriptive
     sort of thing in  And the question is why just
     these, you know?  You know, should you have dropped them
     all, or, you know, included them all.  And it is, you know,
     it is another one of those things.  Where do you stop?  And
     it is just curious, I guess is the answer.
               MR. BUDNITZ:  Well, that is a hard call.  If you
     are looking 1(e)(a)(4), which is 3.3.1.
               DR. SHACK:  Yeah.
               MR. BUDNITZ:  Well, it says at the bottom this is
     not intended to be all inclusive.  So, you know, it was a
     shot, okay.
               DR. SHACK:  Well, the other list wasn't all
     inclusive either.
               MR. BUDNITZ:  Right.  Right.
               DR. SHACK:  You know, there is no all inclusive
               MR. BUDNITZ:  Right.  That is what I said three
     minutes ago.
               DR. SHACK:  Right.  And, you know, the question
     is, where do you stop when you give the example list?  The
     one in (a)(4) seems awfully abbreviated, I guess is sort of
     my just general gut reaction.  And why give such an
     abbreviated list, you know, versus what was in the 8, the
               MR. BERNSEN:  It is distributed to other elements. 
     In other words, if you go back to 1(e)(b)(3), so on and so
     on, it is distributed.  It is still there, but it has been
               MR. FLEMING:  I think it is a valid comment, but
     the discussion that resulted in the solution that we adopted
     in Rev. 12 was the following, that as you go from a general
     list or a larger list to the appropriate list for a
     particular plant, one ends up with the unavoidable
     conclusion that the appropriate set of initiating events is
     plant-specific.  So, we worked on the requirements, the
     supporting requirements, put a lot of emphasis on needing to
     resolve the dependencies, the plant-specific details and so
     forth, to come up with the appropriate list before you are
               The more we worked with more detailed lists, the
     more we got arguments about, well, this is more of a PWR
     list and not a BWR list, and it doesn't belong in my plant,
     and so forth, so I think we all agree that one needs a
     detailed list of initiating events to support the PRA but
     putting a -- we did not want to promote the concept of using
     a standard list of initiating events.
               That is why we did it the way we did it.
               MR. BUDNITZ:  Yes, Karl.  We could, for example,
     have taken the, what? -- 12 or so things on this list, we
     could have made this 62, but then you might convey the
     flavor to the initiated that if you do all those 62 you are
     done, and that is exactly the wrong flavor.
               CHAIRMAN APOSTOLAKIS:  I think we have exhausted
     the subject and it is a broader comment and --
               DR. BONACA:  Yes, we have.
               CHAIRMAN APOSTOLAKIS:  Anything else?
               [No response.]
               CHAIRMAN APOSTOLAKIS:  Okay, Bob.  Thank you very
               MR. BUDNITZ:  Okay.  I'll ring off.
               CHAIRMAN APOSTOLAKIS:  Okay, bye bye.
               MR. BUDNITZ:  Thank you.
               CHAIRMAN APOSTOLAKIS:  And maybe we can also take
     a short break until maybe just before -- until 11:18.
               CHAIRMAN APOSTOLAKIS:  Back in session.  I don't
     have a quorum.
               MR. MARKLEY:  Your quorum -- 13 --
               CHAIRMAN APOSTOLAKIS:  All right.  Okay, back to
     Mr. Fleming.
               MR. FLEMING:  What I wanted to do at this point if
     it is convenient for the committee is to go through some of
     the details of the accident sequence element to bring out
     some additional aspects of the structuring of Rev. 12 of the
     standard and make some key points that we haven't really
     stressed up to now.
               In addition to the three column, three category
     approach, which is of course where the detailed supporting
     requirements are listed, the other feature of this draft of
     the standard was the derivation of high level requirements
     for each of the elements.
               These high level requirements I believe are very
     important, a very important enhancement to the standard from
     the following perspectives that I will get into.
               The process we used to develop them was to start
     with the basic attributes of a PSA -- completeness,
     dependencies, fidelity -- those types of issues and look at
     those in the context of the particular objectives for each
     element, which are also listed for each of the elements in
     the standard and come up with an irreducible set, minimum
     set of requirements for a PRA, for any category of
     application that would have to be met.
               This for example, the slide I have up here right
     now, is on page 38, which is the high level requirements for
     accident sequence analysis.
               MR. EISENBERG:  Page 39.
               MR. FLEMING:  Okay, it may be on page 39 in your
               Table 4.4.2 -- this is the style in which we
     presented all of the requirements for each of the nine
     elements in the standard.
               What we have done is developed from a high level
     perspective really the fundamental requirements that have to
     be met.
               On these requirements we worked very hard, and I
     think the degree of consensus that the project team and the
     other industry groups that peer reviewed and provided input
     to this process, the degree of consensus that was reached at
     this level is much stronger than actually was a reality at
     the functional requirement level.
               I, for one, my own personal opinion is that if a
     peer review team consisted of appropriately experienced and
     competent practitioners in PRA they could take these high
     level requirements and go in and perform a very good, sound
     peer review process to determine the quality of the PRA.
               We wrote these down for several reasons, one of
     which is to show the context and logic for all the
     supporting requirements.
               To the extent that people have optional approaches
     to do common cause, data, whatever, and to the extent that
     someone may not have exactly followed some of the supporting
     requirement tables that we have, the judgment, the yardstick
     on which you should judge the adequacy of an alternative
     approach would be with respect to these high level
               I wanted to mention these, because this is a very
     key addition to what we had in Draft 10.  While many of
     these concepts were in Draft 10, they were kind of buried in
     the textual presentation and we brought them out as very,
     very explicit requirements.
               The detailed requirements are then organized by
     each of these high level requirements.
               I might just pick a page of these for accident
     sequence definition.  As you go into the specific supporting
     requirements for accident sequence, and this happens to be
     several pages in, for Requirement B, which shows you the
     style in which we have presented all these supporting
     requirements, we have carried down into these detailed
     requirements -- at the top of the table a reminder of what
     functional requirement we are supporting.
               This particular page happens to be Functional
     Requirement B on plant-specific CDF and LERF quantification;
     the accident sequence analysis shall provide a sequence
     definition structure that is capable of supporting a plant-
     specific quantification of CDF and LERF by the Level I-II
               That is the functional requirement that all of
     these particular detailed requirements refer to.
               The second thing that we remind people of is that
     as we apply the high level requirement and the supporting
     requirements we are applying them against the attributes
     that are the column headings in these tables, where we
     repeat the particular attributes for this element, which
     provides the scope of applicability of each of the detailed
               As you read these particular -- these are the same
     ones I showed in a previous table -- a common theme here is
     that Category 1 is focused on the dominant sequences and
     contributors to core damage frequency and large early
     release, whereas Categories 2 and 3 have to be extended to
     the risk significant accident sequences and contributors.
               In recognition of the fact that some Category 3
     applications may have to go beyond risk significant accident
     sequence contributors we dropped that caveat with -- you
     know, it is sort of an implicit way of saying something more
     than the risk significant and accident sequences.
               Then as you go down into the specific
     requirements -- now we are into the very, very details of
     the supporting guidelines, one thing that you will see here
     is that for some elements like initiating events and
     accident sequence definition, for example, these elements
     are so fundamental to the overall structure of a PRA that
     you won't see a great deal of delineation of specific
     requirements coming across the columns.
               The example that I threw up here, in most cases
     what you will see in this differentiation is a permission of
     the application of conservative models in Category 1 and a
     lack of tolerance of conservative models in Categories 2 and
               Then based on the judgment of the project team,
     there is in some cases a softening of the language, the
     action statements that we are using that are specifying the
     detailed supporting requirements.
               The choice of the verbs was considered very
     carefully but using the judgment of the project team as to
     whether something ought to be included in the model or
     whether it should just be considered in the model, and that
     is another example of differentiation.
               When you get into other elements like data
     analysis and quantification and HRA you will see quite a bit
     more differentiation of detailed requirements across the
     three columns, and the classic example is what we were
     talking about earlier is that we go from point estimates,
     mean values, to full uncertainty quantification, as one
     particular type of example.
               But then finally we end up for all of the
     elements, and if I am right -- I'll get the particular
     pages -- the particular requirements for documentation.
               In general you will find, and this just happens to
     be the table on page 51, which may be 52 in your package,
     you will generally find the documentation requirements to be
     common across all three elements.
               The practical reason for this is that in order for
     a peer review team to even determine what category of
     application the PRA can support it is necessary to have that
     document so we can measure what is in there, so you tend to
     see particular aspects of that.
               The other thing that we tried to avoid and we
     debated at length in the preparation of this draft of the
     standard, we tried to avoid having let me say "buzzwords"
     like "shall" or "should" or "may" trigger some kind of an
     automatic documentation requirement, so what we tried to do
     is for each element to have very specific documentation
     requirements for that element that is in the judgment of the
     project team is necessary and sufficient for a peer review
     team to come in, read that documentation, determine the
     category of application that that element is capable of
     supporting, and whether it meets the intent of the
               You will see a very, very specific long list here. 
     In fact, I think if you look at these documentation
     requirements you will see that by and large they include the
     documentation requirements that were in Rev. 10 as well as
     additional documentation requirements we had in the
               So that is -- you know, the purpose of this part
     of the presentation was to just walk you through a little
     bit more of the structure to point out the role of the high
     level requirements and the way in which we'd carried down
     the high level requirements and the attributes for each of
     the three categories, so that each time one is reading a
     specific requirement in the tables, they can provide the
     context, they can grasp the context of that requirement and
     what it was intended to achieve in interpreting how far to
     implement it.
               Those represent some of the structural changes and
     enhancements that we tried to put into this draft of the
     standard, again exclusively motivated by the comments that
     we received on Draft 10 to make the standard easier to use
     for a range of applications.
               That pretty much concludes what I planned to
     formally present as far as the standard, so if you have any
     other comments --
               CHAIRMAN APOSTOLAKIS:  It appears to me that
     Category 1 is really separate from the other two and the
     distinction between Categories 2 and 3 is really a very fine
               If you look at the various requirements you are
     imposing, usually you ally both of them to Categories 2 and
     3, with some exceptions.
               Perhaps -- I think this is a very important issue. 
     Maybe the subcommittee can debate it a little bit.  The
     categories are discussed on pages 3 and 4 of the standard in
     terms of examples.
               In Category 2, give examples, typical
     applications, risk informed prioritization of GL96-05,
     periodic valve verification testing requirements, risk
     informed inservice testing, risk monitoring applications,
     quality assurance, and tech spec modifications, it appears
     that most of these, if not all, really rely on importance
     measures to rank SSCs, don't they?
               Then if you go to Category 3, you also have PRA
     products are used to prioritize and rank SSCs with respect
     to safety significance.  Well, if that is the case, then why
     when I do a GQA I have to prioritize, rank SSCs, so which
     category does that belong to?
               On the one hand, you are telling us this is
     Category 2 but then in Category 3 you say that when I rank
     SSCs I have to do a Category 3 analysis.
               This issue came up yesterday in the other meeting. 
     I guess there is an implicit assumption here that the
     importance measures are fairly insensitive to a full
     Category 3 treatment, that you can get a reasonable ranking
     without going to the details of Category 3, which is an
     untested assumption, and perhaps somebody ought to test
               Maybe for RAW that is valid because you make such
     drastic assumptions, although the remaining is still, you
     know, sensitive.  I mean you set a component down but the
     rest of the stuff is at their nominal values.
               I don't know that what you have here on pages 3
     and 4 is the best description of what the categories are and
     whether -- I mean there is no statement here regarding the
     degree of confidence that one has to have in the PRA results
     and how that degree of confidence really dictates how
     sophisticated your analysis should be, but isn't that really
     what it comes down to?
               MR. FLEMING:  Yes, that is very important --
               CHAIRMAN APOSTOLAKIS:  And that is related of
     course to approaching the forbidden region in the diagrams
     of 1.174, increased management attention, that the Staff
               I wonder whether these kinds of thoughts can be
     reflected on these, that what really matters is the degree
     of confidence that is required in the calculations to
     support the application and then you go on to the examples
     and so on.
               DR. SHACK:  The one that struck me as funny here
     was the A4 Category 1, where --
               CHAIRMAN APOSTOLAKIS:  Yews.
               DR. SHACK:  -- in A4 you are looking in
     combinations that may be rather unusual and different, and
     yet it is often in the category where you think you are
     almost looking at the generic PRA and that just struck me as
     a kind of an unusual way for that.
               I would have characterized the A4 applications as
     somewhat different, where you are really looking for some
     unexpected, surprising interactions.
               CHAIRMAN APOSTOLAKIS:  Which comes back to Mario's
     surprise.  Karl?
               MR. FLEMING:  Yes.  I think the particular --
     sorry, I lost my fuzzy there -- back to fuzzy sets.
               I think that is a good comment. I think the reason
     why something like A4 was placed in Category 1 in this
     document is based on the fact that A4 is a rule that was
     imposed on utilities and of course the utilities were
     expected to use their existing PSAs, whatever their existing
     PSAs, they had to implement that rule, but there was really
     no requirement at least from a regulatory perspective that
     they employ the kind of elevation of a PRA, which would be
     expected, like, say, a Reg Guide 1.74 application, so I
     think that is sort of the motivation for putting it down
               It was a rule that they had to fulfill anyway and
     there was not a formal requirement by the NRC that you have
     to do a quality -- you have to upgrade to a certain level of
     quality of PRA before you can implement the rule, so I think
     from a technical standpoint your point is well-taken.
               MR. SIMARD:  We have had a number of
     comments these are not good examples and we need to revisit
               One example though that seemed to resonate at the
     workshop yesterday, Karl, maybe you could summarize the
     discussion about how given that you have a Generic Letter
     88-10 MOV testing program, the distinction between
     Categories 1 and 2?
               MR. FLEMING:  Right.  Yes, we had a very good
     discussion on this and I think it did provide us with some
     good insights on how we can improve the discussion on these
     pages in response to George's earlier comment.
               For example, Generic Letter 89-10, we would take
     the position that use of the PRA to apply risk ranking
     quantifications to your MOV list just for the purpose of
     deciding which ones you are going to test first in
     fulfilling the 89-10 requirements without impacting the
     scope of the valves in 89-10 would be a Category 1
     application, but if you wanted to say that, hey, we only
     wanted -- we are going to exclude certain requirements from
     scope and now we are actually bending or modifying or
     proposing a relaxation of the rule on a plant-specific
     basis, now we are talking more of a risk-informed
     application a la 89-10 in which we would actually have to
     quantify the risk impact of the part of the rule that we
     weren't planning to fulfill.
               While the risk ranking might have been a useful
     prioritization, we would have had to elevate this to a risk-
     significant determination to show that the delta risk
     associated with modifying how we are going to apply the rule
     is justified, so that would be how we would make those
     particular distinctions.
               DR. BONACA:  I would like at some point to go back
     to the Category 1 --
               CHAIRMAN APOSTOLAKIS:  Well, this is the point. 
     This is the time.
               Go ahead.
               DR. BONACA:  Well, it seems to me that one
     fundamental requirement I believe is that the PRA must be
     commensurate with the change that it supports.  That is a
     fundamental element of the whole standard here.
               It seems to me that there is a prejudging by
     saying that Category 1 specifically addresses SSC risk
     significant determination for the maintenance rule and A4. 
     There is a prejudgment here.
               Will it be true that the utility still has a
     responsibility to assess his own PRA, determining in fact
     for example that the component is described and all that
     kind of stuff, and look at the dependencies and if, in fact,
     dependencies are not, then maybe he has to upgrade the whole
               Now the concept is within the standard.  I
     understand that, okay? -- but by reading this under Category
     1 it seems there is almost a prejudgment that anything that
     meets Category 1 it will be adequate for doing this support
     of the maintenance rule.
               Do you feel that it is supportable this way,
     particularly for the issue of dependencies, which were
     sallow at times, as we discussed before, because in the
     application of the maintenance rule that is exactly what is
     happening out there.
               There are very at times innocuous pieces of
     equipment which are removed from service.  They appear to be
     innocuous.  They may be in the air system.  They may be, you
     know, systems which are not safety grade typically.
               MR. FLEMING:  Right. There is one aspect of your
     comment, Dr. Bonaca, that was not intended by the project
     team, and that is the treatment of dependencies.
               I believe that, and we will have to go back and
     check this, but I believe that in the phrasing of the
     requirements for treatment of dependencies which would show
     up in a number of places including initiating event,
     sequence definitions, systems analysis, probably
     quantification and LERF, but as you go through and look at
     the detailed requirements I don't believe that we let the
     PRA staff off the hook, so to speak, in treatment of
     dependencies with the exception that we may permit a
     conservative treatment of dependencies for some of the
     Category 1 applications.
               I don't think that we really intend to avoid the
     need to find all the system interaction and functional
     dependencies and common cause dependencies and so forth even
     for a Category 1 application.
               I think in our upgrade of these pages for the
     final draft that we would consider in light of these
     comments, I don't think that we intend to make these
     assignments of applications on an exclusive basis in the way
     in which it is arrange in the standard right here and right
               There may indeed be an A4 -- applications of A4
     that in the way in which they are implemented at a
     particular plant really call for a Category 2 or 3 of PRA.
               DR. BONACA:  I guess what makes me uncomfortable
     is singling out the maintenance rule and A4 as a specific
     application in Category 1, when I do believe that the
     configurations you may end up with by pulling out equipment
     out of service at power, it would be more challenging than
     other changes more formal that you are making, in part also
     because for more formal changes you do have a more thorough
     process.  They have more time --
               MR. FLEMING:  Right.
               DR. BONACA:  -- and other things, so I think you
     have to be -- I don't know, this just seems to single out
     maintenance rule and A4 as a less challenging situation and
     I don't think it is.
               MR. FLEMING:  There is one other important caveat
     that perhaps we haven't been too clear on, and that is that
     it makes A4 a particularly interesting example to discuss
     because this standard and these requirements are really only
     covering the annual average CDF, LERF part of the PRA.
               All of these new issues that come into play for
     the time-dependent risk monitoring applications that
     differentiate above those different -- this standard does
     not really go into that.  It is really outside the scope of
     our standard, and I think for that reason maybe A4 would not
     be a very good example.
               CHAIRMAN APOSTOLAKIS:  But you have an example
     under Category 2, risk monitoring application.  That is not
     part of the standard?
               MR. FLEMING:  I am saying that when you look at
     the technical requirements for initiating events and
     accident sequence and so forth, we really do not go into the
     additional requirements that you would have to have in there
     to do risk monitoring applications.
               For example, time dependent initiating events --
     the details of what is in the standard right now don't go
     into the additional technical issues that come into play
     when you do time dependent risk monitoring in applications.
               MR. SIMARD:  Well, I think it was a good idea to
     put these examples in because it has brought out some really
     good discussion here, but I think what I am hearing is that
     we need to go back and reconsider whether we have any
     examples at all, because of Dr. Bonaca's point about
     prejudging the outcome.
               The other thing I am hearing from the past day and
     a half is that we need to go back and make even clearer our
     expectations here that, first of all, experience with the
     certification process shows that as you look at the various
     subelements of a PRA you will find for every PRA -- I think
     every PRA that has been looked at some of the subelements
     are grades 2, some are grades 3, some are even grades 4, so
     we have found the spread among existing PRAs that would
     roughly correspond to the three categories.
               Second, we need to make clear that what we are
     talking about now does not describe the PRA, it describes
     the attributes of an application.  And for a specific
     application, our intent is to go through the PRA subelement-
     by-subelement, and for a particular subelement determine
     what level of capability that application calls for.
               So, I think we need to do a better job of
     reinforcing the point that this does not describe the PRA. 
     We are not giving an overall grade, for example, to the PRA.
               DR. BONACA:  I wanted to just point out, first of
     all, I really want to be reasonable.  But let me give you an
     example of what my concern would be.  My concern would be I
     have a very simple PRA and I am going to take out of service
     two safety systems.  Okay.  And my PRA recognizes them, pull
     out one and two.  Then I have some component in a support
     system is not safety related.  Nobody recognizes the safety
     role to it, so, therefore, we take it out of service because
     it doesn't fall into the maintenance rule listing or
     anything like that.  And we know that there are dependencies
     out there.
               Okay.  Now, typically, the operations people are
     pretty smart.  At times, they don't see it.  We are all
     human.  And so that is the scenario under which, you know,
     the maintenance rule still has a lot of experience to -- you
     know, as far as online maintenance to be developed, and so
     it is not such an easy application.  So, anyway, it is just
     a comment.
               CHAIRMAN APOSTOLAKIS:  I would like to come back
     to what Mr. Simard just said.  You said that you look at
     what PRAs are out there and they roughly correspond to one
     of the three categories.  It seems to me that this should
     not be a criterion for defining the categories, because you
     should go one step beyond that and ask yourself, has the NRC
     staff, or have these PRAs of the various categories been
     actually used in some of these applications successfully?
               Because it is true perhaps -- no, I am sure it is,
     that there are Category I PRAs out there, but have they
     actually been used in risk significance determination for
     the maintenance rule?  And has the NRC staff said this is
     good enough?  That should be the criterion, because the fact
     that the PRAs exist out there independently of their use in
     the decision-making process really doesn't tell us very
     much.  So, I wonder whether that is the case, whether
     anybody came here with an IPE that was what we call now
     Category I, they submitted an application and the staff
     said, this is good enough.
               MR. FLEMING:  In the effort that went into the
     original development of the industry peer review
     certification process that was originally sponsored by the
     Boiling Water Reactors Owners Group, now all the Owners
     Groups have picked up on a variation of this process, the
     information that existed at that time with respect to
     different plants' successes and failures with risk-informed
     applications was taken into account in the definition of
     these.  These categories were originally defined in the
     certification process.
               And as a matter of fact, the great success that
     the South Texas project had provided many of the examples
     for the checklist that were developed to differentiate, at
     least with respect to some elements, the category -- what is
     now the Category III applications, for example.  So that,
     the information that did exist with respect to the track
     record of PRAs using the decision-making process, went into
     the original definition of these categories.
               CHAIRMAN APOSTOLAKIS:  The question is whether the
     staff now accepted these applications.  And South Texas
     perhaps is not the best example because they have a very
     good PRA.  Level III, right?
               MR. FLEMING:  Right.
               CHAIRMAN APOSTOLAKIS:  A Category III.  So they
     can support the arguments that they may want to make using a
     very good PRA.  But I think that should be really a good
     test, because just because there are PRAs out there that are
     of the three categories doesn't really mean very much unless
     they have been tested in a real risk-informed decision-
     making environment, which means the NRC staff has reviewed
     them and said, yeah, for this application, this is good
     enough.  And I don't know of any case where the staff did
     not actually go to Category III and raise questions.
               Now, that may be temporary because we are all
     learning, but --
               MR. FLEMING:  But, again, at the time these
     categories were initially defined, there was quite a large
     database of experiences at South Texas and other plants with
     risk-informed applications that had a track record of
     success that provided an information base on which to define
     these categories.
               CHAIRMAN APOSTOLAKIS:  In other words, has anyone
     used the Category I IPE to satisfy the (a)(4) requirements?
               MR. FLEMING:  Well, the definition of that
     category was defined to capture those activities which the
     industry was, in general, using, but was not subjected to an
     additional special peer review process of their PRA to
     verify the application.
               CHAIRMAN APOSTOLAKIS:  The reason why I think this
     is very important is because it is my understanding that
     there is a Presidential directive that all federal agencies
     should use national standards to the maximum degree
     possible, correct?
               MR. MARKLEY:  OMB Circular A-119.
               CHAIRMAN APOSTOLAKIS:  There you are.  So, one
     possible misuse of this might be that, you know, if it
     becomes a national standard, a licensee may come to the
     staff and say, you know, you are not following the OMB
     directive because you have to follow the standard, and the
     standard says that for (a)(4), I can use a Category I
     analysis.  So, now we are really changing the process. 
     Instead of establishing the categories based on a mutual
     interaction between the staff and the industry, now we are
     trying to impose on the staff certain limitations as to what
     they can ask and what they can expect.
               So, I think this is a very critical point here
     when it comes to the categories.
               DR. BONACA:  Because I think it is important to
     note, also, regarding the online maintenance, utilities who
     are making -- who were doing online maintenance before, at
     times they were performing evaluation with their IPEs or
     PRAs, many of them.  So there has been a backfitting and
     they have been using whatever they had.  And so, I don't
     think we want to, you know, make the standard endorse this
     necessarily, force this on the NRC, an acceptance of a
               CHAIRMAN APOSTOLAKIS:  Yeah, I mean it should not
     impose limitations on what the NRC staff may want to do.
               MR. SIMARD:  Can you help me understand your
     concern?  Because if we eliminate any pre-judgment, if we
     eliminate the statement that this particular application
     fits Category I and so forth, all we are doing is -- and,
     again, we are not talking about Category I PRAs.  This part
     of the standard we are talking about describes the
     application, not the PRA.  So, all we are doing is
     recognizing that given applications may require PRA
     subelements of varying capability, that for a particular
     application, you may need a fairly robust treatment, in one
     area of your PRA, and the way you have treated other
     elements of your PRA may not be as important.
               So, all we are doing is setting in place a
     framework without committing the NRC staff to any judgments
     with respect to a given application.  All we are doing, if
     we eliminate the examples here, is saying, for applications
     that have the following categories, here is an appropriate
     level of PRA.
               Now, it is our intent that it is up to the NRC
     staff to make the judgment of particular applications.
               DR. BONACA:  If you are leaving out the examples,
     I thought I would agree with you.  I have no problem at all. 
     The only issue here, in my eyes, was, by de facto, you had
     established that the capabilities necessary to support
     (a)(4) maintenance rule are less than the capability to
     rank, you know, risk prioritizing and less than others.
               MR. SIMARD:  Yeah, no.
               DR. BONACA:  That is really a prevarication of the
     process.  The process --
               CHAIRMAN APOSTOLAKIS:  As long as the NRC staff
     doesn't get its hands tied because of the OMB directive --
               DR. KRESS:  They are already standard.
               CHAIRMAN APOSTOLAKIS:  I'm sorry?
               DR. KRESS:  Go ahead.
               CHAIRMAN APOSTOLAKIS:  And they actually judge and
     say, well, gee, that is what this says, but we really
     believe you ought to do this and this and that to satisfy
     (a)(4), then I don't have a problem.  Yes?
               MR. BERNSEN:  George, I think this discussion was
     very useful.  We need to take it back and consider it,
     because there is no way that this standard is going to put
     out the concept that we are making decisions on where you
     apply it in regulatory space.  These were intended to be
     examples of typical current usage.  And apparently it is not
     clear.  And I think as the discussion proceeded, it isn't
     clear.  We need to reconsider that.
               Now, what Karl presented yesterday in the workshop
     describing the attributes of the different categories was
     very useful.  And, you know, it pointed out that, you know,
     for the Category I, we are talking about cases wehre you are
     using the PRA to support deterministic and you are not
     changing licensing bases and things of this sort, and so on.
               We need to go back and focus on that, and take
     another look at this.  And if these are not good examples of
     current usage, we need to take them out.  And we make it
     clear that this is -- we are not prescribing them.  The
     standard does not prescribe them, that is done by the
     regulator.  We are sensitive to that, so that is a good
               CHAIRMAN APOSTOLAKIS:  I think Section 1.5, pages
     3 and 4, should be revisited with that point of view.
               MR. BERNSEN:  Right.
               CHAIRMAN APOSTOLAKIS:  You know, this is a very
     sensitive issue.  Maybe in the paragraph, the second
     paragraph that talks about the boundaries between the
     categories and so on, bring up the issue of degree of
     confidence and so on.
               MR. BERNSEN:  Right.
               CHAIRMAN APOSTOLAKIS:  But the introductory
     paragraph perhaps should make it clear as to what these
     typical applications are intended to -- the message they are
     intended to convey, and that in no way are binding somebody. 
     I don't know if you can say.
               MR. SIMARD:  Exactly.
               CHAIRMAN APOSTOLAKIS:  Because I think this
     particular section, you know, is very critical in how the
     categories will be viewed later on, because you don't want,
     again, to have people say, gee, it was Category II, and the
     OMB says you have follow it, and all of a sudden the NRC
     staff is on the defensive why they are violating an OMB
     directive and they don't follow a national standard, you
               DR. BONACA:  The other interesting thing is that
     one could contend that in order to evaluate two or three
     different components simultaneously, you need quite an
     advanced degree sophistication, you know, and so you want to
     look at it, too.  Okay.  I just added that.
               CHAIRMAN APOSTOLAKIS:  Okay.  Any other comments
     and categories?  Yes.
               MR. FLEMING:  I wanted to clarify one thing that
     one of my colleagues on the project team wanted me to point
     out, and there is a lot of confusion, I think, or
     opportunities for confusion when one compares these three
     categories in our standard to the categories that were
     originally defined in the industry certification process.
               These categories here refer to the industry
     Categories II, III and IV, and whereas I in the industry
     certification process was the IPE level.  So, I just wanted
     to make a clarification here, is that this Category I is
     already raising the bar, I think to a significant extent
     above what was expected for the IPEs.  So, I just wanted to
     clarify that point.  Category II is further up the bar.
               MR. BERNSEN:  I don't know, we may revisit this
     again, but it is appropriate to bring it up at this stage. 
     We felt that it was useful to have the three categories in
     our standard because it does reflect current usage, and it
     does recognize that there different grades for the various
     elements and supporting requirements.  And we are looking
     for feedback on that, what your reaction is to that, because
     it is an important concept of the standard.
               CHAIRMAN APOSTOLAKIS:  On this particular subject,
     any comments from the NRC staff?
               [No response.]
               CHAIRMAN APOSTOLAKIS:  Public?
               [No response.]
               CHAIRMAN APOSTOLAKIS:  I guess -- Karl, do you
     have any more viewgraphs?
               MR. FLEMING:  No, that is all.
               CHAIRMAN APOSTOLAKIS:  I have comments on the HRA
     and data analysis.  I don't know, Jack, do you have anything
     on the accident quantification?
               DR. BONACA:  On the quantification, yeah.
               CHAIRMAN APOSTOLAKIS:  You do.
               DR. BONACA:  Yeah.
               CHAIRMAN APOSTOLAKIS:  So, I guess we can break
     for lunch now and then pick up the specifics after that.  We
     have comments on specific sections.  I don't think it will
     take more than an hour or so.
               DR. BONACA:  No, I don't have extensive comments.
               CHAIRMAN APOSTOLAKIS:  Gerry.
               MR. EISENBERG:  I just wondered, the question Dr.
     Bernsen raised is a more generic question.
               CHAIRMAN APOSTOLAKIS:  Yes.
               MR. EISENBERG:  Are we going to revisit that after
               CHAIRMAN APOSTOLAKIS:  I thought we were done, but
     at the end there will be -- we will go around the table so
     that the members --
               MR. BERNSEN:  I guess my question is, is the
     discussion clear on this?  Have we had a reaction from you,
     or are we still waiting?  With regard to --
               CHAIRMAN APOSTOLAKIS:  Our reaction to the
               MR. BERNSEN:  Your consideration of the
     acceptability of retaining the three categories and the
     utility of it.
               CHAIRMAN APOSTOLAKIS:  Well, why don't we think
     about it and maybe come back to it.  But at the end I plan
     to go around the table and maybe this is a question to which
     we will have to.
               MR. BERNSEN:  Fine.
               DR. BONACA:  I think there is a lot of questions,
     of course.  And, you know, I want to say that still I can
     see the strength of the high level requirement approach,
     that is a real strength over the previous, because it is
     structured, the top process, how you get into that.
               CHAIRMAN APOSTOLAKIS:  Well, the thing is the NRC
     staff itself, in Reg. Guide 1.174, recognized that the
     degree of sophistication of a PRA varies with the
     application.  That is why we have the shades of gray and
     there is a discussion about sensitivities and model
     uncertainties and so on, as you approach the boundaries.
               So, people do recognize that not all PRAs have to
     be, you know, the perfect PRAs for all applications.
               What you are doing here is you are going one step
     beyond that and you are actually trying to formalize that by
     defining categories.  And, you know, the implications and
     consequences of this kind of thing is something that we are
     all thinking about.  And, again, my concern, as I expressed
     earlier, is how are certain licensees going to use this in
     light of the OMB directive.  And if it is used to impose
     certain constraints on the staff, then I think that would be
     an unfortunate use of the standard.
               So, let's come back to this at the end, around
     2:00 or so, after we finish with the specific questions. 
     But this is certainly something that is extremely important.
               Okay.  So, we will come back at 1:00.
               [Whereupon, at 12:05 p.m., the meeting was
     recessed, to reconvene at 1:00 p.m., this same day.].                   A F T E R N O O N  S E S S I O N
                                                      [1:06 P.M.]
               CHAIRMAN APOSTOLAKIS:  Back in session.  So, we
     will discuss first, some of the items under Risk Assessment
     Technical Requirements, and some of the specific questions,
     and then perhaps go back to a general discussion of the
     standards, and let's plan on finishing maybe like 2:15 or
               Okay, we've discussed already initiating events,
     accident sequence analysis, success criteria.  Any comments?
               [No response.]
               CHAIRMAN APOSTOLAKIS:  System analysis. Success
     criteria, I think Bill Shack may have some comments.  Do we
     know where he is?
               MR. MARKLEY:  Have you seen Dr. Shack?
               MR. SIEBER:  He had an appointment over lunch.
               CHAIRMAN APOSTOLAKIS:  He did?  Well, that brings
     us to Human Reliability Analysis, which is me.
               And that takes us to page 76.  Yes, that's where
     it starts, right, 76.
               On page 78, under C, for pre-initiator HRA, it
     says that the evaluation of errors in pre-initiator human
     action shall be performed using a well defined process that
     recognizes plant-specific nature of the human failure
               Are you referring to the Swain Guttman various
     adjustment factors there?
               MR. MROWCA:  Bruce Mrowca, I'm a Project Team
     Member and also from Baltimore Gas and Electric.  I didn't
     get to the section you were talking about.
               CHAIRMAN APOSTOLAKIS:  Page 78.
               MR. MROWCA:  78.
               CHAIRMAN APOSTOLAKIS:  There's a table there on
     high level requirements for human reliability analysis. 
     Under C, Quantification --
               MR. MROWCA:  Okay.
               CHAIRMAN APOSTOLAKIS:  Plant-specific nature, the
     plant-specific nature of the human failure events, I was
     wondering what that meant.
               MR. MROWCA:  This is for pre-initiating actions?
               CHAIRMAN APOSTOLAKIS:  Yes.
               MR. MROWCA:  My page numbers are different than
     yours.  That's what I'm struggling with.
               MR. BERNSEN:  Our page 76.
               MR. WALL:  Dr. Apostolakis, if I might offer a
     small suggestion, if you use the index number on the
     righthand column, it avoids the confusion about the page.
               CHAIRMAN APOSTOLAKIS:  Righthand column.  How
     about if I give you a table number, 4.4-5.
               MR. WALL:  The index number gives you the right --
               MR. BERNSEN:  But he's talking about the high
     level requirements, Table 4.4-5, page 78 of mine.  Is that
     the same page for you?
               MR. MROWCA:  It's 76 for us.
               CHAIRMAN APOSTOLAKIS:  You dropped two pages, but
     you're not going to tell us which ones.  Quantification.
               MR. MROWCA:  That's the quantification of pre-
               CHAIRMAN APOSTOLAKIS:  Yes.
               MR. MROWCA:  The intent was to reflect the plant
     unique features of test calibration and maintenance, and
     have a process that will identify those tests, maintenance,
     and calibration activities that, one, need to be identified;
     and, then whether they're proceduralized, and some method to
     address the degree of proceduralization and the degree of
     independent degree and checking that's going on in the
     development of those actions.
               It actually is not meant to endorse, again, a
     particular methodology.
               CHAIRMAN APOSTOLAKIS:  The only methodology really
     that is out there and people are using is the NRC Human
     Reliability Handbook when it comes to pre-initiators.  I
     don't know of any other.
               So, in this particular case, it doesn't really
     matter.  It's the post-initiator that is subject to.
               MR. MROWCA:  I've seen many ways to employ a
     single methodology.
               CHAIRMAN APOSTOLAKIS:  Yes, because it has a lot
     of discussion on performance factors, so I was wondering
     whether you were referring to that.
               MR. MROWCA:  Well, the key thing I think we're
     trying to say is that the two attributes that you need to
     consider were proceduralization and independence, and having
     a technique to reflect those attributes.
               CHAIRMAN APOSTOLAKIS:  Okay.  On page 80, which is
     78 for you, I suppose, supporting requirements, Table
     4.4.5(b), under both Category II and III applications, you
     have a parenthesis that says, i.e., latent.
               Now, first of all, we said earlier this morning --
     and I don't know if you were here -- that the definition
     earlier was not quite accurate.
               But this is not something that people do
     routinely, I don't think.  Wouldn't it be worthwhile to
     explain what you mean by latent conditions and latent errors
               I was pleasantly surprised to see it here, but I
     don't know that most people will understand what you mean by
     latent, unless they have worked in the field.
               MR. MROWCA:  Are you making a distinction between
     post-initiators and latent errors?
               CHAIRMAN APOSTOLAKIS:  No, no; this applies to
               MR. MROWCA:  Excuse me, pre-initiators and latent
     errors, is what I meant to say.  They are the same thing,
     right, the way you read it.
               CHAIRMAN APOSTOLAKIS:  Oh, I see.
               MR. MROWCA:  Essentially, I think they're being
     used interchangeably in the standard, for pre-initiators
     being any action prior to the initiating event.
               CHAIRMAN APOSTOLAKIS:  Right, and you refer to
     miscalibration, for example.
               MR. MROWCA:  Miscalibration, maintenance,
               CHAIRMAN APOSTOLAKIS:  What I am saying is that in
     his book, Jim Reason and others defined latent conditions in
     a broader sense, so my mind went to that.  I didn't go to a
     specific human action that disables something before the
               For example, latent conditions may include things
     like prioritizing something or giving it a low priority, so
     even though the Agency is aware of second actions that must
     be taken, they will take them sometime in the future, and
     then something happens before the corrective action is
               And a recent review by Idaho for the Staff here
     identified many of those.  They did a number of root cause
     analyses and confirmed that.
               So, the word, latent, means now something very
     specific in the HRA community.  And I think you should make
     that clear that you are not referring only to the specific
     action of miscalibration.
               MR. SIMARD:  Would you suggest a change then to
     our definition of latent human error?
               CHAIRMAN APOSTOLAKIS:  There is a definition which
     is very specific.
               MR. SIMARD:  Right.
               CHAIRMAN APOSTOLAKIS:  I suggested earlier this
     morning to broaden it.
               MR. SIMARD:  Thank you.
               CHAIRMAN APOSTOLAKIS:  But I'm not sure that just
     listing a definition will do it.  Maybe some elaboration
     somewhere here as to what latent conditions are would help. 
     I'm sure the Staff will be happy to give you the INEL study,
     or at least the viewgraphs that were presented to us, and
     that will give you an idea of where people are coming from.
               MR. SIMARD:  All right, thank you.
               CHAIRMAN APOSTOLAKIS:  One other comment on this
     table:  Oh, on page 82, again I was pleasantly surprised to
     see under HR-C-4, assess the dependency of pre-initiator
     human actions among multiple systems and trains, including
     whether the work process itself introduces a mechanism for
               Well, I don't know of anyone besides me who
     worries about work processes, so this was really very
     pleasant to me.  Is that something that you are doing and I
     don't know about it?
               MR. MROWCA:  Well, I think maybe it's maybe the
     interpretation of what work processes --
               CHAIRMAN APOSTOLAKIS:  The way the term is used at
     the plants?
               MR. MROWCA:  We assess whether there are different
     crews that look at redundant channels, for example, and
     whether there are a couple of mechanisms that's possible
     between those redundant channels.  That's what we do at
     Calvert Cliffs.
               When we look at pre-initiators, we're trying to
     actually identify mainly those things that do take out
     redundant channels, because those were the ones of most
     interest to us.
               CHAIRMAN APOSTOLAKIS:  So you are focusing on the
     number of crews, perhaps?
               MR. MROWCA:  Well, not only the crews, but whether
     the indications of the test will provide indication that it
     was mis-done.  For example, if there is adequate feedback to
     the checker, that he will know that; whether the checker is
     actually embedded into the proceduralized process or
     performing the test or calibration or maintenance activity.
               CHAIRMAN APOSTOLAKIS:  In a maintenance work
     process, for example, you will have maintenance request. 
     There is some prioritization, again, because there are too
     many of those, and there is a scheduling step.
               There are all those things which are before the
     actual execution.  What you are saying, I think you are
     focusing on the execution itself and how many people are
     involved and whether there is feedback.
               MR. MROWCA:  Well, maybe I misunderstand your
     point, but most of the latent failures that I have been
     concerned with are the ones that have occurred as a result
     of maintenance being performed, not as a result of waiting
     for maintenance to be performed.
               And those are typically captured in the
     unavailability conditions that you have in the plant, and
     the length of time that they're in that condition.  And so
     you would see that information in the unavailability data or
     failure data, more than you would see it in developing human
     error probabilities.
               CHAIRMAN APOSTOLAKIS:  I think both the INEL work
     and work at MIT have seen the prioritization process --
               MR. MROWCA:  As important.
               CHAIRMAN APOSTOLAKIS:  As important, yes.  But
     since the words, work process, really means something again
     to many people, maybe you need to define it and explain in
     what context you're bringing it up here.
               MR. MROWCA:  Okay.
               CHAIRMAN APOSTOLAKIS:  And I would encourage you
     to do that.  Page 88, I have something on 88.
               Oh, well, yes.  The issue of model uncertainty is
     really not raised anywhere in this guide, not just the HRA. 
     And it seems to me if one does a Category III PRA, and if
     one goes back again to 1.174, you realize that the issue of
     model uncertainty in some instances may be important.
               And the Staff now explains very clearly in the
     Guide, that those cases where we're near the boundary,
     increased management attention means that we're going to
     look at sensitivity studies and so on, because the Staff
     also recognizes that there is no accepted method for dealing
     with model uncertainty, although expert judgment elicitation
     techniques come into that.
               And in the post-initiator -- I think we are there,
     pre- and post-, so it applies to both -- in the post-
     initiator HRA, the issue of model uncertainty, of course, is
     very important, simply because there are many different
     groups around the world that have developed their own
     models, and we have SLIM MOD; we have ATHENA from here,
     although ATHENA hasn't quantified anything yet.
               And we have ACEP from 1150 and so on.  So it seems
     to me that if there is one place perhaps Level II PRA where
     model uncertainty is really important, it's the post-
     initiator HRA.
               And yet the standard is silent, and what's --
     there is also inconsistency between what you're doing here
     and the level of detail and what you do later for common
     cause failures and expert judgment, but especially common
     cause failures where you actually list five methods for
     handling common cause failures.
               And yet in this chapter, you are completely silent
     as to what methods exist out there.  So that's a broader
     comment for you that, again, we have this inconsistency that
     we discussed also in the context of Bob Budnitz's --
               MR. MROWCA:  In those cases, do you have a
     recommendation that you would prefer to see listed?
               CHAIRMAN APOSTOLAKIS:  I would stay away from
     recommending methods, and I will recommend later in the
     context of common cause failures, that --
               MR. MROWCA:  To remove them?
               CHAIRMAN APOSTOLAKIS:   -- they delete the five
     models.  But that's my personal view, and I'm not going to
     defend it in depth.
               MR. MROWCA:  Okay.
               CHAIRMAN APOSTOLAKIS:  The reason is that, as you
     know very well, these methods -- I mean, there is no single
     method that is acceptable by the group of people, but we
     have to recognize, it seems to me, the fact that there are
     different models out there, and perhaps you are aware of
     some benchmark exercises that were run by the ISPRA
     laboratory of the European Union a number of years back.
               And Mr. Fleming, in fact, participated in at least
     two of those, I believe.  One was a common cause failure.
               MR. FLEMING:  That's right.
               CHAIRMAN APOSTOLAKIS:  And they have nice tables. 
     I mean, the table on HRA is just mind-boggling.  The same
     team using different models get orders of magnitude
     different results, and different teams, of course, using the
     same model also get that.
               So, it's all over.  And that's not your problem,
     but you have to recognize here, it seems to me, that there
     is such significant model uncertainty, and that if the issue
     of recovery actions, for example, is important, important
     enough to attract increased management attention, some
     handling, some sort of sensitivity analysis here, you know,
     using perhaps two or three of these models or arguing that
     the model we're going to use under these assumptions is
     really the bounding one, we need some guidance on this, it
     seems to me.
               And I'm not saying it's an easy task, and I'm not
     saying it was obvious and we missed it, but it strikes me
     that's something that is unique to this particular subject,
     and I think the fact that even this Agency in this era of
     limited resources is spending considerable resources in
     developing ATHENA, shows you that this is an area where
     there is really a lot of activity.
               So I would recommend that you do that, but I would
     also recommend the higher-ups that they put something on
     model uncertainty somewhere.
               MR. FLEMING:  I don't think this will end up being
     an impressive list, but there are a limited number of
     examples of modeling uncertainty that's addressed in the
     standard to some extent, and a couple that come to mind are
     the seal LOCA.
               CHAIRMAN APOSTOLAKIS:  Exactly.
               MR. FLEMING:  The reactor coolant pump seal LOCA,
     which is a modeling uncertainty issue, and the other one is
     the modeling of the electric power recovery process.
               CHAIRMAN APOSTOLAKIS:  Right.  This morning, I
     don't know if you remember, but I suggested to Bob that he
     list a few examples in the expert judgment section of Level
     I issues that would require experts.
               Now, again, why do they require experts?  Because
     of model uncertainty.  So the whole thing ties in very
     nicely, but I think somebody who can influence the whole
     thing, can make -- must make sure that these things are
               You wouldn't go to expert judgment in Level II
     unless you had model uncertainty, right?  If it's a
     parameter issue, it's not a big deal.
               Well, we can move on.  So, my personal view is not
     to list any models.  You know how people -- at least some
     people are.  They might say, well, one of these is
     acceptable and we'll do that.
               If, on the other hand, in the context of model
     uncertainty, you say, well, there is a number of models out
     there, e.g., such and such, and then immediately you say one
     would need to do something with those, then it's okay.
               MR. FLEMING:  Okay.
               CHAIRMAN APOSTOLAKIS:  Yes?
               MR. FLEMING:  Sid just reminded me of something
     else to point out for the record, and that is when we get
     down to quantification, there are specific requirements to
     include modeling uncertainty.
               CHAIRMAN APOSTOLAKIS:  Wonderful.
               MR. FLEMING:  That's in a general sense, not a
     specific sense.
               CHAIRMAN APOSTOLAKIS:  Yes, but this is an area
     where guidance is needed, because, you know, I don't think
     we disagree.
               I have just a minor comment on 87, Table 4.4-5(f),
     the third row, HR-F3, Category 3, include best estimate
     time-dependent HEPs for initiation control, and the previous
     one says best estimate, best estimate.
               It is a crusade of mine to eliminate that
     terminology from reactor safety.  I realize it will be an
     uphill battle when it comes to thermal hydraulics, fighting
     an entrenched establishment.
               But I have yet to find a book on probability and
     statistics that tells me what a best estimate is, and I
     think that is a pretty powerful argument.  And I would
     suggest that you, especially in Category III, avoid the
     words, best estimate, and, in fact, there may be uncertainty
     in that time.
               I remember seeing a paper way back in the early
     80s where people said, the author said that -- well, and I
     think they are right -- that what really matters is not the
     available time.  It's what the operators perceive as the
     available time.
               They're not going to do thermal hydraulic
     calculations there in 20 minutes, so that's what they
     perceive as the available time.
               Then, of course, the authors went out and asked
     operators what the available time would be under a given
     condition, expert judgment elicitation, and the available
     time was overestimated systematically by a factor of three
     or more.
               This confirms another finding from the
     psychologists that people tend to be optimistic when it
     comes to their profession  and their ability of handling
     those situations.
               So my suggestion is to eliminate, at least under
     Category III, the words, best estimate, and maybe state
     something to the effect that the time itself may be
               MR. MROWCA:  Okay.
               CHAIRMAN APOSTOLAKIS:  And then on page 90, this
     is data.  We're moving on to data.  But overall, I thought
     it was a pretty good high level description of the
               MR. MROWCA:  There was a strong attempt to try to
     stay away from methodologies and just stick to
     characteristics that were important.
               CHAIRMAN APOSTOLAKIS:  Which is the theme of the
     standard, right?  Now, I don't know how some of these things
     will affect the quantification, but that's a universal
               I mean, I don't know whether we can really argue
     that the quality of the written procedures is so high that I
     should use rates lower than what another fellow is using,
     but at least it makes people think about it.
               Any other comments on HRA from the audience,
               [No response.]
               CHAIRMAN APOSTOLAKIS:  No?  Data analysis is the
     next chapter, 4.4.6.   On page 90 of mine, where it talks
     about parameter estimation -- and this is repeated later.  I
     guess, Karl, you are the man here?
               MR. SIMARD:  That would be Ian Wall.
               CHAIRMAN APOSTOLAKIS:  Ian, I'm sorry.  Okay, Ian
               Under Requirement C, Parameter Estimation, it says
     uncertainty intervals should address key parameters.  Why
     intervals and not distribution?  That's my page 90, Table
               MR. WALL:  Which index number.
               MR. BERNSEN:  It's high level C.
               CHAIRMAN APOSTOLAKIS:  High level requirements for
     data analysis, Table 4.4-6, or maybe the numbering of the
     tables has changed.
               MR. WALL:  Let me say up front, Dr. Apostolakis,
     that as penance for a lifetime of sins, I was assigned this
     section, even though I lack expertise.
               CHAIRMAN APOSTOLAKIS:  Your's is not to blame,
     it's just to come up with a better product.
               MR. WALL:  I was fortunate in having some
     excellent help as facilitator, including Karl Fleming and
     Stanley Levinson, and other consultants, Shobba Roa, who I
     think you may know for PLG.  So if I cannot answer your
     questions, I will be happy to take notes of them and get an
     answer later on.
               CHAIRMAN APOSTOLAKIS:  So, the question is really
     why -- first of all, are we on the same page?
               MR. WALL:  We're on the same page.
               CHAIRMAN APOSTOLAKIS:  Uncertainty intervals shall
     be addressed for key parameters.  My question is, why not
     distributions, because what are you going to do with the
               MR. FLEMING:  I think I have an insight here. 
     This is a high level requirement which is phrased so that it
     applies to all three categories.  If you go down to the
     detailed categories, you will see that we're insisting on
     uncertainty quantification -- full uncertainty
     quantification for Category III, but less -- you know, more
     of a qualitative examination of uncertainties in Categories
     I and II. 
               So the phraseology up at this level --
               CHAIRMAN APOSTOLAKIS:  For Category I, Karl, you
     really don't ask for any uncertainty, so why would they come
     up with an uncertainty?
               MR. FLEMING:  There are still requirements to
     understand uncertainty from a qualitative perspective, even
     in Category I, so the choice of words up at this level was
     selected to be broad enough.  It says considered.
               It says uncertainty intervals shall be considered. 
     Down in Category III, you will see uncertainty distributions
     have to be quantified.
               CHAIRMAN APOSTOLAKIS:  Why didn't you note that? 
     You confused me, and if you guys want to take action --
               MR. FLEMING:  I think the word, intervals, was the
               CHAIRMAN APOSTOLAKIS:  The word, intervals,
     bothers me.
               Then on page 94, alpha factor, multiple Greek
     letter, basic parameter binomial.  Why?  This is the only
     place where you do this.
               Table 4.4-6(e), third row.  Use one of the
     following models for estimating CCF parameters, and it gives
     a reference, which I bet is NUREG.  And I don't know why all
     of a sudden you decided to be so specific here and list
               I mean, I would rather do what the HRA folks did,
     and say use a model from out there, but be careful and do
     all these things that we're telling you.
               As you know, I don't know that the binomial
     failure rate model is one of the following models, at the
     same level as a multiple Greek letter model.  I would rather
     go with multiple Greek letter model than the binomial
     failure rate model.
               And also, you know, now it begs the question, the
     alpha factor model was developed to correct certain
     statistical things in the multiple Greek letter model, and
     yet we say you can go back and use the multiple Greek letter
     model.  I would take them out.
               It would be consistent with the other chapters,
     and you wouldn't get questions like this, the ones I'm
     giving you now.  Now you're endorsing models.
               MR. FLEMING:  If I might respond to that, I think
     that, yes, I think it is true that there are some
     inconsistencies across the elements in the position they
     took with respect to methods.
               There were a few example like an accident sequence
     definition where we throw out some concepts like event
     sequence diagrams, dependency matrices, and so forth, to
     provide specific examples of a systematic method for a
     certain task in the PRA.
               But in each case, we made judgments.  There is an
     important distinction we need to make between when we
     compare HRA to Common Cause, and that is that, in fact, one
     of the conclusions of the ISPRA benchmark exercises on these
     different elements was that the selection of the method, the
     modeling method itself, was not fundamental to the nature of
     the result.
               It was more in how the model was applied and how
     the data was screened and so forth.  So the variability
     across methodologies among all the ones that are mentioned
     here, are not nearly as strong as they are in the HRA field.
               But we have a tradeoff here.  We can list methods
     that we know are acceptable, and say justify alternatives,
     or we can take this out and then replace it with the
     recreation of these NUREGs that describe all the acceptable
     characteristics that you have to have.
               CHAIRMAN APOSTOLAKIS:  If you took out the
     binomial failure rate model, it wouldn't bother me.
               MR. BERNSEN:  George, that's a carryover from
     Draft 10.  It's a direct statement, quote, out of Draft 10,
     but it's still a valid comment on your part.  I'm not --
               CHAIRMAN APOSTOLAKIS:  Yes, I think you would have
     to think about the issue of consistency from chapter to
     chapter, but coming to the specific thing here, I mean,
     sure, Karl has a point, people have spent a lot of effort
     and resources on developing these.  But the binomial failure
     rate model, I'm not sure belongs in the same category as the
               As a matter of fact, we were told that in a
     different context.  But the San Onofre risk monitor uses a
     multiple Greek letter model.  So you really don't want to
     say just use the alpha factor.     
               I agree with you, Karl, that some statistical
     corrections really never made a big difference, which brings
     me to the other point which I may have missed.  But as you
     know, a lot of the emphasis of the NRC work on this subject
     over the last several years has been on building up a good
     database, and then urging the user to screen all these
               I think, in fact, that goes back to you, Karl, in
     the early days when you started this project.
               To screen these past events as to their
     applicability to the particular facility for which the PRA
     is done.  Is that emphasized anywhere here or have I missed
     it?  In any table here it says go to the data and actually
     screen and be careful when you screen.  I mean all it says
     is usually a list of common cause probabilities.  But the
     probabilities come after a long investigation.
               MR. FLEMING:  For example, if you look at DA-B8.
               CHAIRMAN APOSTOLAKIS:  B8?
               MR. EISENBERG:  DA-B8.
               MR. FLEMING:  DA-B8, which is at the top of one of
     the pages of Table 4.4-6B.  It is probably page 94 in your
     IEEE.  The very top requirement in that.
               MR. SIMARD:  It is the one right before the
               MR. FLEMING:  It is suggesting that for Category
     III applications you have to do this plant-specific
     screening as indicated in the NUREG.  So, yes, it is
     mentioned for Category III applications.
               CHAIRMAN APOSTOLAKIS:  Yes, supported by plant-
     specific screening and mapping.  All right.  If that is good
     enough, that is good enough.  And, again, we have a
     particular reference here.
               MR. WALL:  I would like to note, Dr. Apostolakis,
     that the supporting requirement DA-B8, to which Karl has
     referred, is actually responding to an ACRS comment on the
     previous draft, your comment in one of your attachments. 
     So, we did listen last time and we will listen again.
               CHAIRMAN APOSTOLAKIS:  So, our promises are coming
     back to haunt us, is that what you are saying Dr. Wall?
               MR. FLEMING:  In fact, we refer to this as the
     Apostolakis requirement.
               CHAIRMAN APOSTOLAKIS:  Okay.  Well, so I don't
     know, is your inclination right now -- I know that you
     cannot really commit the committee, but to eliminate all
     four or keep the first three and delete the binomial failure
     rate model?
               MR. FLEMING:  I think those are good comments that
     we will certainly take very seriously.
               CHAIRMAN APOSTOLAKIS:  I would like to see a
     little more emphasis on the screening.  Just saying plant-
     specific screening, maybe that is good enough, I don't know.
               Well, I think I am done with the common cause
     failures, unless -- no, there is more.  There is more.  I'm
     sorry.  There is data analysis.  Yeah, that is part of data
     analysis.  I have more comments.  Page 96, Table 4.4-6C. 
     Again, the caption talks about intervals but we have
     discussed this.
               On the righthand side there are four bullets.  Are
     we on the same table?  I like this one, verify that the
     Bayesian updating does not produce a posterior distribution
     with a single bin histogram.  That way you don't -- in fact,
     I like it so much I think you should elaborate a little bit
     more, because the original data specialization paper did
     not, unfortunately, emphasize this enough, and people take
     test data, for example, and you don't need very many of
     those.  If you have done a few Bayesian calculations, you
     know that very quickly, the posterior distribution becomes
     very narrow, and if the idea was to keep a tail for the
     accidents, for the accident conditions, and the data come
     only from tests, then you have a problem.
               MR. FLEMING:  That's right.
               CHAIRMAN APOSTOLAKIS:  And I think that was the
     intent here, but I think perhaps only a few of us understand
     this if you just read it.  So --
               MR. FLEMING:  True.
               CHAIRMAN APOSTOLAKIS:  And then the fourth bullet
     says, verify the reasonableness of the posterior
     distribution mean value.  I would say, verify the
     reasonableness of the posterior distribution.  Why just the
     mean?  I mean since you have done the work, you might as
     well look at it, which is related to the previous comment.
               MR. FLEMING:  Good comment.
               CHAIRMAN APOSTOLAKIS:  And I think now I am done
     with the data.
               MR. WALL:  Before you leave the data, Dr.
     Apostolakis, I would like to just mention that you may react
     to the fact how much smaller this section is than Rev. 10,
     and I would like to provide some assurance to you that we
     systematically went through Rev. 10 and took each of the
     "shalls" from Rev. 10 and made sure it was handled in this
     section as an action statement.  And this section, I tried
     also, in the square brackets on the lefthand side, to show
     the paragraph in Rev. 10 from which that statement came.
               CHAIRMAN APOSTOLAKIS:  Good.  Thank you.  Well,
     this is an area that is fairly mature now.  The data has
     subsided.  So, unless someone else has a comment on data?
               MR. FLEMING:  George.
               CHAIRMAN APOSTOLAKIS:  Yes.
               MR. FLEMING:  I just wanted to -- Ian brought up a
     very good point, it is something that I just wanted to bring
     to your attention, the committee's attention, is that one of
     the characteristics of Rev. 10 that we tried to address in
     this rev. was that if you go back to Rev. 10 and develop a
     frequency distribution of the number of pages of
     requirements against the elements, there was a strong
     feeling by many of us that it was out of balance.  That Rev.
     10 tended to write the most about the things that we have
     the least concern about in PRA, system analysis, data
     analysis.  There were lots and lots of requirements in here,
     and fewer and fewer requirements in there that are really
               So, we tried to balance, come up with a better
     distribution of the level of detail of the requirements to
     reflect the importance of the PRA element, and that was part
     of what Ian was trying to accomplish this.
               CHAIRMAN APOSTOLAKIS:  One last thing, the user
     will, if they go the literature, which I am sure they will,
     or the PRAs, will find terms such as state of knowledge
     distribution, more recently, epistemic, alliatory. 
     Shouldn't these be in the glossary definitions?
               MR. FLEMING:  If we use them.
               CHAIRMAN APOSTOLAKIS:  And perhaps in the data
     section, say a few words about the terminology.  Because I
     can see someone getting completely lost.  Right now it says,
     you know, develop a probability.  In the definitions, I
     didn't see it.  But I think there is complete silence on
     these things, and I think people will find it useful.
               MR. FLEMING:  In the quantification section, QUD2,
     the ones we were referring to earlier, we do, in fact, use
     alliatory and epistemic.
               CHAIRMAN APOSTOLAKIS:  Yeah, but in the data
               MR. FLEMING:  Therefore, we should have a
               CHAIRMAN APOSTOLAKIS:  So, there are two places,
     one is the definitions and the other in the data section. 
     But when we talk about uncertainty intervals for parameters
     like failure rates, maybe you put a parenthesis and say, you
     know, this is epistemic.
               Dr. Shack, we skipped the success criteria
     section.  Do you have any comments?
               DR. SHACK:  No.
               CHAIRMAN APOSTOLAKIS:  It's okay?
               DR. SHACK:  It's okay.
               CHAIRMAN APOSTOLAKIS:  Fine.
               DR. KRESS:  I had one question on success
     criteria, I guess.  It is sort of the same question I had on
     the fission products.  They call for use of remediation
     things in fission product, but this requires you have a
     model for fission product release and transport, but there
     is no standards, or no requirements related to what that is.
               I had sort of the same question on success
     criteria.  It calls for using realistic thermal-hydraulic
     analysis or whatever to determine the success criteria.  But
     that is about as far as it went.  And I was a little bit
     concerned, well, I can do an awful lot in perturbing the
     results of a PRA by screwing around with success criteria. 
     But it seemed like we didn't talk much about how one
     determines those success criteria and what are the standards
     of the deterministic calculations or the other kind of
     calculations that are used for those.  And it was just a
     comment.  It just seemed --
               CHAIRMAN APOSTOLAKIS:  That is actually related
     also to the available time.
               DR. KRESS:  And available time, yeah.  Yeah.
               CHAIRMAN APOSTOLAKIS:  It all comes together.
               DR. KRESS:  Same -- available time is the same
     sort of issue there.  So, I thought it needed a little more
     discussion or something about that in there.
               CHAIRMAN APOSTOLAKIS:  In fact, different success
     criteria do have an impact, right?
               MR. FLEMING:  Absolutely.
               CHAIRMAN APOSTOLAKIS:  It is not like failure
     rates and common cause failure.
               I think Dr. Kress makes an important point.  I
     think you can tie that to the available time for HRA
     purposes and all that, and maybe say a few words about the
     uncertainties in the so-called deterministic analysis. 
               MR. FLEMING:  Right.  Good comment.
               CHAIRMAN APOSTOLAKIS:  Okay.  Mario.
               DR. BONACA:  In many cases it makes enormous
               DR. KRESS:  Yeah, you can really, you can make
     enormous differences with it.
               DR. BONACA:  Train auxiliary feedwater, the
     reality, the best estimate, a cantilever, a three redundant. 
     Really, if you have a three redundant train, rather than two
     redundant.  So that just is an example, you can derive very
     big differences just by the fact that you can prove that you
     have three redundant system rather than two.
               CHAIRMAN APOSTOLAKIS:  Sure.
               DR. BONACA:  I had some comments on the
     quantification here.  One of them is I believe mostly
     editorial, I brought it up yesterday, it is on page 109.  I
     am not sure it is only editorial.  If you look at the high
     level requirements, they are listed as B and C and they are
     the same.  But if you go into the supporting requirements,
     page 113 and 115, they are different.  So, I think probably,
     C was meant to be something else.
               MR. SIMARD:  Oops, a mistake there.
               CHAIRMAN APOSTOLAKIS:  They are identical, yeah.
               MR. SIMARD:  The official answer to that is
               DR. BONACA:  But the supporting requirements are
     different in their meaning, so maybe you want a different
     heading here under C.
               MR. FLEMING:  I believe that one of them was
     intended to be Completeness and Scope, and the other one
     supposed to be Completeness and Detail, but we will confirm
     that.  But it is obviously a failure of our document.
               MR. SIMARD:  Yeah.  Obviously a failure of the
               MR. FLEMING:  Yes.
               DR. BONACA:  At the beginning I scratched it, and
     then when I went back, and I said, oh, I can't scratch the
     supporting requirement there.
               MR. FLEMING:  That is where I hit "do what I mean"
     button and nothing happened.
               DR. KRESS:  I wish I had one of those buttons.
               DR. BONACA:  On page 110, under index, a couple of
     the tables you have, under Category I, applications, two
     applications and three, the only difference is one is
     understanding, two is a sound understanding, and three is
     sound understanding and quantification, which is fine.  But
     when you get down to the first supporting requirements, and
     often after that, you use the word "estimate CDF and LERF,"
     rather than calculate.  Any reason here?  I mean estimate
     seems to me like such a more vague word that has a lot of
     latitude to it, estimate things.
               MR. FLEMING:  I don't believe we had any profound
     reasoning in the use of that term and it is probably
     superfluous.  It probably could be --
               DR. BONACA:  Well, I just, you know -- since you
     are using it also for a Category III, I thought it may have
     been something else at the beginning.  Anyway, look at it,
     for whatever that.
               On page 111, the top of the page, now if you look
     at Category I, again, it says it should have an
     understanding of the impact of key uncertainties.  And so I
     don't know why the next supporting requirement, QUA9, you
     know, the truncation requirement, would that apply also to
     Category I?  I mean you still want to truncate at a
     sufficiently low enough value that the importance
     calculations are understood.  Even if you don't have the
     same level of scrutiny or precision, you would want to have
               And I had the same comment somewhat on QUA11 here. 
     You are talking about, this is something that I have seen
     analysts do always.  I mean at some point they estimate what
     they have lost in the truncation.  It is a simple check, I
     mean it is not that this is a major undertaking.  My sense
     is that they will be beneficial by just using the word
     "consider," to me says that if you don't consider, you may
     lose the understanding of the impact of key uncertainties.
               So, my suggestion here is just that you review the
     high level requirement against the supporting requirements
     to see that there isn't a logical inconsistency.  And that
     goes down also to QUA12.  I don't know, the screening value
     seems to be pretty large.
               CHAIRMAN APOSTOLAKIS:  I have a minor editorial
     comment off of there.
               DR. BONACA:  So that is -- I'm sorry.
               CHAIRMAN APOSTOLAKIS:  Understanding and
     quantification of the impact of uncertainties, I would say
     quantification of the uncertainties, to avoid.  And that is
               MR. FLEMING:  Yes.
               CHAIRMAN APOSTOLAKIS:  Yes.  I also got the
     comment from Mr. Barton, he is also wondering whether people
     will really understand the distinction between a realistic
     quantification and straightforward quantification.  He asks,
     what is meant by realistic basis?  And I think that is
     related to what Dr. Bonaca just raised.  He actually -- we
     make a distinction here between an understanding and a sound
               These are very fine lines to walk --
               DR. BONACA:  That's why at the beginning I didn't
     critique it.
               I just went down into the supporting to see if
     that, by that clarification --
               CHAIRMAN APOSTOLAKIS:  I really don't know what
     you guys can do about it but I am just telling you what the
     reaction of people who have not lived with this for the time
     that you have lived with it is when they see that a
     distinction between one category and the other is that here
     you are modeling something but here you are doing a
     realistic model.  Here you understand something but here you
     have a sound understanding.
               If you can do something to make it clearer, I
     don't know what you can do but this is a reaction, okay?
               MR. FLEMING:  Just a comment on that.  I think
     that you see to some extent a work in process along the way
     from a point in time when we were trying to figure out the
     logic for, an appropriate logic for differentiating
     requirements across the three categories.
               At one time it was oversimplified to say that in
     Category 3 you shall do something, should, and may, so we
     converted the actions statements --
               CHAIRMAN APOSTOLAKIS:  Right.
               MR. FLEMING:  -- and we don't mean unsound
     understanding if you don't say sound, so I think this is
     good feedback.
               We need to go back and tighten that up.
               CHAIRMAN APOSTOLAKIS:  And that is why I will come
     back to my comment yesterday and this morning that if you
     tie these categories to the decisionmaking process and the
     decisionmaking process is 1.174, and refer to the decision
     criteria of 1.174, at least for me that makes it much
     clearer as to what you mean by Category 3 and Category 2.
               MR. FLEMING:  Right.
               CHAIRMAN APOSTOLAKIS:  The Staff is on the record
     saying that as you approach the boundaries things become
     darker, increased management attention, but when you attract
     increased management attention you better have a Category 3
               MR. FLEMING:  That's right.
               CHAIRMAN APOSTOLAKIS:  That makes it clearer to
               MR. FLEMING:  That's right.
               CHAIRMAN APOSTOLAKIS:  Otherwise, sound versus
     unsound, realistic versus unrealistic -- it's a little
               It may be in the same context -- I mean I can see
     the distinction between Category 2 and 3 in the context of
     1.174.  Category 1, I can't place it, so that is something I
     am sure we will discuss again.
               DR. KRESS:  50.59?
               CHAIRMAN APOSTOLAKIS:  50.59 is always the answer.
               DR. BONACA:  I had one more comment here.
               CHAIRMAN APOSTOLAKIS:  One more comment, yes.
               DR. BONACA:  Which is general.  That's all I've
     got -- which is more of the use of the word "may" rather
     than use like -- let me give you an example -- QUA-14 on
     page 111.
               Under Category 3 you are making it a requirement
     to use the same truncation limit for solving each system in
     the overall sequence CDF, because it is the proper approach.
               When you come down to the other two categories you
     are using the word "may" -- now if I had seen the word
     "should" I would have said that's fine.  It's a
     recommendation.  People don't have to follow it, but "may"
     seems to be a little bit -- almost too loose.
               CHAIRMAN APOSTOLAKIS:  Too weak.
               DR. BONACA:  Too weak.  I mean "may" -- yeah, I
     may also take a walk.
               CHAIRMAN APOSTOLAKIS:  But should you though?
               DR. BONACA:  Just a suggestion though -- it seems
     a little bit, you know, especially for Category 2 --
               CHAIRMAN APOSTOLAKIS:  Too wimpy.
               DR. BONACA:  Category 2 I would see that,
     certainly I would want to see there a "should" and then if I
     go to Category 1, I would almost I will put the word "may do
     without using" by saying, you know, well, okay, but "may" I
     couldn't understand what it meant.
               I can understand the trouble you are going through
     and I wouldn't want to be in your shoes, actually, going
     from where you are going before you had all those "shall" --
     but still I think it would help.
               MR. FLEMING:  Just to comment on that, it was our
     intention but although not completely successful, in
     preparation of this draft to use the word "may" only as a
     permissive.  You had several options.  You could say you
     could do it (a), (b), or (c), but there are some remnants
     and this was one of them of leaving a "may" to be a somewhat
     less than "should" -- which we don't intend to leave in
               MR. BERNSEN:  We also don't want to use
     "should" -- we have had too much trouble with that so we
     will come up with something else.
               DR. BONACA:  Okay.
               MR. WALL:  It may be appropriate to point out that
     in the process of going to action statements we translated
     the "shalls" to a plain declarative verb, "shoulds" to a
     consider -- so it would be use this -- the "shoulds" to
     consider using something, and we left the "mays" as "mays"
     so that was the general thing, the way we presented this.
               Now in the process of other hands doing things,
     some of those "consider usings" may have turned into "mays."
               MR. FLEMING:  That's right.
               MR. BERNSEN:  We'll fix it.
               CHAIRMAN APOSTOLAKIS:  Level II we have covered.
               Process check is one little paragraph.  Anybody
     has a comment on the little paragraph?
               DR. BONACA:  I just want to ask would this process
     check be different from a review that you normally have for
     a standard calculation?  What I mean is that there are very
     specific requirements in QA whereby after performing a
     calculation you have an independent reviewer who reviews it
     and he also takes the responsibility to make a statement
     that says I sign it, I have reviewed it for approach and
     content and I agree or disagree on the following issues.
               Would you see it differently?  It doesn't sound
     like -- it is kind of loose, a little bit here?
               MR. BERNSEN:  This is a bit of a special problem. 
     We have been discussing whether to incorporate things like
     specific QA requirements and some other requirements like
     that as by reference in the standard and decided not to do
     that because it is really up to the user at this stage.
               The general feeling of the project team was that
     even though one might not apply an Appendix B process to
     this, although I am not sure where it would be done,
     parentheses, there was a need to do some level of
     independent review but perhaps not as sophisticated as a
     number of licensees have developed in their design control
               It is certainly permissible under Appendix B to do
     any kind of independent review for design verification. 
     This is not intended to be something else other than that,
     but it was trying to clarify that we wanted some level of
     checking yet we didn't want to impose the formality, because
     it is really up to the user in their program.
               DR. BONACA:  Once the user applies this tool for,
     say, 1.174 application, doesn't it take a role, a regulatory
     role, where there are certain requirements in quality
     assurance, et cetera?
               MR. BERNSEN:  Probably, but that is another venue
     right now.
               DR. BONACA:  I think we should note, however, that
     it is not included.
               MR. BERNSEN:  Yes.  We have done both things in
     ASME standards.  In some cases we have explicitly called for
     some kind of QA and in other cases we have allowed for
     different incorporation of QA requirements.
               In some cases we have not been explicit.
               We are finding our way.  We are primarily
     interested in the PRA here, and not configuration control,
     not QA, not things of this nature.
               Management systems -- we are not trying to address
     those.  We are talking about the techniques that need to be
     applied to the PRA, so this is an attempt to recognize
     something needs to be done, but it is still worth your
               I think if it is not clear, we need to clarify it
     some more.
               DR. BONACA:  When you go to Section 5, on PRA
     configuration control and you say certain things that place
     a burden on processes --
               CHAIRMAN APOSTOLAKIS:  Bob, do you have any
               DR. UHRIG:  This standard is different than any
     other that I have had occasion to deal with in the sense
     that a boiler code basically specifies that you have got
     confidence that this is not going to blow up.
               You use an ASME standard in the specification of
     materials, in a system, you know what you are getting, and
     if you don't get it you can sue.
               DR. UHRIG:  This is totally different and I have
     had a little trouble grasping it.  I think I have got a
     pretty good feel.
               I accept the categories you have got here and they
     make sense and I have read through what you have tried to do
     but I am bothered by George's comment about people using the
     same system and getting totally different results.
               Am I misquoting you?
               CHAIRMAN APOSTOLAKIS:  In some instances, no, you
     are not.  In some instances, comma, no, you are not.
               DR. UHRIG:  We are going to have a chaos out here
     in the regulatory area if Utility A gets different results
     from Utility B and they both have got essentially the same
     plants and the -- I don't know how we are going to resolve
               CHAIRMAN APOSTOLAKIS:  Well, it is not their job
     to resolve it.  Their job is to make sure that the Applicant
     recognizes it and does something about it, but I don't think
     the standard really should resolve issues of model
     uncertainty that exist out there, so my comment was in the
     spirit of make sure that the Licensee realizes that here
     there is a real problem and if they have to do a Category 3
     PRA they should expect comments from the NRC Staff.
               DR. UHRIG:  You have got the same problem
     eliciting expert opinions.  Two different sets of experts
     will give you two different views.
               CHAIRMAN APOSTOLAKIS:  Yes.
               DR. UHRIG:  So it is probably inherent in the
               CHAIRMAN APOSTOLAKIS:  Mario?  I'm sorry.  Karl,
     you had a response?
               MR. FLEMING:  Yes.  I think it is an excellent
     observation.  I think it reflects the state-of-the-art to
     some extent, but I think the other examples that you
     mentioned, buying materials, building a pressure vessel, and
     so forth, it could be simplified in the sense that you could
     talk about the state variables of something you could
     measure in an objective sense, and the problem that we have
     is that we are trying to do something that is a function of
     not the state variables but our state of knowledge about the
     state variables and the variability, the variability stems
     from the fact that our states of knowledge are different.
               CHAIRMAN APOSTOLAKIS:  Not only that but let's not
     forget that PRAs are extremely ambitious.  They are supposed
     to have everything that can go wrong at the plant.  This is
     really a huge task which necessarily leads to this
     situation, so it is not a specific issue we are dealing with
     and we don't have the parameters to measure it and so on.
               I mean ideally it should be the model for the
     plant and it is people and everybody.  Necessarily then you
     are led to this situation.
               I think, Jack, have you been trying to say
               MR. SIEBER:  Yes.  I would just like to comment on
     Dr. Uhrig's comment, and this is something that you and I
     have talked about too.
               When I reviewed the standard I reviewed it, I
     thought it was good, and I understood what it was you were
     doing, but I kept in mind some things that were said at a
     meeting we had on January 27th where you were a presenter,
     Mr. Fleming -- Perry.,
               CHAIRMAN APOSTOLAKIS:  That was the ACRS retreat
     in Florida, where Mr. Fleming and Dr. Perry were invited
               MR. SIEBER:  Right, and a couple of the statements
     that were made was typical industry PRA is too simple and
     often incomplete and sometimes has low probabilities for
     initiating events.
               Another one was typical industry PRAs have
     differing engineering assumptions related to equipment
     performance and phenomenological analysis.
               There were some conclusions out of that with CDF
     and LERF, et cetera, are relative terms and the absolute
     value compares from plant to plant is significantly
     influenced by the engineering assumptions and when used to
     evaluate a change in risk PRAs of varying quality may still
     give valuable risk insights related to plant changes.
               Then Mr. Perry said the current quality of typical
     PRAs is not sufficient to move from risk-informed regulation
     to risk-based regulation.
               I think that everybody can at least intuitively
     say all these factors are correct.
               When I reviewed the standard I tried to review it
     with the thought in mind is will the varying qualify from
     plant to plant of PRAs and the different answers the
     different practitioners would get analyzing the same plant
     with the same initiating events as far as the absolute
     value, will that be helped by this standard?
               I came to the conclusion that with the exception
     of the question of completeness, which is the only thing I
     could pull out of here that was more or less addressed, that
     that problem of inconsistency from one practitioner to
     another would probably remain.
               I don't know whether that is good or bad.  From
     the delta risk standpoint it makes no difference in my view. 
     On an absolute value, which is your Category 3 it does make
     a difference, and I am not criticizing or suggesting that
     you change anything but I would like to hear your comments
     on my way of thinking about it.
               MR. BERNSEN:  Yes.  My reaction is that the
     introduction of this standard and application of it is going
     to lead in time to more convergence.
               MR. SIEBER:  Okay.
               MR. BERNSEN:  I don't think that all by itself it
     will get you there, but there is a lot of cross-pollination
     built into this, if you will, through the peer review
     process, through the questions and responses that we will
     have to answer over the years, through the additional things
     that we are going to have to generate to supplement the
               This is just the beginning.  I mean if you recall,
     we were talking about it before, the original nuclear boiler
     code was a vessel design code for the reactor vessel,
     nothing more.  Look how far this has evolved to respond to
     the need.
               I think that we are going to see this as the top
     level that generates over a period of time a lot more
     formalized and informalized guidance and examples and
     probably training and cross-communication that is going to
     bring things closer together in time.
               It is not going to happen overnight but if we
     don't begin to move in that direction, and starting with a
     basis that people can use based on the kinds of standards,
     kinds of PRAs they have, so we encourage them to begin to
     use it in a logical way, control fashion, we are never going
     to get there, so this is I think a major step in that
     direction, but we have got a long way to go.
               MR. SIEBER:  Well, I see it as just one element. 
     There is the standard and you can publish the standard and
     the Commission can endorse it, but equally important are two
     other elements that are the peer review and certification
     process to me is the one that will have the greatest
     influence on standardizing methodology and in self-criticism
     and criticism of bad or inadequate phenomenological
               The other element to that is the sourcebook or
     paper that really combines what was contained in Version 10,
     okay, as a way to say here are the standard methods that are
     being used.  These are the "how-tos" and make the real
     standard, the obligatory part, and the other one just a
     tutorial, so to speak, and so I see this as really a three-
     pronged effort which one of those is the standard itself. 
     The second one is peer review and certification.  The third
     is publishing the "how-to" portion of it, and that at least
     is the way I see it.
               CHAIRMAN APOSTOLAKIS:  But I do agree with you,
     Jack, that if there is one contribution I am sure there is
     more than one of this standard is in the area of
     completeness, because it raises all the issues that the
     practitioner has to think about.
               MR. SIEBER:  Once they are in the standard you
     can't ignore them.
               CHAIRMAN APOSTOLAKIS:  That's right.
               MR. SIEBER:  Obviously you aren't complying and so
     you have to give due consideration to all the factors that
     are in the standard.  From that standpoint it is a good
               CHAIRMAN APOSTOLAKIS:  In fact, given the state-
     of-the-art, I would oppose a standard that went beyond that
     and actually recommended methods because I don't think
     that --
               MR. SIEBER:  And I agree.
               CHAIRMAN APOSTOLAKIS:  In many areas we are not
               MR. SIEBER:  Well, that is my global comment.
               CHAIRMAN APOSTOLAKIS:  Well, it seems to me that
     Mr. Sieber has broadened the discussion, so maybe we can go
     around the table and see what the members have to say in
     terms of general comments, if members so desire, or we can
     open up the discussion in terms of specific issues and have
     an unstructured formal discussion --
               CHAIRMAN APOSTOLAKIS:  -- of the issue by issue.
               We certainly have to revisit the issue of
     categories because ASME is obviously very much interested in
     knowing the subcommittee's views.
               I don't hear any suggestions, so I will pick one
     method.  Why don't we go around the table.  I think Jack
     just --
               MR. SIEBER:  I already gave mine.
               CHAIRMAN APOSTOLAKIS:  -- did his piece.  Tom?
               DR. KRESS:  Certainly. Thank you.
               Well, a general comment.  I was just a little
     disappointed that the standard chose to limit itself to the
     current definition of CDF and LERF.  The reason I say that
     is I think having limited itself to those it did a pretty
     good job.  I mean I really have no complaints, but I think
     that NRC when they go in to risk inform the regulations they
     are really going to be in the business of looking at fission
     products and the frequency of their releases and controls on
     those and the standard comes up short on discussing fission
               I think more will be needed later by NRC or
     somewhere on the standards associated with that, so that is
     just a statement saying as far as it went I think you did a
     good job.  I think it left out an important part.
               You always have to limit what you do, especially
     if you have got limited time and resources so I think it is
     appropriate to limit it but I was a little disappointed in
               I do think the main issue with the limited set is
     going to be the categories and not so much that it is
     appropriate to have different categories for different
     applications.  I think it is.  I think you almost have to
     buy off on that presumption.
               The problem is going to be how to be sure you are
     fitting the right application in the right category and I
     think the guidance on how to do that might be a little too
     loose, that I can fit things into categories one way or the
     other by some small assumptions.
               I think there might be some thoughts about saying
     if you are unclear about what category to use, use the more
     stringent one.  I really don't see that admonition in there.
               I also share George's view that the categories
     might be tied to some extent to how well you need to know
     the answer and that is either in terms of a predetermined
     confidence level you need in the CDF and LERF, which may or
     may not be something a standard ought to deal with. That is
     something that NRC has to come up with, but then if they
     came up with the confidence level they need, how do they go
     to these categories and say this category will give me this
     confidence level?
               That connection is not made explicitly and I think
     George's suggestion that looking at it in terms of the 1.174
     may be a way to make that link and I did kind of like that
     suggestion, George.
               I thought the definition of LERF that allowed site
     specific flexibility probably goes against the intention of
     the definition of LERF in the first place in that it ought
     to be made independent of the site.
               I am not quite sure of that one yet.  I have to
     think about it awhile but that was my first reaction to
               I thought the requirement to have a full
     uncertainty analysis for Category 3 needed a little bit more
     expansion, and the reason is -- I share George's views -- I
     can't see any plant-specific PRA doing a full uncertainty of
     the NUREG-1150 type. What they will do is do the Monte Carlo
     propagation which is easily done, but I think there's some
     guidance needed on how you deal with the knowledge
     uncertainty.  What do you call it, George?  Is that the
     epistemic?   I didn't see real good guidance on how you deal
     with that in here, because I don't think it is going to come
     out of a routine uncertainty analysis, so I thought that was
     a missing element on how to deal with it first, particularly
     for Category 3.
               That is basically all the general comments I have
     right now, besides the specific ones I had before.
               CHAIRMAN APOSTOLAKIS:  Mike, do you want to say
               MR. MARKLEY:  Yes, I have just got one slightly
     technical and the other two just a formality.
               I guess I am a little bit uncomfortable when I am
     told that something is intended for a general purpose and
     not really for regulatory purposes and then something like
     the maintenance rule A4 is partitioned into Category 1.  I
     know you are going to go look at all that stuff.
               The other two things were just that the member
     comments, I just wanted to reinforce that.  They are just
     their own views.  They do not represent the subcommittee's
     or the full committee's, and I wanted to mentioned the
     schedule for the full committee, and that is from 1:15 to
     3:15 p.m. on Wednesday, July 12th.
               CHAIRMAN APOSTOLAKIS:  In fact, we should talk
     about it before adjourn today as to maybe recommendations
     that the members might have, what you should address,
     because it is only an hour and a half, right?
               MR. MARKLEY:  It's two hours.
               CHAIRMAN APOSTOLAKIS:  Two hours, and points to
     focus on -- let's not forget we should do that before we --
     we usually do it with the Staff, because we don't want to
     repeat everything verbatim that was done today.
               Maybe you will have a chance to think about some
     of our comments today and respond to the extent possible. 
     Dr. Bonaca?
               DR. BONACA:  I gave all the comments before, but
     just to summarize what I view as important in addition to
     the points already  made by the members.
               One is again we talked about the view,
     characterization of Category 1 for maintenance rule and I
     meant to say this morning the fact that some part of the
     guidance and presentation as been pointed out in Category 1
     is primarily decisions based on deterministic analysis
     supplemented with basic insights.
               I am not sure that the maintenance rule right now
     says that as far as the role of risk information to make
     decisions so that adds another thing, saying that this point
     really has to be resolved.
               There is an inconsistence.
               Somewhere else also there is some inference that
     the PRA -- for example under Category 1, page 3, it says PRA
     applications are not expected to impact safety-related SSCs,
     but you are pulling them out of service.  There is an issue
               The second is the point I made this morning and I
     still believe it is important.  The chart which you are
     presenting there, which is important, because visually it
     helps to understand, has the fundamental presumption that
     one can build a model to fit a need, and that is simplistic
     in PRA.
               A complete PRA at a plant is a massive model that
     contains all kinds of information and my suggestion would be
     simply that in the text somewhere you can explain that.  The
     reason or the intent of doing that is only, the intent is to
     say that if the application is limited enough and clear
     enough then it can be used for the purpose, but there is
     some warning there that says don't -- and this is really
     with good intent.
               You may have a very low capability that begins to
     do some, for example, subtle electrical changes, in some
     support systems which are nonquality related, and yet they
     cascade into dependencies which are important.
               The third point I would like to make is I
     recognize that the probably there isn't any PRA out there
     that meets strictly the minimum requirements of Category 1
     but the standard allows it.  I don't know how we go
     around --
               CHAIRMAN APOSTOLAKIS:  Say that again. There is no
     PRA that what?
               DR. BONACA:  That meets only the minimum
     requirements of Category 1.
               CHAIRMAN APOSTOLAKIS:  I thought that Karl said
     that Category 1 in fact is higher level than the IPEs. 
     Isn't that inconsistent with this?
               DR. SHACK:  It's the intention.
               MR. FLEMING:  Again I think the distinction needs
     to be broken down at the subelement level, and so we are
     looking at the individual elements and subelements of the
     PRA.  There are some out there that are Category 0, 1, 2, 3
     4, so --
               CHAIRMAN APOSTOLAKIS:  I see.
               MR. FLEMING:  -- so no full PRA is only Category
     1.  It is a mixed bag.
               DR. BONACA:  The message here is only -- I don't
     want to interfere in the process.  I only want to make sure
     that as a committee you can review that and look at Category
     1 and ask yourself does it provide you a very low standard
     and do you want to support it, yes or maybe no.
               It may be that you feel comfortable with that and
     I trust that the committee can do that.  I just say that if
     I were in your shoes I would do that, just a simple
               The last thing I would like to add is only that we
     did not discuss the issue of peer review.
               CHAIRMAN APOSTOLAKIS:  Yes.  Today we didn't.
               DR. BONACA:  Yes.  But, you know, when I look at
     some of the qualifications for example of the PRA I don't
     understand exactly what kind of latitude you are allowing
     the PRA peer review team qualification, because here it says
     somebody who is knowledgeable in the requirements in the
     standard for the area of review, which means we are all
     knowledgeable enough right now with the standard.
               Have the most experience performing PRA
     activities relevant to the area -- it doesn't say he has
     performed PRA.  It says somebody who is doing some, you
     know, evaluations, and have collective knowledge of the
     plant design, containment design and plant operation.
               Now these general characteristics, I don't believe
     that they are capable -- how capable the members will be of
     performing a true independent and thorough evaluation.
               CHAIRMAN APOSTOLAKIS:  It will come down to what
     the Staff has.  The Staff sees two or three of those that
     are shallow the whole process will die.
               A comment from somewhere?
               MR. HILL:  Just a little bit of a response to your
     concern.  Our original -- as you probably saw on Rev. 10 --
     we used timing requirements, so many years experience, et
     cetera, in various areas, but the point was made that you
     could have that many years' experience and still not know
     what you are doing, so that doesn't necessarily qualify
               CHAIRMAN APOSTOLAKIS:  That's right.
               MR. HILL:  And it is difficult to tap into the
     brain of the reviewer and say how much do you really know. 
     We chose these kinds of statements to be able to say that
     they need to be able to cover all the ground of the NSSS,
     the containment type, the operations, et cetera, and leaving
     it somewhat on the judgment of the team leader.
               That person has to make sure that they have the
     right set of skills and talents available to perform their
               So, yes, it is somewhat vague and somewhat
     general, but I am not sure how you can nail it down because
     every time you try to nail it down with some specific
     somebody can say, well, that specific doesn't prove
               DR. BONACA:  The reason why I am raising this
     issue is because this is a unique area where the work done
     for most PRA in the country have been done by a few
     specialists and then put in the hands of the utilities. 
     From my experience many of these do not have the expertise
     internally.  They have expertise to tinker with some change
     inside, but they don't understand oftentimes some of the
     real subtle issues that were addressed by the professionals
     who built the PRA. That is the only reason why I raised the
               MR. HILL:  And you had another comment about why
     doesn't it say performing requirements of having performed
               We did have those kinds of words in there but we
     came to the rapid conclusion that we don't have people
     performing PRAs anymore.  We have people updating PRAs
     because the PRAs already exist and there probably won't be
     any more performed.
               If we limit it to that, five years from now with
     the career paths we won't have anybody available.
               DR. BONACA:  And I think, I was talking to some
     utility guys last week and they were talking about how to
     build a PRA capability at their facilities, and every single
     one of them wanted an experienced systems engineer who would
     be willing to learn PRA methods, and today we are talking
     about experienced people doing this, experienced people -- I
     don't think college graduates will find a job in the nuclear
     business anymore, because all the jobs are for experienced
               DR. KRESS:  Of course.
               CHAIRMAN APOSTOLAKIS:  I don't know what they have
     to do to enter the field.
               DR. BONACA:  Still this is a unique issue because
     I mean --
               CHAIRMAN APOSTOLAKIS:  It is.  It is, but it
     ultimately comes to the guys behind us.  If they start
     rejecting the quality of PRAs that have undergone the PRA
     review, then there is a problem, because I am sure that they
     will at the beginning at least review, themselves, the
               DR. BONACA:  They are not rejecting thermal
     hydraulic codes.
               CHAIRMAN APOSTOLAKIS:  What?
               DR. BONACA:  But that is a different issue.
               CHAIRMAN APOSTOLAKIS:  Thermal hydraulics isn't
     different.  Thermal hydraulics came from the fountain.
               DR. KRESS:  It was handed down from on high.
               George, I think a good graduate student that
     specializes in PRA at the right institution with the right
     teacher could probably qualify as being experienced.
               CHAIRMAN APOSTOLAKIS:  I don't think so, Tom, but
     very kind of you.  Dr. Uhrig?
               DR. UHRIG:  The only thing I have to add is
     related to a comment you made, concern about this OMB
               CHAIRMAN APOSTOLAKIS:  Yes.
               DR. UHRIG:  I don't think that is a real problem. 
     I think if NRC -- I think it has the right to reject or
     modify specific parts of any of the codes and having been on
     the other side of the fence a time or two, if NRC wants to
     do something they usually get it done.
               CHAIRMAN APOSTOLAKIS:  Still, though --
               DR. UHRIG:  Philosophically it is a concern but --
               DR. KRESS:  Well, they have gotten themselves in
     that trap with the backfit rule.
               DR. UHRIG:  What?
               DR. KRESS:  They are certainly gotten themselves
     in that trap with the backfit rule.
               DR. UHRIG:  Yes, they have.  What I alluded to was
     before the backfit rule came into effect.  That's it.
               CHAIRMAN APOSTOLAKIS:  Okay, thank you.  Bill?
               DR. SHACK:  I think it is a very interesting
     attempt -- I think there is a lot of good, useful
     information on the categorization in your viewgraphs and in
     1-5 I think you beat up enough on that already today, but I
     think you really do have the material to make a useful
     approach to the categorization in the viewgraphs.
               CHAIRMAN APOSTOLAKIS:  Don't worry about
     limitations of space when it comes to 1.5, okay?  Take as
     much as you want.
               MR. BERNSEN:  Well, I am going to go around our
     side too.  One of the points that was made I guess by Tom or
     somebody with regard to the completeness of this standard,
     as we mentioned before, this is the first effort.
               You are probably aware of the fact that ANS is
     writing some parts to the overall PRA and the low power
     shutdown, external events.
               We have had on our plate, work assignment, to look
     at what is needed in the future, and in fact Karl graciously
     agreed I still think to lead our little task group to define
     what we should be doing in the future, so this is not the
     end of the line in the process by any means.  It is the
     first step.
               We recognize -- I think all of us -- that you need
     to go further in terms of detail, in terms of guidance, in
     terms of expansion of scope and things of this sort and that
     will be done and we will work in a coordinated fashion with
     ANS in doing that.
               DR. KRESS:  Will this be like other ASME standards
     that may get updated?
               MR. BERNSEN:  Yes.  It's intended -- this is a
     living document.
               DR. KRESS:  Living document?
               MR. BERNSEN:  That we have done with our O&M, with
     our code sections, with our QA and with all the other
     standards.  They are living documents.  They are maintained.
               I was going to say, Bob, you know, think of this
     standard more in terms of a QA program standard, which
     really didn't give prescriptive requirements on how you do
     things either.
               This is not the first step.  It is somewhere in
     the middle between them but we have had to deal with
     different approaches for standards, but as I say, the main
     thing to keep in mind is this is the first step.  It is
     going to be maintained.  It is going to be interpreted.
               I am sensitive to the concern whether or not --
     the Staff certainly can accept or reject pieces and parts of
     the standard.  It makes it more difficult for them to do
     that if we have something in the standard that says this is
     the way we intended it to be used, so we have got to be
     careful that we don't necessarily lead the pack when we
     should be following.  We will think about that.
               DR. KRESS:  Okay.
               MR. FLEMING:  I just wanted to make one final
     comment about the categories that I don't think we had a
     chance to bring up is that there is I think a quantum leap
     in improvement when we went from one line in the sand to the
     idea of multiple categories, because what we were trying to
     avoid is by having one line in the sand is the unfortunate
     consequence of having everybody expected to get to that line
     and stop, so the idea of having, recognizing the three
     categories, especially Category 3, also points a direction
     for future enhancement of the technology.
               We wanted to try to avoid just this idea of
     meeting the requirement -- what do I have to do to meet the
     requirement, as opposed to what we need to do to advance
     this technology so we can make better decisions.
               To echo something Sid said in response to Jack's
     comments earlier, I do believe that already the
     certification process is already helping this problem of
     variability in the PSA results and I also agree with what
     you say.  It can't be done with any one leg of the stool.
               MR. SIEBER:  Right.
               MR. FLEMING:  It is all three of these -- the
     certification process, the standard and the methodology
     enhancements have to be working together.
               CHAIRMAN APOSTOLAKIS:  Gerry?
               MR. EISENBERG:  I am going to hand off to Ron
               CHAIRMAN APOSTOLAKIS:  Sure.  You are not
     obligated to speak.
               MR. SIMARD:  All the good comments have been
               I would just like to thank you on behalf of the
     project team for some good discussions, some constructive
     suggestions over the past day and a half.
               I think I have heard that we have struck a
     reasonable approach by trying to approximate the spectrum of
     possible applications with these three categories, have
     certainly gotten the signal that we need to do a better job
     of characterizing the attributes of these categories, and
     appreciate the fact that you gave us specific suggestions
     that we can work on, so I just want to thank you.
               CHAIRMAN APOSTOLAKIS:  You are welcome.  Ian,
     would you like to say anything?  No?
               Before we adjourn though, I think there are two
     points to be brought up.
               As a prelude, I always find that sometimes being a
     reviewer gives you a certain perspective that is not always
     right, so I always learn when I have to defend my research
     contracts at MIT before other people who are reviewing me. 
     I get upset a little bit at the beginning when they dare ask
     questions, but then after awhile I realize that this is the
     name of the game, so imagine I sitting over there and I act
               Now why is that relevant to this?
               Well, there was a suggestion made yesterday at the
     workshop which I thought was very good.  The suggestion was
     that the NRC Staff apply this standard to its own work, and
     what came to mind was the SPAR models.
               This committee recommended in the recent past to
     the EDO that the SPAR models be subjected to peer review. 
     The response from the EDO was no, we have had enough peer
     review and they have been used by some Sandia folks -- that
     is good enough.
               It seems to me that we should come back to this
     and if the committee agrees, of course, we should come back
     to it and I think it will be a healthy exercise for the NRC
     Staff to use this approach and maybe try to categorize SPAR
     models and what they can do -- well, I think that will be a
     very healthy exercise.
               DR. BONACA:  We will have to develop that.
               CHAIRMAN APOSTOLAKIS:  If we can demand perfection
     from others --
               DR. BONACA:  We have to develop a new category
     then because --
               CHAIRMAN APOSTOLAKIS:  What?
               DR. BONACA:  I will not mention it.
               CHAIRMAN APOSTOLAKIS:  But I mean the SPAR models
     eventually will be, unless I am mistaken, will be the plant-
     specific PRAs that the Staff will be using not to make
     decisions but as a major input to their decisionmaking
               DR. KRESS:  That's right.
               CHAIRMAN APOSTOLAKIS:  And I don't see why the
     SPAR models cannot be subjected to this particular process.
               Yes?  We get a smile from the Staff.  Are we
     getting anything more?  Oh, there you are.
               MR. CHEOK:  By default I'm it.  This is Mike Cheok
     from the Staff.
               I guess the SPAR models will be used as an initial
     stepping stone into whether something is risk significant or
     not.  All it is is to tell us if something needs to be
     looked at some more, and if something needs to be looked at
     some more, we will look at the licensees for more specific
               CHAIRMAN APOSTOLAKIS:  Well, I guess you just told
     us that the SPAR cannot be Category 3.
               CHAIRMAN APOSTOLAKIS:  Now the question is are
     they Category 1 or Category 2?
               MR. CHEOK:  In my opinion, probably not Category
               CHAIRMAN APOSTOLAKIS:  1.5 perhaps.
               MR. CHEOK:  It is probably below a Category 1.
               DR. KRESS:  Category 1.9.
               CHAIRMAN APOSTOLAKIS:  Below Category 1?
               MR. CHEOK:  Yes.
               CHAIRMAN APOSTOLAKIS:  Well, they just did it. 
     But we may find other examples.
               MR. CHEOK:  That's right.
               CHAIRMAN APOSTOLAKIS:  Thank you very much,
     anyway, for the comment. That is your expert opinion.  Give
     it to Budnitz.  He will give it to us.
               The other one is do the members have any
     suggestions regarding the July meeting?  Should, for
     example, Mr. Bernsen and Mr. Fleming and Mr. Simard come
     here with the same presentations or should they modify them
     a little bit?
               MR. SIEBER:  Condense them.
               CHAIRMAN APOSTOLAKIS:  I mean we can't tell you
     what to do.  I'm sorry.  I am using the word "should" -- you
     are not Staff.
               MR. BERNSEN:  "Should" is a recommendation.
               CHAIRMAN APOSTOLAKIS:  Recommendation, okay, or
     "shall" they -- they should consider.  They should consider.
               DR. KRESS:  I think they ought to consider the
     same stuff, only condensed a bit and maybe focus a little
     bit on the categorization process.
               CHAIRMAN APOSTOLAKIS:  I would agree with that.
               I mean if you could -- I don't know how much time
     you have until then, but maybe take the major comments that
     were made today and give us some preliminary reaction?
               DR. KRESS:  Yes.
               CHAIRMAN APOSTOLAKIS:  That will bring the
     committee up to speed.  I don't know if you realize this is
     not a paying job, for some of you anyway, so it may not be
     enough time, but if you could address some of these comments
     or what you thought was something --
               MR. BERNSEN:  We can do that.  Of course we will
     not have had all the comments.  We won't have a project team
     meeting, so again, just as I said in this meeting where we
     are representing ourselves as knowledgeable --
               CHAIRMAN APOSTOLAKIS:  Sure.
               MR. BERNSEN:  -- committee members and project
     team members, we certainly will be able to I think identify
     some of the issues you have raised and the fact that we are
     going to take them under advisement in some possible ways to
     the extent we can.
               CHAIRMAN APOSTOLAKIS:  For example, I mean when,
     Karl, you presented the categories, you might modify your
     viewgraphs a little perhaps to reflect some of the things
     that you accept and say this is what eventually the document
     will say I think that will promote better understanding.
               Again, you don't have to do this.  These are
     individual comments.
               MR. MARKLEY:  And because we had four members who
     didn't attend I think it is unavoidable to have the
               CHAIRMAN APOSTOLAKIS:  We will have the overview.
               MR. MARKLEY:  Short and brief and concise as you
     can, but the issues are really the important points.
               CHAIRMAN APOSTOLAKIS:  And judging from the
     experience of yesterday and today, the question of how do
     you really define the categories is there, so if they feel
     they have gotten any useful comments today, then maybe they
     should do it.
               MR. EISENBERG:  You mentioned there is a two-hour
     window for that.  How much of that window is the
     subcommittee presentation?
               DR. KRESS:  Normally --
               MR. EISENBERG:  Go ahead.
               MR. MARKLEY:  I'm sorry, Tom.  Normally it is
     about five to ten minutes, just introductory to introduce
     you and then you have the majority of the rest of the --
               CHAIRMAN APOSTOLAKIS:  Well, I can go over the
     major points.
               DR. KRESS:  Normally though, when you have two
     hours you ought to count on about an hour of that is yours. 
     The rest of it is interruptions from us.
               MR. EISENBERG:  I understand -- just to know what
     the ratio was.
               DR. BONACA:  I think they have a very good summary
     presentation.  If you went through that and if you just
     simply acknowledged some of the questions you got, to
     anticipate those --
               CHAIRMAN APOSTOLAKIS:  I will try to sketch some
     of the major points in my introduction, okay.
               DR. BONACA:  -- so that we avoid having to jump in
     again and again on the same issues, and otherwise I think
     the summary presentation was good.
               CHAIRMAN APOSTOLAKIS:  And the nature of that
     meeting is not to go into details and say on line this you
     said that and so on, but we will write a letter, right,
     addressed to the EDO?
               Does anyone have any comments or questions from
     the people around the table or others?
               Hearing none --
               DR. BONACA:  I want to say we had a lot of
     comments, a lot of criticism, et cetera.  Again, you know,
     at least personally -- I don't want to leave a perspective
     that I don't think that there hasn't been progress since
     last year.  I think there has been progress and that is just
     my personal view and that you people should be more than
     encouraged for what you are going through.
               I mean you are sifting through not only our
     comments but so many others and so that is what I wanted to
               DR. KRESS:  Yes.  I second that.  I think you are
     on the right track with this as the kind of standard we need
     to come up with.  I certainly want to thank you guys.
               MR. EISENBERG:  Thank you very much.
               CHAIRMAN APOSTOLAKIS:  It's a very difficult job
     trying to draw the line between various applications and so
     on, and a lot of it is subjective, as we discussed, so we do
     realize that you have a difficult job on your hands.
               The comments are offered, you know, in a
     constructive spirit and hopefully they will improve the
     product, because there is a need out there, even
     internationally.  I am being asked by a lot of people when I
     travel, especially people who are not in the business, and
     they are surprised that there is no standard for doing this
     new thing, so I appreciate your coming here and being
     patient with us and thank you very much.
               I think it was a very good meeting.  We all
     learned something, and on that note this meeting is
               [Whereupon, at 2:44 p.m., the meeting was


Page Last Reviewed/Updated Tuesday, July 12, 2016