Reliability and Probabilistic Risk Assessment and Regulatory Policies and Practices - April 21, 1999

                       UNITED STATES OF AMERICA
                     NUCLEAR REGULATORY COMMISSION
               ADVISORY COMMITTEE ON REACTOR SAFEGUARDS
                                  ***
             RELIABILITY AND PROBABILISTIC RISK ASSESSMENT
                 AND REGULATORY POLICIES AND PRACTICES
                                  ***
     
                        U.S. Nuclear Regulatory Commission
                        11545 Rockville Pike
                                           Room T-2B3
                        Rockville, Maryland
                        Wednesday, April 21, 1999
              The subcommittees met, pursuant to notice, at 8:30
     a.m.
     MEMBERS PRESENT:
         GEORGE E. APOSTOLAKIS, Chairman, Subcommittee on   
     Reliability and PRA
         THOMAS S. KRESS, Chairman,  Subcommittee on   
     Reliability and Probabilistic Risk Assessment
         JOHN J. BARTON, Member, ACRS
         MARIO H. FONTANA, Member, ACRS
         DON W. MILLER, Member, ACRS
         ROBERT L. SEALE, Member, ACRS
         GRAHAM B. WALLIS, Member, ACRS
     PARTICIPANTS:
         E. ROSSI, RES
         J. ROSENTHAL, RES
         B. YOUNGBLOOD, Scientech
         P. KADAMBI, RES
         M. MARKLEY, Cognizant ACRS Staff Engineer.                         P R O C E E D I N G S
                                                      [8:30 a.m.]
         DR. KRESS:  The meeting will please come to order.  This is
     a meeting of the ACRS Subcommittees on Reliability and Probabilistic
     Risk Assessment and on Regulatory Policies and Practices.  It's a joint
     subcommittee meeting.  I am Tom Kress, Chairman of the Regulatory
     Policies and Practices Subcommittee.  Dr. Apostolakis is Chairman of the
     Subcommittee on Reliability and PRA.
         ACRS Members in attendance are:  John Barton, Mario Bonaca,
     Don Miller, Mario Fontana, Graham Wallis and Bob Seale, which pretty
     much is most of the committee except for Dana and Bill Shack.
         The purpose of this meeting is to review the Staff's
     reconciliation of public comments on performance-based initiatives,
     SECY-98-132; and the plan for pursuing performance-based initiatives,
     candidate activities, and related matters; and NUREG/CR-5392, entitled
     "Elements of an Approach to Performance-Based Regulatory Oversight."
         The subcommittees will gather information, analyze relevant
     issues and facts, and formulate proposed positions and actions, as
     appropriate, for deliberation by the full Committee.  That is what we
     always do.  I don't know why we always say that.  I guess it's for the
     public record.
         DR. SEALE:  That is what we always aspire to.
         DR. KRESS:  Yes.  Michael T. Markley is the Cognizant ACRS
     Staff Engineer for this meeting.
         Rules for participation in today's meeting have been
     announced as part of the notice of this meeting, previously published in
     the Federal Register on April 6, 1999.
         A transcript of the meeting is being kept and will be made
     available as stated in the Federal Register notice, so it is requested
     that speakers first identify themselves and then speak with sufficient
     clarity and volume so they can be readily heard.  That means come to a
     microphone, naturally.
         We have received no written comments or requests for time to
     make oral statements from members of the public.  The only comments I
     have, in addition to what I have said before, I think the Staff is
     planning on going to the Commission with a Commission paper on this
     issue, and they want our -- I guess they want our opinion on the plans
     laid forth in this Commission paper and so we will have to bring this
     before the full Commission I guess in the May meeting and write a
     letter, so a letter is expected.
         MR. MARKLEY:  It will be June.
         DR. KRESS:  June?
         MR. MARKLEY:  Yes.
         DR. APOSTOLAKIS:  But I thought we were --
         MR. MARKLEY:  The Commission paper is not going to be
     available prior to the May meeting.
         DR. APOSTOLAKIS:  No, but I --
         DR. KRESS:  Go ahead.
         DR. APOSTOLAKIS:  I thought they had a deadline of May 31st
     to send it up to the Commission.
         MR. MARKLEY:  That's correct.
         DR. APOSTOLAKIS:  So we will write a letter afterwards.
         MR. MARKLEY:  In parallel with the Commission consideration
     of that paper.
         DR. SEALE:  Generally we don't write anything until we have
     the appropriate document, so we know what it is we are recommending.
         MR. MARKLEY:  Right.
         DR. SEALE:  Formally.
         DR. KRESS:  We do have a draft document.  We have something.
         With that, I will call on my Co-Chairman to see if he wants
     to make any comments before we start.
         DR. APOSTOLAKIS:  No, I think we should start.
         DR. KRESS:  Okay.  We should start?  So I'll turn the
     meeting over to -- to whom?
         MR. ROSSI:  Let me start off.  I am Ernie Rossi and I am
     Director of the Division of Systems Analysis and Regulatory
     Effectiveness in the Office of Research.  As was indicated, we are here
     today to talk about plans for performance-based approaches to
     regulation.
         First of all, let me thank you profusely for arranging for
     this meeting on such a short time scale.  We received the Staff
     Requirements Memorandum on SECY-98-132.  It was issued on February 11th
     and in that we were asked by the Commission to provide a plan to the
     Commission by the end of May, so we have a very short time in which to
     develop a Commission paper with a plan on it.
         We did have a public meeting last week.  It was not widely
     attended.  I think we had the right people there, but there are a lot of
     other things going on, as you may know, within the Agency now that
     competes for time with the industry, but we did have that meeting and we
     wanted to have it before we came here.
         With respect to our schedule, as I indicated, the Commission
     asked for the plan by the end of May so we are going to depend largely
     on whether comes out of this meeting today in developing the Commission
     paper and then, as Mike indicated, we will meet again with you in June,
     but that will be after the Commission paper has gone up.
         Now the plan that we are developing is to be a plan for the
     NRC as a whole, so that will include involvement of all the major
     offices and, as I think you will find from our presentation, there are a
     number of activities in the offices that are underway now that we
     believe to be the application of a performance-based approach to the
     regulatory process.
         We are looking forward to the interchange of information
     today and are very interested in your views on this subject.  I would
     like to point out that in your package on the last page there is a set
     of questions that we used in our meeting last week with the stakeholders
     and we developed those questions as the things we thought we wanted
     answers to, to help us in developing the Commission paper this next
     month, so we are very interested in your views on anything you hear
     today but you might want to look at the questions and we would very much
     appreciate it if you would look at the questions in particular and any
     comments you have, either on whether they are the right questions or if
     you could provide answers and your view, that would be of help.
         At this point I am going to turn things over to Prasad
     Kadambi, who is going to give you a presentation on sort of the history
     of this issue and the work that we have done to date, and this will then
     be followed by a presentation by Robert Youngblood of Scientech. 
     Prasad?
         MR. KADAMBI:  Thank you, Ernie.  Good morning.  My name is
     Prasad Kadambi.  I am with the Office of Research in the branch, it's
     called the Regulatory Effectiveness Assessment and Human Factors
     Branch --
         DR. APOSTOLAKIS:  Looks like "REHAB" --
         MR. KADAMBI:  Well, maybe that is part of its function.
         DR. APOSTOLAKIS:  That was good.
         MR. KADAMBI:  I'd like to lay out for you the outline for
     today's presentation.  I will be back to Ernie for the management
     overview.  We will go through some of the historical background, talk
     about the Staff activities.
         In the middle we will have a focused discussion on the NUREG
     CR-5392, which has been announced as part of the agenda and Bob
     Youngblood, who is the principal investigator on that, will speak on
     that subject, and after that I hope we will get very much focused on the
     SECY paper and the response that we have to develop on a very short time
     scale, talk about the SRM and what we need to do to address that, the
     stakeholder meeting input, and then the elements of the plan that we
     have come up with up to now, subject to any feedback we receive from
     you.
         With that, Ernie would you like to just address the points
     on the arrangement of --
         MR. ROSSI:  Yes, I will say a few words.  As you know, the
     Office of Research was reorganized, I guess it was the last week in
     March, and the Division of Systems Analysis and Regulatory Effectiveness
     has two branches.  One of them is the branch that Prasad is in.  The
     other branch has the severe accident work, codes and experimental work,
     and the thermal hydraulic work.
         In the Regulatory Effectiveness Assessment and Human Factors
     Branch, they have several teams.  They have a team for generic safety
     issues, one for human factors, one for regulatory effectiveness, and one
     for operating experience reviews, so that is sort of a quick overview of
     the organization.
         I did mention the fact that there were some ongoing
     activities in the other offices that are performance-based approaches. 
     Probably the largest-scale one and a very important one for the agency
     as a whole is revising the reactor regulatory oversight process and
     inspection program, so that will be going on as we do anything in
     addition to that, and what we intend to do is to learn from it as much
     as we can but that program is well underway and we don't anticipate or
     plan to do anything that would have a major effect on it.
         Obviously I'm sure that NRR will keep in touch with
     everything else that is going on, so if something turns up of value they
     will take it into account, but this is not directed at that particular
     program, and then there are other things in the area of
     performance-based applications like Appendix J and the maintenance rule,
     and there are also some activities underway in NMSS.  We don't have
     anybody here to talk about them today, but NMSS did talk about their
     activities in the meeting last week, so those will be discussed I guess
     in the Commission paper.  Prasad?
         MR. KADAMBI:  Thank you.
         I guess I would like to begin the historical background with
     the SRM that was issued for SECY-96-218.  I think this committee has
     heard about this SRM and the four policy issues that were addressed in
     it.
         The four policy issues were the role of performance-based
     regulation and the PRA implementation plan, plant-specific application
     of safety goals, risk-neutral versus increases in risk and risk-informed
     IST and ISI.
         In the SRM the Commission asked us to consider
     performance-based initiatives not explicitly derived from PRA insights. 
     They also asked how these would be phased into the overall Regulatory
     Improvement and Oversight Program.  It was actually in order to address
     this in part that the SECY-98-132 was prepared.
         DR. KRESS:  Did they spell out what those performance-based
     initiatives were that don't come out of PRA insights?  Did they give you
     a list of them?
         MR. KADAMBI:  Well, the SECY-98-132 actually was, as I see
     it, an earlier version of the plans to do that.  I think we are right
     now in part of that effort to develop a process and a rationale to come
     up with the kind of list I think you are asking about.
         DR. APOSTOLAKIS:  The way I read the SRM is they are asking
     you to include in the PRA implementation plan performance-based
     initiatives, and then they say the Staff should include its plan to
     solicit input from the industry on additional performance-based
     objectives which are not amenable to probabilistic risk analysis, so
     this is slightly different in my mind than what you have there.
         In other words, they want to see a performance-based
     approach to regulation that will be based on PRA when necessary or
     possible, but should not be limited to the PRA insights.
         Now what you are doing here is limit it to non-PRA
     applications or is it a total thing, and some of it is based on PRA,
     some of it is not?
         MR. KADAMBI:  I want to make sure that I am cautious in
     answering your question because I think we may be getting some SRMs
     mixed up over here.  I don't know if I am right, but I think the SRM
     that you read from is the SRM 298-132, whereas the SRM that was
     associated with 96-218, which sort of led us into this work, used
     different language.
         DR. APOSTOLAKIS:  I see.
         MR. KADAMBI:  And what it said is look into these areas -- I
     am paraphrasing -- and either make it part of the PRA implementation
     plan or develop a separate plan.
         DR. APOSTOLAKIS:  That's true, yes.
         MR. KADAMBI:  And, you know, that's when we got started on
     looking at what I will call in shorthand "non-PRA" work and right now
     the plans that we are proposing to develop in response to the most
     recent SRM will hopefully answer some of the questions you just asked
     me.  I am not sure I know exactly what it will cover.
         DR. APOSTOLAKIS:  But given the earlier SRM, it says the
     Commission has approved Alternative 1 with respect to the role of
     performance-based regulation, but applications of performance-based
     approaches should not be limited to risk-informed initiatives.
         MR. KADAMBI:  Right, and we are not planning to limit it to
     risk-informed initiatives --
         DR. APOSTOLAKIS:  Yes, but you are not going to limit it to
     non-risk-informed initiates either.
         MR. KADAMBI:  No.
         DR. APOSTOLAKIS:  So today's subject is what?  The issue of
     performance-based regulation, period.
         MR. KADAMBI:  Correct.
         DR. APOSTOLAKIS:  And in some places PRA may be useful and
     in others it may not.
         MR. KADAMBI:  Correct.
         DR. APOSTOLAKIS:  We are not limiting it to non-PRA?
         MR. KADAMBI:  Correct.
         DR. APOSTOLAKIS:  Okay, good.
         MR. ROSSI:  One thing though that in my opinion I think the
     PRA part of this is probably much better developed and on the road.  We
     will consider the whole thing as a whole so we are kind of more
     concerned about the other part that is non-PRA based, so that is
     probably what we are thinking about but we have to make sure that it is
     all integrated --
         DR. APOSTOLAKIS:  Exactly.
         MR. ROSSI:  -- and so forth, and we will do that in one way
     or another.
         DR. APOSTOLAKIS:  Okay.
         MR. ROSSI:  And it also has to be totally integrated across
     the agency with the other offices.
         DR. APOSTOLAKIS:  Good.
         MR. KADAMBI:  The next step along the way was really part of
     the strategic assessment and rebaselining initiative, DSI-12.  There was
     a paper issued and subsequently a COMSECY, which took a rather
     comprehensive approach including performance-based concepts into Staff
     activities.
         Now the way these things I believe were integrated into the
     Staff's activities was through the NRC Strategic Plan, which said we
     will implement risk-informed and, where appropriate, performance-based
     regulatory approaches for power reactors, so that as I see it gave the
     direction to the Staff saying this is what you should be doing.
         DR. APOSTOLAKIS:  Has the Commission ever defined
     performance?
         MR. KADAMBI:  Well, okay, if I can go on, maybe -- I believe
     that we have the best definition up to this date in the white paper, and
     that, if you don't mind, I'll put answering that question off until we
     get to that.
         DR. APOSTOLAKIS:  All right.
         MR. MARKLEY:  Just for the benefit of the public and the
     members here, that white paper is SECY-98-144.  Correct?
         MR. KADAMBI:  That's right.
         DR. APOSTOLAKIS:  This is the Commission paper where they
     defined risk-informed -- yes.
         MR. KADAMBI:  In fact, the next two bullets speak to what
     happened in the June, 1998 timeframe is when the Staff issued
     SECY-98-132, "Plans to Increase Performance Based Approaches in
     Regulatory Activities."  The Staff also issued 98-144, a white paper on
     risk-informed, performance-based regulation, but in that SECY paper, the
     white paper was in a draft form and was offered to the Commission to
     deliberate and confirm the Commission's views on it.
         The SRM for SECY-98-144 was actually issued on March 1,
     1999, so we are talking in a sense of things that happened bunched
     together in time and then there was a long period where not much
     happened, sort of a punctuated equilibrium, as I see it.
         As Ernie mentioned, in February, 1999, we got the SRM for
     SECY-98-132, and it directed the Staff to prepare plans for
     performance-based initiative after obtaining stakeholder input, and that
     is part of the exercise that we are currently involved in.
         DR. APOSTOLAKIS:  What is the message from this history? 
     What are these bullets telling us?  I mean there are a couple of SECY
     documents, Commission SRMs, so then that message is what?
         MR. KADAMBI:  You know, at least to me the message is that
     we have got to use all the direction that has become available from the
     Commission to put together a credible plan that will help us meet the
     Strategic Plan objectives.
         MR. ROSSI:  I think the message from the history is
     primarily just background information, to tell people what has happened
     in the past and bring everybody up to the same point as to what has
     transpired.  I don't know that there is any particular message other
     than people ought to know how it started and what has been written on
     it, primarily.
         DR. KRESS:  I guess the message I get is that the Commission
     is very serious about performance-based regulation and wants --
         MR. ROSSI:  There is also that message in there also, yes. 
     That is why they asked for a plan by the end of May.
         DR. APOSTOLAKIS:  Have you seen an evolution in the
     Commission's thinking about this issue over the last two years or so,
     or -- because I read the SRMs and it seems to me they keep saying the
     same thing.
         MR. KADAMBI:  I believe that with the issuance of the white
     paper in final form, the evaluation that I observed happening through
     the strategic assessment and rebaselining process and the issuance of
     the DSI-12 papers, I observed a certain evolution and I think it has
     come to some kind of a fruition in the white paper.  That is the way I
     am viewing this process that we have been involved in and trying to use
     the white paper essentially to the extent that it will help us guide the
     work.
         So focusing a little more on the SECY paper itself,
     SECY-98-132, at the time that the SECY was prepared we were also working
     with the excellence plan that the Staff had sent up to the Commission
     and in it there was one of the strategies that was described was in the
     excellence plan -- a Strategy 5, which sought to look at regulations and
     regulatory guidance and some other things to improve effectiveness and
     efficiency, which were described as the components of excellence, so
     SECY-98-132 tried to take advantage of that effort and also address the
     Commission's direction on performance objectives which are not amenable
     to probabilistic risk analysis.
         The SECY again did not focus only on reactors.  It took a
     comprehensive approach.  The ACRS issued a letter on April 9th, 1998,
     and asked the question how and by whom performance parameters are to be
     determined and deemed acceptable and how to place acceptance limits on
     them.
         DR. KRESS:  Sound like good questions.
         MR. KADAMBI:  Well, that is the reason why I have got it up
     there, to let you know that we are still thinking about it.  We may not
     have all the answers but we have heard you.
         The EDO responded on May 4th and the response basically said
     the white paper is going to help us and there is this effort on DSI-13,
     which is industry, interaction with industry.
         DR. APOSTOLAKIS:  Speaking of good questions though, I look
     at the last viewgraph, questions for the stakeholders, it doesn't look
     like you asked them that question.  Did you?
         MR. KADAMBI:  We did ask the stakeholders these questions.
         MR. ROSSI:  No, he's talking about did we specifically ask
     the stakeholders the question of how and by whom performance parameters
     are to be determined and deemed acceptable, and how to place acceptance
     limits on them.  No, we did not ask that as a specific question.
         What we I guess are looking for from the stakeholders is
     specific views on regulations and other regulatory activities beyond
     what is being covered in the risk-informed work that we might try to
     apply performance-based techniques to, and so I think that question is
     kind of an implied question in what we did ask.
         I guess to date we haven't gotten very much specific input
     from the industry on what things they would like to see more
     performance-based.
         I think, Prasad, in the agenda to the meeting, you had some
     ideas of areas.
         MR. KADAMBI:  Yes.  What I wanted to point out is that I
     think there is an expectation that we would meet with you before we went
     out to the stakeholders, but because of the crush of events, because of
     the way things have developed in sort of spurts we have not been able to
     do that.  We are coming to you after we met with the stakeholders.
         In SECY-98-132 the Staff presented a suggestion that a
     separate policy statement on performance-based approaches may be
     beneficial but the Commission did not address this question.
         DR. KRESS:  Did you want us to address that in our letter,
     that particular question?
         MR. KADAMBI:  Well, I think as it will come out later, our
     approach right now is to use the white paper as the policy statement, in
     effect.
         DR. KRESS:  I see.
         MR. KADAMBI:  We have always recognized that resource
     requirements presents a really difficult challenge, and it still
     represents a major consideration in our work in this and at the same
     time we also recognize that there are many research needs, many
     unanswered questions that we have to tackle, so that came out in that
     SECY paper also.
         DR. WALLIS:  What do you mean by research?  How do you do
     research on a subject like this?
         MR. KADAMBI:  I guess the way I would look at it is it's the
     kind of research that will help us provide a technical basis that can be
     used by the program offices if they identify candidates that they would
     like to convert to a performance-based approach.
         DR. WALLIS:  Well, research to me, sir, means having
     hypotheses and testing them and seeing what works, and research in this
     kind of field seems to mean dreaming up something and then arguing about
     it with other people.
         MR. ROSSI:  I think this area that I think you described
     your view of what would be done.  I would say it a little bit
     differently.  I would say that what we need to do is we need to give
     careful thought and discussion on where we can apply performance-based
     approaches, and the ACRS letter up there had one area of how to place
     acceptance limits on the things that we define as the parameters that
     ought to be used to judge performance.
         I think that is an area where it takes some analysis and
     thought as to how you would go about developing the acceptance limits,
     so there would not be any experiments or anything of that sort, but
     there would be careful thought and discussion.
         I think you described it in a slightly different way but I
     would say careful thought and discussion and try to develop a consensus
     on what parameters ought to be used and how to place acceptance limits
     on them, and what particular regulatory activities could this be applied
     to.
         DR. WALLIS:  So what you are describing is not a scientific
     process, it is a political one.  It is not let's find out by testing and
     by looking at what has been done before what actually works on some
     basis which could be called scientific.  It is actually putting people
     together who have some stake in arguing or wrangling or discussing,
     whatever you want to say, until people get tired and agree to something. 
     That is a political process.
         MR. ROSSI:  Well, we would try to determine parameters that
     could be used for judging maybe something like quality assurance, and
     then how would you tie that analytically to safety in some way.  That is
     the kind of thing that we would look for.
         DR. MILLER:  What is the process of determining those
     parameters?  Is there an organized process or we just sit around the
     table determining what they are?
         MR. ROSSI:  Well, I think at this point in time we are at
     the point where we are trying to determine what activities might be
     amenable to the process in the first place, and then once we do that we
     would have to -- I mean we are trying to develop a plan for doing many
     of the things that you are asking us questions about.
         DR. BONACA:  The NUREG CR-5392 seems to suggest in the
     beginning an approach to do that.  Is it going to be presented?
         MR. ROSSI:  Yes.
         MR. KADAMBI:  That's part of the presentation this morning.
         DR. MILLER:  This NUREG is part of your research, right?
         DR. KRESS:  Correct.
         MR. ROSSI:  Yes.  That would be an example of research,
     right.  Yes, that is absolutely right.
         DR. KRESS:  That's what passes for research here, Graham. 
     Under your definition, would you have categorized Einstein's special
     theory of relativity as research or not?
         DR. WALLIS:  It's a hypothesis which is testable.  The
     diamond tree is an interesting idea but I don't see it as a testable
     hypothesis.
         DR. MILLER:  Why not?  Why isn't it testable?
         DR. WALLIS:  It is testable if it is used in various
     circumstances and shown to be somehow better than something else.  Then
     I guess you have some measure of whether it is good or not.  That is
     what I am looking for.
         DR. FONTANA:  Excuse me.  I think that the test would be,
     once you have worked this out, is to try it out on a pilot plant with a
     test that maybe lasts 10, 15 years and keep doing the other ones the old
     way, and then if a problem keeps cropping up, you may be able to see it.
         DR. BONACA:  Actually I thought that NUREG had a lot of good
     thoughts in it and I think the important thing is to almost roll up your
     sleeves and just go to an exercise and put in some of those boxes some
     of the activities that you have done in the past and are thinking of not
     doing any more, and then see how the thing works.
         DR. KRESS:  To pull off a pun, there were a few gems in
     there, weren't there?
         DR. BONACA:  Well -- I don't know --
         DR. MILLER:  Mr. Chairman, you can move ahead on that one.
         MR. KADAMBI:  The last point is that we did not incorporate
     this work into the PRA implementation plan, so ask of now we are really
     dealing with something that is different from the PRA Implementation
     Plan.
         DR. MILLER:  So that never will be put in the PRA
     Implementation Plan, or is it just for the --
         MR. KADAMBI:  Never is a long time.  I don't really know.
         DR. MILLER:  There is no plan to put it in there in the near
     future?
         MR. KADAMBI:  That is not part of what we are proposing.
         DR. SEALE:  Is there an intent someplace else in the bowels
     of Research to take another look at the PRA Implementation Plan?  After
     all, it's about three years old or so and most of the goodies have been
     shaken pretty hard.  In fact, some of the things are counter to the
     original statements in the plan, and I was wondering if you were going
     to take a look at it.
         MR. ROSSI:  Well, it does get periodically reviewed.  I
     believe it gets reviewed every quarter and revised and sent up to the
     Commission, right? -- so you would have that.
         Again, some of the questions you are asking I think are
     things we need to think about when we develop our plan. We simply don't
     know the answer to them. I mean depending on how we go, we certainly may
     want to add things into the PRA Implementation Plan because the one
     thing we do want to do is make sure that this thing is all looked at in
     a coherent way.
         We don't want two things going off in semi-different
     directions that overlap so these are all things we have to consider when
     we develop our plans.
         DR. SEALE:  George, we may want to ask these people to bring
     us up to date on the latest status, the details of the PRA
     Implementation Plan at some point.
         MR. MARKLEY:  I think Dr. Seale's question was a little bit
     different.  If you look at the PRA Implementation Plan today it is full
     of more completed items than it is future items, and that is part of it. 
     They don't see the roadway that it is going down, next step if you want
     to call it.
         MR. ROSSI:  I don't think we have the people that are able
     to talk in a lot of detail about the PRA --
         DR. SEALE:  I appreciate that.
         MR. ROSSI:  -- but let me say one thing.  There is the
     effort on risk-informing Part 50 that we went to the Commission and
     described how that might be done, and so there are the beginnings of
     things to do, more major things in the risk-informed area.
         I am sure plans will be developed for anything that is
     undertaken, and how it is factored in to the PRA Implementation Plan I
     can't really tell you.
         DR. SEALE:  Okay.
         DR. MILLER:  You are going to go through this one now,
     Prasad.  There's been a statement these two are saying the same thing,
     98-144 and 132.  Can you point out places where they may not be saying
     the same thing?
         MR. KADAMBI:  Actually, I did not mean to imply that they
     are saying the same thing as such, but they are generally -- you know,
     they are directed towards addressing the same sorts of activities, I
     believe, and as I see it they are taking a step at a time in a direction
     that today we can do a better job of articulating some of the plans that
     will get us where we want to go than we were able to at that time.
         The white paper, as I mentioned, that went up to the
     Commission had a draft of this statement on risk-informed
     performance-based regulation.  I can't tell you exactly how it was
     changed by the Commission's deliberations, but it was changed, I
     believe, so I can't really address the question to what extent is it the
     same or different from 132, which was issued in June of 1998.
         DR. MILLER:  And I assume that you will elaborate on Bullet
     5.  I find it intriguing.  I'll let you go ahead.
         MR. KADAMBI:  Okay, but the white paper does articulate a
     number of principles and I believe these have been very useful in our
     thinking --
         DR. MILLER:  The four principles that are articulated --
         MR. KADAMBI:  I'll get to that also.
         DR. MILLER:  Is it in the overheads --
         MR. KADAMBI:  Actually, the very next overhead I talk about
     the characteristics of --
         DR. MILLER:  Those are the principles?  Did you say a
     number?
         MR. KADAMBI:  Well, that is definitely as far as
     performance-based is concerned those are the principles, but I think in
     the white paper it addressed other things, you know, the current
     regulatory framework
         DR. MILLER:  Those are not principles.
         MR. KADAMBI:  Those are attributes -- you know, the
     Commission said these are four attributes.  As I see it, they would
     become necessary but not necessarily sufficient conditions for a
     performance-based approach.
         I mean that is sort of getting ahead, but --
         DR. WALLIS:  Could I ask you -- I'm sorry.  Is this
     something invented by NRC or is it the history of society using
     performance-based approaches to regulate something else, so that we know
     what works and what doesn't work?
         MR. KADAMBI:  I am not sure it was invented by NRC but as
     part of the work on NUREG CR-5392, there was a literature search done,
     and I think Bob may address some of the other places that he has found
     where the concepts of performance and results oriented principles may
     inform the activity.
         DR. MILLER:  I guess maybe it is performance-based
     regulation used in other regulatory processes.
         MR. KADAMBI:  Yes.  In fact, one of the statements that was
     made at the stakeholders' meeting was it would be wrong to think that
     somehow the performance-based approach started with the SECY-96-218 or
     whatever, you know, that I alluded to earlier, that there are many other
     things that have been performance-based and in fact SECY-98-132 talks
     about how we would consider ALARA as a very performance-based concept
     that has been around awhile and that has worked quite well.
         DR. WALLIS:  There's isn't a case study, though, like
     aircraft maintenance or something, where someone can show that
     introducing performance-based methods improved --
         MR. ROSSI:  I believe the maintenance rule was intended to
     be -- that the NRC?
         DR. WALLIS:  That's again NRC.  It's always been NRC.
         MR. ROSSI:  Okay.
         DR. WALLIS:  Is there anything from outside, as a reality
     check?
         MR. KADAMBI:  Again, the only one that I am aware of and I
     don't necessarily know everything that is going on is the work that we
     sponsored from the Office of Research in 5392.
         DR. MILLER:  So I think Dr. Wallis has a good question.  In
     other arenas, say aircraft and so forth, we don't have any evidence of
     performance-based?
         MR. ROSSI:  Bob, you have done work in this area on a
     literature search.  Maybe you are the right one to try to address this
     question.
         MR. YOUNGBLOOD:  Yes.  It wasn't going to be part of my
     slides, but the EPA has regulatory processes that are performance-based
     in the sense that for monitoring of emissions you can sort of afford to
     go by how well people are doing in controlling them if they are actually
     being measured, so you can afford in that case to have kind of a
     feedback loop and take a performance-based approach to monitoring that
     kind of thing.
         That example is in the back of the report.
         And one or two similar examples are in the back of the
     report.
         DR. MILLER:  Those are in the appendices?
         MR. YOUNGBLOOD:  Yes.
         DR. MILLER:  I'm sorry, I didn't get the appendices.
         MR. YOUNGBLOOD:  It is not a focus of what I am talking
     about here because, although the report wasn't about reactors and isn't,
     and I think the thoughts apply more broadly than reactors, I think that
     for reactor oversight, there are things about reactors that make other
     industries hard to compare.
         DR. MILLER:  Isn't Part 20 somewhat performance-based?
         MR. KADAMBI:  Yes.  And ALARA is part of that.
         DR. MILLER:  Well, ALARA comes out of Part 20.
         MR. KADAMBI:  Right.
         DR. SEALE:  Well, the new leak rate, Appendix J, is highly
     performance oriented.
         MR. ROSSI:  I think the question, though, that Dr. Wallis
     had, had to do with its use in other arenas in nuclear regulation.
         DR. KRESS:  Chemical.
         MR. KADAMBI:  It may well be, and I would suspect that with
     the GPRA, Government Performance and Results Act, there is a push for
     applying it in many areas that perhaps i the past had not considered it.
         If I can go on with this, I guess the bottom line on this
     slide is that I considered this white paper as being very important to
     our work.  I believe that it provides what I call a touchstone for
     implementing the Commission's direction.  We plan to use it as an
     equivalent to a policy statement.
         Of course, we have been aware of the broad outlines of the
     Commission's thinking on DSI-12 and that is what, in a sense, informed
     the work on 5392 also, and, you know, there are some clear indications
     that the Commission wants us to view this kind of work as being
     comprehensive and not directed only towards one regulatory arena.  But I
     believe what they are also saying is right now we are in a better
     position to use risk-informed regulation, regulatory concepts and we
     need to position ourselves better in order to be ready for
     performance-based concepts.  That is the message I get out of reading
     the white paper.
         And RES, the Office of Research, I believe, you know, we
     want to contribute to position the agency in a better state of readiness
     to employ these concepts to the extent that would be useful in actual
     rulemaking or developing regulatory guidance.
         The revision to the reactor regulatory oversight process
     that Ernie referred to, I think is a prime example of something that is
     going on right now from which we can learn the lessons and we in a
     better position to offer the kinds of guidance that people may find
     useful.
         DR. MILLER:  Back to bullet 5 there, Prasad, that implies
     that we are ready to do risk-informed and we are not ready to do
     performance-based, is that what that means?
         MR. KADAMBI:  Right now I think, as a formalized approach,
     you know, although people say that performance-based approaches have
     been used, I am not sure that we can pull together the sort of a
     formalized process that we can offer to the staff as guidance, and that
     is one of the elements in our plan.
         DR. MILLER:  In other words, we need something equivalent to
     a 1.174 for performance-based, is that?
         MR. KADAMBI:  Well, I am not sure what we will find will be
     the best way to do it, but, you know, certainly, I think 1.174
     represented sort of a watershed event in the developments on the
     risk-informed area.  And I hope that we can learn lessons from that
     developmental process also as we go forward with performance-based work.
         DR. APOSTOLAKIS:  Well, since you mentioned 1.174, it seems
     to me that it would not take a great effort to develop a similar
     approach to performance-based regulation, because 1.174 really states
     principles and concerns and expectations.  It doesn't get into actually
     doing risk-informed analysis of particularly regulatory issues.  And you
     already have some principles or attributes here, and what you are trying
     to do, judging from the documents I have read, is you are trying to go
     way beyond what 1.174 did and actually see whether you can define some
     performance criteria somewhere.
         But if you keep it at the level of 1.174 it seems to me you
     already have most of what you will need to write something like that. 
     The only principle that 1.174 uses that you may not have, and you
     probably have to think about, is the delta CDF, delta LERF kind of
     criterion, which may or may not apply to your case.  It may not be
     applicable at all.
         But other than that, in terms of principles, I mean if you
     think about it, 1.174 says be careful, make sure that you comply with
     the regulations.  Make sure you don't compromise defense-in-depth too
     much.  And, you know --
         DR. KRESS:  It is those last two that you get in trouble
     with going to performance-based regulation.  Complying with the
     regulations, I mean that is the whole idea, we are trying to change it.
         DR. APOSTOLAKIS:  Yes.
         DR. KRESS:  And the other thing is the defense-in-depth
     principle.  You have got to do something better than what is in 1.174.
         DR. APOSTOLAKIS:  No, but if you keep it at the level of
     1.174, I think we have the material.
         DR. KRESS:  Well, you keep it a level of determining
     principles and things..
         DR. APOSTOLAKIS:  As a starting point.  Yeah, and what you
     should worry about.  I mean a lot of the stuff in the NUREG that
     Scientech has prepared talks about it, you know, that they took the NEI
     example and they raised some questions and so on.  So you can turn it
     around and write a set of principles for performance-based regulation
     that would be I think at the same level as 1.174.  And perhaps you
     should think about doing something like this to have a product on the
     way to more detailed work later.
         DR. MILLER:  And don't try to solve all the problems at
     once.  The high level principles, then go from there.
         DR. BONACA:  I feel exactly the same.  What I mean is that I
     think you have a lot of information right now and you have even some
     approaches that I believe are quite useful in a pragmatic way to begin
     to build something.  I mean without an example in front of us, you know,
     I am lost in -- such difficulties seem to be represented by what you are
     presenting here.  I believe that some progress can be made pretty
     quickly.
         For one, I believe that whenever there was no concern that
     the failure to meet the performance indicator would result in
     unacceptable consequences, past regulation went to performance-based. 
     For example, ALARA is a typical example of that, where failure to meet
     certain goals are not catastrophic, I mean simply you don't meet those
     goals.  So that is why the regulation was performance-based.
         And so I am trying to say that, you know, I agree that there
     are some many practical elements, okay, from past regulation on what you
     want to do, to begin to build something there, and the diamond, this
     gem, was a good example of how you could go about that.  And, you know,
     I wonder if there is an effort going on right now to attempt to do
     something of that type.
         DR. APOSTOLAKIS:  There isn't.
         MR. KADAMBI:  Well, what tempers my thoughts in this area
     is, although I was watching it from the outside, the Reg. Guide 1.174
     was a huge commitment of resources.
         DR. APOSTOLAKIS:  Yes, but that was because it was done for
     the first time.  Now, we shouldn't -- I mean now we are experienced.  I
     mean things should flow relatively easier.
         And let me give you an example again with 1.174 that shows
     that when you actually get into a specific issue, you do other things
     that what is in 1.174.  1.174 talks in terms of the five principles that
     feed into the integrated decision making, and one of the criteria, as I
     said earlier, is delta CDF and delta LERF.  And then you go to another
     Regulatory Guide that deals with graded quality assurance and you see
     that delta CDF and delta LERF are not used.  In fact, another approach
     is proposed there that classifies components according to the importance
     measures.
         So here is a good example of having a document of principles
     and then when you go to an actual application, you realize that that is
     not sufficient and you do something else.  You do more.
         DR. MILLER:  Or you do less.
         DR. APOSTOLAKIS:  You do less in one area and you do more in
     another area.  So I don't see --
         DR. MILLER:  Identifying the principles.
         DR. APOSTOLAKIS:  I don't see what is different with
     performance-based regulation.  You can pool all this knowledge that you
     have with the attributes that you have, four attributes and so on, put
     them in the form of principles, discuss the issues, just like 1.174
     does, you know, incompleteness, you know, all that stuff that they have
     there.  I am sure you have other issues here.  And, again, the NEI
     example that Scientech worked out is very enlightening in that respect. 
     And then that is a first document that sets the principles for
     performance-based regulation.
         And then you continue now looking at specific cases and what
     specific issues arise.  Because that way we will we have progress.  And
     I don't think it is going to take as much effort as 1.174, which really
     was created basically out of nothing.
         DR. MILLER:  So you could use the principle, the attributes,
     and then we already have experience with the maintenance rule and so
     forth, and test those attributes, go along with Professor Wallis'
     approach and see if those things we already have experience with are the
     attributes that are sufficient.
         MR. KADAMBI:  Well, I think everything you say makes a lot
     of sense and, certainly, the advice would be very useful, but I have
     also got to make it fit into the SRM that we have to respond to within
     the resource limitations that we have.
         MR. ROSSI:  Well, I think all the ideas we are hearing are
     things that we need to consider how to address in the SRM.  I mean we
     might not want to commit in our plan that we present to the Commission
     to doing what you are suggesting.  But, certainly, we will want to think
     about whether that is the right thing to do and make a conscious
     decision at some point in time on how you write it down and how you have
     Reg. Guides and so forth.
         And it might be appropriate at this time if you put up the
     next slide, because the next slide seems to be the one that really talks
     about performance-based regulation in some substance.
         DR. MILLER:  Well, the attributes for me, it is on the next
     slide, those captured a lot of thinking, I thought.
         MR. ROSSI:  Yeah, those are the ones that we need to focus
     on.  And the think that we need to do is to -- is this the one?  Yeah,
     this is the one.  What we would really like, or what I would like -- I
     shouldn't say we, because others may not agree, is to have some good
     area that we could apply these principles to that involved PRA to a
     lesser degree than some of the other things that are going on in the
     agency and work it through as a pilot somehow, and apply all of these
     principles to that area.
         And I think in the meetings we have had with stakeholders,
     we have tried to seek suggestions of where we might do this, and I am
     not sure we have gotten any substantive suggestions as yet.  One of the
     reasons we may have not gotten any further suggestions is that the
     stakeholders, particularly the industry, may feel a little reluctant to
     get involved in another major area at this point in time.  That could be
     the thing.
         But this I believe is the description of exactly what we
     want to do, and what we would like to do is to find appropriate
     regulatory -- appropriate rules or regulatory guidance where we could do
     exactly what is up there on this slide.
         DR. FONTANA:  Let me ask a question to try and clarify the
     direction in which your thinking is going here.  You have got a
     performance-based regulation of some kind, it leaves a lot of
     flexibility on the part of the licensee to develop his procedures and
     such-and-such.
         DR. APOSTOLAKIS:  It doesn't say a lot.  It says have
     flexibility.
         DR. FONTANA:  Well, whatever.  Whatever it is, it is
     supposed to.  Now, the question is, particularly related to that last
     bullet that you have there, the last empty bullet, the question is, the
     failure to meet a performance criterion should not in itself be a major
     safety concern.  Okay.  So that means that someone has to determine what
     the consequences of not meeting a particular procedure or whatever,
     whatever, whatever.
         Now, the question is, who does that analysis that indicates
     what the consequence of not meeting that particular requirement is?  Is
     that going to be the licensees?  In which case, if all the licensees are
     doing it differently, then the NRC is going to have to be able to track
     a lot of different ways of doing it.  Or will the NRC specify, if you
     want to call it that, a methodology by which one could do these
     analyses, and, therefore, save themselves a lot of work because the
     analyses are pretty similar?  Which way are you thinking?
         MR. KADAMBI:  Well, I am not sure that we have come to the
     point where we can offer thoughts that address that question.
         Ernie, did you want to say something?
         MR. ROSSI:  I was going to say that I guess the best example
     that we have is the implementation of the maintenance rule, and I didn't
     come prepared to talk about all the details.  But I think, my
     understanding of what was done is that that is a performance-based
     regulation and it says you have to have criteria to judge whether a
     component is being maintained in the proper way.  And when you get to
     that criteria, it ought to be based on something that you do exactly
     what the fourth item up there is.
         And with respect to who does it, what I believe happened on
     the maintenance rule was we started to prepare a Reg. Guide in the NRC
     and then the industry prepared their own document, and then at some
     point we endorsed the document prepared by the industry, so that it was
     done pretty standardly across the industry.
         DR. FONTANA:  The industry being NEI.
         MR. ROSSI:  NEI, right.  And then NEI got the industry's
     buy-in on their guidance document.  I believe that -- and, again, I am
     speaking off the top of my head, and there may be others that know more
     in the room.
         DR. WALLIS:  Well, let's take this --
         MR. ROSSI:  But that is kind of a model for how -- that I
     think addresses your question.  So it is an example of what was done in
     one case.
         DR. FONTANA:  Thank you.
         DR. WALLIS:  Let me suggest something.  You said it would be
     getting to a level of great detail.  And one could say the only thing
     that matters is CDF.  The only performance indicator that means anything
     is CDF.  So I will have a CDF meter, it will run all the time, and I
     will have fines if it goes too high, and I will have rewards if it goes
     too low, and that is all I need.
         DR. MILLER:  You have great confidence you can measure CDF
     that well?
         DR. WALLIS:  I have more confidence in some sort of vision
     of what I am aiming at, at this sort of level, than getting lost in all
     the details and saying we don't know how to procedure because we haven't
     had all the meetings.
         DR. KRESS:  In essence, Graham, if you look at the Scientech
     report, one of their concepts, which I like, by the way, was they define
     margins, in terms of the last bullet, in terms of what is the
     conditional CDF.  And that, in essence, if you use that as that
     criteria, that is, in essence, doing what you said.  It is just that you
     measure CDF by looking at surrogates.
         DR. WALLIS:  If you have a vision of something like this, it
     is how you might have the ideal performance indicator, --
         DR. KRESS:  I think that is one of the insights to come out
     of that.
         DR. WALLIS:  -- then you can work out the detail.
         DR. BONACA:  And I think that is the process by which that
     comes down in the diamond, which makes all the sense.  What troubles me
     is that if you really look for perfection and completeness, you never
     get there.  Okay.  Right now, you get to look at a situation right now,
     right now the industry has been using for 10 years INPO performance
     indicators, which is really what NEI is talking about.  And they are
     proposing it, they are monitoring those, performance-based.  Some of
     them in my book are insufficient and I think the diamond evaluation
     presents that in a very articulate way.
         Now, the NRC, it seems to me, is monitoring the industry
     below the beltline of the diamond, well below down there.  Okay.  So
     something is happening out there, which is industry is monitoring up
     here and is not good enough.  NRC is monitoring down here, except in
     some cases like ALARA, where they go up to results.  And there is a
     pretty good workable way in which you can come in between, and I think
     that the Scientech report shows that to be an effective implementable
     way.
         Now, I agree that we will never get completeness or proof,
     this is not research that will come out with perfection.  Okay. 
     However, I think there is such an opportunity for progress there, that I
     am kind of puzzled by the shyness of -- you know, I don't see the
     movement in that direction.
         DR. KRESS:  I agree with you.  I agree.
         MR. ROSSI:  Well, I think the plant performance assessment
     and oversight process is working towards exactly what he has described
     there, because they are looking at the things that can lead to core
     damage, like initiating events and system reliabilities and trying to
     build a framework for monitoring things.  So they are doing -- that is
     another good example of doing this.
         They really have risk-informed thresholds and risk-informed
     things to look at, so that is another example that is pretty far
     developed at this point in time and is getting a lot of input from the
     stakeholders in industry so that it is bringing them closer together.
         DR. KRESS:  Mario, I agree with practically everything you
     said, and in fact if you look at these four attributes, basically you
     could say each one of them with the possible exception of the third one
     applies to our current regulations.  It's just a matter of what level
     you are at in determining your performance indicators.  It is a matter
     of where are you going to put that level.  I think the diamond tree
     process does allow you to organize your thoughts.  It's a good way to
     look at your thinking.
         DR. BONACA:  Absolutely.
         DR. WALLIS:  Those bullets apply to anything.  Those bullets
     apply to academic education, when you measure the student performance.
         DR. KRESS:  That's the problem.  They are not good
     principles to design to.
         DR. BONACA:  That's one of the reasons why I said that
     before.  I feel the sense of urgency because the industry is still
     looking at the top of the diamond there with those limited performance
     indicators.  They are saying I am performing well.  The NRC is looking
     down and they say you are not performing well, and there is lack of
     consistency or a common set of measurements that we can use
     mathematically to help agree on what is good performance and bad
     performance.
         DR. APOSTOLAKIS:  Let me propose this, and tell me why you
     can't do it.  As a prelude, Professor Wallis asked earlier what are
     other industries doing.  It just occurred to me that the fire protection
     community has proposed in some countries like New Zealand, I believe,
     and maybe Sweden, I'm not sure, performance-based regulations, you know,
     for buildings.  The performance measure they use is really equivalent to
     our risk measures.
         They are using individual risk as a performance measure. 
     Now you try to calculate individual risk when you have a fire in your
     high-rise building.  There's a lot of uncertainty, a lot of judgments,
     and yet that is what they use.
         I believe that in our agency using, in our business using
     the top level goals is out of the question, because the agency has
     already stated that there are other things it worries about.  We have a
     good example in the oversight effort where we have the cornerstones,
     okay, so the agency is not only interested in core damage frequency, for
     example, it also doesn't want to see very frequent initiating events, it
     doesn't want to see high unavailabilities of mitigating systems and so
     on.
         MR. ROSSI:  But those things are tied to core damage.  I
     mean they have a relationship and so they are derived from that concept.
         DR. APOSTOLAKIS:  And I totally agree, yes, but what I am
     saying -- I am just arguing why core damage frequency, for example,
     could not be the only performance measure.  Why can't I write then
     2.2174 -- whatever -- for performance-based approaches which says, the
     first three bullets, has the same principles -- you see, what's missing
     here is the expert panel, the integrated decision-making, which was not
     mentioned anywhere in your documents.  That is what I like about 1.174,
     that all these things feed into an integrated decision-making process.
         Do you remember that figure, with the principles feeding
     into the decision-making process?  Where it says delta CDF, delta LERF I
     replace that box by five other boxes that say the performance measures
     will be the frequency of initiating events -- and I will give you a
     delta F -- so if you are above this, your performance is not good.  The
     mitigative system unavailability and I will give you a delta Q, so you
     can't exceed that or you take action.  Emergency preparedness, and I'll
     give you some criteria for that -- so I replace it by these four or
     five, the cornerstones, and then I have the equivalent of 1.174.  That
     states very clearly what I want to do -- and why not?  Why can't I do
     that?  And that satisfies Professor Wallis's concern.
         It is high level, reasonably high level.  It satisfies
     defense-in-depth requirements, because of course we have already set
     the -- that is a statement of defense-in-depth at that level.
         DR. BONACA:  But the larger issue that Tom was talking about
     is not addressed here.
         DR. KRESS:  Neither that nor whether or not you have a
     sufficient number of indicators.
         DR. APOSTOLAKIS:  The margin issue will be in the deltas.
         DR. BONACA:  How do you deal with, assume that you have a
     failure of -- I don't know -- high pressure or decay heat removal
     system?  That is one where Criterion Number 4 says it is unacceptable to
     monitor at that level because, first of all, it is difficult to measure
     the frequency so low, so it would be meaningless, but the other thing is
     once you have a failure --
         DR. APOSTOLAKIS:  A failure of what?
         DR. BONACA:  I don't know.  The example they make I believe
     is RHR.
         DR. APOSTOLAKIS:  No, but the cornerstones are at the lower
     level.  The cornerstones are at the mitigating system level.  The
     example of NEI is at the higher level.
         DR. KRESS:  Not NEI -- not NEI, the Scientech report.
         DR. APOSTOLAKIS:  The Scientech report says you can't talk
     only about the temperature limit, because you have to worry about losing
     component cooling water system.  You have to worry about losing AC
     power.  But each one of these has systems in it and you will monitor the
     unavailabilities of the diesels, you will monitor the unavailabilities
     of the component cooling water system, so the criteria are at the lower
     level than the NEI.
         DR. KRESS:  You have already decided what level of the
     diamond tree you are going to --
         DR. APOSTOLAKIS:  I didn't.
         DR. KRESS:  Without a principle.
         DR. APOSTOLAKIS:  The agency did.
         DR. KRESS:  I know, but that was -- they didn't have the
     principle to guide them.  They just chose things.
         DR. APOSTOLAKIS:  I don't know that you can prove in a
     mathematical way that you have to go with initiating events and so on. 
     This is a value judgment.
         DR. KRESS:  Absolutely, and that is one of the
     characteristics of performance-based regulation.  It doesn't show up
     there.
         DR. APOSTOLAKIS:  But it has to be a value judgment.
         DR. BONACA:  You may have a good use there for an expert
     panel.
         DR. KRESS:  Well, there needs to be a process by which to
     guide that value judgment.
         DR. APOSTOLAKIS:  And the oversight program showed us the
     process.  It's not that we didn't talk -- they said we have reactor
     safety.  We have -- what was the other one? -- radiation protection --
     and I don't remember now.  There were three or four of them.  Do you
     remember the hierarchy they developed?
         DR. KRESS:  I think Mario and I are saying that the diamond
     tree process does give you a structured way to make those value
     judgments.  It tells you what to look at.  Now it doesn't tell you how
     to look at.  It doesn't tell you how to look at them.  It tells you what
     to look at.
         DR. APOSTOLAKIS:  And I think that is what the oversight
     group has done.  They didn't call it a diamond, but they developed a
     tree --
         DR. KRESS:  Sure.
         DR. BONACA:  Sure.
         DR. APOSTOLAKIS:  -- and it is a value thing, so there is no
     proof, and the ACRS did not object to it.
         MR. ROSSI:  It seems to me that in the implementation or in
     the writing and the implementation of the maintenance rule pretty much
     what you described has been done, because what they do in the
     maintenance rule is they first identify the most risk-significant
     components in the plant and then they apply a process where they monitor
     how well those components perform and then when they are not performing
     well enough they go back and look to see whether maintenance is the
     reason, so that is being done there, and I think that that certainly is
     a good approach, and the tie to core damage frequency is that they have
     to go through in the maintenance rule and determine the risk
     significance of the components.
         That is the tie, and that ties it to core damage frequency.
         DR. APOSTOLAKIS:  Right, but it's at such a high level
     though.
         DR. BONACA:  Yes, well the thing I like about the approach
     that we are presenting in the document is that is actually the approach
     to determine whether the performance indicator, which is
     performance-based, is acceptable or not.  If it isn't acceptable, you go
     a step below until you find out that will be acceptable, because it will
     have all these four attributes met.
         DR. APOSTOLAKIS:  I didn't see that, by the way.
         DR. BONACA:  I don't think that you have many cases where
     you have to go so deep, but you may have some, and I think it's just an
     exercise and again, if you are not afraid of completeness, okay, then
     you can go to the exercise.
         DR. KRESS:  The weakness there was you determined its
     acceptability by some criteria.  One, can you measure it at that
     level --
         DR. BONACA:  That's right.
         DR. KRESS:  -- but two is what its conditional core damage
     frequency is, and that requires a PRA.
         DR. BONACA:  That requires a PRA, yes.
         DR. KRESS:  You'll read some places where you have to deal
     with it a different way, and they talked about it.  They recognize that.
         DR. BONACA:  But there is information at the plant-specific
     level to extract that information and make the judgement, and again a
     panel, expert panel, would be critical to do that tailoring.
         DR. APOSTOLAKIS:  That is why you need the equivalent of
     1.174, to set those principles, that these four attributes will be
     declared as satisfied by a panel.
         DR. BONACA:  Yes.
         DR. APOSTOLAKIS:  So you need a principle like that, and I
     think we should separate values from technical arguments.  I mean there
     is no technical basis for saying yes, I want the frequency of initiating
     events to be low.  This is a value judgment.  The agency wants that low
     because of public outcry, potential public outcry and so on.
         Now once you have declared that as your objective, then
     defining the proper performance indicators to make sure all that happens
     is a technical issue.  Okay?
         So first of all, this work was done I guess in parallel to
     the oversight program so that's why we don't see the impact from that,
     right?
         MR. KADAMBI:  There were many things happening at the same
     time and there were many people that were common to both of them but
     what we see over here are the products of discrete efforts --
         DR. APOSTOLAKIS:  Yes.
         MR. KADAMBI:  -- and we have to pull them together.
         DR. APOSTOLAKIS:  I don't blame anybody for that.  I mean
     that is the way things happen, especially if you want to do a lot of
     things in a short period of time, but it seems to me that having a short
     document that very clearly states these principles emphasizes the
     integrated nature of the decision-making process, identifies the issues
     pretty much like what Scientech did, what you should worry about,
     because if you go to 1.174 it doesn't tell me what to do about model
     uncertainty.
         All it tells me is that I have to worry about it.  It
     doesn't tell me what to do about incompleteness, but it tells me, look,
     we may ask you that, but it is a statement in black and white that you
     have to worry about it.
         Now in some instances it is not important.  In others it may
     be, and I think a short document like that will be useful, and the
     second point is we cannot afford to have different values in one
     project, in the oversight project, you know, where they declare the
     cornerstones as being the driving force, and a different set somewhere
     else, so either as an agency we decide that the cornerstones are indeed
     the way to go, or if you guys don't like it for example, then you have a
     debate with the oversight team and say we don't like this or we want
     more there and less here, so we will have an integrated approach at some
     point.
         But I think the statement of principles and what needs to be
     done and what you should worry about would be a useful document to have,
     and it will also show progress in this area, because a lot of the stuff
     that I read in the documents you sent us really tries to go way down to
     the detail and solve the problem, which I think is not wise, similar to
     what 1.174 -- 1.174 did not try to solve the risk-informed regulatory
     problem.
         DR. WALLIS:  I would like to say what is happening here.  I
     mean you'll notice the discussion has come up from the ACRS, so what's
     happened is that the ACRS is thinking about the problem and coming up
     with ideas which look good.  What we hear from you is you haven't got
     much input from industry, you've got to do research.
         You looked at this problem for a year or something.  I would
     expect you to have had all the ideas that we have had in the first few
     weeks or days and to come up with some sort of a much more mature thing.
         We have looked at the problem and this is what needs to be
     done.  These are our ideas and the reason we are doing it is because we
     don't hear it from you.
         MR. FLACK:  If I could make a comment to George's earlier
     comments.  This is John Flack from the Office of Research.  I am also in
     the same branch working on the same problem.
         Getting to the cornerstones, we did do that.  In fact,
     something very similar -- they called it a football and not a diamond
     tree, and they did go through the thought process to develop at what
     level you would start to collect information for the PI.
         When we are talking about 1.174 we have to remember that
     1.174 had a certain purpose, and that was to make a change in the
     licensing basis of a plant, and in there one of the principles was also
     performance monitoring, so it does capture this element, and when we are
     talking about another guide, I am trying to think of where this
     application would be.  What did we not capture in 1.174, for example.
         The intent of what Prasad is presenting is we are intending
     to do is stand back and look at our regulations in a global sense, not a
     specific plant or a change to a specific plant, which was the intent to
     1.174 -- to change something on a specific plant, but to look more
     globally across the regulations to see if there were areas where we
     could be less prescriptive, and that was where this was coming from.
         Now the concept of developing another guidance for
     performance-based, I am still trying to understand where that would go
     with respect to changes to the licensing basis that would not be
     captured today by 1.174.
         DR. APOSTOLAKIS:  The 1.174 does not address these things,
     so that is what I am saying, that you would have -- I don't know if it
     has to be a regulatory guide, but a document.
         MR. FLACK:  Yes, a document of some sort, right.
         DR. APOSTOLAKIS:  But if you want to go this way, here is
     the way to do it at a fairly high level, the same level that 1.174 is
     on.  You will have to have these attributes.  You will have to have an
     integrated decision-making process, and you have to worry about certain
     things.
         For example again, I really like Chapter 2 of the Scientech
     report, where they took -- I think it is Chapter 2 -- the NEI example
     and analyzed it, talked about the cut sets and so on.  Something like
     that would be very useful to have.  It will not change the direction you
     are going.  All I am saying is it will be an intermediate document that
     will solidify, so to speak, all the thinking that has gone into this
     without trying to go all the way down and solve the whole problem, which
     is the way I think you are going now.  You are really trying to find
     performance goals at a fairly low level.
         DR. KRESS:  I think -- to say it another way -- I don't
     think George is saying take anything explicitly out of 1.174, but use it
     as a guidance as to how to structure the approach to this thing.
         DR. APOSTOLAKIS:  Yes.
         DR. KRESS:  1.174 has very little to offer in the sense of
     things you can take right out of it.  It is a process --
         DR. APOSTOLAKIS:  Exactly.
         DR. KRESS:  -- and a way to develop --
         DR. APOSTOLAKIS:  And you have already done 90 percent of
     the thinking.  This is not new to you because you have already thought
     about it.
         So pull all that together, create a document, and then the
     other thing, John, that I am not sure about, is this idea that you are
     trying to do it in a generic way and identify performance indicators
     that would apply to the whole industry.
         The question is why didn't 1.174 try to do that?  Why not
     try to a risk-informed, develop a risk-informed approach that would be
     industry-wide?  It didn't do that.  It said no, the licensee should do
     that.  It is too plant-specific, and then we saw that in the graded --
     and too issue-specific.
         In the GQA -- if you read the GQA Regulatory Guide, you find
     things in there that are not in 1.174.  There is an implication that if
     you do these things, delta CDF and delta LERF will remain acceptable,
     but we never really quantify it, and 1.174 doesn't say anything about
     that, so the question really is is it feasible to do what you are trying
     to do?
         MR. KADAMBI:  Well, let me break that question into two
     parts.  You began with the question, why can you not do this?  And I
     can't answer the question why we cannot do this.  It seems like
     something we could do, but we would have to be in a much better position
     that we are today in terms of what is going on in the cornerstones area
     and everything.
         So, you know, I think what I hear from you is we need to
     pursue what Ernie said we will be doing, which is making this a much
     more integrated effort and learn from that.
         DR. APOSTOLAKIS:  But let me offer you another thought.  I
     think there is an implicit assumption here -- I may be wrong, but I
     think there is an implicit assumption that these four attributes should
     lead to performance criteria that will be equivalent in some sense to
     what we are doing now.
         MR. ROSSI:  Well, to give an equivalent level of safety I
     think is correct.  But what this is intended to do -- I mean basically
     what performance-based approaches do is they get you out of the mode of
     looking at the procedures for doing the maintenance, and they get you at
     looking at how well is the equipment working.
         DR. APOSTOLAKIS:  Right.  I understand.
         MR. ROSSI:  And so you want to do that in other areas.  But
     the idea is you get at least the same level of safety, maybe even an
     improved level of safety, but you get a much more focused effort and
     more flexibility on the part of the licensee.
         DR. APOSTOLAKIS:  Right.  And I would call that a bottom-up
     approach.  Right now we have a certain level of safety by doing certain
     things.  Now, we look at an area, and we say, well, gee, we really don't
     need to do this group of things here because we can set the performance
     criterion a little higher and let the licensee worry about it.
         In a top-down approach, like the oversight program did, you
     ask yourself first, what are my objectives?  We don't ask the objectives
     here, what the objectives are and work down.  We say, if my objective is
     to limit the frequency of initiating events, then what would be the
     performance criteria that would do this for me?  Okay.  So the approach
     is philosophically different.  And that is the kind of thing that,
     again, a principles oriented document should address.  What exactly are
     we trying to do?
         MR. MARKLEY:  I think that is part of what John was talking
     about here.  It sounds to me like they don't have a regulatory decision
     in mind.  I mean if you are looking at Reg. Guide 1.174, it is a change
     to the current licensing basis.  Here it doesn't appear to be a
     particular decision that they have in mind for it.  So it is nice to
     have this information, but what are you going to do with it?
         MR. ROSSI:  Well, our intend would be, again, to look at can
     we change rules so that you focus less on procedures and more on
     performance and give a license.  So it is actually rule changes and
     change to the guidance that we are looking at here.  So that is the
     decision.
         MR. MARKLEY:  1.174 doesn't address changing rules, per se,
     it is difference means of meeting the rules.
         MR. ROSSI:  Right, I know.
         DR. APOSTOLAKIS:  Right.  But a document like that, in this
     context, would address the question I just raised.  What are we trying
     to do?  Are we trying to maintain equivalence with the existing system? 
     Or are we changing our approach and we are going now top-down and we are
     saying the objectives are -- the cornerstones or maybe something else, I
     don't know.  But, finally, we have to settle on that.  We can't have a
     set of cornerstones for the oversight program and another set for
     somebody else.
         MR. ROSSI:  Well, I think in principle that we would focus
     on the cornerstones wherever they can be focused on.  Now, there may be
     some areas like fitness for duty rule and that kind of thing, where you
     can have a performance-based approach, and you can't tie it directly to
     the cornerstones.  And I think that is the areas that we are looking at.
         A lot of this stuff I believe, when you tie it to the
     cornerstones, does come out of the effort to risk-inform the
     regulations.  I mean I think that will come out of that part of it.  And
     so that is why we are trying to expand it in a coherent way to things
     like fitness for duty, maybe more into how you do the quality assurance
     and quality control, and there were some other areas where you can't as
     easily tie it to the cornerstones.
         DR. APOSTOLAKIS:  And I appreciate that.  I mean it is not
     that I think the cornerstones are the answer to everything.  But these
     four attributes, I guess my comments really refer to the last three
     words of the fourth bullet, "immediate safety concern."  Maybe we can
     change those, I mean instead of calling it immediate safety concern, use
     something like objectives or something.
         But right now, this doesn't tell me what would be of safety
     concern.  Scientech assumed that it was core damage, and I don't know
     that is the case.
         DR. WALLIS:  Isn't it very simple?  This is
     performance-based, it is not independent of risk-informed.  They go
     together.
         MR. ROSSI:  They go together, yes.
         DR. WALLIS:  Risk-informed, to me has to mean a measure of
     risk.
         MR. ROSSI:  Yes, they go together, no question.
         DR. WALLIS:  And I understand that is something like CDF. 
     So the only question is, what can you measure that tells you something
     about how they are doing on CDF?  That is the only question.  Why is it
     so complicated?
         DR. APOSTOLAKIS:  If CDF is your objective.
         DR. WALLIS:  Well, that is what risk-informed mean, isn't? 
     If it means something else, then tell me.
         DR. APOSTOLAKIS:  No, but the agency is on record as saying
     that it is not only the core damage that they are interested in.  They
     don't want to see --
         DR. WALLIS:  But instead of facing the main question, we
     have sort of spread it out into all the details.
         DR. APOSTOLAKIS:  No, I agree with you.
         DR. WALLIS:  Let's go to the heart of the matter.
         DR. APOSTOLAKIS:  The main question is, what are the
     objectives of that?
         DR. WALLIS:  But it seems clear to me -- maybe I am wrong,
     but isn't it to implement this CDF as a measure of risk and to figure
     out --
         DR. APOSTOLAKIS:  Well, maybe a level lower than CDF, but it
     is certainly up there.
         DR. BONACA:  But, again, if you look at the example in the
     context of the bullet number 4, you know, the example is when you lose
     RHR in mid-loop operation, okay, and only rely on operator action to
     recover.  Okay.  Now, what they are saying is that if you rely on an
     indicator to measure your performance, and you fail just that time, that
     is not good enough, for a couple of reasons, but the main reason is you
     can't rely purely on these recovery actions as a means of, you know,
     staying away from core damage, and that is the point of margin that we
     are talking about.
         So that is a case where, Graham, they show that just looking
     at performance, okay, is not good enough, because it doesn't give you
     the margin there.
         DR. APOSTOLAKIS:  But, Mario, what you just said I think is
     consistent with what Graham and I have been saying.  They used, as a
     measure of how close to an undesirable situation, CDF.  That is what
     Scientech did.
         DR. BONACA:  Yes.
         DR. APOSTOLAKIS:  And yet that is not mentioned here.
         MR. ROSENTHAL:  This is Jack Rosenthal, I am Branch Chief of
     the Reactor --
         SPEAKER:  REA.
         DR. APOSTOLAKIS:  We know who you are.
         [Laughter.]
         MR. ROSENTHAL:  I am still learning how to say my name.
         DR. APOSTOLAKIS:  We know who you are.
         DR. MILLER:  That is your title today.
         DR. SEALE:  We know who you are, Jack.
         MR. ROSENTHAL:  Clearly -- clearly, we want to learn from
     the oversight process.  We don't want to duplicate the process.  I think
     that if you look at the underlying history of the documents, in the area
     that is risk-informed, the agency is moving forward, taking steps, it
     has plans to risk-inform Part 50, et cetera.  We will learn from that
     exercise.
         My concern is how do we take on these other programmatic
     activities that are not necessarily amenable to delta core damage and
     that the bigger contribution to the agency would not be to duplicate an
     already identified effort, but to take on some of these other roles such
     as fitness for duty, training, quality assurance, commercial dedication,
     procurement processes.  There is a whole, in my view, perhaps burdensome
     infrastructure that could be replaced by more performance-based goals. 
     And that is --
         DR. MILLER:  CDF can't be a measure of that.
         DR. APOSTOLAKIS:  No, but it can not --
         MR. ROSENTHAL:  I am sorry, what was your --
         DR. APOSTOLAKIS:  CDF can not be a measure of that.
         MR. ROSENTHAL:  Right.  And that is why we want to come up
     with other attributes.  And one of the things is, is this the right four
     attributes?  What other attributes should we be looking for?  Do you
     agree with that?
         I think of the discussion that you have been having on do we
     look at the core damage frequency, what we do at the equipment and
     reliability level is very important to us.  My own view is that you
     should do things which are measurable, although not necessarily reported
     to the NRC, nor would you necessarily require that it be reported ever,
     but that it is directly measurable.  And you could do more than count,
     maybe you want to count and divide by something.  But you might not want
     to get into a fancy numerical scheme.
         But that is my own view, and I would like to hear your views
     on -- well, you called it, Mario, the beltline.  About how far down do
     you think we should go?  And get away from the initiating frequency
     example we are all too familiar with, and let's talk about some
     hypothetical training or some other programmatic activity.  At what
     level should we get going?  And that is where the guidance would help
     us.
         DR. APOSTOLAKIS:  And I understand that.  I guess what I am
     saying is these four attributes are not sufficient to define performance
     criteria -- or goals, actually, not criteria -- goals.
         And one question, one issue that they do not address is what
     objectives you have.  What is the immediate safety concern?  Now, in the
     PRA case, I think it is easier if you have a PRA.  You can take the four
     cornerstones.  If you don't have a PRA, then you can still have
     objectives and see how the other thing.
         MR. ROSENTHAL:  Yes.
         DR. APOSTOLAKIS:  What did you say?
         DR. FONTANA:  Excuse me.  Go ahead.
         DR. APOSTOLAKIS:  Did you say "guess"?
         MR. ROSENTHAL:  No, "yes."
         DR. APOSTOLAKIS:  Oh, "yes."  Okay.  Now, so the objectives
     I think is important.
         Second, and I think Jack just put his finger on it,
     calculable.  Do you go with the fire protection community's approach,
     you know, the individual risk?  I mean there is so much judgment going
     into it.  Is that really a valid indicator?  Unavailability is very
     simple, you divide two numbers.  Nobody questions that.  So where do you
     draw the line?  That is a real issue here.  Because when we say "or
     calculable," actually, there is much more to it, calculable and perhaps
     believable, or with some confidence.
         DR. FONTANA:  Well, it doesn't --
         DR. APOSTOLAKIS:  So, I still think a document like 1.174
     should put all this together, and it will be very useful.
         DR. FONTANA:  Well, I think those attributes, I think are
     very good attributes.  I think they are subsidiary to a higher level
     objective, like George says.
         The thing is there is much -- a slip between the cup and the
     lip, in that these are all very good, but when you start to apply them
     in a real case, then you see the problems that are involved in trying to
     make them work.
         Now, the question is, what do you have in mind in using
     these for test cases?
         MR. KADAMBI:  All I can offer at this point is it is part of
     our plan to sort of look into this sort of question.  Where might the
     pilot projects come from?
         But, Ernie, you were going to say something.
         MR. ROSSI:  Well, yeah, what I was going to say is that we
     have had a couple of meetings now, and this could even be considered as
     a third meeting, where we have the question, and it is one of the
     questions that is in your handout there -- what specific rules might we
     use to apply this technique to?  Because what we have is we have a
     fairly -- a very robust effort underway to apply PRA to the regulatory
     process and risk-inform the regulations.  And at lot of that already
     includes performance-based approaches like in the oversight process.
         So what we would like to do is find examples that the
     industry would support to apply this process to outside of the other
     things that are going on, but are coherent with it.  And as yet, we
     haven't heard any specifics at this point in time.
         DR. FONTANA:  But the test case I think would have to
     include some of the things that Jack raised.
         MR. ROSSI:  Right.
         DR. FONTANA:  Things that are not measurable.  By
     measurable, I mean it may not come out with a number, but it is
     something that could indicate a qualitative assessment of like -- what
     will one more screw-up do to me?
         DR. KRESS:  Yeah, but, Mario, let me comment on that. 
     Because if you can apply a PRA, then I think things will fall out and we
     can figure out where to go.  And, technically, you can get there.
         DR. FONTANA:  That's fine.
         DR. KRESS:  Those things you can't apply a PRA as analogous
     to our problem with the process versus product in the INC.  With a PRA
     you can determine the product, the change in risk, the change in
     something.  You don't -- and I think Jack is saying, well, we don't want
     to go to a new rule network where we correlate these things like QA and
     management to a change in the soft score even.  That is going a little
     bit too far.  And I think what you have to do in those is you have to
     focus on the process again, at least I would think, when you get down to
     that level.
         MR. ROSENTHAL:  If I can offer, we can go back and look at
     Part 20 and say, has that been a -- what is the story, has that been a
     success?  And has it been implementable, measurable, et cetera?  We can
     look at Appendix J.  The story there will be a mixed story, but we can
     look at our experience with that.
         DR. MILLER:  When you say look at, does that mean you are
     going see if these attributes are applicable?
         MR. ROSENTHAL:  How many people -- right.  Can you measure
     it?  I mean, you know, is it working?  Do we have the intended
     containment integrity that we wanted when we moved from a prescriptive
     to a more performance-based rule change?  Are we still maintaining
     containment integrity?  Is it an implementable rule?  How much of the
     industry has adopted the rule?  So we can learn from the past.
         We can learn from the maintenance rule.  Okay.  And we can
     learn from the performances -- the new oversight process, and we intend
     to do that.  What we would like -- and that is in our plan, to go back
     and learn from those past experiences.
         We would like to also identify, and we have it listed as
     pilot programs, some conceptual areas where we could -- it would be
     pilot exercises where we would identify some other rule maybe, some
     training, although there doesn't seem to be enthusiasm.  I would just go
     to fitness for duty as an example, where maybe you say that the amount
     of drug testing at a plant should be proportional to the problems that
     they find at the plant.  You know, could come up with some scheme.  I am
     only using this as an example to push us away from the reactor one.
         And then having nominated some new rule to be changed from
     performance to -- from prescriptive to performance-based, then use that
     as a test case, a pilot case.  And we would love to hear suggestions for
     rules or Reg. Guides that would be candidates for this pilot.
         DR. MILLER:  It seems like you ran through a pretty broad
     gambit right there.
         MR. BARTON:  There are some good ones right there.
         MR. ROSENTHAL:  Those are the ones I know about.  So what
     about the ones I don't know about.
         DR. SEALE:  Jack, you made a reference a while ago to the
     idea that there were some performance indicators that might not be
     reported to the NRC.
         MR. ROSENTHAL:  No, what I said was that -- I'm sorry.  That
     -- well, one, we would like to hear your views on the first bullet.  Is
     that an appropriate attribute that would be measurable?  And then I just
     offered up that on a personal basis, in my own view, I don't even think
     they have to be things that are reported, but rather things that at
     least the licensee could measure.
         DR. SEALE:  Very good.  That is the point I was driving at.
         DR. FONTANA:  Well, they ought to be measurable where you
     can measure them, but there are a lot of other things that you can't.
         DR. SEALE:  Could I make my point?
         DR. FONTANA:  Oh, I am sorry.  I thought --
         DR. SEALE:  The aspirations of the industry I think are that
     as we go to a more risk-informed and performance-based regulatory
     process, that there is also going to be a disentanglement of the NRC
     inspection process and a lot of things with the day-to-day operation of
     the plant.  At the same time there is a recognition within the industry,
     and I think this is a fair statement, that there are things that they
     need to keep track of that are not on the level of the NRC's radar
     screen.
         It strikes me that whatever process they come up with there
     is somehow a rich area to be mined for the kind of information that you
     are talking about.  What is the integral effect, or the cumulative
     effect of these non-radar screen detectable, measurable, trackable and
     verifiable and comparable across the industry between plants and so
     forth?  What is the sum of those experiences that somehow might take a
     form that would be useful as an overall performance indicator for you?
         Now, obviously, you can't answer that question till you know
     what these things are.  But we also know, for example, that there are
     people in INPO who have said we are interested in other endpoints for
     assessing satisfactory performance besides core damage frequency.  If it
     is that late, forget it.  I mean we are already dead in the water. 
     Certainly, that plant is dead in the water.  We need to find things that
     are sensitive at a level that is low enough to tell us that there are
     things that need to be done before we get to the core damage frequency.
         One of the things that bothers me is if you get too high on
     this list, your feeling the problem through awfully thick mittens.  You
     are not feeling the problem at a low enough level.
         Now, I agree at the same time you can't be all nerve endings
     down there.  That is the problem we have had in the past.
         DR. MILLER:  The key question is which level you want to be.
         DR. SEALE:  Well, there is somewhere in between.
         DR. WALLIS:  Well, the key thing, what do you first?  You
     can't say we won't core damage frequency because we are worried about
     drug testing.  If you know how to do CDF, you do it.  You don't want
     until you can --
         DR. SEALE:  We need to ask ourselves what are these other
     things.
         DR. WALLIS:  But you do, you bear it in mind.  But you do
     the high priority thing you can see how to do first, and get on with it.
         DR. APOSTOLAKIS:  I think we should clarify something.  Core
     damage frequency itself, if it reaches a certain level, is not too late,
     it is a frequency of an event.  It is the core damage itself you don't
     want.  Let's not confuse the two.
         DR. SEALE:  Yeah, but there are a lot of things that have to
     with unavailability.
         DR. APOSTOLAKIS:  Sure.
         DR. SEALE:  And the adequacy of the maintenance and all that
     sort of thing, which are a lot easier and more sensitively detected at a
     lower level of screening, if you will.
         DR. APOSTOLAKIS:  I agree.  I agree that there are issues
     that, you know, you can not see the core damage frequency.
         DR. SEALE:  And what are those things that the industry
     people are looking at?  They keep coming to us.  I think you need to go
     look at them, too.
         DR. BONACA:  Let me just say about that, because that is
     really a point, right now, if I built a hypothetical diamond there and I
     try to fill all these boxes, I would say that almost every box is being
     tracked somewhere inside a plant, literally.  There is information being
     gathered.  I mean there are hundreds of indicators out there.  What is
     terrible for the plants is that they don't know oftentimes what it
     means.  They know specifically that if I measure this, I get a feedback
     on that particular piece of equipment.
         But the confusion they are having, they have no guidance
     from anywhere on what counts and what doesn't count.
         DR. APOSTOLAKIS:  No, as to what the objectives are.
         DR. BONACA:  What the objectives are.
         DR. SEALE:  Yes.
         DR. BONACA:  So what happens is that then the corrective
     action program may be extremely important, but if the corrective action
     program is overwhelmed by the search for comments missing on some
     documents, then you just don't have focus on what counts.  And I think
     the objective here on our part should be the one of developing some
     guidance on how do you go back from the very important, and how far you
     monitor.  And at times you may have to get down to the corrective action
     program.
         DR. SEALE:  What are the diagnostic requirements of the
     corrective action program?
         DR. BONACA:  But the point I want to make, again, the
     resources are being expended today.  The question is, where do you draw
     this line that is not just a straight beltline out there, but is
     somewhere, you know, just jagged to reflect insights from PRA?  Really. 
     You know, how do you draw the line?
         DR. KRESS:  I am not being a very good junkyard dog, I am
     letting the meeting get away from us.  I regret it, but I do have to
     declare a recess at this time because we have some other obligations we
     have to meet.  You know, I hate to break in right in the middle of the
     discussion, but I am going to declare a recess for 15 minutes, until
     10:30, at this time, at which time we will come back and I will try to
     ride herd on this bunch a little better.
         [Recess.]
         DR. KRESS:  We need to get started again.  George may be
     delayed a little while, but he will join us as soon as possible.  In the
     meantime, I think we can continue with your presentation.  Let's see, I
     guess we are on what, the second slide?  Okay.
         MR. KADAMBI:  We have been through a few.  Thank you, Mr.
     Chairman.  I would like to resume the staff's presentation.
         Where we left off was on the discussion of these attributes
     which are taken directly out of the white paper, and, again, I want to
     reiterate that the white paper has a lot of stuff in it.  It tries to
     pull together many of the ideas that will help us with risk-informed,
     performance-based approaches to regulation.
         Having gone through the white paper, I tried to capture on
     one slide what seemed to me to be rather important messages coming out
     of it.  The first one is that the deterministic approaches should be
     changed but should be done incrementally.  This isn't something the
     Commission is looking for to do wholesale.
         DR. KRESS:  Let me ask you about that, do you think that is
     possible?  You know, it seemed like if you followed the advice in the
     Scientech report that that is going to be very difficult to do that way. 
     You always have to take the regulations as a whole.
         MR. KADAMBI:  It is not for me to argue with the Commission.
         If the Commission tells you do that, I guess you have to do
     it is what you are saying, yeah.
         DR. WALLIS:  It reminds me of the approach to Kosovo, you
     set out your objectives ahead of time.
         MR. KADAMBI:  I want less to argue with the Commission.
         DR. WALLIS:  Then you say that, yes, but you should say we
     -- our hope is that in ten years we will have 90 percent
     performance-based or something, some sort of a goal out there.  If you
     are saying you are tentative and incremental, then that is a pretty poor
     way to start a major program.
         MR. KADAMBI:  I think really at this stage doing it
     incrementally makes more sense to me than to undertake something,
     because although there was a suggestion made that maybe we are trying to
     solve all problems at the same time, really, what we are trying to
     develop, I believe is some sense of confidence that there is in fact a
     success path that we can chart, you know, before we really bite off too
     much more than we can chew.
         DR. WALLIS:  I think we all agree with that.  If just put
     incremental upfront instead of the long range objective upfront, it
     gives the wrong message.
         DR. KRESS:  Well, he is just extracting the message he got
     out of the white paper I think.
         MR. KADAMBI:  Yes.  I am just trying to reflect --
         DR. WALLIS:  I guess I have the benefit of independent
     thought.
         DR. SEALE:  I think there is another value to Graham's
     suggestion, too, and that is it would be nice to take the Commissioners'
     temperature on this issue.  Do they expect 90 percent in 10 years, or 5
     percent in five years, or what?
         DR. KRESS:  I think you have to see if this thing works
     first, before you make a wholesale change in the regulations.  I kind of
     agree with this, you try it where you can.  You know, it is too risky to
     change the whole regulatory structure all at once.
         DR. SEALE:  I agree with that, Tom, but on the other hand, I
     think we also will have to agree that not very long ago we reached a
     point when some of us were asked whether or not we thought the NRC staff
     was prepared to move to risk-informed regulation, and I think the answer
     they got was they are as prepared as they are ever going to be until you
     tell them that they will.
         DR. KRESS:  I see.
         DR. SEALE:  So the will, a clear-cut statement of the will
     of the Commission is an important attribute in motivating that process
     as well, and it probably would be a good idea to remind the
     Commissioners of that occasionally.  And, also, to pass on to the
     collective Commission, as it evolves with different members and so
     forth, a clear statement of what the aspirations of the existing
     Commission is that launched the Commission -- the staff on this
     undertaking.  End of speech.
         DR. FONTANA:  When you look at the top to bottom, the
     objectives to the bottom all the way down through the operations, down
     to the final preparation of procedures, ultimately, they are all
     prescriptive, aren't they?  Those procedures that are usually written by
     the licensee are prescriptive.  They almost have to be, right down to
     the very bottom line.  And the question we are asking here is, where
     along this spectrum the NRC does its regulation?  Am I helping any? 
     More confusing?
         MR. KADAMBI:  Well, if I can --
         DR. FONTANA:  In other words, the prescription is always
     there at some level if you go deep enough, you know.
         MR. KADAMBI:  I think there are many prescriptive elements
     associated with procedures and things like that.  But I think today what
     we think of as procedures and the processes that happen at a nuclear
     power plant, at least now we are thinking about it in something more
     than a one-dimensional or three-dimensional form.  We are actually
     thinking about hierarchies where, you know, one level may be more
     important than another.  So, to me, that is something that is new and we
     need to see, you know, how we can apply that more effectively.
         So you can be prescriptive at one level and it may be quite
     appropriate, but another level it may not be as appropriate.  And having
     a way to select the right level of prescription for a given level is
     part of the question.
         DR. FONTANA:  Part of the problem is that the NRC, the
     regulator, does not run the plant.  The licensee runs the plant.  And
     the question is, at what level does the regulator regulate and monitor
     things so that any mistakes that the licensee may make are not -- are at
     least one step removed to an unacceptable event.
         MR. KADAMBI:  Maybe that leads to the second bullet.  I
     believe what that is telling us is that the traditional approach, you
     know, has been successful, that it has insured no undue risk to public
     health and safety in the use of nuclear materials.
         DR. KRESS:  And you know that how?
         MR. KADAMBI:  Well, I am taking the message from the white
     paper.  Again, I --
         DR. WALLIS:  What is your performance measure of no undue
     risk?  I mean this whole thing is about performance indicators, so how
     do you measure this no undue risk?
         MR. KADAMBI:  Well, I can read you the specific words out of
     the white paper.
         DR. WALLIS:  No, that is by decree, that is by assertion,
     that is not by measure.
         MR. ROSSI:  Well, let me just say a couple of things about
     the word undue risk.  There is another term that is frequently used,
     which is adequate protection.  And I think we all understand that that
     term is not as precisely defined as perhaps we would like, and we may
     never be able to define it precisely, but it is my understanding that
     the Office of Research is going to try to work on that to get a better
     definition of it.  So I am not sure that we can do much in terms of
     trying to discuss it today.
         DR. WALLIS:  But one of your specifications for any new
     system or framework presumably is that whatever the measure of no undue
     risk to public health is, it is maintained.
         MR. ROSSI:  Yes.
         DR. WALLIS:  I don't see that here.  I guess maybe it is
     somewhere here.
         MR. ROSSI:  I would say that that would be --
         DR. WALLIS:  That would seem to be the upfront --
         MR. ROSSI:  Probably a working definition of what we might
     want to do is we might want to make things less prescriptive and more
     performance-based without lowering the level of safety.  Now, I don't
     know whether we can measure that or not.  We probably could.
         DR. WALLIS:  I think you need a measure of safety if you are
     going to do that.
         MR. ROSSI:  We could measure a delta I think.
         DR. KRESS:  I think the answer to the question we asked is
     it is an after the fact determination by the use of the IPEs and PRAs,
     which are incomplete.  You know, we don't have a full answer to that
     question, but the indications are that it is true.  And, you know, I
     think you can proceed on the basis that that is a true statement, even
     though we really don't have a full -- what you need is a complete PRA
     for every plant, and compare it with, say, the safety goals for each
     plant and then you could say, yeah, that is true.  And I think the
     indications are that it is true, but I don't think we really know that.
         DR. WALLIS:  Maybe the problem is in the word risk.  I mean
     it is ensure no undue risk.  That means the public was never at risk. 
     What you probably may mean is that history has shown that there has been
     no undue harm and, actually, in terms of real measure, not probabilistic
     type risk, but in terms of real measure.  Is that what you mean?
         MR. KADAMBI:  I am not going to attempt to explain what the
     Commission meant when using those words.
         DR. WALLIS:  It is a bit like the nuclear war thing.  I mean
     we haven't had a nuclear war so everything has been fine.  But at times
     we have been at pretty untolerable risk maybe.  So it is really a
     question of what your measure is.
         DR. KRESS:  I think they actually meant risk.
         DR. WALLIS:  They meant risk rather than --
         DR. KRESS:  And I actually think they meant risk in terms of
     the safety goals.
         DR. WALLIS:  So you didn't mean actual -- that people have
     not been harmed.  You meant --
         DR. KRESS:  Well, I am interpreting, too.  But I think they
     meant risk, and risk in terms of the safety goals..
         DR. FONTANA:  Well, you know, going back to the history of
     the early days, I am not sure they did.  I think they didn't want that
     --
         DR. KRESS:  There might have been a time when they might
     have, but I think at the current level of regulations and oversight, you
     and probably say this with some assurance as a true statement, based on
     what you see in the IPEs.
         DR. WALLIS:  You mean that Three Mile Island was not an
     event that posed risk to public safety?
         DR. KRESS:  Oh, no, I didn't mean that.
         DR. WALLIS:  Well, that is tradition.  I mean tradition goes
     back to '79.
         DR. KRESS:  I said currently, at the present time, based on
     the current regulations and the current oversight.  And in my mind, the
     only real measure of that we have is the IPEs.
         MR. ROSSI:  Well, you have the accident subsequent precursor
     program I think, too.
         DR. KRESS:  Well, that is another way to look at it, too.
         MR. ROSSI:  And the trending, yeah.
         DR. KRESS:  Yes.
         MR. ROSSI:  Because they tell you how close we came on
     individual events and how many of those there were.
         DR. KRESS:  So I should not have left that out, because I
     think that is a very important part of it.
         DR. FONTANA:  Well, that is going back kind of after the
     fact, after the plants were designed pretty much, and modified as such.
         Originally, I am almost sure that they basically used the
     defense-in-depth approach more from the point of view of the multiple
     barriers, then it kind of built up as it went along.  The question, I
     think you are really getting to the third bullet, the question isn't so
     much how performance-based approaches affect defense-in-depth, I think
     much more of a question is, how much of a margin do you have left?  That
     is hard to measure and hard to determine.  I think you can almost always
     show you have got some defense-in-depth somewhere.
         MR. KADAMBI:  Well, if I can stay with the second bullet,
     really, the implication of the first sentence is, why do anything at
     all?  And that is I think addressed in the second bullet.  That is the
     efficiency, consistency and coherence of the regulatory framework can be
     improved using risk-informed and/or performance-based approaches.
         DR. WALLIS:  That is a statement of faith?  That is a
     statement of faith.
         DR. KRESS:  Well, I think you will find that statement in a
     couple of ACRS letters also.
         DR. WALLIS:  That is a statement of faith --
         DR. KRESS:  Of course.
         DR. WALLIS:  -- until somewhere has a calculation or an
     experience that shows that, yes, indeed, we implemented something and
     the efficiency increased.  It is a statement of faith.
         DR. KRESS:  It is a statement of faith, I agree with you. 
     And I wouldn't even disagree that the word performance-based adds to the
     coherence, it may even detract from it.  But I certainly agree with the
     statement in terms of risk-based, risk-informed.
         DR. WALLIS:  You see, the thing is this seems to be
     backwards, cart before the horse.  Someone has decreed we should do
     something and then it said it is going to have these benefits.  The way
     that I have been taught to problem solve is you first identify the
     problem.  The agency is inefficient.  Let's have a measure of
     inefficiency.  Let's figure out to fix it.
         You know, this -- your exercise seems to be in the other
     direction.  Let's decree we will do something, and then decree it is
     going to fix efficiency, and let's keep working, and then eventually,
     after 20 years, we might have a proof that efficiency has gone up.
         DR. FONTANA:  Well, you know, deficiency being defined as
     the desired output as compared to the amount of effort that it takes to
     do that.  The question is, is the existing safety of reactors adequate? 
     And I think we would say it would.
         So what you are getting at in terms of efficiency is
     reducing the cost on both the part of regulator and the part of the
     licensee, I think.
         MR. ROSENTHAL:  There is two contemporaneous examples that
     have been discussed with the ACRS in other formats.  One is risk-based
     ISI and the other is risk-informed -- risk-informed ISI, risk-informed
     IST.  In both cases we are saying, let's take ISI, we are saying we know
     that we can do a more efficient job and increase safety also by
     inspecting the right piping, dropping the inspections on the stuff that
     is silly, and you will have greater efficiency for the regulated
     industry.
         DR. KRESS:  There is certainly evidence of that.
         MR. ROSENTHAL:  That is a concrete example that I think has
     been fleshed out.  The other contemporaneous one, I would argue, is
     risk-informed IST, where we can see where we would have potential safety
     improvements and improved efficiency.  Now, those are both
     risk-informed.  I don't have a good performance-based example for you.
         DR. KRESS:  Well, I think there is certainly evidence out
     there to support that bullet.  It is not wholly faith-based.  And not
     only that, I think we are being told that the Commission has decided
     that these are good objectives and that this is a way to achieve them,
     you know.  So, we kind of act on that some way anyway.
         DR. FONTANA:  Well, the thing about efficiency, and I take
     this to mean lower cost, if safety is the same -- of course, there could
     be situations where applying your effort on more important things could
     actually increase safety, but let's assume that you are trying to keep
     it the same and reduce the cost, the -- I had completed the thought that
     I was getting to, I will get back to you.  I found a flaw in my train of
     thought.
         MR. KADAMBI:  It is not yet clear how performance-based
     approaches may affect defense-in-depth.  That I think is an important
     question we will have to address and keep at the back of our minds as we
     -- or even in the front of our minds as we go through some of these
     things.  But I feel I can state, based on the work that we have done so
     far, and what I have seen, that indicates that defense-in-depth can in
     fact be strengthened if it is done right.
         DR. KRESS:  I think you need a pretty firm definition of
     what you mean by defense-in-depth in order to address it properly when
     you go about changing the regulations, and I am not sure we have one. 
     There is one in the 1.174 which comes close to being a good definition,
     but I think it is still incomplete.
         The definition of defense-in-depth has to some way involve
     the uncertainties in your measures of performance, and a
     defense-in-depth need should somehow be related to how uncertain your
     performance measures are.  So somehow you have to connect the two.  I
     think when you come down to acceptance criteria on your performance
     indicators, and when you come down to this level at which you don't want
     to have margin to unacceptable consequences, I think defense-in-depth
     enters into the uncertainty you have in both of those determinations. 
     And somehow you have to make that connection.
         MR. KADAMBI:  The other thought that I had was that these,
     the attributes that are for performance-based regulatory approaches that
     are given in the white paper really provide the basic elements of a
     screening test of some sort, which, you know, we are tasked to develop
     as part of our plans.  So I think, you know, the white paper is again
     serving a useful purpose, I believe.  I would like to know if people
     disagree with that.
         DR. KRESS:  Well, I think the sense of this group is that,
     as I have heard it, these are good descriptive attributes that can
     almost be applied to anything.  I don't see how you can go from there to
     a real screen.  Something different, I think it has sort of a screen. 
     Certainly, they are necessary attributes, but not sufficient.  We will
     need something more to go with it.
         MR. KADAMBI:  Okay.
         DR. FONTANA:  What I was trying to get at before, is there
     really an additional cost here?  Because while you are implementing
     these new approaches, you still have to maintain the old system.  So,
     does the regulator and the licensee really have to do both in parallel
     at the same time, and, therefore, during the transitional period, the
     thing is costlier than it would have been before?  How long is this
     going to last?
         MR. KADAMBI:  I would say those are the kinds of lessons we
     would learn from the maintenance rules.
         DR. FONTANA:  What have you learned so far?
         MR. KADAMBI:  I haven't looked enough.  Ernie may know more
     about it.
         MR. ROSSI:  Well, we were told last week by NRR in the
     public meeting we had that there was a fairly sizable cost upfront in
     the maintenance rule.  And that could be the case in anything we do in
     this area.  However, I think that what we would do is proceed with
     specific examples and pilots and see what the costs are.
         I think if the costs turn out to be more to do it than we
     gain back, and it is only a question of being less prescriptive and not
     a question of safety, then we will proceed differently than if there was
     a big reduction in cost.  But that is something we have to take into
     account, no question about it.
         DR. WALLIS:  Could I go back to what Ernie said I think a
     long time ago, that there had been very little input from industry on
     this?  And it seems this is being done by decree from the Commission,
     but the people who really stand to benefit, surely, are the utilities
     and the industry.  Why aren't they knocking on the door and saying get
     on with it?
         MR. ROSSI:  Well, I think they are knocking on the door and
     saying get on with risk-informing Part 50, and get on with some other
     things.  And the situation may be that this is at this point in time
     just lower on their priority.  That is what I suspect, because there are
     lots of things going on.
         And, again, I have said this before, but I will say it
     again, there are things being done on performance-based approaches,
     because there is the reactor oversight and performance assessment
     process that is being revamped, as well as the inspection program.  So
     those are major areas where performance-based approaches are being used,
     and risk-informed approaches are being used, and they have a significant
     effect on the industry.
         DR. SEALE:  And the industry has expressed significant --
         MR. ROSSI:  Interest in those.
         DR. SEALE:  Interest and displeasure with the past situation
     where they have been micromanaged they feel.
         MR. ROSSI:  Right.
         DR. WALLIS:  Yes, but as soon as you suggest another scheme,
     there seems to be a great deal of reluctance to go ahead and let you
     implement it.
         MR. ROSSI:  Well, this isn't intended to be another scheme,
     this is intended to a furtherance of what is already going on, in my
     opinion.  I mean it has to be made coherent and carried out in a
     coherent way with the risk-informed approaches.
         DR. KRESS:  And I think NEI has waded into this, so they are
     interested in it.
         DR. SEALE:  It is like cod liver oil, it tastes bad when you
     take it, but the long-term effect is appropriate.
         DR. KRESS:  What is cod liver oil?
         DR. SEALE:  That is back in the olden days.
         DR. KRESS:  I know what it is.
         MR. KADAMBI:  Okay.  Let's see, the next bullet addresses
     the third attribute in the list of attributes in the white paper for
     performance-based approaches.  The specific words the Commission has
     used are, "Licensees have flexibility to determine how to meet the
     established performance criteria in ways that will encourage and reward
     improved outcomes."  You know, I guess I am just not sure exactly how to
     use the word, or what meaning to attach to the word "reward."  This is
     going to be something that we will have to figure out.
         DR. KRESS:  Less oversight is the reward.
         MR. KADAMBI:  The next bullet addresses attribute 4 and, in
     my mind, looking at that, it seems to be -- the best definition that I
     have seen for getting at this concept of margin, margin to unacceptable
     consequences, which can, you know, be margin in physical parameters,
     temperature pressure, et cetera, but, you know, in my mind it could also
     be time.  You know, it could be if there is the right margin built in
     there.  It could be time for operator action.
         DR. KRESS:  I think you need a coherent principle on that
     margin.  I think that is going to be the key element in
     performance-based regulation.  It is going to be the hardest thing to
     determine, how much margin you have to what.  So you have to define what
     an unacceptable consequence is first.  That is George's saying, in your
     principles, you ought to put what your objectives are.  And your
     objectives are to do this, this and this.  It might very well be CDF,
     but it also could be other things.
         And then, once you establish those, you have to determine
     what are acceptable levels of those that you never want to exceed.  And
     then the margin is going to be hard to determine between there and where
     you put your performance indicator.  And you will need some sort of
     guiding principle there, and that is the place where you really need it. 
     And that margin, as I said before, has to be related to your uncertainty
     in determining the relationship between this performance indicator and
     this objective, what you are trying to achieve, and it has to be related
     to that somewhere.  And that is, to me, the trickiest, hardest technical
     issue you need to face right now.  And you need a coherent principle on
     that.  That is where I would really put some thought into it.
         I like, for example, the Scientech example of conditional
     CDF.  I mean that is a reasonable one.  It might be conditional
     something else, if you have other objectives.  But you have to say what
     is the uncertainty in that measure of conditional CDF, and your margin
     has to be related to that uncertainty.
         DR. WALLIS:  Well, the biggest uncertainty is probably the
     human performance.  That is the one that is toughest to estimate, even
     without uncertainty.
         DR. KRESS:  Oh, I agree.  But you have to figure out how to
     deal with it some more.  And that, in a sense, is what I would be
     calling part of your defense-in-depth.
         MR. KADAMBI:  Yes, I understand what you are saying, and I
     also agree, it is one of the more interesting issues coming out of this.
         DR. WALLIS:  I think most defense-in-depth is actually put
     in because humans can screw up.  If you think about why you have air
     bags and safety belts and so on, although a car is a very steerable,
     controllable device, if used properly.  It is because of human error
     that you put these defense-in-depth items in there.
         MR. KADAMBI:  Okay.  The last one is the white paper clearly
     says that the licensees must continue to comply with regulatory
     requirements.  But, of course, the requirements may change as a result
     of performance-based approaches.
         DR. KRESS:  That almost doesn't need to be said.  I mean you
     are always going to have that as a principle of regulation.
         DR. WALLIS:  That is a big assumption.
         DR. KRESS:  Well, you know, that is why you inspect and
     audit and do things.
         MR. KADAMBI:  Let's see, I think at this point I can get
     down to earth a little bit and talk about, you know, what we have been
     doing in the Office of Research.  I mentioned that SECY paper 98-218,
     you know, I guess the SRM on that came out January or so of '97, and we
     got started thinking about the subject.  And in June of 1997, the Office
     of Research initiated the research project that resulted in the NUREG
     CR-5392.
         It was actually done with a very small amount of money which
     would have evaporated if we had not come up with some idea to use it.
         The intent of the research project was really rather simple,
     you know, to do a literature search and use what comes up in some kind
     of case studies.  And we focused on the Commission's direction on
     performance-based objectives not amenable to PRA.  Bob Youngblood was
     the principle investigator on that project.  And the work was actually
     completed in April of 1998, and this I will point out is well before
     some of the work that is now going on on the revisions to the reactor
     regulatory oversight process.
         But the final report on it, the printed version, didn't get
     issues till January 1999, mostly because of -- there is a new
     publication process that tries to use newer technology and we sort of --
     this was the first report that went through this new process and it got
     caught at many points along the way.
         DR. KRESS:  Do you mean just printing it and putting the
     document together?
         DR. MILLER:  It was in Office '97 or Office '98, or
     something crazy like that.
         MR. KADAMBI:  Well, we had to put it on CD-ROM and, you
     know, lots of things that were associated with that.
         DR. MILLER:  New technology slowed down the work.
         MR. KADAMBI:  Since we had it in electronic form, we also
     put it up on the Internet on a conference site, and this way we are
     hoping to get involvement of the technical community in furthering this
     discussion, and hopefully get stakeholders involved who don't normally
     come to stakeholder meetings and things like that.  We have got about
     four comments up till now, not very many.  But, you know, it is
     interesting.
         DR. WALLIS:  Well, let me hope that waiting for other
     people's ideas doesn't prevent you from having your own.
         DR. SEALE:  There is another thing here.
         DR. WALLIS:  No, seriously.  I think if you lead the way and
     say, look, these are the things we are thinking about, and ask for
     reactions from stakeholders, that is a good way to proceed, rather than
     waiting.  I am sure you are not just waiting.
         MR. KADAMBI:  We are not waiting.  I mean this is something
     that is out there and I monitor the site, you know, once a day and see
     if somebody has weighed in with --
         DR. WALLIS:  George is gone, but George got all sort of
     involved in thinking about ways to think about this.  I would think that
     you must have done this sort of thing, you could almost give an
     exposition of the way you have thought and analyzed the situation.  That
     is what I would like to hear, too, really, rather than waiting for some
     stakeholder to -- because they always think about the difficulties and
     the hazards.
         MR. KADAMBI:  Yeah, maybe after I get the plan done, you
     know, then we can --
         DR. SEALE:  Yes.  The other thing that would be interesting
     is if there are any comments specifically related to DSI-13.  Now, I
     know when you did them, you went out and got public comment on the
     direction setting issues.  But now here is an application and it is in
     the context of a specific situation where it would be interesting to see
     if there is any significant difference in the response to the idea in
     DSI-13 in detail, as opposed to in principle.
         MR. KADAMBI:  Yes, I think that is something that we could
     do and it is really included in our plans because the Commission very
     specifically mentioned DSI-13 stakeholders.
         DR. SEALE:  That's right.  And I think they need to get
     feedback on that particular issue.
         MR. KADAMBI:  Yes.  We will keep that in mind and we did
     keep that in mind when a stakeholder meeting was held on September 1st,
     1998 in Chicago on the role of industry and, you know, performance-based
     approaches was one of the agenda items, but we didn't hear almost
     anything at all on that.  But this is basically the background on the
     work that went into the NUREG CR-5392, and at this point, I guess I
     would like to ask Bob Youngblood to go into that if there are no --
         DR. KRESS:  Yes.  Let me intercede here just a minute.  I
     hate to do real damage to our agenda.  George Apostolakis, in
     particular, wanted to be here during this presentation of Dr.
     Youngblood's.  He, unfortunately, is detained in a meeting with one of
     the Commissioners right now.
         DR. SEALE:  They are just back.
         DR. KRESS:  Wonderful.  That takes care of my problem.  I
     was going to suggest a real --
         MR. BARTON:  But we all got fired, so we can't comment on
     your --
         DR. KRESS:  Is that good or bad?
         DR. APOSTOLAKIS:  Everything is fine.
         DR. KRESS:  George, we have just now arrived at the point
     where we are going to hear Dr. Youngblood's presentation on the
     Scientech study.
         DR. APOSTOLAKIS:  Okay.
         DR. KRESS:  So we will proceed with it then.
         MR. KADAMBI:  Thank you, Mr. Chairman.  I guess I will
     request Bob to take over now.
         DR. APOSTOLAKIS:  Do you plan to go over each one of these
     viewgraphs, Bob?  It is going to take forever.
         MR. YOUNGBLOOD:  That is up to you.  I have brought more
     than I strictly need to show, and I will try to go as fast as I can.
         Could I ask what time we are shooting for a break?
         DR. KRESS:  Well, I think two members of the subcommittee,
     or three at least, have to get out of here by 1:00 to 1:30.  So I don't
     know.  I can certainly abbreviate the lunch period, cut into it, make it
     very short.
         DR. APOSTOLAKIS:  Twenty minutes perhaps.
         DR. KRESS:  Yes, even 20 minutes.
         DR. APOSTOLAKIS:  Is that okay with you?
         MR. YOUNGBLOOD:  You are suggesting I shoot for 20 minutes?
         DR. KRESS:  Oh, no, no, no.
         DR. APOSTOLAKIS:  Can you?
         MR. YOUNGBLOOD:  I can shoot.
         DR. KRESS:  I am suggesting that we shoot for about 1:00 to
     1:15 as an ending time.
         DR. MILLER:  With a 20 minute lunch.
         DR. KRESS:  With a 20 minute lunch break.  And so if you can
     do the subtraction.
         MR. YOUNGBLOOD:  Okay.  Yes.  It is not clear to me that all
     time is mine necessarily.
         MR. KADAMBI:  No, I do need some time to focus on the SRM
     for the SECY paper and our ideas that we now have for the plan that we
     have to submit, develop and submit.
         DR. WALLIS:  What I would like to go away from this meeting
     with is the hope, or the impression that someone has some ideas which
     are workable and is likely to contribute something towards a solution. 
     If you could show us why what you have done is workable and contributes
     to a solution of the problems, then that would be great.  I don't want
     to get involved in the details so that message gets lost.
         DR. KRESS:  How many members will disappear around 1:00?
         DR. APOSTOLAKIS:  Four, they are losing quorum.
         DR. KRESS:  We are losing.  Well, we only have to have two
     people for quorum.
         DR. APOSTOLAKIS:  But it is not fun anymore.
         DR. KRESS:  Well, we lose a lot of --
         DR. APOSTOLAKIS:  Why don't we let Bob start and we tell him
     which viewgraphs he can skip.
         MR. YOUNGBLOOD:  Okay.  I am Bob Youngblood, one of the
     authors of this report.  I would like to mention a couple of other
     names, in particular, because I will be talking about things that they
     in particular brought to this report.  Bob Schmidt did the piece of
     modeling that has been mentioned several times in connection with the
     heat removal at shutdown, and Niall Hunt is the guy that introduced the
     rest of us to the diamond tree idea.
         And in case I forget to say that later, I would like to
     point out that when he did that, he was working on plant availability
     issues as a plant guy.  He was at Baltimore Gas & Electric at the time,
     and they saw this as a tool to promote plant availability.  And as you
     may have seen in the report, you can go down into great detail of
     operational stuff.  People either can or do track a lot of things, and
     it was that background that he brought into this issue.
         If I were issuing this report today, I think I would choose
     a word different from "oversight."  Oversight to some people means
     inspection and enforcement and, as several of you have pointed out, that
     is not what this has been about.  This is about changing requirements.
         DR. APOSTOLAKIS:  Is that the same as changing the licensing
     basis?
         MR. YOUNGBLOOD:  I think it would have -- in my mind, yes, I
     don't know, you could chop logic on that, but, yes.
         DR. APOSTOLAKIS:  So I would have to apply 1.174?
         MR. YOUNGBLOOD:  Well, no, you would go -- you would change
     requirements, and then it would entail a change in licensing basis.  But
     you wouldn't be leaving the rules alone and changing the licensing basis
     the way 1.174 does.  You might just be wiping out whole categories of
     requirements.
         DR. KRESS:  The licensing basis would probably still remain
     the FSAR, and I don't think that would likely change.  It might.
         MR. YOUNGBLOOD:  I think you could imagine tech spec
     changing as a result of this..
         DR. KRESS:  Tech spec changes, definitely, yes.
         MR. YOUNGBLOOD:  So this report is really about two topics. 
     One, the much discussed attribute, the fourth attribute where your
     monitoring criterion shouldn't become -- shouldn't be set so high that,
     by the time you trip it, you have got a real problem.  And I will talk a
     little bit about the example that we did to shed light on that.
         The other area was how to do better at considering areas in
     which PRA does not appear to do a good job, and that is sort of a clumsy
     formulation of it.  But the general idea is that -- the way I would
     explain it to myself is that, when I read reports discussing significant
     events, and as a PRA guy, ask myself whether work that I had done or
     even seen really captured what was going on in that event, then maybe
     the answer is no or a partial no or something.  So there is a lot going
     on in event reports or even some information notices and that kind of
     thing that you don't really see well done in PRA, and that was issue
     two, and that is the issue that drove in part -- one of the two reasons
     I think that the diamond tree came out strongly.
         DR. APOSTOLAKIS:  It seems to me you have to be careful with
     the language here, Bob.
         MR. YOUNGBLOOD:  Yeah, I probably should.
         DR. APOSTOLAKIS:  In your report, page 3, there is a
     sentence, "It is well known that PRAs do not typically model everything
     that is important to safety."  Do you really believe that?
         MR. YOUNGBLOOD:  Oh, yeah.  But that is actually -- that
     sentence is actually meant to address a different issue.  There is a lot
     of components that -- that sentence, when written, was meant to refer to
     a lot of components that are not -- that don't have basic events in the
     fault tree, is what that sentence meant.
         DR. APOSTOLAKIS:  But why?  I mean if they are important to
     safety, why aren't they there?  I don't understand it.  Because there
     was a meeting, in fact, the workshop that you had on April whatever --
     14, where Mr. Lochbaum I think said that PRA is fiction.
         MR. YOUNGBLOOD:  Well, a lot of things were said that day.
         DR. APOSTOLAKIS:  Are you supporting that statement?
         MR. YOUNGBLOOD:  No.  No, I think it is a different thing. 
     Well, just compare how many things are there on a Q list and how many
     things are there in a PRA?  The number is different.  That is all I
     really meant there.
         DR. MILLER:  It seems like Lochbaum is certainly rather
     dramatic in his statement, but it seems like -- I agree with George that
     his statement and this one are kind of consistent.  Lochbaum is saying
     it is fiction because we believe that everything in a PRA is
     safety-related, and you are saying the same thing.
         DR. APOSTOLAKIS:  And then Mr. Riccio, public citizen,
     stated that PRAs do not reflect reality.  And here we have a NUREG that
     sort of confirms that.
         MR. YOUNGBLOOD:  Well, the sentence, I guess I will take
     hits on the language of the sentence.  But what I meant by it I think is
     correct.  There are instrumentation systems that plant people care about
     that you don't see modeled.  There may be a basic event in a PRA that
     says probably people will respond to this and do the right thing, and
     that event may tacitly take credit for kinds of instrumentation that you
     don't then have a little tree reflecting the possibility that all these
     instruments will fail.
         You don't call out every piping run in a typical PRA.  You
     can go back and do that if you want to, but the thing just explodes.
         DR. APOSTOLAKIS:  No, but it seems to me that when you do
     walkdowns and you do space-dependent, common cause failure, potential
     common cause failure analyses, you worry about these things.
         MR. YOUNGBLOOD:  You worry about them, and then, having
     worried about them, maybe you decide that you don't need to put them in,
     either because they don't contribute significantly to the train you are
     modeling, and, in addition, they don't take down multiple trains when
     they go.  You don't see cables in most PRAs, but people care about them.
         DR. APOSTOLAKIS:  Fire.
         MR. YOUNGBLOOD:  Well, sure.
         DR. APOSTOLAKIS:  When you do fire PRA, you do.
         MR. YOUNGBLOOD:  Yes.  Not cable by cable, not tray by tray
     necessarily.
         DR. APOSTOLAKIS:  But I am not convinced that the cable by
     cable, each little cable is important to safety.
         MR. YOUNGBLOOD:  No.  No.
         DR. APOSTOLAKIS:  See, that is the thing, there is a
     systematic approach for identifying what is important to safety.
         MR. YOUNGBLOOD:  Yes.
         DR. MILLER:  You raised an example of instrumentation.  What
     instrumentation that is not modeled is important to safety?  Just as an
     example.
         MR. YOUNGBLOOD:  Well, just as an example that appears in
     the report, there is level instrumentation that played a role in the
     event analyzed that -- well, of course, you could say shutdown PRA is
     pretty general to begin with, in some cases.  I don't know whether that
     instrumentation would have appeared, and that strikes me as something,
     if you care about level instrumentation in that operational mode, you
     would highlight it as important.  I am not sure that things that are
     like that are always in PRAs.
         DR. KRESS:  It may appear implicitly in how much credit you
     give for operator action or something.
         DR. APOSTOLAKIS:  Right.
         MR. YOUNGBLOOD:  Yes.  To go back to important to safety for
     a moment, I don't -- this is probably the wrong day to get off onto
     that, but it seems to me that PRAs actually don't systematically
     identify what is important to safety, they are about identifying
     vulnerabilities.
         DR. APOSTOLAKIS:  That is a pretty strong statement. 
     Typically, they don't do that.  I mean then what do they do?  And what
     is safety?  I mean, you know, remember, people are conditioned by the
     regulations of 40 years to worry about a lot of things, and that is why
     we are moving to risk-informed regulation, to make sure we worry about
     what is important.
         DR. MILLER:  If this statement has some validity, that would
     mean that --
         DR. APOSTOLAKIS:  It is way too strong.
         DR. MILLER:  If I go to your level example, if my level
     instrument fails, indeed is a safety-related instrument, it causes my
     delta CDF to change by so much, but it is not modeled, I won't see it.
         MR. FLACK:  Yeah, I think that -- this is John Flack.  I
     think we agree.
         DR. MILLER:  Am I going too far with this statement?
         MR. FLACK:  Yeah, I mean the staff generally agrees with
     that, is why we are moving along the risk-informed regulatory process. 
     When I see it is not complete, I think of security, those kinds of
     things that are not explicitly modeled, because, at this point, that is
     a limitation of the PRA.
         DR. APOSTOLAKIS:  That is a difference statement.
         MR. FLACK:  This is different, and that is the way I think
     we should take that statement at this point.
         MR. YOUNGBLOOD:  I, in fact, didn't necessarily mean to be
     -- I think that PRA does what it does, but there is a lot of things you
     care about that it doesn't explicitly have, is all I meant.
         DR. APOSTOLAKIS:  But those things you care about are not
     necessarily important to safety.  This is way too strong.  It is well
     known -- not to me -- that PRAs do not typically model everything that
     is important to safety.  Well, I take exception to that.  Anyway.
         DR. KRESS:  Everything is a pretty encompassing word,
     George.  He just leaves out one thing and that statement is true.
         DR. APOSTOLAKIS:  No, I don't believe so.  I don't believe
     so.  I think you worry about a lot of things that are not necessarily
     important to safety, because the regulations tell you you should worry
     about them.  But to say that the PRAs typically do not model them, I
     would like to see specific examples to be convinced.
         DR. MILLER:  Is this meant to apply to non-plant related
     issues, I mean issues like security and so forth?
         DR. APOSTOLAKIS:  Well, if it is incomplete, it is
     incomplete.  Yeah, but then there is a statement upfront that this thing
     has not been modeled.
         DR. MILLER:  So, in other words, this statement may be
     valid, but all those things that aren't modeled are stated pretty clear
     they are not modeled.
         DR. APOSTOLAKIS:  Yes.  You know what you have not modeled. 
     As John said, you know, security issues and so on, you have not modeled.
         MR. YOUNGBLOOD:  Well, when I put it in --
         DR. APOSTOLAKIS:  Yeah, let's go on.
         MR. YOUNGBLOOD:  When I put it in, I was thinking of a level
     of detail issue, and I am not sure it is on today's agenda.  It may be
     worth another discussion.
         DR. APOSTOLAKIS:  My point is that if Mr. Lochbaum has said
     this thing before, that the reason why you cannot have risk-informed
     regulation is PRAs, you know, he hasn't put it as strongly as in the
     last workshop, that they are fiction, but he said, you know, they are
     not realistic.
         MR. BARTON:  Well, he says fiction because of its
     incompleteness.  That was his basis for saying it was fiction.  And I
     would agree that a lot of PRAs are not, you know, quote, complete. 
     Everything is not modeled.
         DR. APOSTOLAKIS:  And that is why it is risk-informed
     regulation.
         MR. FLACK:  That's right.  That's exactly right.
         DR. APOSTOLAKIS:  But there is a big difference that and
     saying that, typically, they do not model everything that is important
     to safety.
         DR. WALLIS:  George, I think we have to move along.
         DR. APOSTOLAKIS:  That is what I think, too.
         MR. YOUNGBLOOD:  Okay.  Still on this slide, we took as
     input to this report that -- essentially, that risk-informing would be
     done.  In other words, this report was supposed to be about how to go to
     performance-based and not about how to do risk-informed.  So we didn't
     argue, really, whether it should be risk-informed.  We didn't really
     argue whether performance-based was a good thing.  And we assumed that
     there would be what I called here policy inputs, namely, somebody would
     decide, is CDF the thing we worry about?  You know, what do we mean by
     defense-in-depth and how do we want to see it reflected?  That sort of
     thing.  That is one big category of input to the process that we talked
     about.
         And another big category of input is, how sure do you want
     to be that you are getting satisfaction of those?  So those two kinds of
     inputs, how much safety and what do you mean by it, and how sure do you
     want to be that you are getting it.  Those are sort of dials on the box.
         DR. KRESS:  Those are typically policy issues.
         MR. YOUNGBLOOD:  Yeah.
         Finally, I would say at some point that the process is
     intended to apply more broadly than just to reactors although today I
     think we will naturally gravitate toward reactors.
         What I am going to try to do in this presentation is talk a
     little bit about the two issues mentioned on the preceding slide, go
     through steps to develop an alternative set of regulations in a way that
     tries to address those issues, and then throw out some ideas.
         This one, Number 4, has been discussed several times here
     today implicitly.  The word "performance" has meant different things to
     different people.  I have begun to try to use the word "outcome
     oriented" to describe something is more performance-based if it is more
     pegged towards outcomes.  The issue of whether your performance
     criterion has been set at too high a level I think is natural to discuss
     on this slide and on the left I am showing here ASP.  That stands for
     the Accident Sequence Precursor Program.
         It seems to me that many events that come out of that
     program has having seemed significant are events that sort of pop up at
     the system level.  A component is not typically an accident sequence
     precursor but complete loss of a system perhaps is and so earlier when
     you were talking about tracking things at the system level, if ASP
     events are typically system failures and if system failures seem like
     events with not enough margin left in them, then maybe train would be a
     better idea, and I must say I don't plan to get into this level of
     detail in the presentation but I came away from this agreeing or
     believing as other people have believed before in performance indicator
     programs that train level indicators would make a lot of sense for many
     of these things.
         DR. APOSTOLAKIS:  If I look at the second column here,
     examples of performance measures, they are all really PRA measures,
     aren't they?  It is all frequencies, and I thought your emphasis was on
     non-PRA performance measures.
         Can you have a non-PRA performance indicator that uses
     frequencies?
         MR. YOUNGBLOOD:  Maybe I don't tend to think of non-PRA
     examples.  I guess I was trying to think of numbers.
         DR. APOSTOLAKIS:  So this is an overall approach.  It does
     not emphasize not amenable to PRA issues that we need to develop
     performance measures for?
         DR. KRESS:  I think he gets to that.
         DR. APOSTOLAKIS:  But here we don't.
         MR. YOUNGBLOOD:  Yes, you certainly don't see it on this
     slide, and --
         DR. APOSTOLAKIS:  I mean the NEI example again in their
     appendix is not PRA-based.  They don't use any frequencies.  They just
     say maintain the temperature below 158 degrees.
         MR. YOUNGBLOOD:  Well, there may be an element of how often
     did you trip that --
         DR. APOSTOLAKIS:  But they don't say that.
         MR. YOUNGBLOOD:  Yes.
         DR. APOSTOLAKIS:  They say that every single time you do it,
     the temperature has to be, you know, below this level because then you
     have sufficient margin, so this particular slide does not address that. 
     This is really PRA-driven.
         DR. KRESS:  Agreed -- except below the line there, there are
     things --
         DR. APOSTOLAKIS:  Frequency of loss of function?
         DR. KRESS:  Well, that is the only one --
         DR. APOSTOLAKIS:  -- rates --
         DR. KRESS:  Maintenance effectiveness, maintenance rule --
     nothing under institutional factors.  Those clearly are not PRA.
         DR. WALLIS:  Would you describe what this is supposed to
     show?  The question is what level is appropriate so I am looking for
     which levels in this picture are appropriate.
         MR. YOUNGBLOOD:  Well, the process -- I don't believe in a
     generic answer to that question.
         DR. WALLIS:  How does the figure answer the question?  That
     is what I am trying to get at.
         MR. YOUNGBLOOD:  The figure really poses the question.  It
     is where on here -- we will return to this figure later --
         DR. WALLIS:  Horizontally or vertically or what?
         MR. YOUNGBLOOD:  A higher level -- do you want to monitor,
     for example, at the system level or at the function level or at the
     train level or down here?
         DR. WALLIS:  Oh, it's vertical segregation you are talking
     about here.
         MR. YOUNGBLOOD:  Yes.
         DR. MILLER:  Before we are done, we are going to have a
     criteria or at least proposed criteria on where to choose the
     appropriate level?
         DR. KRESS:  The process, I think.
         MR. YOUNGBLOOD:  Yes, the process, and it would depend on if
     you had a CDF objective.  Well, you'll see -- I hope you will see.
         DR. WALLIS:  I don't see how you could possibly put human
     actions down below.  Humans are the most likely to make errors.  The
     functions are more likely to be performed or the devices than by the
     people.
         MR. YOUNGBLOOD:  There are actually human actions that
     belong at that level, because they are like components.
         DR. WALLIS:  Absolutely.
         MR. YOUNGBLOOD:  This thing is meant to be sort of
     implementation of programs, so we don't -- if this work goes on and goes
     on in this vein and goes on using, trying to use a hierarchy like this,
     we will get better at labelling those tiers, but that tier called Human
     Action here is really about things like performing maintenance.
         DR. WALLIS:  Putting a barrier across the middle, it kind of
     implies that anything below that line is less important?
         MR. YOUNGBLOOD:  Well, actually there are figures.  This
     figure is derived from figures that appear later, where that barrier has
     some other significance as it would be better for the plant if we didn't
     try to monitor below the level of that barrier, but I just forgot to
     eliminate it from this.
         DR. WALLIS:  That's Mario's Wasteland, is that --
         MR. YOUNGBLOOD:  Yes, exactly.
         DR. WALLIS:  So human actions don't exist except below the
     waist?
         MR. YOUNGBLOOD:  Belt, yes.
         DR. KRESS:  Let's keep it clean.
         MR. YOUNGBLOOD:  So in the example, NEI had proposed an
     example where basically an outcome would be keeping temperature below
     some number, and it seemed natural to model that to see what it would
     mean if you did try to monitor that way.
         DR. KRESS:  Actually, I thought that was very illustrative.
         DR. APOSTOLAKIS:  But again, I took the NEI example and the
     Scientech analysis and again if you look at the previous figure, the
     Scientech analysis says that the NEI approach or at least the example
     they gave deals with more or less normal operation, that they want to
     keep the temperature below 158 degrees and they did not consider the
     possibility of having a rare occurrence like loss of component cooling
     water, system or station blackout and so on.
         But in a performance-based system wouldn't you also have
     some performance criteria on these systems?  So maybe, you know, they
     should have continued the example to consider these contributors and say
     now if I have station blackout, I have to lose the diesels, and I have
     lost the grid of course.  Now I will have information on the grid.  I
     can't do much about it, but for the diesels I can.
         MR. YOUNGBLOOD:  Yes.
         DR. APOSTOLAKIS:  So I will have performance criteria, so I
     will have my precursor.
         MR. YOUNGBLOOD:  That's right.
         DR. APOSTOLAKIS:  What they are doing is perhaps incomplete
     in the sense that they did not finish the example, but it is not
     inconsistent with what you did.
         MR. YOUNGBLOOD:  I think if we had applied our process to
     that model of shutdown we would have come out with the kind of thing
     that you are talking about.  What I carried away from the example is
     that pegging the criterion in temperature space is not a good idea, but
     putting performance requirements at perhaps the train level on RHR and
     on the key supports is the kind of thing that I would expect to come out
     of our process.
         That is I think below the level that example wanted to see
     them go.
         DR. APOSTOLAKIS:  Again it depends on how you phrase it,
     because they can come back and say, well, if we really wanted to
     implement this, you know, we would realize that it is impossible to keep
     the temperature below 158 degrees under all conditions so we will have
     to consider now the conditions where this might now happen, and that
     would have led them to the analysis you have done.
         MR. YOUNGBLOOD:  Fine.
         DR. KRESS:  But the question is what does NRC now go in and
     look at to assure themselves that the safety case is maintained?
         DR. APOSTOLAKIS:  Yes.
         DR. KRESS:  If they are only going to look at this
     temperature, I don't think they are doing their job.
         DR. APOSTOLAKIS:  I am not sure they said only.  That is the
     thing anyway.
         MR. YOUNGBLOOD:  Setting aside what NEI really meant, I
     thought that they were trying to reach for the purest outcome based
     possible thing, where the outcome really is keeping the temperature low.
         DR. APOSTOLAKIS:  They also were making the point, I think,
     that was the main point of that appendix, that a performance criterion
     can be defined in the deterministic space.  It doesn't have to be
     frequency driven.  I think that was the main message there, that by
     monitoring the temperature you have a performance criterion that is, you
     know, monitoring a physical variable rather than frequencies and
     unavailabilities.
         Anyway, I don't disagree with what you have done.  I am just
     saying that I don't think that example was intended to be a complete
     demonstration of how you would do it, and what you have done is very
     reasonable.
         But again, though, Bob, everything that you are doing here
     seems to be PRA-driven, isn't it?  You are talking about conditional
     probability of core damage given heat-up.  You give this example here. 
     If I don't want to use a PRA, what would I do?  I would just count D, P,
     F, G and say, you know, I have so many boxes therefore I am in good
     shape?
         MR. YOUNGBLOOD:  Well, this example is PRA-driven, but I
     think if I can -- well, in reactor space where we are talking about
     safety functions, it is really hard for me not to think in terms of PRA,
     but if we were talking about some other kind of radiological facility
     where the descriptions of scenarios that we were worried about would not
     be embedded in what you would call a typical PRA, then --
         DR. APOSTOLAKIS:  How would you analyze fitness for duty
     using that?
         DR. KRESS:  Well, that is the whole point, I think.
         DR. APOSTOLAKIS:  What is the whole point.
         DR. KRESS:  Well, you have to think in terms of PRA in terms
     of your high level objectives because they are the things that PRAs do,
     but the point is you get to a point where the PRA is not useful to you
     anymore.  Then what do you do?  And this is what they are leading up to.
         DR. BONACA:  One thing there is that NEI gives you a
     scenario with some deterministic criteria.  We are all puzzled by that. 
     Is it adequate?
         DR. KRESS:  Where do the criteria come from?  What does it
     have to do with safety?
         DR. BONACA:  But since the criteria you making is core
     damage, the only tool you have really to analyze that is PRA, and that
     is really what they have done.
         DR. APOSTOLAKIS:  You are changing the scope, Mario.  The
     Commission's --
         DR. BONACA:  No, wait, wait, wait, wait.  What it is doing
     is performing an analysis using PRA to determine whether or not the
     cut-off point to monitor is adequate.
         DR. KRESS:  You just can't arbitrarily pull those things out
     of the air.  You have to have some reason for having that.
         DR. BONACA:  And you can't use -- they have to use PRA
     criteria to see if core damage, if your margin to core damage here is an
     adequate measure, and really, I'll be frank with you, I don't
     understand, I do not like the NEI approach.  I did not understand as
     well the weakness of it until I saw a PRA based --
         DR. KRESS:  It's really pointing out the weakness and what
     you will find out now is your thinking will lead you down to things that
     are important to core damage but you don't know how to deal with them
     with a PRA, and then they are going to ask the question well, what do we
     do about these?  I think that is going to be the important question and
     they have some ideas on it but it's not complete.
         They haven't given you a recipe yet.
         DR. BONACA:  I would add that PRA will be fundamental in my
     book in drawing the line through the diamond.
         DR. KRESS:  Yes, you use the PRA wherever you can in this
     process.  You'll find out you can't use it for all things.
         DR. SEALE:  It doesn't answer all the questions.
         DR. KRESS:  Yes.
         DR. APOSTOLAKIS:  I don't disagree with what you said, but
     first of all, there is an assumption here that what is going to happen
     is an initiating event, so if I want, if I am monitoring -- again, my
     cornerstone is to make sure that the mitigating systems have an
     unavailability that is acceptable, so a mitigating system now is Box D,
     okay?
         It is Box D.  Now the margin I have is all the other boxes . 
     In other words if that system is unavailable in order to go to core
     damage I have to have an initiating event occurring.  I have to have E,
     F, G occurring and all that stuff, so it is not just a one-way street,
     right?
         I can take an accident sequence, monitor a few things, and
     then take one thing out and everything else is a conditional probability
     of core damage.
         DR. KRESS:  I don't get your point.
         DR. APOSTOLAKIS:  That is it not just an initiating event. 
     I mean it is not just such a nice sequence all the time.
         MR. YOUNGBLOOD:  Oh -- no, it's not.
         DR. APOSTOLAKIS:  Okay, fine.  I have no problem with it.
         MR. YOUNGBLOOD:  Actually this slide is a good place to
     re-introduce the idea of the precursor event.  I think what you were
     just describing basically is accident sequence precursor analysis, but
     it came to seem to me to be natural to think about, to always be
     thinking about the precursor idea, that in some sense the conditions
     that we are monitoring are not precursor in the sense of the program,
     which had a threshold and if you were worse than that, then you were a
     precursor, but all of this is really about monitoring precursor
     conditions in some sense, and as you say, it doesn't always go like --
     sorry?
         DR. KRESS:  That is an excellent way to look at it.
         MR. YOUNGBLOOD:  And as you say, it doesn't always go this
     way and that is why we would be maybe monitoring train level indicators
     out here and so forth.
         DR. WALLIS:  The whole subject is performance-based, so does
     this figure help me to understand what you do in a performance-based
     world?
         MR. YOUNGBLOOD:  This figure doesn't help you much.  It is
     really keyed to -- you know, this axis up here corresponds loosely to
     the vertical axis in the last figure that we were talking about, and it
     is really keyed to that example where the NEI criterion corresponded
     essentially to this and the point of this slide, which is to justify --
     which is really about the numbers on the following slide is that
     sometimes the thing that took you to this point also affected the
     mitigating systems.
         DR. WALLIS:  But really everything here has something to do
     with performance.
         MR. YOUNGBLOOD:  Yes.
         DR. WALLIS:  If F goes down, that is not negligible
     performance indicator.
         MR. YOUNGBLOOD:  Right.
         DR. WALLIS:  -- selection is still there --
         MR. YOUNGBLOOD:  Yes.  Yes, it is, and in fact that is a
     comment I meant to remind myself to make.
         DR. WALLIS:  And humans could intervene at any time
     presumably to screw up one of these by taking some inappropriate
     actions.
         MR. YOUNGBLOOD:  Right.  That's right, and the process that
     I will get to eventually would lead you to be interested in all of those
     things and not just the things to the left.
         While I'm still on this slide, different scenarios,
     different cut sets, might have boxes E and F on the left.
         DR. WALLIS:  Right.
         MR. YOUNGBLOOD:  That's a potential confusion introduced by
     the slide, but ultimately if you had a criterion where some boxes say --
     if Box G is always on the right no matter in all the scenarios and where
     you've drawn the line, then it sort of follows to me from a figure like
     this that there will be things that you won't even think about doing in
     a performance-based way because there are some things that are just
     always going to be to the right of any reasonable line that you draw,
     and so a point I meant to make here is that I can't imagine for -- where
     you have a severe consequence that you're trying to prevent, I cannot
     imagine a purely performance-based approach.
         DR. WALLIS:  Well, the margin is -- the margin must have
     some units of measurement presumably.
         MR. YOUNGBLOOD:  Well, down here it was probability.
         DR. WALLIS:  Oh, it's probability?
         MR. YOUNGBLOOD:  That's right.  Down here it's probability,
     and up there it was sort of loosely probability but correlated with
     narrative things happening in the scenario.
         DR. WALLIS:  So it's not really margin.  There's always a
     finite probability of core damage.  It's how much you've lost in CDF or
     something?
         DR. BONACA:  I think it's probability and uncertainty,
     however.
         MR. YOUNGBLOOD:  Yes.
         DR. BONACA:  Well, because that was a consideration there,
     right?  I mean, you had no other boxes that have to do with human
     recovery.
         MR. YOUNGBLOOD:  Yes, that's true.
         Let's see if I can get quickly through this.  This is
     showing -- the column labeled here initiating event frequency is really
     the frequency of getting the kind of heatup that corresponds to that
     criterion, and that's based on a little piece of logic that had boxes A,
     B, C in it basically.  So it's a core damage model with some attention
     paid to actually quantifying an intermediate step in the sequence. 
     Otherwise it's a normal model, but we made it put out both this
     frequency as a synthesized quantity and core damage and then probability
     of core damage given that as synthesized quantities.
         Actually all of these numbers are -- all of these
     conditional probabilities are high by the standards of the precursor
     program, I think.  And in that sense it's, you know, all of them sort of
     trip that flag.  But within this group of all large numbers, some of
     them are still a lot bigger than others, and that's just intended to
     make the point that the character of the event, what actually mattered
     is significant and not just whether you got to that temperature.
         So to summarize those points, a calculation was done to
     evaluate, just show what it might mean to be using heatup as an
     outcome-oriented performance.  We did the things in these bullets.  We
     saw a big variation in conditional probability of core damage, and I
     think it would be -- this is formally I think the way you address that
     fourth attribute.
         I mean, maybe it's a big -- you wouldn't go to that level of
     detail all the time, but I think that's a thought process, you know,
     what does your criterion mean and are there important areas of that
     criterion where you don't really have the margin that you would like.
         Okay.  The other kind of question that --
         DR. KRESS:  The margin that you would like, you view that as
     a policy issue --
         MR. YOUNGBLOOD:  Yes.
         DR. KRESS:  That the Commission or somebody would just have
     to set based on their comfort level.
         MR. YOUNGBLOOD:  I'm not sure that they would have to
     articulate it as margin, but how sure do you want to be that if we
     follow this scheme we won't drift into a higher --
         DR. KRESS:  When you say how sure, you're incorporating the
     concept of uncertainties in the determination --
         MR. YOUNGBLOOD:  Well, both are you getting enough data and
     are you getting it in time, and I really -- how sure do you want to be
     has all kinds of things mixed in.  Let me -- I should have just given
     you a simple yes, actually.
         DR. KRESS:  Okay.
         MR. YOUNGBLOOD:  Okay.  Other kinds of issues.  I believe I
     mentioned earlier that Niall had been working on the diamond tree in
     connection with plant-unavailability issues.  Basically the top half of
     the thing is a goal tree, and the goal tree is pretty well known, and
     then the bottom half is less hardware-oriented things that affect the
     goal tree.  So the key word there is "hierarchy."  If you think clearly
     about the levels of that hierarchy, actually just a list of those levels
     has come to seem significant to me.
         DR. APOSTOLAKIS:  I don't understand the title.  What does
     the diamond tree have to do with PRA?
         MR. YOUNGBLOOD:  Uh --
         DR. APOSTOLAKIS:  Drop the title and it's okay.
         MR. YOUNGBLOOD:  Okay.
         DR. APOSTOLAKIS:  They're entirely different things.  The
     diamond tree tells me, you know, what influences what.
         MR. YOUNGBLOOD:  Well, the title, half of it, just the
     goal-tree part, some people would say that the top half of it is like
     restating your PRA model and success space.
         DR. APOSTOLAKIS:  But the PRA doesn't have any goals.  PRA
     does an assessment of what you have.  Here you're talking about values
     and goals and objectives.  I mean, there are two different things.
         MR. YOUNGBLOOD:  Well, I guess even without talking about it
     as a goal, though, you choose to model certain states in the PRA. 
     They're your top events of your --
         DR. APOSTOLAKIS:  Sure.
         MR. YOUNGBLOOD:  Fault tree, they're the end states of your
     event tree.  You don't maybe talk about them as goals, but they are
     things that you address.
         DR. APOSTOLAKIS:  So if you look at the flow of a PRA
     initiating events, plant damage states, and so on and so on, you can say
     that's the top part of the diamond tree.  I think the title is
     misleading.  I mean, the diamond tree does certain things.  It's
     goal-oriented.  And I don't know that the PRA does not address
     performance issues as well as this does.
         DR. KRESS:  I don't have that much trouble with the title
     because I think I view this as somewhat of an influence value.
         DR. APOSTOLAKIS:  Well, the PRA is not intended to do this,
     Tom.  So to say that it doesn't do it well, I mean, yes.
         DR. KRESS:  Well, you didn't let me finish my statement. 
     You look at all the things that influence achieving your top-level
     objective here, and you notice when you do that that there are things
     that have a possible strong influence on achieving this top-level
     objective that are not well addressed by PRA, and that's these things
     down here at the bottom.
         DR. WALLIS:  But isn't PRA much the same thing?  I mean the
     PRA is a tree as well, it's just not called a diamond tree.  It's a PRA
     tree.  It has the same sort of idea, what influence --
         DR. KRESS:  There are some things in common, that's for
     sure.
         DR. APOSTOLAKIS:  Let's understand what addressing means. 
     So you think that just putting down that programs affect activities you
     have addressed the issue?  Well, I can do that every day.
         DR. KRESS:  Oh, no, no, this is a way to say oh, look at
     these things -- I perceive that they affect each other.  Now I've got to
     do something about them.  That's addressing it.  You're just identifying
     things that are not well addressed.
         DR. APOSTOLAKIS:  Right.  And what I'm saying is that the
     diamond tree does not address them either.
         DR. KRESS:  No, it just identifies them.
         DR. APOSTOLAKIS:  Well, PRA has identified them for years. 
     It says, you know, we don't model organizational factors.
         DR. KRESS:  Yes, but you identify them as, sort of in an ad
     hoc way you say we know organizational factor are important, but we
     don't deal with them.  This thing sort of puts it down on paper and says
     yes, they're important, and they affect these things and --
         DR. MILLER:  It gives you a structure --
         DR. KRESS:  It gives you a little structure, more structure
     to this thinking.
         DR. MILLER:  A structure approach identifying those issues
     which are not --
         MR. YOUNGBLOOD:  I think that I'm getting the message at
     several levels that there may be some indication here that I was trying
     to slam or criticize PRA, and that certainly wasn't the intention.  If
     we could return to this point in a few slides, I think it will be very
     natural for me to clarify what's meant and maybe for you to discuss that
     too.
         DR. APOSTOLAKIS:  Maybe you can say methods for identifying
     performance issues that are not well addressed by typical PRAs.  That's
     different.  Because I don't see any performance issues that are not well
     addressed by typical PRAs here.  Because you're not addressing them
     either.  You're just saying they're important.  And any PRA analyst will
     tell they're important too.
         MR. BARTON:  Moving right along.
         MR. YOUNGBLOOD:  Let's return to that.  This is just a
     picture of a diamond tree, really, it's one I could find that looks like
     a diamond.  As far as I know, the guy that thought this up was Niall
     Hunt, but it's actually -- he doesn't bother to write things down in
     papers, so it's not an easy story, but he, Mohammed Modarres, and Marvin
     Roush were working on a tool to improve plant availability, of which
     safety is a piece.
         DR. MILLER:  This is an outcome of the goal-tree work?
         MR. YOUNGBLOOD:  Yes.
         DR. APOSTOLAKIS:  Of what?
         DR. MILLER:  Modarres and Hunt did what they called
     goal-tree work to analyze systems.  This is an outgrowth of it.
         DR. WALLIS:  There's nothing in here about regulation?
         MR. YOUNGBLOOD:  Not yet.
         DR. WALLIS:  But regulation is going to -- diamond tree's
     going to help you to tell you where regulation should intervene?
         MR. YOUNGBLOOD:  That's the idea.
         DR. WALLIS:  Okay.
         MR. YOUNGBLOOD:  The next slide we've seen before.  I'm
     showing it now again because the levels of function system train and so
     forth I find it useful to keep writing those down and then putting
     things opposite them and drawing belt lines and so forth.  That list is
     just sort of the spinal column of the diamond tree.
         DR. WALLIS:  Where is the spinal column on the tree?  I
     don't understand --
         MR. YOUNGBLOOD:  Well, if you --
         DR. WALLIS:  I don't understand the link between the
     conceptualization of the diamond and the conceptualization of the level.
         MR. YOUNGBLOOD:  It means that you have a lot of flexibility
     in how you -- I don't know if it's going to work to show these together
     or not, but if at some point on here, maybe a goal corresponds in some
     to some function and then you have subgoals being systems that can
     perform that function --
         DR. WALLIS:  Okay.  So there is a relationship --
         MR. YOUNGBLOOD:  There is a relationship.  And on the next
     slides actually I've taken out that list of -- that same list of levels
     of the diamond tree and sort of then festooned the page with bullet
     items from an AIT report on a particular event at a particular plant,
     the idea being that all of these are criticisms made by that team of --
     they're either observations or criticisms.  They're things that that
     team chose to write up in the context of some event.  And some of them
     are observations that some system either didn't work or was degraded. 
     Some of them are just observations that there were a lot of components
     that were inoperable or leaky -- leaky valves.  There were human actions
     that were criticized.  There were criticisms of performance at the
     supervisory level.
         DR. KRESS:  Your point here is that these bullets were
     things that are indicative of poor performance --
         MR. YOUNGBLOOD:  Yes.
         DR. KRESS:  And that your hierarchy here tends to capture
     all those at one place or another somehow.
         MR. YOUNGBLOOD:  Yes.  It seems to me that if -- that
     properly done, and we don't really have standard guidance on how to do
     these, but if you -- that in principle you could do a tree structure
     like this and design a hierarchy that was reasonably unambiguous about
     where you would put things.  And then everything that made any sense to
     talk about in a performance context you could find a place for it.
         DR. BONACA:  And by the way, I mean, this is isn't a real
     part of the root cause analysis.  I mean every time you have loss of
     function, you go through steps of that kind and you get back always, if
     you do a proper job, to supervision or some -- because you want to look
     if there is a failure at that level.  Hopefully you find failures
     intermediately that do not propagate down into organizational issues. 
     But it's interesting how that's the same structure of a root cause.
         MR. YOUNGBLOOD:  Well, that actually it's very likely that
     Niall got it there.
         DR. BONACA:  Absolutely.  In fact, you know, I mean, you
     know, you want to work it out to the point where you're looking at the
     organization factors, and the question is where do you stop.  And
     hopefully you always stop somewhere higher up in this sequence.
         Anyway, just an observation.
         MR. YOUNGBLOOD:  Well --
         DR. WALLIS:  Now is it your intent here that the human
     actions and the supervision appear in the bottom half of the diamond
     tree then?  Is that --
         MR. YOUNGBLOOD:  Some human actions --
         DR. WALLIS:  Seems to be below the line in this.
         MR. YOUNGBLOOD:  Well, as you -- that's right.  Human
     actions actually, as you pointed out earlier, really belong at several
     levels on this thing in a functional sense.
         DR. FONTANA:  I think that the diamond tends to confuse
     things.  This gives you a sequence -- conceptual sequence going from the
     bottom to the top, and things that affect it some from the side.
         MR. YOUNGBLOOD:  Yes, this is -- well --
         DR. FONTANA:  I think this is a little clearer.  To me,
     anyway.
         MR. YOUNGBLOOD:  So you think this is --
         DR. FONTANA:  Clearer.  More clear --
         DR. MILLER:  Isn't this a diamond, though?
         MR. YOUNGBLOOD:  Well, this -- actually the -- any -- this
     is just a list of categories, and then I sort of festooned this page
     with those things, but the placement on the page other than -- the
     horizontal placement on the page means nothing.  It means --
         DR. FONTANA:  That's right.  The vertical does.
         MR. YOUNGBLOOD:  Yes, the vertical is intended to; yes.
         DR. FONTANA:  Feeding in at various --
         DR. KRESS:  It becomes a diamond because there's a number of
     these things.
         MR. YOUNGBLOOD:  Yes, yes, but again this is just stuff from
     one AIT report --
         DR. KRESS:  Yes.  Right.
         MR. YOUNGBLOOD:  Maybe there would be some other event where
     all the stuff was here and it wouldn't look like this at all.
         DR. KRESS:  There is another dimension on this vertical
     axis, and that's the level of perceived intrusiveness in regulations --
         MR. YOUNGBLOOD:  Yes.  Yes, And we're --
         DR. KRESS:  And I think they need to keep this in mind, too.
         MR. YOUNGBLOOD:  That's right.
         DR. KRESS:  The further down you go, the more intrusive you
     are.  And that's why I think you put human actions where they are,
     although they -- you know, they tend to fit a lot of places, but --
         DR. FONTANA:  I think your intrusiveness is perpendicular to
     the paper.
         DR. KRESS:  No.
         DR. BONACA:  Oh, no, no, no.  It is deep down into the
     human -- I mean -- and I think human actions and supervision is, well,
     really at the base of that, but is also much less scrutable, because
     it's much more complex, behind layers of observation that you make from
     the top down.  Anyway, but that's interesting, that true sequence of
     root cause.
         MR. YOUNGBLOOD:  Well, so, two comments I need to make about
     this.  One is that when I say that PRA doesn't, you know, issues not
     well addressed by PRA, what I mean is that items that I can put on this
     diagram are not basic events in a fault tree.  That's really all that I
     meant to imply, and I don't mean to criticize fault-tree analysis in
     saying that.
         DR. KRESS:  That's actually what I took it to mean.  I don't
     know why George had --
         MR. YOUNGBLOOD:  Well, if I ever have another shot at it --
         DR. WALLIS:  Because all of these are human actions.
         DR. SEALE:  Some people have to have their feelings on their
     sleeve.
         DR. WALLIS:  All of these are human actions, so what you're
     pointing out is that human faults are not very well modeled by PRA.
         MR. YOUNGBLOOD:  Well, specific human actions are, but if
     you had to go back to a higher-level example, if there really is a large
     number of valves that leaked and unavailability of parts is kind of a
     maintenance backlog, and somehow they weren't catching, if there was a
     rash of not capturing test -- failures after maintenance, in a sense,
     PRA -- some cut set in a PRA might have a large number of leaky valves
     in it, and at that level it's in there, but in a typical PRA you
     wouldn't necessarily take the trouble to model an underlying
     common-cause event of all those, or you might.
         You might not do it, but this comment of the AIT was sort of
     trying to signal programmatic conditions.  The fact that if they think
     they saw -- by large I think they meant more than they would have
     thought and that meant that something underlying wasn't working right
     and PRA I think does some job of capturing the fact that if a large
     number of valves leak you may have a functional failure but the actual
     prospect of that happening maybe isn't terribly well quantified in all
     cases.
         DR. KRESS:  Well, it's difficult --
         MR. YOUNGBLOOD:  Yes.
         DR. KRESS:  -- to put an importance function on human
     action.  What we are after is how important are these things to the
     final risk product and somewhere down there you have to have it modelled
     in your fault tree in order to get an importance factor out of it.
         You are just saying a lot of those things just aren't
     modelled in the fault tree and how can you get an importance factor out
     of it, so how do we know how to deal with it in performance space.
         MR. YOUNGBLOOD:  Yes, and I think we have a structured way
     of deciding that that stuff is important and that PRA plays an important
     role in that once we decide what challenges the thing has to meet and
     what systems we want.  We can go back and walk them down and pick up the
     stuff that is not in the PRA, but the PRA is more guide to success paths
     in that sense than it is an actual list.
         DR. APOSTOLAKIS:  Sure.  I think the PRA does a pretty good
     job from human actions and higher.  From human actions and below you
     have other models like Athena is trying to put some structure there. 
     Jim Wreathal has published figures that show how supervision and line
     management deficiencies and so on -- so there are models down there.
         MR. YOUNGBLOOD:  Yes.
         MR. ROSENTHAL:  Let me remind you, what we are trying to do
     is figure out what we should do with performance-based regulation and
     part of that was saying just where should we go, at what level should we
     look in figuring out how we might use the square.
         Now if I am talking about, and we have had this
     discussion -- part of my new title is Human Factors.  It is something I
     care about.  Nevertheless, if we have a performance goal of .95 diesels,
     how did I get .95?  Because that was a risk-informed number that got me
     the .95 and that becomes my performance goal, and I am measuring .95
     diesels.  I am meeting my goal.  I know that I might stop at that level
     even though I know that human performance is affecting the
     unavailability of the thing.  For those things which we think that we
     could measure at a performance level that achieves our thing, we won't
     look lower, and so I don't have to look at human performance as it
     affects my diesel if my diesel's performance is adequate.
         Now if my concern is how this crew performed in a high
     stress, rare event, dynamical situation, well, yes, then I have to worry
     about how am I going to come up with something.  I could monitor that,
     but we are trying to apply that rigor to it.
         DR. BONACA:  Of course you have always the challenge that
     programmatic and organizational failures may affect common cause because
     of the very nature of those and the moment which you commit not to look
     at those issues then you have to have some other reliance, and certainly
     one could be that you rely for example on a proper corrective action
     program that is looking at root causes effectively.
         I mean that is important at some point to get some judgment
     there that you are not waiting until something happens to make a
     judgment that you had some deep-seated common cause programmatic
     failure.
         Okay.  I can see the challenge that you have anyway in
     moving up to the top part of that.
         MR. YOUNGBLOOD:  I would like to make one more comment about
     this slide is there's a whole dimension of requirement that this doesn't
     address and that is documentation, because the way this thing is wired,
     it is about stuff other than documenting.  It is about stuff other than
     showing the regulator what is going on.  It is just the sheer safety
     performance that this tries to capture, so we could bend this or adapt
     it in some way to do that, I suppose, but I just want to make the point
     that that is a dimension that is not on here.
         With that, I think I will try to pick up speed.  Before
     getting to the process steps, there is one more idea that needs to be
     thrown out, the idea of allocation.  This topic has to do with the fact
     that you wouldn't necessarily want to set your performance goals simply
     by the numbers that you had used in your PRA.  If you come in with a
     core damage frequency of 10 to the minus 5 and you believe that the mean
     failure probabilities you used are ambitious, maybe you think they are
     right, that doesn't mean that that is the thing that you want to use to
     establish -- to think about performance here, because here it seems to
     me -- of course, this is policy -- but it seems to me that there is a
     level of safety you would like to drive to and the thing you need to
     answer is what numbers do I need in order to achieve that, and that that
     is a distinct exercise that people -- and I believe of course that the
     licensee would have the lead on doing that allocation.
         Another idea that we sort of saw in the literature search
     and that I think has been discussed here in other meetings is that it is
     useful to recognize a distinction between how you license the plant from
     a design basis point of view and what you care about overseeing or
     requiring in the way of operational measures.  I think there is a
     distinction that is useful to remember there.
         There is a slide here now about --
         DR. WALLIS:  This is one of the disconnects that runs
     through the NRC all the time is the conservative depictions of things
     that make it okay for being sure you are outside somewhere.  It doesn't
     help you at all when you are faced with some real situations.
         MR. YOUNGBLOOD:  Yes.  Yes, there are in a sense two things,
     two things wrong with it.  It makes you worry about the wrong things and
     it may not make you worry about the right things, but I remember one
     meeting here where South Texas came -- I am sure they have been to
     many -- but they were making the point that actually it is good to have
     robust systems in your design basis, so they weren't in favor of sort of
     junking that, but then recognize that something different goes on later.
         This slide is simply -- in case the point needed making, the
     point is that in risk analysis you have modelled your CDF in terms of
     initiating events and mitigating system failures that the allocation
     process -- somebody decides what we want to live with and what these
     numbers need to be in order to get there, and it is those numbers that
     have some bearing on setting performance monitoring rather than the
     numbers you may have used in your PRA.
         DR. FONTANA:  Does it lead to a legal problem if it has
     different than a design basis?  I am asking these guys, I guess.
         MR. ROSSI:  I don't know.
         MR. KADAMBI:  I would think that those are the kinds of
     problems that once we understand the application of performance-based
     approaches a little bit better and actually apply it in regulatory
     space, we would have to face some of those questions.
         DR. APOSTOLAKIS:  This allocation process, is it going to be
     done here on a generic basis or isn't it highly plant-specific?
         MR. YOUNGBLOOD:  I would think that it would be plant
     specific.
         DR. APOSTOLAKIS:  So the plant would do it?
         MR. YOUNGBLOOD:  At the very least plant-type specific.  I
     could imagine types of plants having generally similar allocations
     within a type.
         MR. ROSENTHAL:  Well, let me get back to -- if you would put
     the former slide up -- you get back to the point that Dr. Rossi has been
     making.  In the maintenance rule it is the plant that sets the goals for
     their equipment at the system, function and train level and component
     level, so it is the plant that is doing the P of C given the maintenance
     rule, so that is already in our regulations.  They are doing it.
         MR. YOUNGBLOOD:  Okay.  Let's see how quickly I can get
     through the steps.
         By report process here I mean that there is a series of
     steps in the report that I will try to walk through very quickly.
         The output of those steps is intended to be a set of
     performance areas where really a scheme, a performance-monitoring, a mix
     of performance monitoring requirements and prescriptive requirements
     that put together is supposed to lead to the level of safety that you
     articulated before you entered the process.
         Step one, "Build Safety Case."  You could say that that has
     some relationship to risk-informing, and safety case is a phrase in
     fairly common use, and I've sort of indicated here what I mean by it. 
     But as part of that, you're allocating performance, deciding what
     success paths you want to take credit for, you want to take credit for
     bleed and feed, fine, put it here.  The objectives CDF has called out
     here, I think that if you were going to have a requirement, some kind of
     requirement on some kind of defense-in-depth other than CDF, I think
     that's where you would put that in.  And again, that's something that
     the licensee does.
         The next step, at least conceptually build a diamond tree, I
     put conceptually here to make clear that, you know, a diamond tree could
     be a vast thing.  I'm not sure -- we haven't really learned what piece
     of it we need to draw or what rules of thumb to shorten it.
         DR. KRESS:  If you have more than one safety objective, you
     would have a separate diamond tree for each one?
         MR. YOUNGBLOOD:  Well, no, I think the way goal tree people
     think about it is you'd have them all -- because some performance
     elements might support more than one objective, and your allocation, as
     you allocate performance, you'd want to be thinking about everything --
         DR. KRESS:  Combine them.
         MR. YOUNGBLOOD:  Yes.  And then -- I'll actually walk
     through this.  I'll wave my hands at the next slide to describe what
     step 3 means, but given the level of performance that you want, and you
     could of course write numbers on your -- if you had some train level
     availability goal that you had allocated, you could sort of scribble
     that on your diamond tree.
         So now you have a picture of performance and a description
     of what performance you need from various nodes to get the performance
     you want.  Now go downwards through the tree, try to put performance
     monitoring requirements as high on the tree as you can, and if it
     doesn't work there, for some reason, take a step down, because you've
     got to get the assurance somewhere, and you keep working downward
     through the --
         DR. WALLIS:  Can I ask something here?  I looked at your
     tree, and you've got all sorts of boxes which say maximize reliability,
     minimize probability of failure.
         MR. YOUNGBLOOD:  Oh, now what page are you looking at?
         DR. WALLIS:  Operation of controls.  Well, these are
     impossible extremes.
         MR. YOUNGBLOOD:  Oh, you're looking at --
         DR. WALLIS:  You can't maximize.  It's a meaningless
     statement.  The tree is--
         MR. BARTON:  Where are you, Graham?
         MR. YOUNGBLOOD:  He's looking at the report.
         DR. WALLIS:  The tree is full of exhortations.  Maximize the
     probability the operator will quickly detect.  You want the probability
     to be 1?  That's maximizing the probability is make it 1, just a
     meaningless statement.
         MR. YOUNGBLOOD:  Yes.
         DR. WALLIS:  It's full of statements like that.
         MR. YOUNGBLOOD:  Yes.  I think some of the diamond trees
     are.
         DR. WALLIS:  I'm sorry, it's just that --
         MR. YOUNGBLOOD:  No, that's okay.
         DR. WALLIS:  Just don't use the word "maximize probability"
     like that, it's a red flag.
         MR. YOUNGBLOOD:  Yes.
         DR. FONTANA:  Don't use the word "minimize" either.
         MR. YOUNGBLOOD:  This is a little gaudy, and maybe it's
     underdeveloped --
         DR. WALLIS:  Let me tell you why I got into that, because I
     said look, after all this sort of religion and picture conceptual let's
     see what a real case looks like, and so I looked at your real case, and
     it's full of these, again, these exhortations, which I don't see are
     useful.
         MR. YOUNGBLOOD:  Those are.  I don't -- there are things
     about those trees that I don't necessarily like, but as I say, there's
     not a standard format, and they may not have been drawn -- well, they
     weren't drawn from several -- I'll just -- let me just stop.
         DR. WALLIS:  Well, let me get back to the question I asked
     you an hour ago, is to show us the usefulness of the approach.
         MR. YOUNGBLOOD:  Well, what I think you would like to see,
     and what I would like to do, is a pilot.  And we haven't done that, and
     we can maybe, well, if I move faster than I did, maybe we could have
     talked about what ingredient a pilot might have.
         This slide is intended to be sort of marked up.  If you
     wanted this box to be a 10 to the minus 5 box, 10 to the minus 5
     probability of failure of some function, then -- and you could get that
     performance by some combination of these lower boxes, maybe one of these
     boxes would be a 10 to the minus 3 box, and the other would be a 10 to
     the minus 2 box, 10 to the minus 4, 10 to the minus 1, assuming, of
     course, that they're independent, you step down deciding how much
     performance you want and how you can get it.  And you might decide at
     some point, as you decompose your performance, you might decide that
     actually this box you can actually establish that it's performing at
     that level by some sort of monitoring scheme.
         DR. KRESS:  At this point you're talking about what the
     licensee would do.
         MR. YOUNGBLOOD:  The licensee did the allocation.  I think
     the licensee would do the allocation, I would think, but it would have
     to be iterated with how the regulator saw the monitoring scheme.
         DR. KRESS:  The licensee might come down and say I want to
     emphasize the performance of this thing, but NRC might say no, we want
     you to put some on this other because of certain defense-in-depth
     considerations or something.
         MR. YOUNGBLOOD:  Well, yes.  I guess defense-in-depth would
     have entered before you got to this stage in the safety case, and you
     would have allocated performance in a way that met whatever those
     objectives were, CDF, defense-in-depth, and, yes, at that stage it would
     have forced you to do a different allocation.  So, but given those
     constraints, the licensee would do the allocation.
         DR. APOSTOLAKIS:  I have a problem with the use of the word
     "allocation," because this implies that I'm going to design a system and
     build a reactor now.  And what we have in reality is 104 plants out
     there operating.  So the utility will not really allocate anything.  The
     utility will come back and say -- if they have to do this, they will say
     well, at this level at least my PRA does something, it tells me
     something, tells me what the unavailabilities of the functions are,
     trains and components.  This is the way it is.  I'm not allocating
     anything.  And presumably it's acceptable because I'm allowed to
     operate.
         Now the question is which parts, which boxes here which I
     have already assessed can I monitor and establish performance criteria
     for.  That's really the question, not allocating performance.  The
     performance is already there.
         MR. YOUNGBLOOD:  The best estimate -- or I shouldn't use
     that phrase here in this audience -- no, the mean failure probabilities
     are already there in some sense.  I believe that the numbers that go on
     this diagram are more generous numbers than that, that some people --
     for example, I mean, an implication of this is that if you invoke
     something on this scheme, the regulator will get interested in it.
         DR. APOSTOLAKIS:  No, but my point is really one of
     approach.  I think the approach you are taking is I'm starting with a
     clean slate and I'm allocating and I'm looking at things and so on and
     I'm saying no, you already have what is already out there.  So you look
     at it, you say well, this is the function unavailability they have for
     this plant, system trains, components, now what is it that I can use as
     a performance measure to convince myself that this is what's going to
     remain next year.
         DR. BONACA:  These are criteria.  These are not
     calculated -- what he means, I understand what you are saying, you're
     right, you know, PRA, but assume that for example you're expecting a
     certain availability from a component, okay?  You're not going to set
     your goal to that availability.  Probably you are going to give yourself
     some margin to that.
         DR. APOSTOLAKIS:  Sure.
         DR. BONACA:  And I think that would be for the licensee to
     propose it --
         DR. APOSTOLAKIS:  Yes.
         DR. BONACA:  And then you get some box out there where you
     say well, you know, I can't monitor that, I would rather just go down
     below a level and deterministically commit to something else, okay?
         DR. APOSTOLAKIS:  But the starting point should be what I
     have now.
         DR. BONACA:  I agree with that.
         DR. APOSTOLAKIS:  I'm not allocating anything.  I mean, this
     is it.  This is what I have.  This is what's feasible.
         MR. YOUNGBLOOD:  I think I would admit that as one way to
     implement this, but I have imagined that some licensees who have a lot
     of alternative core cooling schemes in their PRAs would not want to have
     all of those schemes pop up here and get requirements on them that they
     would maybe say well, you know, to meet 10 to the minus 4 I don't
     actually need all those success paths, I don't want all those criteria,
     you know, being monitored against all those criteria.  I'll just bet the
     ranch on a couple of good success paths that I'm sure I can satisfy the
     more generous number that I can live with and still meet 10 to the minus
     4.
         So I saw it as a way of giving discretion because, as Mario
     says, it's criteria we're talking about here and not modeled risk.
         DR. APOSTOLAKIS:  Well, I guess my complaint was that I
     didn't hear you emphasize enough the fact the facility already exists,
     and that you already have a lot of these numbers.
         DR. SEALE:  Yeah, but it has brought you down to the point
     where you now can deregulate certain blocks on this diagram based on an
     assessment of what the overall contribution to risk is.
         DR. APOSTOLAKIS:  But I am not allocating anything.
         DR. SEALE:  I didn't say that.  I said you can deregulate.
         DR. APOSTOLAKIS:  Yeah, I agree.
         DR. SEALE:  I don't put as much inspection in it.
         DR. APOSTOLAKIS:  Yeah.
         DR. BONACA:  The use of the word "allocating" is confusing.
         DR. APOSTOLAKIS:  I think allocating means designing
     something, and I am saying this is what I expect from this system, from
     that system.
         MR. YOUNGBLOOD:  Okay.  The next three slides are just
     figures in the report and they are just there for purposes of
     illustration.  This is really where that beltline came from.  What is
     shown on this particular slide is probably a bad scheme, the most
     intrusive scheme imaginable, where the regulator is looking at stuff at
     all levels of the tree.
         DR. WALLIS:  I don't know that that is necessarily bad as
     long as he doesn't do it all the time.  It seems to me it be very good
     if the regulator says, at any time I can come in and make a spot check
     of something.  That keeps the licensee on his toes.  That is not
     necessarily a bad thing.  Otherwise, he will just work on the things
     which are emphasized.  So it is not necessarily intrusive to have the
     ability to check things, as long as it is not done to excess.
         DR. APOSTOLAKIS:  That is not what NEI wants, though.  They
     want you to do that only if some performance measures --
         DR. SEALE:  Or only if you indicate that in a certain
     maintenance operation, the level of CDF has risen to a point where you
     need more assurance that you have got another --
         DR. APOSTOLAKIS:  Well, Graham says that we should reserve
     the right to go and --
         DR. SEALE:  Yeah.
         DR. APOSTOLAKIS:  And NEI says no.  NEI says only if you
     have a good reason to do that.  That is a different, I mean --
         DR. WALLIS:  Everything else which isn't somehow measured in
     this way can go to hell and you don't have any right to go and find out?
         DR. BONACA:  Although the special program that was
     presented, I mean had all kinds of flexibility for --
         DR. WALLIS:  It has got to have flexibility on both sides, I
     think.
         DR. SEALE:  Okay.  Now, you have found us an issue where we
     can have some comments.
         MR. YOUNGBLOOD:  I hope it is semi-clear by now that a
     process could lead to a big mix of performance things that you could
     monitor and some prescriptive requirements to reinforce performance,
     where monitoring wasn't going to work.  One thing I would just like to
     throw out briefly, because it appeared in the report, some people took
     an interest in it, is the idea that abductive inference might be a good
     way to think about the problem.  The issue is we are getting -- we get a
     boat load of numbers and how do we think about what they mean?
         And what this, all that this slide means to suggest is that
     in some ways it is analogous to medical diagnoses and that work is going
     on in that community to formalize how they think about getting a lot of
     information.
         DR. WALLIS:  You still have regular checkups whether you are
     sick or not.
         MR. YOUNGBLOOD:  Yes.  Yes, there is your monitoring.  So, I
     have a couple of summary slides here, I am not certain how -- we have
     talked about things so much that I can probably zip right through it,
     and then maybe we could a take word to talk about what a pilot might
     look like.
         The process was biased towards finding the most
     performance-based possible scheme, and, again, we didn't argue the
     merits of that or debate it, we just tried to make a scheme that would
     drive that way.  And we tried to look at areas -- well, there is that
     phrase again.  We tried to look at those areas that I don't have a good
     phrase for, that don't leap out of PRA as having been well treated.
         The process involves, first of all, formulation of a safety
     case, and I could just sort of toss that off in a page because this was
     about performance-based and not risk-informing, but, obviously, there is
     a lot behind making a safety case.
         Then we described a process for identifying a combination of
     monitoring measures and other measures that would somehow combine to
     satisfy the objectives that you had articulated.
         The example we did suggests that the physical parameter
     temperature is not necessarily a good way to monitor a functional
     performance.  No need to dwell on that.  Diamond trees --
         DR. APOSTOLAKIS:  Is the message here, Bob, that the
     prevailing view that one can have performance-based regulation without
     it being risk-informed is really false?  Because I think
     risk-information was an essential part of your presentation.
         MR. YOUNGBLOOD:  It was.  I took the view, I think in
     principle you can have performance-based without risk-informing.  And I
     guess, formally, I could say, well, maybe your safety case wasn't
     risk-informed either, but somehow you could talk about performance. 
     Without a complete PRA, I think of availability, you could even work
     with availability and have it not be risk-informed.  You could just work
     with a gibberish set of unavailabilities, and then go about
     performance-based implementation of that.
         DR. MILLER:  We have performance-based without
     risk-information for a long time.  What you are really saying is risk
     insights help you do performance-based regulation.
         DR. APOSTOLAKIS:  No, he is saying you can't do it.
         MR. YOUNGBLOOD:  Well, the process I described had a safety
     case in it that, to my mind, began with a PRA.
         DR. APOSTOLAKIS:  The example that NEI did and Scientech
     reworked convinces me that every time I see now a deterministic
     performance criterion, I will have to become very skeptical.
         DR. KRESS:  What is this now?
         DR. APOSTOLAKIS:  Well, excuse me, but that is what the
     example shows.  The examples shows that what NEI proposed made no sense
     in the probabilistic framework, that you had to go deeper and look at
     the initiators and the conditional probability of core damage and so on. 
     So, I mean one message I am getting from this work is that you can't
     really have performance-based regulation without risk-informing it.
         MR. KADAMBI:  I would like to make sure that the message of
     this report, you know, is what I believe it should be, which is that
     there is a methodology that we had not become aware of, even though it
     was really the diamond tree was done on behalf of NRC, actually, some
     time ago.  So this was something that we became aware of that could be
     useful in thinking about performance-based approaches, but the broader
     thing is that it would help us in dealing with the broad range of
     regulatory applications that we in this agency have to worry about,
     including all the materials issues.
         It could well be that there are many NMSS licensees who
     don't have any kind of risk analysis at all who are -- you know, we may
     be able to regulate them better if we use a performance-based approach,
     as opposed to what is likely right now would be a very prescriptive kind
     of approach.  I think in principle you can have performance-based
     regulation without any risk analysis and this did not -- nothing in this
     work I believe contradicted that.
         DR. APOSTOLAKIS:  No, I thought the example they did
     contradicted that.
         MR. KADAMBI:  Well, that was just an example --
         DR. APOSTOLAKIS:  But you didn't show another example that
     did work.
         MR. KADAMBI:  You know, as I mentioned, this is a very
     modest contract.  It spanned less than a year and we really were not
     able to answer a lot of questions and in fact if you look at that report
     in the Foreword we try to identify many important questions that we were
     not able to answer at all.
         MR. YOUNGBLOOD:  No, but the fourth attribute that you don't
     want to have a problem by the time you trip this criterion, I think it
     can fairly be asked whether there is a way to explicitly address that
     bullet without a risk dimension, and that is certainly the way I
     approached it.
         DR. APOSTOLAKIS:  Well, you can address it perhaps by
     looking at how many additional ways you have to achieve the same thing
     without looking at the probability.
         MR. YOUNGBLOOD:  Yes.
         DR. APOSTOLAKIS:  The pre-PRA way of looking at things, you
     know, the number of events in a minimal cut set without the
     probabilities.
         MR. YOUNGBLOOD:  But that would to me -- scenarios being
     part of risk though, I mean that's -- if you had a logic model --
         DR. KRESS:  That is still being risk-informed, wouldn't it?
         MR. YOUNGBLOOD:  Yes.  It wouldn't be PRA-based --
         DR. KRESS:  Where you get into trouble is where you get down
     to the level of talking of things like QA and safeguards and management
     issues and things like that, which this concept is just not going to
     help you at all and you will have to decide on how you are going to set
     performance indicators and levels there, if indeed you feel like that is
     something that has to be part of your performance measures.
         Like I say, use the PRA where you can but when you get down
     to these levels you have got to look for something else.
         DR. APOSTOLAKIS:  I agree with that, but the whole
     presentation assumed that you had a PRA -- everything has probabilities
     in it.
         DR. KRESS:  Not the whole presentation because they --
         MR. KADAMBI:  To some extent that is true, but I believe the
     fire protection example did not specifically use a PRA, did it, Bob?  I
     can't remember the details.
         MR. YOUNGBLOOD:  You are pushing me.  Certainly --
         MR. KADAMBI:  That may not be a good example --
         DR. APOSTOLAKIS:  It is not.  We did review the NFPA
     standards and I don't think the committee liked it.  You know, they
     thought that they were going to propose two parallel paths, one
     risk-informed and one not risk-informed and the one that was not
     risk-informed was nothing, so I really have doubts that you can have a
     performance-based -- I mean you can go to a little thing, somewhere, an
     issue and say, gee, if I went a risk parameter I don't have to do the
     rest.  Okay, maybe, but is that really what we are talking about when we
     say --
         DR. BONACA:  I think I agree with you, George.  I believe
     that right now we are looking at everything, the whole diamond -- many
     part of that.  What NEI is saying -- we want you to look only at some
     high performance indicators up there and that's it, and the bridge to
     doing that, it's risk information in my mind, okay?
         DR. APOSTOLAKIS:  Yes.
         DR. BONACA:  And it is again I think the only way to enable
     us to make those judgments is to say we are not going to look at
     causative factors because we can monitor functions and have sufficient
     margin is to have the understanding, and I agree with you that without
     risk information we're not going to get that.
         DR. APOSTOLAKIS:  Well, maybe we should tell the Commission
     that, because they keep separating the two.
         DR. BONACA:  Clearly I believe that many of those
     performance indicators, that you can poke holes in them the same way you
     did here, okay, with pretty simple analysis.  I mean it was
     well-structured but it was pretty simple.  The PRA analysis showed that
     that was inadequate.
         DR. APOSTOLAKIS:  I thought that was a great part of your
     presentation.  Lest silence is perceived as agreement, I disagree
     completely that the diamond tree hierarchy is a useful way to organize
     discussion.  It is really an influence diagram that you are trying to
     develop and it will be declared as useful only after you actually do it. 
     I mean it is so easy to talk at that level, you know, programs affect
     this -- but try to do it and you will see how difficult it is to develop
     an influence diagram or a diamond tree which is a renaming of the thing,
     to actually show all these influences.
         You are talking about modelling a complex industrial
     facility and you want to bring into it everything that management does
     that affects other things.
         I mean if you could do that, it would be a great guide, but
     at the high level of course it is a useful way.  I mean actually the
     most useful part in my opinion was the vertical stuff you showed --
     function, system, component down.  The tree itself, I think the report
     over-advertises the usefulness of the diamond tree, and talks about it
     as if it is the special theory of relativity and we have to find out who
     proposed it first.  I mean if you go to the report, there is a whole
     paragraph as to who proposed it first.
         I think it is a simple idea.  Decision analysts have called
     it an influence diagram.  The top level is the value tree.  The bottom
     level is the decision part and I think you should reduce the emphasis on
     it.  I mean you can claim that it is a useful way to organize discussion
     only after you have done it and demonstrated it can be done, because the
     most difficult part in any decision program is built in the influence
     diagram and you renamed it.  That is all you have done, and you say it
     is useful.  Yeah, sure, conceptually it is very useful, but try to do
     it.
         DR. KRESS:  On that note --
         DR. APOSTOLAKIS:  It sounded a little harsher than I wanted
     it to but a lot of my colleagues were impressed by the diamond, and I
     wanted to make sure that the record shows that there is disagreement.
         DR. BONACA:  But you are not unimpressed.  You say simply
     that there is --
         DR. APOSTOLAKIS:  I don't think it works.
         DR. BONACA:  -- renaming or something else.
         DR. APOSTOLAKIS:  Yes, and that something else is very
     difficult to do.
         DR. MILLER:  But it still may be a very useful to organize
     discussion, even though you can't do it for an actual situation.
         DR. APOSTOLAKIS:  No, I think the vertical thing that you
     showed is much better -- but that's okay.  I mean it helps people.
         DR. FONTANA:  There's more ways to skin a cat.
         DR. BONACA:  But the vertical accounts for the very nature
     that you start with an objective and it opens up and by the time you get
     to the middle --
         DR. APOSTOLAKIS:  People have called this a master logic
     diagram --
         DR. BONACA:  I don't care how you call it --
         DR. APOSTOLAKIS:  -- and decision analysis is decision tree,
     value tree -- I mean --
         DR. MILLER:  All of them are hierarchical --
         DR. APOSTOLAKIS:  A lot of stuff, hierarchical approaches,
     hierarchical decomposition of a problem -- because, you know, this may
     fall in the hands of a non-nuclear person and then we are really
     undermining our credibility if they see something that is very familiar
     to them advertised as a new discovery.
         DR. WALLIS:  George -- I agree with George mostly.  Now the
     thing that concerns me is this is an important thing that the Commission
     wants to get done and you seem to be still trying to figure out how to
     get to first base.  I don't see a plan to implement anything or anything
     like it, and you are still arguing about how you might conceivably think
     about the problem.
         MR. YOUNGBLOOD:  That has a -- can I just jump in and make
     one technical comment before the Staff takes over and responds to that,
     that the theory, the observation that we should really do it for some
     problem to really show, that has been a consistent theme, and I
     certainly agree with that.
         The report as written contemplates a soup-to-nuts, do the
     whole thing at once mentality, because I think it is hard to be sure
     when you go about risk-informing, it's hard to just take a piece of the
     risk analysis and then believe that you have done it --
         DR. WALLIS:  You're right.
         MR. YOUNGBLOOD:  -- done it right, so in moving forward with
     a pilot, which I of course would love to do, in an incremental way.  It
     would be nice to identify a piece of the problem that moderately cleanly
     decouples from the rest and then do that, so figuring out a piece that
     decouples from the rest would be an important step, and I think if we
     had already done that step, we could have already done the things that
     you are talking about.
         DR. APOSTOLAKIS:  It seems to me, Bob, that if you indeed go
     ahead with the pilot, it would behoove you to look at the literature and
     influence diagrams.
         MR. YOUNGBLOOD:  Oh, yes.
         DR. APOSTOLAKIS:  That is where the action is.  That is
     where people have spent time understanding what is going on and
     developing mathematical theories and so on.
         MR. YOUNGBLOOD:  Actually we didn't not look at it, and I
     agree we may have written more than we should have about the diamond
     tree, partly because hardly anybody had heard of it, and --
         DR. APOSTOLAKIS:  Well, I had.
         MR. YOUNGBLOOD:  Well, you are one of maybe six people.
         DR. APOSTOLAKIS:  The authors plus me, perhaps?
         I don't want to finish on a negative note.  I thought the
     analysis that you did of the NEI example was very good.  I learned a lot
     from it, and I think your discussion of the hierarchical level
     independently of the diamond tree was actually very good, but not very
     new to me, but the other stuff was really new and I really enjoyed it.
         MR. YOUNGBLOOD:  Good.
         DR. APOSTOLAKIS:  I thought in other words there are good
     parts to this report.  I don't know which ones you wrote, Bob, but --
     the Executive Summary --
         MR. YOUNGBLOOD:  No, no --
         DR. APOSTOLAKIS:  -- but part of the diamond tree I must
     say, you know, if we keep it among ourselves perhaps it serves a
     purpose, but I can see decision theorists looking at this, saying, you
     know, the nuclear business is going its own way.  I have done the same
     thing and I'd never quote it that way.  It was a nice tree with a
     decision node at the bottom, the value at the top, the objectives, and
     now that I think about it, it looks like a diamond, yes.
         DR. KRESS:  Well, George, one thing that I liked about it
     that hasn't really come out is that it provides a way to be sure you are
     looking at the performance and covering every branch of this influence
     diagram in some way.
         DR. APOSTOLAKIS:  You use the right words and I liked the
     same thing.
         DR. KRESS:  Okay.
         DR. APOSTOLAKIS:  But I must point out that what Bob showed
     us, with the vertical lines there, he could have done it, in fact he did
     it, without the diamond tree.
         MR. YOUNGBLOOD:  Yes.
         DR. APOSTOLAKIS:  But that was a useful part.
         DR. BONACA:  But the diamond just describes it -- I am
     looking at this purely as a pragmatic tool to help me making some
     judgments, okay?
         DR. APOSTOLAKIS:  It is not pragmatic until it is applied.
         DR. BONACA:  I understand that.  In some examples that I saw
     and some that I have been familiar with, I just played a little bit and
     it was pretty useful.  That's all I am looking at.  Certainly I have not
     tested whether or not it is complete, if it is totally effective.  That
     is beyond my interest at this stage.
         You know, what is important to me, however, is that it
     showed me in a quantitative way why my suspicion of some of those
     criteria I saw in the NEI proposal --
         DR. KRESS:  Shows that you were right to be suspicious.
         DR. BONACA:  Yes, in a quantitative way.
         DR. KRESS:  At this point I want to turn it back over to --
         DR. APOSTOLAKIS:  Wait, wait, wait --
         DR. BONACA:  On the components -- you know, quantitative
     means simply looking at the contribution, looking at the concept of
     margin --
         DR. APOSTOLAKIS:  Yes, I agree.  That was the great part,
     and the diamond tree has nothing to do with it.
         DR. KRESS:  I think at this point we are running out of
     time, and we need to hear Mr. Kadambi's plans for the Commission paper,
     so --
         MR. ROSSI:  Could I break in just one minute, because we
     clearly did not come here with a plan that we wanted you to review, but
     Prasad, do you have on a viewgraph the questions for the stakeholders
     that you could put up?
         MR. KADAMBI:  Yes, I do.
         MR. ROSSI:  Okay.  We did not come here with a plan.  We are
     at the point now of collecting information to respond to the
     Commission's request in their SRM I guess by the end of May.
         Now at the beginning of the day I indicated that we had a
     number of questions and I asked people to sort of take a look at those
     and see if we are asking the right questions when we are trying to
     develop our response to the Commission.
         Now, here are the questions that we have focused on and
     rather than read them, maybe you ought to read them yourselves and then
     make comments on them.  Because I think much of this has been the
     subject of discussion today, in particular, the -- well, all of these, I
     think.
         I think your comments, Dr. Apostolakis, have, in many cases,
     been addressed to questions that are up there.
         DR. APOSTOLAKIS:  Well, a procedural matter now, obviously,
     you had some input from the subcommittee, you know, the transcript is
     available, or will be available.  What is the plan now, that you will
     develop a plan and the next time we will see it will be June?
         MR. ROSSI:  Yes.  That is the plan.  And that will be after
     we send a paper to the Commission.
         DR. APOSTOLAKIS:  So you feel you have enough now guidance,
     input from the committee?
         MR. ROSSI:  Well, we have as much as we are going to be able
     to get.
         DR. APOSTOLAKIS:  That is a pragmatic view.
         MR. ROSSI:  Right.  I mean we have everything that we are
     going to be able to get at this time.
         DR. APOSTOLAKIS:  But that will be a plan, so we can still
     talk about it in June.
         MR. ROSSI:  Yes, it will be a plan and it may be something
     that ties a great deal to what is going on in the agency in the area of
     risk-informing things.
         DR. APOSTOLAKIS:  Fine.
         MR. ROSSI:  And, as a matter of fact, Prasad, you might put
     up the planning for performance-based approaches.
         DR. WALLIS:  I have comments on the question.  Do you want
     comments on the questions?
         MR. ROSSI:  Sure, that's fine.
         DR. WALLIS:  They seem to me very preliminary type
     questions.  I mean I would have difficulty responding to any of these
     without a better idea of what you guys are up to, what you have in mind
     in the form of regulations which are performance-based.  These are
     questions based on some hypothetical thing I have difficulty
     visualizing.
         So if you would come up more -- perhaps more of a
     discussion, more of a specific thing that is visualizable of what
     performance-based regulation might be more like, and how it might
     specifically change my life, then I would have a better way of answering
     the questions.
         MR. ROSSI:  Well, we have the example of the maintenance
     rule.  We have Appendix J.
         DR. WALLIS:  Maybe say I just don't know it.  Maybe you
     should make that reference then.  You should say here are some examples
     of performance-based.  If this were extended to some other regions, or
     something.
         MR. ROSSI:  Yeah, as a matter of fact, the other viewgraph
     that I was suggesting he put on the screen, I think makes that point.
         DR. WALLIS:  Okay.
         MR. ROSSI:  Well, why don't you put the one, planning for
     performance-based approaches, Prasad.
         MR. KADAMBI:  Okay.
         DR. WALLIS:  So the two would go together then?
         MR. KADAMBI:  Well, these are essentially the elements of
     the plan that we have come up with right now.  This was going to be my
     last slide, as, you know, the Commission paper itself presenting these
     with some schedules as being our plan at this point.
         If I can just maybe quickly talk through these.  The
     Commission wanted to make sure that we were well integrated into other
     things that are going on, so whatever we do, we do together with the
     other offices and make this a truly agency-wide effort.  And to do this
     we need to appropriately recognize where the similarities are with the
     revisions to the regulatory oversight process, the risk-informed
     revisions to Part 50 and other NMSS activities which I don't fully know
     yet, but we will be finding more out about, and make sure that we are
     not duplicating things that are going on, that is going on elsewhere.
         We want to learn from the prior experience of the
     maintenance rule and Appendix J and, you know, this will be part of the
     plan to incorporate the prior experience into it.
         We want to participate in pilot projects, those that are
     going on elsewhere, and those that we might want to initiate.
         I'm sorry, did you --
         DR. APOSTOLAKIS:  I think -- yeah, sure, I mean this is
     important to do.  But there are some basic questions.
         DR. WALLIS:  I see, a list of activities.
         DR. APOSTOLAKIS:  Yeah, these are activities.  Like the
     question we asked in our regional letter, you don't seem to think it is
     important enough, but shouldn't you be asking people and trying to
     address that question.  You know, who would set them up and how?  And
     the other question that I asked earlier, what is performance?  And the
     reason why I am asking is because the NFPA standard that was advertised
     as performance-based had examples that said, yeah, you look at the two
     pumps and if the distance between the two is X feet, then it is fine,
     that is a performance criterion, which I didn't expect it to be a
     performance criterion, it is not design.  Right.
         So, performance, it seems to me, has an element of time in
     it.  You know, you monitor something over time that might change, right,
     not how you design the plant.  So a definition of performance someplace
     would be useful.
         And then the objectives.  What are we trying to achieve with
     this?  And that I think will come when you try to integrate your work
     with what is going on in other parts of the agency in terms of
     objectives.  Do you subscribe to the cornerstone approach?  Do you want
     something else?  You know, these are the kinds of issues that should be
     debated right now, because if you disagree, then those guys that are
     working in that area should know about it.  Because the last thing we
     want is, you know, to try to risk-inform the regulations and then five
     years from now we have a complete mess in our hands with different
     objectives in different parts of the agency, and then, of course, we
     will blame risk-informed regulation.
         MR. KADAMBI:  Well, no, I mean that is the reason why I
     began with, you know, make sure that this is an agency-wide effort.
         DR. APOSTOLAKIS:  Okay.
         MR. KADAMBI:  We want to keep it that way.  But relative to
     some of the other questions, one of the activities definitely that we
     will be doing is developing guidelines, you know, to identify and assess
     issues and candidate performance-based activities.  I mean these are
     actual specific ones.
         DR. APOSTOLAKIS:  Now, before the guidelines, don't you
     think you need a set of principles?
         MR. KADAMBI:  Well, I mean it may be that --
         DR. APOSTOLAKIS:  You have heard that before, haven't you?
         DR. MILLER:  It seems like that what we talked about this
     morning is where I was starting all this.  As George says, lay out a set
     of principles in some sort of document, and then start on this.
         MR. KADAMBI:  If the ACRS recommends that that is the way we
     go, --
         DR. APOSTOLAKIS:  The ACRS cannot recommend anything today
     because the ACRS is not here.  This is a subcommittee.
         MR. KADAMBI:  I understand.
         DR. MILLER:  If you started with a blank sheet a paper and
     go over the process you have outlined there, you are going to end up
     with a sheet of paper full of things that aren't very well coordinated. 
     You need to provide leadership of some type here.
         MR. ROSSI:  Well, we are certainly going to start with the
     attributes of performance-based regulation.  I mean there is no question
     about that, and we have talked about that today.  And we are going to
     talk about the work that has gone on in terms of the revisions to the
     reactor regulatory oversight process.  And there, they do indeed have
     objectives laid out.  They do indeed depend on the cornerstones.  They
     depend to a large extent on the use of PRAs in what they do.  So we will
     look at the thins that are going on now.
         Now, we believe that those -- we have no reason whatsoever
     to believe that any of that is misdirected in any way whatsoever.  So
     what we are trying to do, I think, is see whether there are other things
     that we should be doing in the area of performance-based activities over
     and above what is ongoing and is using PRA.  And what we heard today I
     think that we got a lot of input in this area.  We didn't get any, and
     have not as yet gotten any specific suggestions on what we would focus
     on if we were to look at something where risk was not a major
     contributor and to developing the performance criteria in whatever we
     monitored.  I don't think we have gotten that.  And the things that are
     covered by risk I think are already going on.
         DR. MILLER:  You are looking for an example where risk is
     not going a valuable input?
         MR. ROSSI:  Well, yeah, because we have got many things
     going on in the area where risk is a valuable input.  And, generally,
     where risk is a valuable input, the objective focuses on the core damage
     frequency.  It works back to initiating events and mitigating systems,
     et cetera.  I guess in NMSS area, it focuses on doses, it works back.
         DR. MILLER:  But that is not a risk.
         MR. ROSSI:  Well, I suppose it is -- I think that is what
     they focused on.
         DR. MILLER:  I think that is more performance, isn't it?
         MR. ROSSI:  Well, yeah, that may be.
         DR. MILLER:  Go back to Part 20.
         MR. ROSSI:  Performance, right.  Yeah.  It may be less on
     risk.
         DR. MILLER:  It says you have to maintain these things.
         MR. ROSSI:  Now, it may very well be, apparently there are
     some members here today that feel that way, that the areas where you
     don't use risk in developing the performance-based approach are limited. 
     I mean that may be the answer.  I don't know.  That appears to be at
     least one member's view, perhaps others.
         DR. KRESS:  Of course they are limiting, but the question
     is, are they important enough to develop some guidelines on how to treat
     them in a performance place?  I think if we look at the influence
     diagram and see what is important even in risk-based, it probably would
     have to include -- yeah, they are limiting, but they are important. 
     Probably you need to develop some guidelines on how to treat them in
     performance-based.
         DR. SEALE:  Well, whether you call them diamond diagrams or
     risk diagrams or whatever name you want to give them, and whether you
     are going to be worried about being accused of stealing them from
     somebody else, or starting over, or whatever, I don't care, the thing
     that came out of this that I saw was a lot better basis on which to
     begin to identify the level and the kinds of things that intervention
     oriented performance indicators would have to address.
         DR. KRESS:  Well, I think we are talking about things like
     QA, training, institutional factors, safety culture, safeguards.
         DR. SEALE:  Specific to certain systems.
         DR. MILLER:  Well, why don't you try training?  There is a
     lot of information there, and a lot of performance criteria in training.
         DR. KRESS:  Well, once again, I think you are going to have
     trouble determining performance indicators that are different than just
     the process, because they have the appropriate process in place.
         DR. SEALE:  Yes.
         MR. ROSSI:  Well, you can look at examination results.
         DR. KRESS:  Examination results.
         MR. ROSSI:  Simulator results and that kind of stuff.  But
     that may be already being done.  I mean that --
         DR. KRESS:  My point is --
         DR. SEALE:  That is not where the screw-ups are.  The
     screw-ups are with the guy on the maintenance floor.
         MR. ROSSI:  We have the maintenance rule to look at that,
     and the maintenance rule is indeed performance-based and, presumably, if
     that is where the problem is, it shows up in monitoring that is done for
     the maintenance rule.
         DR. SEALE:  Yes, yes.  It is not in the training as such.  I
     mean not what we normally think of as the training program.
         MR. ROSSI:  Right.  Right.
         DR. SEALE:  Is it in the maintenance areas.
         DR. KRESS:  Well, I think there are some important issues in
     just performance orienting the risk-informed part.
         DR. SEALE:  Yes.
         DR. KRESS:  I think there are still some important questions
     that were raised here and need to be asked.  I don't know if you guys
     are the right ones to address those in this particular program.  But
     there are some good questions that came up that I think need addressing. 
     And maybe that is -- maybe you ought to look at that, too.
         MR. ROSSI:  Well, one of the things that I think we can say,
     based on our interactions with industry represented is we haven't -- I
     guess we haven't heard anybody stand up and give an example of where
     they believe there is some particular regulation, or set of regulations
     that are overly prescriptive, that could be made more performance-based
     and reduce the burden without reducing safety, that are not being
     covered in the efforts that are going on in risk-informing the
     regulations.  I don't think we have heard that.  So that is an important
     piece of input, I believe.
         DR. KRESS:  Yeah, when you do hear that, you will need some
     principles to guide you on deciding whether or not --
         MR. ROSSI:  That's right.  But then the question is, how
     much effort do we put into it until we have a real problem to work on?
         DR. KRESS:  Ahead of time until you have something.  Yes,
     good question.
         Well, I think we have reached the witching hour.  Are you
     just about through with this?
         MR. KADAMBI:  Yeah, I am really -- as I mentioned, this was
     going to be my last slide anyway.  One of the, I guess, important points
     to be made on this slide is we are, you know, thinking about modest
     resources on this part of the program right now.  And we do foresee that
     if something useful comes out of the plan, it will become incorporated
     into the normal agency activity.  So it will either become
     institutionalized or it will be sunset, depending on whatever comes out
     of it.
         DR. KRESS:  At this point I don't anticipate a letter
     because subcommittees don't write letters, and probably our next time we
     will hear about this is when you do have some sort of a Commission
     paper, or a draft plan.  Maybe we can look at it then and make some more
     comments, and actually have a letter.
         DR. SEALE:  And that will be June.
         DR. KRESS:  And that I think would be at least June.
         MR. KADAMBI:  We intend to meet our schedule relative to the
     Commission paper, which will be done and submitted to the Commission by
     the end of May.
         DR. KRESS:  The end of May.  Yeah, you need to go ahead and
     do that.
         MR. KADAMBI:  Right.
         MR. ROSSI:  And we did get what I feel to be a lot of good
     comments today, and a lot of good input that we can think about.  And
     the kind of things that we got in our other public meetings, even though
     there were some kind of what I would say, lack of input, I think that
     sends us a message also.
         DR. MILLER:  I think an exercise maybe the staff, I like to
     see the staff do, just in your offices, take your attributes and apply
     them to a current performance-based process like some part of the
     maintenance rule.  Just see how they stack up.  Not something you are
     going to write up, just force yourself to do something real.  Because
     all right now Professor Wallis is saying, we are doing a lot of talking,
     but we haven't done much.
         DR. KRESS:  Well, I think he did that with the NEI proposal
     as a rule.  I mean that basically was an example of doing that.
         MR. ROSSI:  Yeah, I think that has been done, to apply the
     attributes.  And I think you can apply them fairly quickly.
         DR. MILLER:  But the NEI proposal was not an ongoing
     performance-based process.
         DR. KRESS:  But it was the form of what a rule might have
     been, that is a way to look at it.
         DR. FONTANA:  Are we about done?
         DR. KRESS:  Yes.  I am getting ready to --
         DR. FONTANA:  I looked up, one of your slides had abductive
     reasoning, and I looked it up in the dictionary, and a third definition
     is a syllogism where the major premise is certain, but the minor
     premises are probable.  That goes back to 1670 or 1700 years.  But the
     first definition is illegal carrying away of a person.  Which one of
     those are you --
         DR. SEALE:  Where is this?
         DR. KRESS:  It is the fourth definition, you need another
     dictionary.
         DR. FONTANA:  Abductive reasoning.
         DR. SEALE:  Abductive, I see.
         DR. KRESS:  With that, I am going to declare this meeting
     adjourned.
         [Whereupon, at 1:02 p.m., the meeting was concluded.]
 

Page Last Reviewed/Updated Tuesday, July 12, 2016