United States Nuclear Regulatory Commission - Protecting People and the Environment
Home > NRC Library > Document Collections > ACRS > Meeting Schedule and Related Documents > 1999 > Human Factors - September 19, 1999

Human Factors - September 19, 1999

                       UNITED STATES OF AMERICA
                     NUCLEAR REGULATORY COMMISSION
               ADVISORY COMMITTEE ON REACTOR SAFEGUARDS
                                  ***
                        MEETING:  HUMAN FACTORS
     
                        U.S. NRC
                        Conference Room 28-1
                        Two White Flint North
                        11545 Rockville Pike
                        Rockville, Maryland
     
                        Friday, November 19, 1999
     
         The committee met, pursuant to notice, at 8:30 a.m.
     MEMBERS PRESENT:
         GEORGE APOSTOLAKIS, Chairman, ACRS
         DANA A. POWERS, Member, ACRS 
         THOMAS S. KRESS, Member, ACRS
         JOHN J. BARTON, Member, ACRS
         JOHN D. SIEBER, Member, ACRS
         MARIO V. BONACA, Member, ACRS
         ROBERT E. UHRIG, Member, ACRS
         ROBERT L. SEALE, Member, ACRS.                         P R O C E E D I N G S
                                                      [8:30 a.m.]
         DR. APOSTOLAKIS:  The meeting will now come to order.
         This is a meeting of the ACRS Subcommittee on Human Factors. 
     I'm George Apostolakis, chairman of the subcommittee.  The ACRS members
     in attendance are Mario Bonaca, John Barton, Robert Seale, Dana Powers,
     Jack Sieber and Tom Kress.
         The purpose of this meeting is to review a proposed revision
     to NUREG 1624, Technical Basis and Implementation for a Technique for
     Human Event Analysis, ATHEANA, period and assist staff research
     activities related to human reliability analysis, pilot application of
     ATHEANA to assess design basis accidents and associated matters.  The
     subcommittee will gather information, analyze relevant issues and facts
     and formulate proposed positions and actions as appropriate for
     deliberation by the full committee.
         Mr. Juan Piralta is the cognizant ACRS staff engineer for
     this meeting.  The rules for participation in today's meeting have been
     announced as part of the notice of this meeting previously published in
     the Federal Register on October 14, 1999.  The transcript of this
     meeting is being kept and will be made available as stated in the
     Federal Register notice.  It is requested that speakers first identify
     themselves and speak with sufficient clarity and volume so that they can
     be readily heard.
         We have received no written comments or requests for time to
     make oral statements from members of the public.  We have to recess at
     11:45, because I have to go to another meeting, and then, we will
     reconvene again at maybe 12:45, okay?  So, if you can plan your
     presentation around that schedule, that will be good.
         We will now proceed with the meeting, and I call upon Mr.
     Mark Cunningham, for a change, to begin.
         [Laughter.]
         DR. APOSTOLAKIS:  Was there ever a meeting where Mr.
     Cunningham was not the first speaker?
         [Laughter.]
         DR. APOSTOLAKIS:  We all ask.
         MR. CUNNINGHAM:  Probably one or two in the last 20 years;
     not much beyond that, it seems.
         Good morning.
         DR. APOSTOLAKIS:  And Dr. Uhrig just joined us, for the
     record.
         MR. CUNNINGHAM:  All right; on the agenda, I've got a couple
     of items to begin with this morning.  First is just an overview of what
     we're doing.  The second is topics related to international efforts. 
     I'd like to put the international efforts, to delay that a little bit
     and discuss it after the ATHEANA presentation, because I think the
     context is much better after you've heard more about ATHEANA and the way
     we're treating human errors and things, unsafe acts, I'm sorry, that
     sort of thing.
         But anyway, by introduction, we have I guess by and large
     one big topic and a couple of smaller topics to discuss this morning. 
     The big topic is the work we've been doing over the last year or so to
     the ATHEANA project to respond to the peer review that we had in Seattle
     awhile back, June, okay?  That's the main topic for the day, so we'll
     talk about that; we'll talk about the structure of ATHEANA, what the
     objectives of the project are and then have an example.
         One of the things we've been doing over the last year is
     demonstrating the model in an analysis of a fire accident scenario in a
     plant that gets involved with this self-induced station blackout, a
     SISBO plant, if you will.  After that, we'll come back and talk about
     two smaller topics.  One is a base proposal, which are basically our
     international efforts in the human reliability analysis.  We had some
     work underway for the last couple of years with CSNIs, PWG-5, Principal
     Working Group 5, and you had errors of commission; we also had a CUPRA
     program related to trying to relate risk -- bring into risk analysis the
     impact of organizational influences.  So, I'll talk briefly about those
     later on in the morning or right after lunch or something like that.
         DR. APOSTOLAKIS:  Oh, I forgot to mention that Mr. Sorenson,
     a fellow of the ACRS, will make a presentation on safety culture after
     lunch, and we would appreciate it if some of you guys stay around and
     express comments and views.  This is an initiative of the ACRS, and
     certainly, your views and input would be greatly appreciated.  So don't
     disappear after the ATHEANA presentation.
         MR. CUNNINGHAM:  We won't.  Most of us won't.
         DR. APOSTOLAKIS:  Good.
         MR. CUNNINGHAM:  With that, I'll turn it over to Katharine
     Thompson.  Katharine is the project manager of the ATHEANA project in
     the office built by two support people, John Forester from Sandia and
     Alan Kolaczkowski from SAIC.  We've got some others in the audience,
     too, but we'll get back to that in a minute.
         DR. THOMPSON:  Good morning, and it's my pleasure to be here
     this morning to discuss ATHEANA with you for the first time, I guess.  I
     know you've heard a lot about it.
         DR. APOSTOLAKIS:  We should invite you more often,
     Katharine.
         [Laughter.]
         DR. POWERS:  Well, George, I will point out that the first
     speaker before the committee usually gets asked a fairly similar
     question.
         DR. APOSTOLAKIS:  Yes; go ahead, Dana.
         DR. POWERS:  What in the world qualifies you to speak before
     this august body?
         [Laughter.]
         DR. THOMPSON:  I have orders from my manager.
         [Laughter.]
         DR. POWERS:  No, I'm serious; could you give us a little bit
     of your background?
         DR. THOMPSON:  Oh, sorry; I have a Ph.D. in industrial and
     organizational psychology.  I've been at the NRC for about 10 years.  I
     was in NRR and human factors for a few years, and then, I went as a
     project manager for the Palo Verde plant.  I've been over here in the
     research and assessment branch for about 5 or 6 years, and I've been
     working on ATHEANA for the past about 5 years.
         DR. POWERS:  What in the world makes you think that this
     body will understand anything you have to say?
         [Laughter.]
         SPEAKER:  We'll be slow in delivery.
         DR. THOMPSON:  Okay; just a brief outline of the
     presentation.  I'm going to be discussing the overview and a brief
     introduction.  Dr. John Forester will be going through the structure of
     ATHEANA and how it's done.  Alan Kolaczkowski will be talking about the
     fire application, and then, I'll be back to talk about some conclusions
     and some follow-up activities.
         We're not going to talk about the peer review in the
     interests of time, but in the back of your handout, you have all of the
     slides and discussion of the peer review, so you can look at that in
     your own time.
         DR. APOSTOLAKIS:  Unless we raise some issues.
         DR. THOMPSON:  Unless you raise some issues.
         I guess the first question that always comes is why do we
     need a new HRM method?  And so, we've talked about this and looked at
     accidents that happen in the industry and other industries, events that
     have happened, and certain patterns and things come to the surface. 
     What we're finding is that a lot of problems involve situation
     assessment; that scenarios and the events deviate from the operator's
     expectation.  Perhaps they were trained in one way on how to approach a
     situation, and the scenario didn't happen that they were trained on.
         We've seen that plant behavior is often not understood, that
     multiple failures happen that are outside the expectations of the
     operators, and they don't know how to respond to this or how to handle
     it properly.  They weren't trained on how to follow these scenarios. 
     And we also know that plant conditions are not addressed by procedures. 
     A lot of times, these things don't match.  The procedures tell them how
     to go through a scenario, but yet, the scenario isn't matched with the
     procedures at hand, so that they may do something that's not in the
     procedures; that could, in fact, worsen the conditions.
         And these types of things aren't handled appropriately in
     current ERAs and HRAs, and so, we need to address these problems with
     situation assessment and how the plan is understood by the operators.
         DR. APOSTOLAKIS:  Now, this thing about the procedures is
     interesting.  Isn't it true that this agency requires verbatim
     compliance with procedures, unlike the French, for example, who consider
     them general guidelines?
         DR. POWERS:  Guidance.
         DR. APOSTOLAKIS:  Yes; it's like traffic lights somewhere
     else.
         So how -- what are we going to do with this?  I mean, should
     the agency change its policy?
         MR. CUNNINGHAM:  We are probably not the best people to say,
     but I don't think that's the policy of the agency, to follow -- require
     verbatim compliance with the procedures.
         DR. BARTON:  George, the agency requires that you have to
     procedures to conduct operations, to handle emergencies, et cetera. 
     Some procedures are categorized in different categories:  continuous
     use, reference, stuff like that.  But there really isn't --
         DR. APOSTOLAKIS:  There is no --
         DR. BARTON:  -- a requirement that you do verbatim
     compliance.
         DR. BONACA:  Although the utilities --
         DR. BARTON:  Utilities have placed compliance, strict
     compliance, on certain groups of procedures, and they have also policies
     that say if you can't comply with the procedure, what you do:  stop and
     ask your supervisor, change the procedure, et cetera.  But I don't think
     there are any regulations that say you have to follow procedures
     verbatim.
         DR. APOSTOLAKIS:  Although we have been told otherwise,
     though.  What's that?
         DR. BONACA:  Control room procedures, however, in an
     emergency, EOPs, for example, there is following verbatim to line by
     line.
         DR. APOSTOLAKIS:  But these are the ones that Katie is
     talking about, right?  EOPs?
         DR. BONACA:  Yes.
         DR. APOSTOLAKIS:  Not procedures for maintenance.  I mean,
     you're talking about --
         MR. CUNNINGHAM:  Again, I don't know that it's a requirement
     of the agency that they follow line-by-line the procedures.  It's my
     understanding that it's not.
         DR. UHRIG:  It was 20 years ago at one point, but that was
     believed.
         MR. CUNNINGHAM:  Okay.
         DR. BONACA:  Well, the order in which you step through an
     emergency procedure is very strict.  I mean, at least -- I don't know if
     it is coming from a regulation, but it is extremely strict.  You cannot
     -- I mean, the order of the steps you have to take; that's why you have
     the approach in the control room with three people, and one reads the
     procedure; the others follow the steps.
         DR. APOSTOLAKIS:  Yes, that's true.
         MR. CUNNINGHAM:  Again, I think all of that is very true.  I
     just don't think it's a requirement -- it's not in the regulations that
     they do that is my understanding.
         DR. APOSTOLAKIS:  You say you are not the appropriate
     people.  Who are the appropriate people who should be notified?
         MR. CUNNINGHAM:  I'm sorry?
         DR. APOSTOLAKIS:  Maybe that will do something about it.
         MR. CUNNINGHAM:  Who --
         DR. APOSTOLAKIS:  Who in the agency is in charge of the
     procedures and compliance?
         MR. CUNNINGHAM:  It's our colleagues in NRR, obviously, and
     where exactly in the last reorganization this ended up, I'm not quite
     sure.
         DR. APOSTOLAKIS:  Okay.
         MR. CUNNINGHAM:  But the issue of whether or not there is
     verbatim compliance is an NRR issue that --
         DR. SEALE:  It might be interesting to discuss this with
     some inspectors in the plant.
         DR. KRESS:  Whenever we've heard -- one of these things that
     always seems to show up.  I'm sorry; I can't talk to them and listen at
     the same time, but it seems to me like there was almost an implied -- on
     these procedures.
         DR. APOSTOLAKIS:  Yes.
         DR. KRESS:  Whether it's real or within the regulations or
     not.
         DR. BONACA:  But that certainly has been interpreted by now
     by the licensees.  I mean, for the past 10 years, especially -- even the
     severe accident guidelines, in some cases, where you look at the
     procedures, they are very strictly proceduralized, I mean.  And you
     check to see that people do not even in the simulator room do not invert
     the order of the stuff.
         DR. THOMPSON:  Yes, but a lot of that came from the
     analysis, because following the procedure requirement, it's the next
     step that you must deal with, I don't ever recall a regulation requiring
     verbatim compliance.  We had company policy about certain procedures.
         DR. APOSTOLAKIS:  Okay.
         DR. THOMPSON:  Okay; so what we know from all of these
     reviews of accidents and events is that situations in the context
     creates the appearance that a certain operator action is needed when, in
     fact, it may not be and that operators act rationally; they want to do
     the right thing; they try to do the right thing, and sometimes, the
     action is not the appropriate action to take.  The purpose for ATHEANA,
     then, is to provide a workable, realistic way to identify and quantify
     errors of commission and errors of omission.
         There are three objectives of ATHEANA.  First is to enhance
     how human behavior is represented in accidents and near miss events.  We
     do this by looking at the decision process involved, how people -- their
     information processing abilities and how they assess a situation, and we
     also integrate knowledge from different disciplines.  We look -- we have
     technology factors, engineering risk assessments.  We try to incorporate
     many different areas of knowledge there.
         DR. POWERS:  I guess I'm struck by how this view graph would
     have been written by somebody -- who developed human error analysis
     methodologies they use now.  They probably would use this view graph and
     just change the title, right?  Everybody that advances the human -- our
     reliability analysis program says he's going to make it realistic; he's
     going to integrate perspectives of ERA with plant engineering,
     operations training, psychology, risk-informed and have insights.  I
     mean, this is true of any conceivable human error analysis.
         MR. CUNNINGHAM:  In theory.  Now, we could go back perhaps
     in another session and talk about how much did other methods really
     accomplish this, and I think what you see, and you hear stories of how,
     in the poorer qualities HRAs, if you will, how this is implemented in a
     way that, in fact, the issues such as psychology and operations and
     training and things like that are handled on a rather -- one way to put
     it is a crude way, and one way would be just a mechanical way or
     something like that.
         DR. POWERS:  You know, I mean when you look for things like
     your hallowed Navier Stokes equations, people come up with --
         DR. KRESS:  Hallowed, not hollowed.
         DR. POWERS:  That's right, hollowed.
         [Laughter.]
         DR. POWERS:  The fount of all wisdom, and you call it the
     big bang; everything else was just thermohydraulics.
         [Laughter.]
         DR. SEALE:  And a little chaos thrown in.
         [Laughter.]
         DR. POWERS:  You know, in your equations, you say, well,
     we'll make an approximation.  We may have zeroeth Ns, and you can see
     that there is no dimensionality in the zeroeth approximation, and then,
     you have first order ones and second order ones and third order ones,
     and it's very clear when somebody is getting more realistic and
     incorporating more terms.  How am I going to look and see that this
     ATHEANA program is more realistic?  You know, what is it that says
     clearly that this is more realistic than what was done many, many years
     ago for the weapons programs?
         MR. CUNNINGHAM:  I guess in my mind, there would be a couple
     of clues.  I guess one would be how well we can mimic, if you will, or
     reproduce the real world accidents that Katharine started talking about,
     and again, those are the accidents that are, if you will, I think of
     them as the more catastrophic accidents.  If you look back and see,
     investigate human performance in catastrophic accidents, how well does
     this model -- I don't want to say predict but work with those types of
     events?
         DR. KRESS:  You're not talking about neutral.
         MR. CUNNINGHAM:  No, I'm talking about in general.  I can
     think of --
         DR. KRESS:  Can you transfer that technology to technology?
         MR. CUNNINGHAM:  Yes, I think you can, and that's kind of
     one of the subtle, underlying presumptions is that the human performance
     in catastrophic accidents can be translated across different industries,
     highly complex, high-tech industries, if you will:  aircraft, chemical
     facilities and that sort of thing.
         DR. APOSTOLAKIS:  I think there is a message here,
     Katharine:  use your judgment as you go along, and skip the view graphs
     that are sort of general and focus on ATHEANA only.  Do not raise
     anything until you come to the specifics.  Otherwise, you're going to
     get discussions like this.
         [Laughter.]
         DR. APOSTOLAKIS:  So can you go on, and we'll come back to
     these questions?
         DR. THOMPSON:  Skip the next one, John.
         This is just to show you the basic framework of ATHEANA and
     to underscore again -- well, we use different ones here; that's the left
     part.  Psychology, engineering -- this is something we've been working
     on.  The left-hand side shows you the elements of psychology, human
     factors engineering that are folded into the framework.
         DR. APOSTOLAKIS:  Go ahead.
         DR. THOMPSON:  And then, it flows into the PRA logic models
     and the rest.  You've seen this before.  John is going to talk more
     about this in the future, so I don't want to spend too much time on this
     right now.
         DR. APOSTOLAKIS:  I have a couple of comments.
         DR. THOMPSON:  Okay.
         DR. APOSTOLAKIS:  I have complained in the past that
     error-forcing context is a misnomer, and then, I read your chapter 10,
     which tells me that there may be situations where the error-forcing
     context really doesn't do anything.  So I don't know why it's forcing. 
     I notice that some of the reviewers also said that it's probably better
     to call it error-producing, error -- I don't know, some other word than
     forcing, because you, yourselves say in chapter 10 that the probability
     of error, given an error-forcing context, is not one, may not be one.
         DR. THOMPSON:  Right.
         DR. APOSTOLAKIS:  Second, I don't understand why you call
     them unsafe actions.  I fully agree that the human failure event makes
     sense, but until you go to the human failure event, you don't know that
     the action is unsafe.  I mean, you insist -- in fact, you just told us
     -- that people aren't rational, and I'm willing to accept that.  So the
     poor guy there took action according to the context, which led to a
     human failure event.  So I don't think you should call it unsafe.  I
     mean, human actions -- don't you think that that would be a better
     terminology?
         And then, finally, coming back to Dr. Powers' question, I
     give you my overall impression of the report.  I think the greatest
     contribution that ATHEANA has is the extreme attention it paid to the
     plant, at the plant conditions; that there is an awfully good discussion
     of how the plant conditions shape the context.  But I must say that
     chapter 10 was a disappointment.  The quantification part, I didn't see
     anything there that really built on the beautiful stuff that was in the
     previous chapters.  In fact, it just tells you go find a method and use
     it.
         It's a little harsh, but, I mean, in essence, that's what it
     says.  I mean, I have this context.  I spent all this effort to find the
     error-forcing context.  And then, all you are telling me is now, you can
     use half.  You can use, you know, slim model if you like.  I thought I
     was going to see much more.  I mean, this thing of error mechanisms has
     always intrigued me, why you bother to use it.  And then, in chapter 10,
     you don't use it, which is sort of what I expected.  I mean, I can't
     imagine anybody quantifying error mechanisms.
         So I don't know if this is the proper place to discuss this,
     because it's jumping way ahead, but I'm just letting you know that
     chapter 10, I thought, was a let-down after the wonderful stuff that was
     in the previous chapters.
         MR. CUNNINGHAM:  Yes, I think we are getting a little ahead
     of --
         DR. APOSTOLAKIS:  Yes, okay.
         MR. CUNNINGHAM:  I mean, after John and Alan talk for
     awhile, we can come back to this.
         DR. APOSTOLAKIS:  But, I mean, one part of the answer to Dr.
     Powers is that this is really the first HRA approach that really paid
     serious attention to the plant conditions, and I think that is very,
     very good, very good, but we are really -- we are not just speculating
     now.  You guys went out of your way to see how this circle there, plant
     design, operations and maintenance and plant conditions shape the
     context.  I've always had reservations about the error mechanisms, but I
     deferred to people more knowledgeable than I.
         But chapter 10 now makes me wonder again.  So, but the
     terminology, I think, is very important.  I'm not sure that you should
     insist calling it error-forcing context when you say in chapter 10 that
     -- I don't remember the exact words but, you know, sometimes, you know,
     it doesn't really matter.  How can it be forcing it?
         Yes, John?
         MR. FORESTER:  Do you want me to comment on it?
         DR. APOSTOLAKIS:  I want you to comment on this.
         MR. FORESTER:  I suggest we come back and --
         DR. APOSTOLAKIS:  Great.
         MR. FORESTER:  -- the natural progression of the talk will
     get us to chapter 10.
         DR. APOSTOLAKIS:  Okay; fine, fine.
         MR. FORESTER:  Sometime today so --
         DR. APOSTOLAKIS:  Do you have any reaction to the comments
     on the terminology?  I mean, last time, you dismissed me.  Are you still
     dismissing me?
         [Laughter.]
         DR. THOMPSON:  We'll come back to it.
         MR. FORESTER:  We will come back to it.
         MR. KOLACZKOWSKI:  The answer is yes.
         DR. APOSTOLAKIS:  Well, then, that gives me time to find
     your exact words in chapter 10.
         [Laughter.]
         DR. APOSTOLAKIS:  Okay.
         DR. THOMPSON:  This slide going real fast.  I wanted to just
     briefly recognize the team, because they all did a wonderful, wonderful
     job, and it, again, underscores the different disciplines we've brought
     to this program.  We've got psychologists, the first three,
     specifically.
         DR. APOSTOLAKIS:  Always pleased to see names that are more
     difficult to pronounce than my own.
         [Laughter.]
         MR. KOLACZKOWSKI:  I don't see any such names here.
         [Laughter.]
         DR. THOMPSON:  He's referred to as Alan K., because I can't
     pronounce it either.
         [Laughter.]
         DR. THOMPSON:  Engineers, risk assessment experts,
     psychologists, human factors, so we've brought all of the disciplines to
     this project that we need.
         DR. APOSTOLAKIS:  By the way, I hope you don't misunderstand
     my comments.  I really want this project to succeed, okay?  So I think,
     you know, being frank and up front is the best policy.  So I must tell
     you that it was not a happy time for me when I read chapter 10.
         MR. CUNNINGHAM:  We appreciate that over the years, we've
     gotten a lot of good advice from the various subcommittees and
     committees here, and we appreciate that and take it in that vein, even
     though we may take your name in vain occasionally.
         [Laughter.]
         DR. POWERS:  We are probably in good company.
         DR. APOSTOLAKIS:  Now, you know why Mr. Cunningham is always
     there --
         [Laughter.]
         DR. APOSTOLAKIS:  -- every time we meet.  He knows how to
     handle situations like this.
         [Laughter.]
         MR. FORESTER:  Yes; I am John Forester with Sandia National
     Laboratories, and I'm, I guess, the project manager, the program
     manager.  I work for Katharine, and I'm the project leader for the team.
         DR. APOSTOLAKIS:  She's not Kitty anymore?  Is it Katharine
     now?
         MR. FORESTER:  Katharine, yes.
         DR. APOSTOLAKIS:  Okay.
         [Laughter.]
         MR. FORESTER:  For this part of the presentation, I'm going
     to discuss the structure of ATHEANA, and what I'd like to do is focus on
     the critical aspects and processes that make up the ATHEANA method.
         DR. APOSTOLAKIS:  So, you skipped the project studies.
         DR. THOMPSON:  I'm sorry; I'll get back to that at the end
     when we talk about the completion.
         DR. APOSTOLAKIS:  Okay.
         MR. FORESTER:  Okay; ATHEANA includes both a process for
     doing retrospective analysis of existing events and a process for doing
     prospective analysis of events.
         DR. KRESS:  A retrospective?  Is that an attempt to find out
     the cause?
         MR. FORESTER:  Right, an analysis of the event to find out
     what the causes were and, you know, ATHEANA has had a process or a
     structure, at least, for doing that for quite awhile, to be able to
     analyze and represent events from the ATHEANA perspective so that you
     can understand what the causes were and also, by doing that in this kind
     of formal way, you'd have a way to maybe identify how to, you know, fix
     the problems in a better way.
         DR. KRESS:  And you can use that retrospective iteratively
     to improve some of the models in the ATHEANA process?
         MR. FORESTER:  Yes; you know, the idea was that by doing
     these retrospective analyses, we learn a lot about the nature of events
     that had occurred and then can take that forward and use it in the
     prospective analysis.
         DR. APOSTOLAKIS:  But today, you will focus on prospective
     analysis.
         MR. FORESTER:  That is correct; yes, I just want to note
     that one of the recommendations from the peer review in June of 1998 was
     that we had the structure for doing the retrospective, but we did not
     have an explicitly documented process for doing the retrospective, and
     we have included that now, okay?  And we do see that as an important
     part of the ATHEANA process in the sense that, you know, when plants or
     individuals go to apply the process, they can look at events that have
     occurred in their own plant and get an understanding of what the kinds
     of things ATHEANA is looking for, sort of the objectives of it, and that
     way, it will help them be able to use the method, in addition to just
     learning about events in the plant and maybe ways to improve the process
     or improve the problem, fix the problem.
         Okay; now, we do see in terms of the prospective analysis,
     as George said, we're going to focus on that mostly today.  We do see
     the process as being a tool for addressing and resolving issues.  Now,
     those issues can be fairly broadly defined in the sense of we're going
     to do an HRA to support a new PRA, but we also see it as a tool to use
     more specifically in the sense -- for example, you might want to extend
     an existing PRA or HRA to address a new issue of concern; for example,
     maybe, you know, the impact of cable aging or operator contributions to
     pressurized thermal shock kind of scenarios or fire scenarios.  So it
     can be used in a very effective manner, I think, to address specific
     issues.
         Also, maybe, to enhance an existing HRA or, you know,
     upgrade an existing HRA to be able to -- for purposes of risk-informed
     regulation submittals and things like that.  So it can be a very
     issue-driven kind of process.
         The four items there on the bottom are essentially sort of
     the major aspects of the tool, and I'm going to talk about each one of
     those in detail, but in general, the process involves identifying base
     case scenarios; sort of what's the expected scenario given a particular
     initiator and then trying to identify deviations from that base case
     that could cause problems for the operators.
         Another major aspect of the --
         DR. KRESS:  Are those the usual scenarios in a PRA that
     you're talking about?
         MR. FORESTER:  The -- well, no, the base case is sort of --
     I'll go into more detail about what the base case scenario actually is,
     but it is what the operators expect to occur, and it's also based on
     detailed plant engineering models, okay?  So maybe you'll lift something
     from the plant FSAR, but I'll talk about that a little bit more.
         And again, another major aspect of the revised method is
     that we try to clarify the relationship between the deviations, the
     plant conditions and the impact on human error mechanisms and
     performance shaping factors.  So we tried to tie that together a little
     better, and I think we've created at least a useful tool to do that
     with.  And then, finally, the other major aspect is the integrated
     recovery analysis and quantification, and I would like to say Kitty has
     already pointed out that I'll kind of go through the general aspects of
     the process, and then, Alan is going to give us an illustration of that
     process, okay?
         [Pause.]
         MR. FORESTER:  Okay; I think as we mentioned earlier, sort
     of the underlying basis for the prospective analysis is that most
     serious accidents occur when the crew sort of gets into a situation
     where they don't understand what's going on in the plant.
         DR. APOSTOLAKIS:  Is this Rasmussen's knowledge-based
     failure?
         MR. FORESTER:  Yes, I guess it would be.  It's where the
     procedures don't maybe fit exactly; they may be technically correct, but
     they may not fit exactly, and, well, even in the aviation industry or
     any other kind of industry, what you see in these kind of serious
     accidents was that they just didn't understand what was going on. 
     Either they couldn't interpret the information correctly.  I mean, in
     principle, I guess it could have been responded to in a rule-based kind
     of way, but they didn't recognize that, so it did put them into a
     knowledge based kind of situation.
         DR. KRESS:  When I read that first bullet, I'm thinking of
     nuclear plants because it comes from the broad plan.
         MR. FORESTER:  Yes; that's true, but there have been some
     events.  I mean, they haven't led to serious events, necessarily, and
     even beyond TMI and --
         DR. KRESS:  Yes, but that's one data point.
         MR. FORESTER:  I mean, there are other events, though, that
     haven't gone to core damage or, I mean, that haven't really led to any
     serious effects.
         DR. KRESS:  But you're getting this information from --
         MR. FORESTER:  Yes, yes; okay.
         DR. KRESS:  Because in designing nuclear plants, we talk
     about conditions not understood.  We've gone to great pains to get that
     out. I'm sorry; I'll just quit talking.
         [Laughter.]
         MR. FORESTER:  It does seem, even in the nuclear industry,
     you know, there are times where people do things wrong.  I mean, it
     doesn't lead to serious problems, but people do, you know, they bypass
     SPASS --
         DR. SEALE:  You know, it really goes back to George's
     comment about human error.  Human error is a slippery slope.  It's not a
     cliff.  And, in fact, when human error occurs, the angle of that slope
     will vary from error to error, and while you may talk about TMI as a
     case where you led to an accident, I bet you you could find a dozen
     where people did something, recognized that they were on a slippery
     slope, and recovered, and that seems to me, that should be just as
     useful an analysis, an identification to do in your ATHEANA process as
     was the TMI event, because it's the process you're trying to understand.
         MR. CUNNINGHAM:  No, I think that's right; you learn from
     your mistakes.  You also learn from the mistakes you avoid.
         DR. SEALE:  And the ability to recover is important
     knowledge.
         MR. CUNNINGHAM:  Yes; there's a lot of work that's been done
     about TMI; an operator response to initial events, and as you said,
     there is still the residual that they don't understand, and that's where
     we can get into very severe accidents, even after all that training.
         DR. POWERS:  It seems to me that a double-ended guillotine
     pipe break, that's a severe accident that a crew would understand
     absolutely what it was doing in a double-ended guillotine pipe break.
         DR. KRESS:  So we are never going to have one.
         [Laughter.]
         DR. POWERS:  If we had one, you would damn well know what
     happened.
         [Laughter.]
         DR. POWERS:  You wouldn't be able to mistake it for much. 
     It seems like what you're saying may be true for accidents that are of
     real concern to us, but it's going to run counter to the DBAs.  The
     DBAs, you know what's going on, and it doesn't seem like it applies to
     the DBAs.
         MR. CUNNINGHAM:  DBAs are obviously very stylized accidents. 
     DBAs themselves are very stylized accidents, and the training, you know,
     25 years ago was fairly stylized to go with those accidents.  We've made
     a lot of progress since then in taking a step back from the very
     stylized type of approach, but you can still have accidents or events. 
     The one that comes to mind for me is the Rancho Seco event of -- I don't
     know -- the early eighties or something like that, where they lost a
     great deal of their indication; another indication was confusing and
     that sort of thing.  It's not a design-basis accident, but it was a
     serious challenge to the core, if you will.
         DR. APOSTOLAKIS:  Isn't, John, I don't see anything about
     the values of operators, the references; again, the classic example is
     Davis-Bessie, you know, where the guy was very reluctant to go to bleed
     and feed and waited until that pump was fixed, and the NRC staff, in its
     augmented inspection team report, blamed the operators that they put
     economics ahead of safety.  The operators, of course, denied it.  The
     guy said, no, I knew that the pump was going to be fixed, but isn't that
     really an issue of values, of limits?  It's a decision making problem.
         MR. CUNNINGHAM:  Right.
         DR. APOSTOLAKIS:  Where in this structure would these things
     -- are these things accounted for?  Is it in the performance shaping
     factors, or is it something else?
         MR. FORESTER:  Well, one place it comes through is with the
     informal rules.  We try to evaluate informal rules.  And if there's sort
     of a rule of, you know, we've got to watch out for economics, I mean, in
     their minds, it may not be an explicit rule, but in their minds, they're
     not going to do anything that's going to cost the utility a lot of
     money.  That's one way we try to capture it.
         There's also -- we try and look at their action tendencies. 
     We have some basic tables in there that addresses both the BWR and PWR
     operator action tendencies, what they're likely to do in given
     scenarios.
         DR. APOSTOLAKIS:  But if I look at your multidisciplinary
     framework picture that you showed earlier, I don't see anything about
     rules.  So the question is where, in which box, you put things like
     that.
         MR. FORESTER:  Well, I guess it would probably be sort of
     part of the performance shaping factors.
         DR. APOSTOLAKIS:  I'm sorry, what?
         MR. FORESTER:  Well, overall, the impact of rules would sort
     of be -- or of what you're describing here, and I used informal rules as
     how we get at that in terms of the framework, it would certainly be
     covered under part of the error forcing context, essentially.
         DR. APOSTOLAKIS:  But this is the performance shaping
     factor, part of the performance shaping factor?
         MR. FORESTER:  I think it -- I guess it would also -- I'm
     not sure we'd directly consider it as a performance shaping factor.
         DR. APOSTOLAKIS:  What is a performance shaping factor in
     this context?  Give us a definition.
         MR. FORESTER:  Well, procedures, training, all of those
     things would be -- the man-machine interface, all those would be --
         DR. APOSTOLAKIS:  Technological conditions?  Is that
     performance-shaping factors?
         MR. FORESTER:  Stress and --
         DR. APOSTOLAKIS:  So the error forcing context is the union
     of the performance shaping factors and the plant conditions.  Is that
     the correct interpretation of this?
         MR. FORESTER:  That's a correct interpretation.
         DR. APOSTOLAKIS:  So clearly, values cannot be part of the
     plant conditions, so they must be performance-shaping factors.  I mean,
     if it's the union --
         MR. KOLACZKOWSKI:  I'm Alan Kolaczkowski with SAIC.  Yes, if
     you want to parcel it out, if you want to actually put tendencies of
     operators or roles into a box, it would best fit in the performance
     shaping factors, yes, but the reason why I think we're struggling is
     that we recognize that to really define the error-forcing context, you
     have to think about the plant conditions and all the influences on the
     operator in an integrated fashion, and it's hard to parcel it out, but
     if you want to put it in a box, I would say yes, it's affecting the
     performance shaping factors.
         DR. APOSTOLAKIS:  That's what the box says:  all of these
     influences --
         MR. KOLACZKOWSKI:  I understand.
         DR. APOSTOLAKIS:  -- are the PSFs, because there's nothing
     else.
         MR. FORESTER:  Well, it could be more specified, I would
     say, in the sense that part of what you're bringing up is augmented in
     the organizational factors, maybe even team issues, things like that,
     which are going to be -- which are certainly going to contribute to the
     potential for error.  Those are not explicitly captured.  In some sense,
     they could be looked at as part of the plant conditions, and they could
     also be looked at as performance shaping factors.
         DR. APOSTOLAKIS:  Now, this sector on the left, what do you
     mean by operations?
         MR. FORESTER:  Just the way they do things there, the
     procedures, their modus operandi, I guess, as to the way they run the
     plant.
         DR. APOSTOLAKIS:  Is what other people call safety culture
     there?
         MR. FORESTER:  I think that's more --
         DR. APOSTOLAKIS:  No, but that's part of it.  there's an
     error there on the left, plant design, operations and maintenance.  I
     remember the figure from Jim Reason's book, where he talks about line
     management deficiencies and valuable decisions.  Are you lumping those
     into that circle, or are you ignoring them?  I mean, the issue of
     culture --
         MR. FORESTER:  We have not explicitly tried to represent
     those yet.
         DR. APOSTOLAKIS:  But this is a generic figure, so that's
     where it would belong, right?
         MR. FORESTER:  I'm not sure I would normally necessarily
     pigeonhole it there.  It's all part of that whole -- the whole error
     force in context and what feeds into the error force in context.
         DR. APOSTOLAKIS:  But the error force in context is shaped
     by these outside influences.  It does not exist by itself.  You have
     these arrows there.
         MR. FORESTER:  Right.
         DR. APOSTOLAKIS:  So this is an outside influence, so, for
     example, if I wanted to study the impact of electricity market
     deregulation, that would be an external input --
         MR. FORESTER:  Yes.
         DR. APOSTOLAKIS:  -- that would affect the performance
     shaping factors and possibly the plant condition.
         MR. FORESTER:  Yes; that is correct.
         DR. APOSTOLAKIS:  Okay.
         MR. CUNNINGHAM:  That is correct.
         DR. APOSTOLAKIS:  So all of these are external influences
     that shape what you call error force in context.
         MR. CUNNINGHAM:  That's right.  This is a very conceptual
     description of the process.
         DR. APOSTOLAKIS:  Yes.
         MR. CUNNINGHAM:  And it's probably a little broader than
     ATHEANA is today, but again, if we could go back and get into ATHEANA as
     it is today, it might help --
         DR. APOSTOLAKIS:  Okay.
         MR. CUNNINGHAM:  -- some of the others understand what we're
     going through here.
         MR. FORESTER:  Well, given what we've identified as the
     nature of serious accidents, we think a good HRA method should identify
     these conditions prospectively, and we have several processes that we
     use to do that.  Mr. Chairman, I'm going to talk about these in more
     detail, to identify the base case scenarios, and again, these are
     conditions that are expected by the operators and trainers given a
     particular initiating event.
         They may want to identify potential operational
     vulnerabilities, and these might include operators' expectations about
     how they think the event is going to evolve.  It could include
     vulnerabilities and procedures; for example, where the timing of the
     event is a little bit different than what they expect.  The procedure
     could be technically correct, but there could be some areas of ambiguity
     or confusion possibly.
         And then, based on those vulnerabilities, at least part of
     what we use is those vulnerabilities, then try and identify reasonable
     deviations from these base case conditions, to sort of see if there are
     kinds of scenarios that could capitalize on those vulnerabilities and
     then get the operators in trouble.
         DR. APOSTOLAKIS:  So I think it's important to ask at this
     point:  what were the objectives of the thing?  It's clear to me from
     the way the report is structured and the way you are making the
     presentation that the objective was not just to support PRA.
         MR. FORESTER:  Not just to support PRA, no; I guess that's
     maybe how we started out, but I think the method itself can be used more
     generally than in PRA.  I think it needs to be tied to PRA because of
     some of the ways we do things, but no, certainly, it could be used more
     generally.
         DR. APOSTOLAKIS:  What other uses do you see?
         MR. FORESTER:  You can do qualitative kind of analysis, so
     if you're not doing a PRA, you don't need explicit quantitative
     analysis.  So with, for example, in the aviation industry, there is not
     a whole lot of risk assessment done as far as I know on what goes on in
     the airplane cockpits, but that doesn't mean that you couldn't use this
     kind of approach to develop interesting scenarios, potentially dangerous
     scenarios, that you could then run in simulators, for example, or in the
     nuclears, you can run these things as simulators and give operators
     experience with them and see how they handle the situation.
         DR. APOSTOLAKIS:  So this would help with operator training?
         MR. FORESTER:  I believe it would, yes, because there is a
     very explicit process.
         DR. BONACA:  I think we have the fundamental elements of
     root cause, for example, and so, that would help with that.
         MR. SIEBER:  I think it also helps in revising procedures,
     because you have a confusing procedure, and it doesn't really give you
     the -- but this technique helps you pinpoint --
         DR. APOSTOLAKIS:  This is an important point that I think
     you should be making whenever you make presentations like this, because
     the sole objective is to support the PRA, and I think a legitimate
     question would be are you sure you can quantify that?  Maybe you can't,
     but if your objective is also to develop health of operator training and
     other things, then, I think it's perfectly all right.
         DR. BONACA:  I think the value of this, you know, when I
     looked at this stuff is that -- was in part, I mean, some of the issues
     are based on the mindset that the operators have.  Here, you have a
     boundary where they believe they have the leeway not to follow
     procedures; for example, the issue of not going to bleed and feed was
     very debated in the eighties, because it seemed like an option was that
     severe accidents, something, and if you look at the procedure, before
     1988 or so, there was no procedure to do bleed and feed.  I mean, simply
     said, if you have a dry steam generator, do something.  One thing you
     could do was bleed and feed.
         Well, then, leave it to the judgment of the operator to do
     so.  Well, today, you go into it.  We learned that that was a mistake. 
     So we said the only thing you can do is bleed and feed, so do it, and
     you put it in the procedure now, and they follow it now, but it took a
     long time for the operators to convince them to go into it.  I mean,
     they didn't like that.
         So I'm saying that in a model like this, it would help to
     talk about some of the shortcomings.
         MR. SIEBER:  I'm pretty well convinced that even if you
     didn't have a PRA, you could profit from looking at how --
         DR. APOSTOLAKIS:  And all I'm saying is that those
     statements should have been made up front, because the review, then,
     doesn't say what you are presenting, and I would agree.  I agree, by the
     way, that this is a very valuable result.
         DR. SEALE:  It's interesting, because the utility of this
     method actually begins in terms of influencing procedures and so forth
     before it gets terribly quantitative, and yet, it's the ultimate
     objective, presumably, or let's say the most sophisticated use of it is
     when it gets quantitative so that you can use it in the PRA, but it
     strikes me that it might be when you talk about these other uses to
     actually identify the fact that in its less quantitative form, it's
     still useful --
         MR. CUNNINGHAM:  Yes.
         DR. SEALE:  -- in doing these other things, and that
     supports the idea, then, that you can evolve to your ultimate objective,
     but you have something that's useful before it ever becomes the final
     product.
         MR. CUNNINGHAM:  That's very useful.  We've talked about
     that and those types of benefits, but we could make it clearer.
         DR. APOSTOLAKIS:  Okay; can we move on?
         DR. KRESS:  Before you take that slide off --
         DR. APOSTOLAKIS:  We have two members who have comments.
         Dr. Kress?
         DR. KRESS:  The three sub-bullets under two, if I could
     rephrase what I think they mean, you start out with some sort of set of
     base case scenarios, and you look at that scenario and look at places
     where the scenario could be described wrong, and it could go a different
     way somehow all through it, so those are the vulnerabilities or place
     where it could go differently than you think or might even go different. 
     And then, the abbreviations are the possible choices of these different
     directions a scenario might go; it looks a whole lot to me like an
     uncertainty analysis on scenarios, which I've never actually seen done. 
     So it looks to me like a continuum.  I don't know how you would make
     this a set of integers.
         MR. CUNNINGHAM:  We'll talk about that later.
         DR. KRESS:  You'll talk about that later?
         MR. FORESTER:  Yes.
         DR. KRESS:  Okay.
         MR. CUNNINGHAM:  We want to get to that later.
         DR. KRESS:  Okay, so I'll wait until you do.
         DR. APOSTOLAKIS:  Mr. Sieber?
         MR. SIEBER:  I have a question.  When I read through this, I
     had a sort of an understanding of what the performance shaping factors
     were.  It's all the things that go into the operator, like training, the
     culture of the organization, mission of the crew, formal and informal
     rules, et cetera.  That to me makes this whole process unique to each
     utility, because the performance shaping factors are specific to a unit. 
     And this stuff is not transferable from one plant to another; is that
     correct?
         MR. FORESTER:  That is absolutely correct.
         MR. CUNNINGHAM:  The process would be transferable but not
     the results.  That is correct.
         MR. SIEBER:  So you just couldn't take some catalog of all
     of these potential possibilities for error and move them into your PRA,
     and anything that had any relevance to anything --
         MR. CUNNINGHAM:  The potentials and the experience base are
     useful inputs, but they are not substitutes for the analysis of an
     individual plant.
         MR. SIEBER:  Well, when you're doing, then, a retrospective
     analysis, you have to do it with the crew who was actually on the shift,
     and you will reach a conclusion based on that crew, not necessarily that
     plant; certainly not some other plant; is that correct?
         MR. KOLACZKOWSKI:  That would be the best track, correct.
         MR. SIEBER:  Thank you.
         MR. FORESTER:  So, sort of the next critical step after the
     issue has been defined and the scope of the analysis is laid out is to
     identify the base case scenario.  So we've got to go into a little bit
     more detail about exactly what we mean by base case scenario.
         Usually, the base case scenario is going to be a combination
     of the expectations of the operators as to how the scenario should play
     out given a particular initiating event.
         DR. APOSTOLAKIS:  So these are key words.  You're analyzing
     response to something that has happened.
         MR. FORESTER:  Yes.
         DR. APOSTOLAKIS:  You have a nice description in chapter 10
     of the various places where human errors may occur.  Essentially,
     they're also saying there that we recognize that the crew may create an
     initiating event, but that's not really the main purpose of ATHEANA.
         MR. FORESTER:  Right; that's -- yes, the crew could
     certainly create an initiating event, but they still have to respond to
     it once they create it.
         DR. APOSTOLAKIS:  Right; so, the understanding is what an
     event three, now, in the traditional sense, and the operators have to do
     something.
         MR. FORESTER:  Right.
         DR. APOSTOLAKIS:  Okay.
         MR. FORESTER:  Okay; so, we're looking at that kind of
     scenario, and it is the expectations for operators and trainers as to
     how that scenario should evolve, what, sort of, their expectations are,
     combined with some sort of reference analysis.  Again, that could be
     some sort of detailed engineering analysis of how this scenario expected
     to proceed, and again, that could be something from the FSAR.
         DR. KRESS:  Would the structure of ATHEANA allow you to do
     essentially what George says it doesn't do, and that is go into how an
     initiating event is created in the first place, if it's created by an
     operator acting of some kind?
         MR. FORESTER:  Well, certainly, we could --
         DR. KRESS:  Because you're starting out with normal
     operating conditions.
         MR. FORESTER:  Right; well, in terms of what the process
     does right now, it doesn't really matter whether the initiating event
     was caused by an operator or someone working out in the plant or some
     sort of hardware failure.
         DR. KRESS:  I know, but I was trying to extend it to where
     we could do some control over initiating events by looking at the --
         MR. FORESTER:  Well, we didn't explicitly consider that, but
     certainly, you could, you know, begin to examine activities that take
     place in the plant and sort of map out how those things could occur and
     then sort of use the process to identify potential problems with those
     processes that take place in the plant that could cause an initiating
     event, so it certainly could be generalized in that way.
         DR. SEALE:  That's an interesting point, because we always
     worry about completeness of the PRA, and this is another way to cut into
     the question of what are the possible scenarios that can be initiated
     and do my intervention mechanisms, cross-cut those scenarios to give me
     relief.
         DR. KRESS:  Well, my concern was initiating event
     frequencies are kind of standardized across the industry, and they're
     not plant specific.  They probably ought to be.
         DR. APOSTOLAKIS:  I think this operator-induced initiate is
     more important for low-power and shutdown point.
         DR. KRESS:  Yes, that's where I had -- that's what I was
     thinking of.
         DR. APOSTOLAKIS:  But anyway, if they do a good job here,
     that's a major advance, so let's not --
         DR. KRESS:  Let's don't push it yet.
         MR. KOLACZKOWSKI:  I was just going to comment that, for
     instance, if you could have as the base case scenario how an operator
     normally does a surveillance procedure, and then, you could look at the
     vulnerabilities associated with that in terms of how well is he trained? 
     How well is the procedure written?  Et cetera.  And then, the deviations
     would be how could the surveillance be carried out slightly different,
     such that the end result is he causes a plant trip, so we still think
     the process could apply.  It is true that in the examples right now
     provided in the NUREG, we don't have such an example, but we don't see
     why the process would not work for that as well.
         DR. APOSTOLAKIS:  Because in those cases, the fact that you
     have different people doing different things is much more important, and
     ATHEANA has not really focused on that.  Dr. Hullnager observed that,
     too.  So, I mean, the principles would apply, but it would take much
     more work, which brings me to the question:  what is the consensus
     operator model?  Are you talking about everybody having the same mental
     model of the plant?
         MR. FORESTER:  Yes; well, and the same sort of mental model
     of how the scenario is going to evolve.  So, if you ask a set of
     operators and trainers how they would expect a particular scenario to
     evolve in their plant, you would get some sort of consensus.  We try and
     derive -- the analysts would try to derive what that consensus was.
         DR. APOSTOLAKIS:  Now, again, one of the criticisms of the
     peer reviewers was that you really did not consider explicitly the fact
     that you have more than one operator, that you sort of lumped everybody
     together as though they were one entity.  So in some instances, you go
     beyond that, and you ask yourselves do they think, do they have the same
     mental model of a facility, but the so-called social elements or factors
     that may affect the function of the group are not really explicitly
     stated; is that correct?
         MR. FORESTER:  It is in some ways, in the sense that when
     you look at a crew perform, you can identify characteristics of how
     crews tend to perform at plants.
         DR. SEALE:  You can find the alpha mayo, huh?
         DR. APOSTOLAKIS:  By the way, John, you don't have have to
     have done everything.
         MR. FORESTER:  And I was going to say, we have not
     explicitly considered --
         DR. APOSTOLAKIS:  Okay; good; let's go on.
         MR. FORESTER:  -- the two dynamics, okay?
         [Laughter.]
         MR. FORESTER:  But it's not totally out of it is what I'm --
     the point I was --
         DR. APOSTOLAKIS:  I agree.
         MR. FORESTER:  Okay.
         DR. APOSTOLAKIS:  Because you're talking about consensus
     over the model.
         MR. FORESTER:  That is correct.
         DR. APOSTOLAKIS:  So it's not totally --
         MR. FORESTER:  Right.
         DR. KRESS:  I am still interested in the consensus operator
     model.  Excuse me for talking at the table but --
         MR. KOLACZKOWSKI:  That's okay; we understand why.
         DR. KRESS:  But, you know, I envision you've got two or
     three sets of operators, so you have maybe -- I don't know -- 10 people
     you're dealing with, and they each have some notion of how a given
     scenario might progress.  My question is really, do you have a technique
     for combining different opinions on how things progress into a consensus
     model?  Do you have some sort of a process or technique for doing that
     that you can defend or an interim entropy process or something?
         MR. FORESTER:  We don't have an explicit process for that. 
     I think the analysts were going to base their development of the base
     case scenario on what they understand from what the operators are
     saying; from what trainers are saying; what they see done in the
     simulators when they run this kind of initiator in the simulator, how
     does it evolve?  Again, you have reference case.
         DR. KRESS:  It's a judgment.
         MR. FORESTER:  It is a judgment.
         DR. KRESS:  Of who is putting together --
         MR. FORESTER:  Yes, it is.
         DR. KRESS:  -- your model.
         MR. FORESTER:  Yes.
         DR. KRESS:  Okay.
         MR. FORESTER:  Okay; well, there's what we see as the
     critical characteristics of the base case scenario, the ideal base case
     scenario is going to be well-defined operationally; the procedures
     explicitly address it; those procedures are in line with the consensus
     operator model; well-defined physics; well-documented.  It's not
     conservative, and it's realistic.  Again, we're striving for a realistic
     description of expected plant behavior, so that then, we can try and
     identify deviations from those expectations.
         One thing I do want to note, that part of what is done
     usually in developing the base case scenario is to develop parameter
     plots, so that if a given initiating event occurs, we try and map out
     how the different parameters are going to be behaving, but the
     expectations of the parameter behavior will be over the length of the
     scenario, because that's what the operators deal with.  They have
     parameters; they have plant characteristics that they're responding to. 
     So we try and represent that with the base case.  And not every issue
     allows that, but in general, that's the approach we want to take.
         DR. POWERS:  You have based those ideal scenarios on the
     FSAR, you have you looked at how they deviate from the FSAR?
         MR. FORESTER:  That's right; okay; the next step, then, is
     to see if we can identify potential operational vulnerabilities in the
     base case.  The idea is to try and find sort of areas in the base case
     where things are not perfect, and there could be some potential for
     human error to develop.  We look for biases in operator expectations, so
     if operators have particular biases, maybe they train a particular way a
     lot, and they've been doing that particular training a lot; the idea is
     to look at and try to identify what it is they expect and see if those
     expectations could possibly get them to trouble if the scenario changed
     in some ways, if things didn't evolve exactly like they expect them to.
         DR. APOSTOLAKIS:  So you are not really trying to model
     situations like the Brown's Ferry, where they did something that was not
     expected of them with the control rod drive pumps to cool the core?  You
     are looking for things that they can do wrong, but you're not looking
     for things that they can do right to create -- because I don't know that
     that was -- what was the base case scenario in that case, and what it is
     it that made them take this action that would raise the core?
         MR. FORESTER:  I'm not sure I understand the -- no, no, yes,
     the Brown's Ferry fire scenario.
         DR. APOSTOLAKIS:  Yes, the fire.  They were very creative
     using an alternative source of water.
         MR. KOLACZKOWSKI:  George, like PRA, this is basically, yes,
     we're trying to learn from things that the operator might do wrong. 
     This is in PRA; we try to -- we treat things in failure space and then
     try to learn from that.  But we certainly consider the things that the
     operator could do right, and particularly when we get to the recovery
     step, which we'll get to in the process, in the case of the Brown's
     Ferry fire, one of the things that the -- if we had now -- were doing an
     ATHEANA analysis, if you will, of that event, a retrospective analysis,
     one of the things you would recognize is that there was still a way out,
     and that was to use the CRD control system as an injection source, and
     that would be a recognized part of the process.
         But, yes, just like PRA, we are basically trying to find
     ways that the scenario characteristics can be somewhat different from
     the operator's expectations, such that the operator then makes a mistake
     or, if you will, unsafe act, as we call it, unsafe in the context of the
     scenario, and ends up making things worse as opposed to better, and
     then, we hope to learn from that by then improving procedures or
     training or whatever, based on what the analysis shows us the
     vulnerabilities are.  So --
         DR. APOSTOLAKIS:  The emphasis here is on unsafe acts.
         MR. KOLACZKOWSKI:  That's what I ended up trying to figure
     out is what could be the unsafe acts?  What could be the errors of
     commission or omission?  How might they come about, and then, what can
     we learn from that to make things better in the future?
         DR. SEALE:  But it still would be useful to understand what
     it takes to be a hero.
         MR. KOLACZKOWSKI:  I agree it's still part of the recovery.
         DR. BONACA:  In all of the power plants, that's what people
     refer to as tribal knowledge, especially discussions of the operators in
     the crews and among themselves:  what would you do if this happens and
     so on?  That would demonstrate the ways to get there, and in some cases,
     they lead you to success, like, for example, the example you made here,
     they would proceduralize and yet, they succeeded.
         In the other cases, I've noticed things that they have that
     they were talking about that would never lead to success; for example,
     the assumption that, you know, you dry your steam generator, and now,
     you do something to put some water in it; well, it doesn't cool that
     way.  You've got to recover some levels before you can do that.  So the
     question I'm having is is there any time to -- or is there any
     possibility?  I guess you can incorporate the type of information into
     this knowledge, right?  You would look for it.  Is there any extended
     process to look for it that you would model with ATHEANA?
         MR. CUNNINGHAM:  I think we'll come back to that.
         DR. BONACA:  The reason that I mentioned it is that that is
     -- you know, if you look at a lot of scenarios we have in accidents, it
     has a lot of that stuff going on.  As soon as you get out of your
     procedures, it comes in, and people do what they believe that --
         DR. APOSTOLAKIS:  In other terms, this is called informal
     culture.
         MR. FORESTER:  That's right, and we are taking steps to
     address those things; we certainly do.
         DR. KRESS:  I'm sorry to be asking so many questions, but
     I'm still trying to figure out exactly what you're doing.  If I'm
     looking at, say a design basis accident scenario, what I have before me
     is a bunch of signals of things like temperatures, pressures, water
     levels, maybe activity levels in the various parts of the plant as a
     function of time.  This is my description of the progression of events. 
     Now, when you say you're looking for deviations that might cause the
     operator to do something different than what -- are you looking for
     differences that might exist in those parameters?  The temperature might
     be this at this time, or the water level might be this?
         MR. FORESTER:  It might change at a faster rate than this.
         DR. KRESS:  It might change at a faster rate than you
     expect.  So, those are the indicators you are looking at.
         MR. FORESTER:  Exactly.
         DR. KRESS:  And you're looking at how those might possibly
     be different from what he expects and what he might do based on this
     difference.
         MR. FORESTER:  Right.
         DR. KRESS:  Okay; thank you.
         MR. FORESTER:  Okay; so, there are essentially several
     different approaches for identifying the vulnerabilities is what we have
     up there.  Again, we want to look for vulnerabilities due to their
     expectations.  We also want to look at a time line or the timing of how
     the scenario should evolve to see if there is any particular places in
     there where time may be very short, so if the scenarios are a little bit
     different than expected, then, there should be some potential for
     problems there, again, focusing on the timing of events and how the
     operators might respond to it.
         We also then tried to identify operator action tendencies,
     so this is based on what we call standardized responses to indications
     of plant conditions.  Generally, for PWRs and BWRs, you can look at
     particular parameters or particular initiators, and there are operator
     tendencies given these things.  We try and examine places where those
     tendencies could get them in trouble if things aren't exactly right.
         And then, finally, there is a search for vulnerabilities
     related to formal rules and emergency operating procedures.  Again, if
     the scenario evolves in a little bit different way, the timing is a
     little bit different than they would expect, there is some chance that,
     again, even though the procedures may be technically correct, there may
     be some ambiguities at critical decision points.  Again, we try and
     identify where these vulnerabilities might be.
         And once we've identified those vulnerabilities, we go to
     the process of identifying potential deviation scenarios.  And again, by
     deviations, we're looking for reasonable plant conditions or behaviors
     that set up unsafe actions by creating mismatches.  So again, we're
     looking for deviations that might capitalize on those vulnerabilities,
     and we're looking for physical deviations, okay, actual changes in the
     plant that could cause the parameters to behave in unusual ways or not
     as they expect, at least.
         In this step of the process, we're also developing what we
     call the error-forcing context.  We're going to identify what the plant
     conditions are.  We want to look at how those plant conditions may
     trigger or cause to become operable certain human error mechanisms that
     could lead them to take unsafe actions and also begin to identify
     performance shaping factors like the human-machine interface, recent
     kinds of training they had that could have created biases that could
     lead them, again, to take an unsafe action.  So part of the deviation
     analysis is to begin to identify what we call the error-forcing context,
     and ATHEANA has search schemes to guide the analysts to find these real
     deviations in plant behavior, and again, we are trying to focus on
     realistic representations.
         Part of the deviation analysis does involve, also, again,
     developing parameter plots that try and represent what it is the
     operators are going to be seeing and what is going to be different about
     the way this scenario would evolve, the deviation scenario would evolve
     relative to what they would.  So these four basic search schemes that we
     use to identify potential characteristics for a deviation scenario,
     there are similarities between these searches; there is overlap.  They
     use similar tools and resources.  There are a lot of tables and
     information in the document to guide this process, but in general, we
     recommend that each step is done sequentially, and by doing that, some
     new information could come out of each step.
         DR. APOSTOLAKIS:  John, this is a fairly elaborate process,
     and shouldn't there be a screening process before this to decide which
     possibly human actions deserve this treatment?  This is too much.  Am I
     supposed to do it at every node of the event three?  If I look at an
     event three, for example, and it has some point, you know, go to bleed
     and feed, I know that's a major decision, major human action.  I can see
     how it deserves this full treatment, but there are so many other places
     where the operators may do things here or there.
         Surely, you don't expect the analysts to do this for every
     possibility of human action.  So shouldn't there be some sort of a
     guideline as to when this full treatment must be applied and when other,
     simpler schemes perhaps would be sufficient?  Because as you know very
     well, one of the criticisms of ATHEANA is its complexity.  So some
     guidelines before you go to the four search schemes, so right after, as
     to which human actions deserve this treatment --
         MR. FORESTER:  Correct.
         DR. APOSTOLAKIS:  -- would be very helpful.
         MR. FORESTER:  Well, just a couple things.  One is there is
     an -- you know, if you identify a particular issue that you're concerned
     with, then, you can identify what particular human failure events you
     might be interested in, okay, or unsafe actions, so the issue may help
     you resolve some of that in terms of what you would like to respond to. 
     If that's not the case, if you are dealing more with a full PRA, you're
     trying to narrow down what it is you want to look at, then, we do
     provide some general guidance in there for how to focus on what might be
     important scenarios to initially focus your resources on.
         DR. APOSTOLAKIS:  But you're not going to talk about it.
         MR. FORESTER:  No, I hadn't planned on talking about that
     explicitly.  It's -- you know, I mean, you can say that, you know, it's
     the usual kind of things, I guess, in terms of looking for -- trying to
     prioritize things, you know, do you have some short time frame kinds of
     scenarios?  We have a set of characteristics; they're not coming to mind
     right at this second, but a set of characteristics that were used to
     prioritize those scenarios to focus on.
         On the other hand, I think that the process itself, the
     search for the deviation scenarios, you are reducing the problem,
     because you're trying -- you're narrowing down to the problem kind of
     scenarios.  Okay; once you've identified, you know, an initiator, for
     example, and maybe you're going to focus on several critical functions
     that the operators have to achieve to respond to that initiator, then,
     what the process does is it focuses the analyst in on the problem
     scenario.  So the process itself reduces what has to be dealt with. 
     We're not trying to deal with every possible scenario; we're trying to
     deal with the scenarios that are going to cause the operators problems.
         MR. KOLACZKOWSKI:  Let me also add, George, though, I think
     if you were going to apply this to an entire PRA, if your issue was I
     want to redo the HRA and the PRA, I would say that no matter what HRA
     method you used, that's a major undertaking.
         DR. APOSTOLAKIS:  Yes, but you are being criticized as
     producing something that only you can apply.
         MR. KOLACZKOWSKI:  I was going to say -- thanks, Ann -- I
     think you'll see, as we go through some more of the presentation and
     show you the example, the method now has become much more -- excuse me,
     methodical, and the old method that you saw in Seattle, it has changed
     actually quite a bit from that method now.  It's far quicker to use as
     long as you don't want to get caught up in all of the little minute
     documentation.  You can actually do an entire scenario, set of
     sequences, probably in a matter of hours to a day kind of thing.
         MR. FORESTER:  Once you've done a little bit of front end
     work on this.
         [Laughter.]
         MR. FORESTER:  So again, though, I do think the process
     itself -- you're looking for the deviation scenarios; I think that
     narrows the problem solving.  Is that -- you know, the prioritizations
     -- okay; okay, we have four basic searches.  The first search involves
     using HAZOP guide words to try and discover troublesome ways that the
     scenario may differ from the base case.  So again, we try and use these
     kinds of words to ask questions like, well, is there any way the
     scenario might move quicker than we expect it to or faster?  Could it
     move slower?  Could it be more, in some sense, than what they expect,
     given a particular initiator?  For example, maybe given one initiator,
     you also have a loss of instrument error.  So now, it's more than it
     was.
         Another example might be in one of our examples in the
     document is we're a small loca, close to a small loca, but it's actually
     more than a small loca; yet, it's not really a large loca either.  So,
     again, we begin to look -- one way is to use these HAZOP guide words
     simply as ways to investigate, you know, potential ways that the
     scenario might deviate from what is expected, and the -- we're
     interested in the behavior of the parameters, once again:  are the
     parameters moving faster than we expected in things like that?  So
     that's one way we do the search.
         Another search scheme is then to identify that given the
     vulnerabilities we already identified, maybe with procedures and
     informal rules, are there particular ways that the scenario might behave
     that could capitalize on those vulnerabilities?  Should the timing
     change in some way to make the procedures a little bit ambiguous in some
     ways?  That type of thing.
         Third, we look for deviations that might be caused by subtle
     failures in support systems, so this is sort of the way the event occurs
     and the way something else happens might cause the scenario to behave a
     little bit differently.  They might not be aware that there is a problem
     with the support system.  So again, a subtle failure there could cause
     them problems in terms of identifying what's happening.
         DR. APOSTOLAKIS:  Are you also identifying deviations that
     may be created by the operators themselves, by slips?
         MR. FORESTER:  Yes, I don't see why we couldn't do that.  I
     mean, to arbitrarily examine what kinds of slips are possible at this
     point in time, I'm not sure we've done that explicitly, but that's
     certainly an option in terms of doing the deviation search.
         DR. APOSTOLAKIS:  Because it has happened.
         MR. FORESTER:  That's going to get pretty complex but --
         DR. APOSTOLAKIS:  It has happened.
         MR. FORESTER:  It has happened; that's true.
         DR. APOSTOLAKIS:  That isolated systems simply by their own
     problem, but then, it takes about a half an hour to recover.
         MR. FORESTER:  Yes; I guess, you know, if we found some
     vulnerabilities or we found some inclinations or some situations where
     they might be focusing on particular parts of the control room or
     something or on the panel, part of what we do examine are performance
     shaping factors like the human-machine interface that could contribute
     to the potential for an unsafe action, and in examining those things, we
     would determine that there is some poor labeling or something that
     creates the potential for a slip, that would certainly be figured into
     the analysis.
         DR. APOSTOLAKIS:  So it could be, but it's not right now.
         MR. FORESTER:  No; I guess I shouldn't have said it that
     way.  I think it is.  As I'm saying, once you've identified potential
     deviations, part of the process is involved in looking at the
     human-machine interface; looking at other performance shaping factors
     that could contribute to the potential of the unsafe action.  So, and
     that is part of the process.  That is explicitly part of the process, to
     examine those things.  So you might, then, identify, you know, it would
     take someone knowledgeable about the way the control room panels and so
     forth should be designed to maybe identify those problems, but
     presumably, you'll have a human factors person on the team.
         DR. KRESS:  I'd like to go back to my question about the
     continuous nature of deviations.  Let's say you have a base case
     scenario, and you've identified in there a place along the time line
     that's a vulnerability and that the operator might do something, and
     then, when he does that something, it places you in another scenario
     that's different than your base case.
         MR. FORESTER:  Right.
         DR. KRESS:  And then, there are things going on after that,
     and there may be different vulnerabilities in that line than there were
     in the base case.
         MR. FORESTER:  That's true.
         DR. KRESS:  And there's an infinite number of these.  I just
     wonder how you deal with that kind of --
         MR. FORESTER:  Well, we try and deal with it during the
     recovery analysis, when we move to quantification, when we try and
     determine whether -- what the likelihood of the unsafe act might be. 
     Once they've taken that action, we then try and look at what kind of
     cues would they get, what kind of feedback would they get about the
     impact that that action has had on the plant; you know, what other
     things; how much time would be available; what other input could they
     receive in order to try and recover that action.
         DR. KRESS:  So you did tend to follow the new scenario out
     --
         MR. FORESTER:  Right.
         DR. KRESS:  -- to see what he might be doing.
         MR. FORESTER:  Exactly.
         Okay; and before I go to the last search scheme, I'd like to
     go to the next slide.  Actually, we sort of cover it on the next slide
     anyway so --
         DR. KRESS:  When you say search, what I'm envisioning is a
     person sitting down and looking at event trees and things and doing this
     by hand.  This is not automated.
         MR. FORESTER:  It's not automated at this point, no.
         DR. KRESS:  You're actually setting --
         MR. FORESTER:  It could be automated, yes, and we hope to be
     able to automate it, provide a lot of support for the process.
         DR. BONACA:  You know, I had just a question.  You know, it
     took a number of years to develop the symptom-oriented procedures, and
     they really went through a lot of steps from what you're describing
     here.  In fact, it was a time-consuming effort that lasted years, and
     they had operators involved.  Have you looked at them at all to try to
     verify, for example, the process you are outlining here?  Because they
     did a lot of that work that could be useful.
         DR. KRESS:  It sounds very similar to that.
         DR. BONACA:  Yes; I mean, they have to go through so many
     painstaking steps; you know, is this action or recommendation in the
     procedure confusing?  I wonder if you had the opportunity --
         MR. FORESTER:  Well, part of our process involves doing flow
     charts of the procedures, specifically investigate where the ambiguities
     could occur.  So we go through that process.
         Now, in terms of have we actively tried to look at, you
     know, validating the existing procedures?  No, we haven't taken that
     step.  But I think the general consensus is is that there are -- the
     procedures are not perfect; that things don't evolve exactly -- I mean,
     there can be timing kinds of issues, and there can be combinations of
     different kinds of parameters that can be confusing.
         DR. BONACA:  So I think that probably, they would exercise
     at one point with one set of procedures is what rules would be a good
     foundation for a code like this and furthermore would give you some
     indication of the strengths you may have in the process here of
     identifying things or only the key points that -- for example, the key
     points that were then central to the discussions of an owners' group, so
     that they can identify in this process what they were, and they actually
     go through the same situations.  So there is a lot that can be learned
     to verify the adequacy of a tool like this.
         MR. CUNNINGHAM:  No, that's a good point.  We'll follow up
     with that somewhere along the line here.
         MR. FORESTER:  Okay; on the next slide, one thing I wanted
     to emphasize that a major part of the first three searches while we're
     looking for the expectations, and they're using the guide words to sort
     of characterize the way the scenarios could develop, we're also trying
     to evaluate what the effect of those deviations, what the effect of the
     deviations could be on the operators.  What we wanted to determine is
     the way particular parameters behave or the way the scenario was
     unfolding, could that trigger particular human error mechanisms that
     could contribute to the likelihood of an unsafe act?
         Also, are there other performance shaping factors that could
     then, based on the characteristics of the scenario and potential human
     error mechanisms, are there performance shaping factors that could also
     contribute to that potential for an unsafe act?  So we're doing that at
     the same time we're developing the actual deviations, and one thing
     we've done, which I'll talk about here somewhere, I think -- maybe not
     -- is to try and tie particular characteristics of the scenario:  are
     the parameters changing faster than expected?  Or are two of them
     changing in different ways?  And try to identify how the characteristics
     of the scenario could elicit particular types of error mechanisms: 
     could it cause the operators to get into a place where they're in kind
     of a tunnel vision kind of state?  They're focused on the particular
     aspects of the scenario, or do they have some kind of confirmation bias
     developed, or based on their expectancies, they have, you know, a
     frequency bias of some sort.
         And then, we try and tie the behavior of the scenario, the
     characteristics of the scenario, to potential error mechanisms and then
     relate specific performance shaping factors to the potential for the
     error.  We have tried to provide some tables that make that process a
     little easier, so we have -- essentially, we have made an effort to try
     and tie those factors together much more explicitly.
         So getting that process, then, the fourth scheme, the fourth
     search, is to sort of do a reverse process.  If once you identify
     potential error types and tendencies or operator tendencies that could
     cause the human failure events or unsafe facts of interest, then, you
     simply use conjecture to try and ask are there any kind of deviations
     that could make these things occur, that have the right characteristics
     that could make these things occur.  So it's sort of coming from the
     other direction rather than starting with the physical characteristics;
     you just kind of start with the human tendencies and see if there are
     deviations that could cause that.
         So with those four searches, we think we do a pretty good
     job of identifying a lot of potential deviation kinds of
     characteristics.  Then, once that's --
         DR. APOSTOLAKIS:  Does everyone around the table understand
     what an error mechanism and an error type is?
         MR. FORESTER:  Well, error types are fairly straightforward,
     in the sense that it's just things that they could do that could lead to
     the unsafe fact, like make a wrong response; skip a step in a procedure;
     normal kinds of -- it's not a real sophisticated kind of concept there;
     it's just things that they could do.
         Error mechanisms, we're referring to, you know, essentially
     things within the human, general processing, human information
     processing characteristics, what their tendencies are, maybe some
     processing heuristics that they might use; not everything is going to be
     a very carefully analyzed, completely systematic kind of analysis. 
     They'll use bounded rationality, so people have sort of general
     strategies for how they deal with situations.  Now, most of the time,
     those kinds of situations, those kinds of strategies can be very
     effective, but in some situations, the characteristics of the scenario
     that may, where those particular tests may apply, may lead to an error,
     because they're misapplied.  So that's how we're characterizing error
     mechanisms.
         DR. APOSTOLAKIS:  Is the inclusion of error mechanisms in
     the model what makes it, perhaps, a cognitive model?  I've always
     wondered about these things.  Because you have included these error
     mechanisms, you can claim that now, you have something from cognitive
     psychology in there?
         MR. FORESTER:  Well, we have the error mechanisms.  We also
     have the information processing model, you know, the monitoring and
     detection process; the situation assessment.  The human error
     mechanisms, to some extent, are tied to those particular stages of
     processing, so, you know, we try and include all of that.  In fact, the
     use of the tables that address the error mechanisms is broken down by
     situation assessment and monitoring.
         DR. APOSTOLAKIS:  We are going very slowly.
         MR. FORESTER:  Okay; well, I'm just about done.
         Once you have identified all of the deviation
     characteristics, basically, you've got to put them all together and
     identify the ones that you think are going to be the most relevant,
     okay?
         We can look to that.  And the final slide is, again, we just
     want to emphasize that once we have identified what we consider a valid,
     a viable deviation scenario that has a lot of potential to cause
     problems for the operator, and we analyze that, we want to quantify the
     potential for the human failure event to occur or the unsafe actions. 
     We can directly address the frequency of plant conditions; standard
     systems analysis to calculate that.  We can get the probability of
     unsafe act and the probability of nonrecovery at the same time given the
     plant conditions and the performance shaping factors.
         We look at this thing in an integrated way, and we do want
     to emphasize that, that we carry out the scenario all the way out to the
     very end, in a sense, to the last moment, when they can have a chance to
     do something.  We consider everything that's going on, and then,
     ideally, in my mind, in terms of quantifying that, we have the input of
     operators and trainers.  Once you -- for example, if you can set up the
     scenario on a simulator, you can run a couple of crews through that. 
     You may not necessarily -- you're not using that to estimate the
     probability, but what I like to look for is what it is the operators and
     trainers, what they think will happen when their crews in the plant are
     sent through that scenario.
         If everyone pretty much agrees, oh, yes, you know, if that
     happened like that, we would probably do the wrong thing, then, you have
     a very strong error-forcing context, and quantification is simple.  For
     situations where that is not the case, where there are disagreements
     about what happened or not or a lot of high expectation that the actual
     unsafe actions would take place, then, we do not have a new or a special
     approach for dealing with that problem for a couple of reasons:  one,
     none of the existing approaches are completely adequate as they are. 
     For one thing, we have no empirical basis from psychology to support
     those kinds of quantifications, those kinds of estimates.  It just
     doesn't exist.
         Nor do we have an adequate existing database of events that
     we can base it on.  So, getting that situation, our suggestion for now
     is to try and use existing methods.  However, I think there are some
     things that we could do to improve our existing quantification process. 
     You know, part of what we're recommending is maybe use SLIM.  Well, the
     problem with SLIM, of course, is you don't have adequate anchors.  It's
     hard to determine what the anchors might be so you can actually use a
     SLIM kind of process.
         So one thing we'd like to investigate, I think, is how we
     could identify some maybe anchor kinds of events; we could characterize
     the events that we could pretty substantially determine what the
     probability of that event was; characterize that event in some way, at
     least maybe a couple of events on the continuum, so that then, when we
     characterize events using the ATHEANA methodology, we would know roughly
     where they fit along that continuum.  Okay; so, that's one improvement
     that we could make that we haven't made right now.
         DR. BONACA:  One question I have is that in your
     presentation, you are discussing the operator, but there are operators
     who operate.  One thing is to talk about the operators in the control
     room who have been trained on system-oriented procedures, and there,
     it's pretty clear how you can define the problem.  The problem is that
     they're following a procedure to the letter, and then, if there is some
     area where we have misdesigned the procedures, then, we mislead them,
     and they may have to initiate something that they're not used to, and
     that's all kind of stuff.
         Life is pretty clear that in the operators in the plant,
     they follow procedures to do maintenance, for example, it seems to me
     that the way you would train those kinds of operators would be very
     different from the ones in the control room, because there, they have
     their options on the procedures, on how you use them and so on and so
     forth.  Also, the operators are at the mercy of other operators doing
     other things with other systems.  I think even if you talked about how
     they would --
         DR. APOSTOLAKIS:  They haven't done that.
         MR. FORESTER:  No.
         DR. BONACA:  So when you're talking about operators, you're
     talking about the ones --
         DR. APOSTOLAKIS:  A single entity.  A single entity.
         DR. BONACA:  Yes.
         DR. APOSTOLAKIS:  In the control room.
         MR. FORESTER:  In the control room, that is correct.
         DR. APOSTOLAKIS:  I have a few comments.  This is the only
     slide on quantification?
         MR. FORESTER:  Yes.
         DR. APOSTOLAKIS:  So I will give you a few comments.
         MR. KOLACZKOWSKI:  Except for the example.
         MR. FORESTER:  Yes, we do have the example.
         DR. APOSTOLAKIS:  Okay; on page 10-7, coming back to my
     favorite theme, item two, the error-forcing context is so non-compelling
     that there is no increased likelihood of the unsafe act.  If you really
     want the error-forcing context, the error-forcing context is so
     non-compelling that there is no increased likelihood -- I really don't
     understand your insistence on calling it forcing.
         MR. FORESTER:  Well I guess --
         DR. APOSTOLAKIS:  You don't have to comment.
         MR. FORESTER:  We've also been criticized for using the term
     error at all, okay?  But the point we want to make is operators are led
     to take these unsafe actions.
         DR. APOSTOLAKIS:  Forcing -- and later on, you say that the
     probability, even if it's very relevant, will be something like 0.5.
         MR. FORESTER:  Yes; I know Strater uses error-prone
     conditions or error-prone situations, so there are other terms.
         DR. APOSTOLAKIS:  You saw here the HEART methodology.  Have
     you scrutinized it?  I'll give you some things that bother me.  On Table
     10-1, there are generic task failure probabilities, so that first one is
     totally unfamiliar; performed at speeds with no real idea of likely
     consequences, and there is a distribution between 0.35 and 0.97.
         Then, it says that in Table 10-2, HEART uses performance
     shaping factors to modify these things, and the first 10-3 is
     unfamiliarity.  So now, I have a generic description of a totally
     unfamiliar situation that I have to modify because I'm unfamiliar with
     it, and the factor is 17.  It's the highest on the table.  So I don't
     know what that means.  Either I was unfamiliar to begin with, and
     second, there is a distribution in Table 10-1.  Am I supposed to
     multiply everything by 17?  What am I doing?  Am I multiplying the 95th
     percentile by 17?  Am I multiplying the mean by 17?
         MR. FORESTER:  It's just the action.  It's just the
     probability for the action.
         DR. APOSTOLAKIS:  It's not explained.
         MR. FORESTER:  We didn't really claim to completely explain
     HEART in there.  We're trying to provide some guidance.
         DR. APOSTOLAKIS:  You need to scrutinize it, I think, a
     little better.
         MR. FORESTER:  I think you're right, and a lot of the
     categories are not always easily used.  It's not a perfect method.
         DR. APOSTOLAKIS:  And then you say that one of the modifiers
     is a need to unlearn a technique and apply another that requires the
     application of an opposing philosophy.  I'm at a loss to understand how
     you make that decision, that somebody has to unlearn something and apply
     something else.
         And then, there is a modifying factor of five if there is a
     mismatch between the perceived and the real risk.  I don't know what
     that means, risk.  If I were you, I would throw this out of the window. 
     You don't have to take all these great stuff you presented in the first
     18 view graphs and then present this thing.  You should do your own work
     here, in my view.
         As I said earlier, I thought that the quantification part is
     not at the same level of quality as the rest of the report.
         MR. FORESTER:  Agreed.
         MR. KOLACZKOWSKI:  Agreed.
         DR. APOSTOLAKIS:  You are throwing away a lot of the details
     that you took pains to explain to us.  There are no error mechanisms
     here anywhere.  And I fully agree, by the way, with what you said about
     the difficulty and, you know, there has to be some sort of a judgment
     here.  There is no question about it, and this committee will be very
     sympathetic to that, but not this kind of thing.  And this is old,
     right?  The reference is from 1988, way before ATHEANA came into
     existence.
         The thing that is really startling is that it is not very
     clear how the error-forcing context is to be used.  They mention SLIM. 
     I thought I was going to see here an application of SLIM with the
     problems that you mentioned.  Everybody has those problems; where you
     would remedy one of the difficulties or weaknesses of SLIM, namely,
     which performance shaping factors one has to consider.  And I think your
     error-forcing context or whatever you call it in the future is ideal for
     producing those.  I mean, you have done such a detailed analysis.  Now,
     you can say, well, a rational application of SLIM would require perhaps
     a set of independent ESFs or mutually exclusive -- I don't know what the
     right term is -- and these are derived from the error-forcing context we
     just defined in this systematic way, and no one will blame you for that,
     because, I mean, if you've worked in this field for a month, you can
     realize that the numbers will never be, you know, like failure rates,
     where you can have data and all of that, and the anchors, I think you
     pointed out, is an extremely important point, and perhaps you can do
     something about it to give some idea.
         But this guy who developed HEART had no heart.
         [Laughter.]
         DR. APOSTOLAKIS:  His task is unfamiliar, and then, they
     modified because I'm unfamiliar with the situation?  I mean, what is
     this?  And a factor of 17, right?  You increase the probabilities by
     approximately 17.
         [Laughter.]
         MR. FORESTER:  The only advantage to that method is this guy
     did claim that a lot of these numbers were based on empirical data.
         DR. APOSTOLAKIS:  And you know very well --
         MR. FORESTER:  Yes, well, okay --
         DR. APOSTOLAKIS:  -- what that means.
         [Laughter.]
         DR. APOSTOLAKIS:  Now, another thing -- so I'm very glad
     that you are not willing to really defend to the end chapter 10.
         MR. FORESTER:  No.
         DR. APOSTOLAKIS:  It's probably something you wouldn't be
     working on.  Okay; I'm very happy to hear that, I must say, because I
     was very surprised when I saw that.
         Now, the -- actually, some discussion is really great.  The
     figures there, there is some type of figure 10-1 is repeated twice. 
     Well, that's okay.  There was one other point that I wanted to make
     which now escapes me -- oh, this -- all the information processing
     paradigm is not here, right?  You are not really using that.
         MR. FORESTER:  Well, we're using --
         DR. APOSTOLAKIS:  All of this stuff, I didn't see it playing
     any role, at least the way it is now.
         MR. FORESTER:  It's not explicitly represented; you're
     right.
         DR. APOSTOLAKIS:  The way it is now; okay.
         MR. FORESTER:  In our minds, it's represented.
         DR. APOSTOLAKIS:  Oh, I know that the mind is a much broader
     term.
         Okay; I'm very glad for that.
         Okay; the dynamic element, and I believe Hullnagel commented
     on that, too.  We were doing in a different context some retrospective
     analysis recently at MIT of two incidents.  One was at Davis-Bessey; the
     other was the Catawba.  And what you find there is that there are some
     times, critical times, when the operators have to make a lot of
     decisions.  There's no question about it.  That's why you ask about the
     training, and, I mean, you don't really want to attack each one with a
     full-blown analysis.
         MR. FORESTER:  Right.
         DR. APOSTOLAKIS:  But in one of the incidents, I think it
     was the Catawba, there were two critical points.  One was 6 minutes into
     the accident; the other 9 minutes.  Where they had to make some critical
     decisions, and the contexts were different, there was a dynamic
     evolution.  In other words, at 9 minutes, they had more information;
     they were informed that something was going on, so now, they had to make
     an additional decision.  This specific element, the dynamic nature of
     the EFC, is not something that I see here, and perhaps it's too much to
     ask for at this stage of development, but it appears to be important,
     unless I'm mistaken.
         In other words, is the error-forcing context defined as a
     deviation from what's expected?  And for this sequence, it's once and
     for all?
         MS. RAMEY-SMITH:  No.
         MR. CUNNINGHAM:  No.
         DR. APOSTOLAKIS:  No, so you are following the evolution and
     the information that is in the control room, and you may have to do this
     maybe two or three times at two or three different --
         MR. KOLACZKOWSKI:  Exactly, George.  We present this as a
     very serial type of process.  Your point is well taken.  You really have
     to iterate and iterate.  I think in one of the examples that we have for
     the loss of main feed water event, one of our deviation scenarios is X
     minutes into the event, all of a sudden, the spray valve on the
     pressurizer is called for, and it sticks.
         DR. APOSTOLAKIS:  Right.
         MR. KOLACZKOWSKI:  That changes the scenario; it changes the
     operator's potential response, and that's carried through.  So I think
     we try to do that.
         DR. APOSTOLAKIS:  Okay.
         MR. KOLACZKOWSKI:  But clearly, we're still discretizing the
     situation into pieces of time, yes.
         DR. APOSTOLAKIS:  Okay; good, so, the dynamic nature of that
     is recognized; that's good.
         Now, a recovery in this context, my impression is it means
     recovering from errors that they have made, not recovery in the sense
     that the average person or the plant person will use it to recover from
     the incident.  They are two different things, aren't they?
         MR. KOLACZKOWSKI:  Well, ultimately, we're worried about it. 
     The core damage is the situation we're worried about.  We're ultimately
     worried about recovering the scenario.  So, as I said, it will go back
     to a success path.  But part of that recovery may be overcoming a
     previous error or unsafe act that the operator has performed.  So now,
     something has to come in that changes his mind about what I did an hour
     ago, I now recognize was a mistake, and now, I need to do this.  So,
     that could be part of the recovery, but ultimately, we're looking at
     recoveries of the scenario, yes.
         DR. APOSTOLAKIS:  So both.
         MR. FORESTER:  Both.
         DR. APOSTOLAKIS:  Okay; well, fine; if the main thing was to
     realize that you, yourselves felt that chapter 10 needed more work, so I
     have no more questions.
         DR. POWERS:  But I may still.
         DR. APOSTOLAKIS:  I'm sorry; yes.
         DR. POWERS:  As you're willing to say that the system is
     more complicated, how do we decide that it's better?
         DR. APOSTOLAKIS:  In my view, as I said earlier, the
     emphasis on context, the extreme attention that they have paid to
     context is a very good step forward.  Other HRA analyses, they do some
     of it but not -- the quantification part, I am not prepared to say that
     it is better, but I am glad to see that they are not saying that either. 
     But I think this detailed analysis that you see, there are other
     argumentations in scope, but that's expected.
         I think it's a good step forward.  It's a very good step
     forward.  If I look at the --
         DR. POWERS:  Maybe the question is just different.  The
     analysis is more complicated.  Therefore, you wouldn't have to be
     sparing in your application of it.  How would we know when this
     complicated system --
         DR. APOSTOLAKIS:  I asked them that question and
     unfortunately, they got upset.
         [Laughter.]
         DR. POWERS:  And when can I do something else, and what is
     that something else that I should do?
         DR. APOSTOLAKIS:  I think the message is very clear,
     gentlemen, that you have to come up with a good screening approach.  You
     can't apply this to every conceivable human action.
         MR. CUNNINGHAM:  That's right, and if we need to better
     describe how to do that and take that on, we've already talked about
     that as an issue in terms of next year's work or this year's work, that
     sort of thing.
         DR. APOSTOLAKIS:  Speaking of years, Hullnagel points that
     out, and I must say I'm a little disturbed myself.  This project started
     in 1992, 7 years.  Do all the members feel that this is a reasonable
     amount of time for the kind of work they see in front of them?
         DR. KRESS:  Well, we'd have to know whether this work was
     continuously done and how many people --
         DR. APOSTOLAKIS:  Mr. Cunningham is here.  He can explain
     that to us.  Were there any --
         DR. POWERS:  Well, come on, George.  It's difficult, is it
     not, to manage the NRC?  And besides, on the performance that they want
     --
         DR. APOSTOLAKIS:  No, but on the other hand, if I'm
     presented with a piece of work, I mean, how much effort has been
     expended on it is a factor in deciding whether the work is good or not.
         DR. POWERS:  It is?  That stuns me.  It certainly is not in
     the thermal hydraulics community.
         [Laughter.]
         DR. APOSTOLAKIS:  After such a powerful
     argument --
         [Laughter.]
         DR. APOSTOLAKIS:  I defer humbly to -- I withdraw my
     question.
         DR. SEALE:  The thing is that the entropy is always
     increasing, whether you do a damn thing about it or not.
         [Laughter.]
         DR. KRESS:  Only in closed systems.
         DR. BONACA:  One thing that I'd like to -- I like the
     process, et cetera.  Still, it seems to me that the process doesn't
     distinguish, for example, between the French situation and the American
     situation.  In the U.S., we have extremely detailed procedures that the
     operators will live by, and literally 10 years were expended to put them
     together, going through a process which was as thorough as this and
     involved all kinds of people, from the operators to engineers to
     everybody else.  And it seems to me that -- I'm trying to understand if
     I go to review a possible situation that develops in an accident under
     the French plan, where, in fact, there isn't a structural procedure; I
     understand how I would have used it.
         In fact, I would use it to see if the operator was
     discussing the elements and what kind of errors he will make.  I would
     make a hypothesis.  But in the U.S., I would tend to say that applied in
     a way to review the procedures that they followed to see what errors he
     would make in the U.S. and to eliminate all of those elements that are
     then focused purely on the many possible -- see what I'm trying to say? 
     I don't see any of the --
         MR. KOLACZKOWSKI:  Yes.
         DR. POWERS:  It seems to me that it would be that way
     because of the tie to the DBAs.  When you tie them to the DBAs, you've
     only got one measure.  You say, gee, I can use this just to make sure my
     -- but I think that when you go into the severe accident space, and you
     have multiple failures, this network of deviations, there is an infinite
     net that they show, and it changes character.
         DR. BONACA:  It does.  There are new procedures.  It's
     totally different.  They're not at all looking at these DBAs.  They're
     looking at the air pressure, temperature condition, et cetera, is moving
     in this direction; what are you going to do?
         DR. BARTON:  And you still have underlying error.
         DR. POWERS:  But still you have underlying a failure, and
     when you go to multiple failures --
         DR. BONACA:  You do, and it makes an assumption that, you
     know, you are going to a key procedure, because you have conditions that
     will require your ECCS to come up, for example, so there are some entry
     decisions you make, but then, especially for the EPGs, for BWRs, they're
     extremely symptom-oriented.  I mean, at some point, you forget where you
     came from.
         DR. POWERS:  Even with the symptom-oriented, you do things
     that apply to an area that ultimately get you to what's wrong.
         DR. BONACA:  I understand, but again, if it was a plant X,
     and they would use this, the first thing I would do, I would go through
     this process to understand where my procedures were invested billions of
     dollars; you're correct.  That's really what happened.  I mean, if it
     followed literally, then, it would be different in certain respects from
     the application that we make for -- where I have no prescribed way, and
     so, I may discover that that's why I led the operator in the situation
     we are in.
         Now, I don't know if this had to have a different
     perspective when you apply it to our plants, which are going through
     very structured procedures.  It seems to me every scenario would be
     still open if you review it in a way where everything is possible, and
     yet, you're ignoring the existence of the framework, which is exactly
     the pattern of the steps you're suggesting here.
         MR. CUNNINGHAM:  I guess my reaction is that I think we
     would have to kick that around among the team as to implications of the
     French style versus the American style and that sort of thing.  I just
     -- I don't think we've thought much about that.
         DR. APOSTOLAKIS:  It may require a designer approach.
         We will recess for 12 minutes, until 10:35.
         [Recess.]
         DR. APOSTOLAKIS:  We have about an hour and 5 minutes, so
     you will decide how best you want to use it.  It's yours.
         MR. FORESTER:  Okay; I think what we'd like to do is present
     an example of application of the method to some fire scenarios.  This is
     part of another task that we have to apply ATHEANA to fire scenarios. 
     We want to sort of do a demonstration of the methodology for fire
     applications, and Alan Kolaczkowski is going to present this.
         DR. APOSTOLAKIS:  We have this or we don't have this?  We
     don't have it.  No, we don't have the report.
         MS. RAMEY-SMITH:  It hasn't been written.
         MR. KOLACZKOWSKI:  My name is Alan Kolaczkowski.  I work for
     Science Applications International Corporation.  George, I'm one of the
     new team members.  I've only been around for about a year and a half so
     --
         DR. KRESS:  You're saying we can't blame you.
         MR. KOLACZKOWSKI:  Blame?  No, I guess you can't.
         Okay; well, you've heard at least in the abstract now what
     the methodology involves, and again, I think the important points is
     that -- and I think George articulated this very well -- is that we're
     really trying to look at the combination of how plant conditions can,
     based on certain vulnerabilities either in the operator's knowledge
     about how the scenario might proceed, weaknesses in the procedures,
     whatever, how those two things may come together in a way that if the
     scenario is somewhat different from, if you will, the base case scenario
     that maybe the operator is prone to perform certain actions which would
     be unsafe in light of the way the scenario is actually proceeding.
         I want to demonstrate now, actually, the stepping through
     the process that will make some of these things and some of these
     abstract ideas perhaps a little bit more concrete, step through it by
     actually showing you an example, and as John pointed out, what I want to
     do is take you through a set of a couple of fire analyses that we've
     done, and as Ann pointed out, this report is currently in process in
     terms of being put together.
         So, the first slide, what I'd like to point out here really
     is focus primarily on the third bullet, unless you have questions on the
     others, and that is if you look at current HRA methods and the extent
     that they look at fire events, and certainly, this had to be done as
     part of the IPEEE program by the licensees, et cetera, what you find is
     that a lot of the current HRA methods look at the human reliability
     portion of the issue pretty simplistically.  Most of the IPEEEs, if you
     look at them, what they've done is they've taken their human error
     probabilities from the internal events, and they might put a factor of
     five on it and say, well, the stress is probably higher because there's
     a fire going on, and there's a bunch of smoke, et cetera, and that's
     what we're going to use for our human error probabilities.
         And there really is, for the most part, not a hard look at
     what is the fire doing?  How is the equipment responding?  Might some of
     those responses be erratic?  How might that change the way the operator
     responds during the scenario, et cetera?  That kind of look at what the
     human is doing is typically not looked at.  It's treated pretty
     simplistically, for the most part.  And so, we thought that this was an
     error that would be very fruitful for ATHEANA to look at in order to
     look at the context of fires and how scenarios from fire initiators
     might affect the way the operators will respond as the fire progresses
     and so on and so forth.  So that's kind of why we looked at this.
         DR. APOSTOLAKIS:  What is SISBO?
         DR. POWERS:  Self-induced station blackout.
         DR. APOSTOLAKIS:  What?
         DR. POWERS:  Self-induced station blackout.
         MR. KOLACZKOWSKI:  I'm going to describe that in the next
     slide, I believe.
         So we decided that this was a pretty fruitful area to look
     at, and that's why we chose this one as a good example to present here
     in front of the committee.
         DR. POWERS:  Do we have a good phenomenological
     understanding of how the fire affects equipment and other things?
         MR. KOLACZKOWSKI:  I guess I don't know how to measure good. 
     I think we have some general ideas, but that's part of the problem is
     that fires can affect equipment in many, many different ways, which can,
     therefore, make scenarios be somewhat different than what we expect, and
     it's these kinds of deviation scenarios that we're talking about.
         MR. CUNNINGHAM:  In parallel with our work on human
     reliability analysis, we have a separate program that's looking at the
     issue of modeling of fires in risk analyses.
         DR. POWERS:  They repeatedly tell me that they can't really
     predict what -- that that's why their research needs to go on --
         MR. CUNNINGHAM:  Yes.
         DR. POWERS:  -- is because they don't know what kinds of
     things will happen to equipment.
         MR. CUNNINGHAM:  That's right; both are viable subjects,
     reasonable subjects for research.
         DR. POWERS:  And I have had the licensees in saying the
     vicious and evil thing about the NRC staff, because they take too
     conservative a position on fire-induced changes and things like that.
         MR. CUNNINGHAM:  Again, we have another program.  Part of
     the reason for picking the fire example was to try to bring some of
     these -- bring the two programs a little closer together.
         MR. KOLACZKOWSKI:  The next slide, as you're going to see in
     a moment, we picked two particular scenarios to look at, but first, you
     have to understand a little bit what the plant design is like, at least
     in a general sense, for dealing with fires and what this SISBO concept
     is, because we did decide to look at a so-called SISBO plant.
         This cartoon, if you will, is meant to at least show you
     what the separation is typically like in a nuclear power plant for
     dealing with fire, and then, as I said, I want to introduce the SISBO
     concept.  You can see here that if you look at the cabling equipment in
     the plant and so on, typically, for Appendix R purposes and so on, in a
     very simple, two-division kind of plant, you end up with separating the
     cables in the various cable trays and having certain walls and rooms and
     fire barriers, et cetera, between equipment such that all the division A
     equipment is located somewhat separately and at least are protected from
     a fire standpoint from division B equipment, and we see that displayed
     in this cartoon.
         Of course, plants have now a remote shutdown panel
     associated with them.  Usually, that remote shutdown panel has a limited
     amount of instrumentation and controls associated with it for
     controlling one of the divisions of equipment for shutting down the
     plant safely should the operators have to leave the main control room,
     which might be the case for fire in the control room area as well as, as
     you'll see in a moment, if it's a SISBO plant, there are other reasons
     why they may leave the main control room as well.
         So anyway, we have this standard separation between the two
     divisions, and that separation, to the extent possible, is maintained
     all the way up through the cable spreading room, the relay room, the
     main control room, where we have the various fire barriers and so on and
     so forth.  As I indicated, we have this remote shutdown panel, the idea
     being that if we need to leave the main control room, we go down to the
     remote shutdown panel as well as other local areas in the plant, and we
     operate this -- what's called dedicated areas of equipment or division A
     equipment, and typically, what's done is that there is a set of switches
     there on the remote shutdown panel, and that's just shown as one switch
     in this little cartoon, that are thrown such that we become now isolated
     from the main control room so that shorts, hot shorts or other
     electrical problems that might be propagating up through the main
     control room won't come down to the remote shutdown panel.
         And now, we hook in the remote shutdown panel directly with
     the equipment out in the field, and then we safely shot down the plant
     from there.  What's unique about the SISBO idea is that some plants, in
     order to respond to various requirements in Appendix R and other
     fire-related requirements for dealing with potential hot shorts and so
     on, have taken on this so-called self-induced station blackout approach,
     in which basically, what happens is the plant, once the fire gets so
     severe that they feel that they are losing control of the equipment
     because of erratic behavior, potentially because of hot shorts,
     whatever, they essentially de-energize all of the equipment in the
     plant, and at the same time, energize only either the alternate area
     equipment if the fire is in a dedicated area zone, or they would go down
     to the remote shutdown panel and operate the dedicated area of equipment
     if the fire is in an alternate equipment zone and then re-energize that
     equipment off that diesel.  And then, they operate just that particular
     set of equipment to safely shut down the plant.
         So essentially, they put the plant into a loss of power
     situation and then re-energize either A-bus or B-bus and then use just
     selected equipment off of that bus that they think is not being affected
     by the fire.  Of course, the advantage of that is that now, hot shorts
     can't occur in the A equipment, let's say if that's where the fire is,
     because you've got it all de-energized, and so, you won't have a
     spurious opening of the PORV or something like that that could make the
     scenario much worse.  So that's kind of the concept behind the SISBO
     idea.
         Next slide.  Now, for illustrating the ATHEANA process, what
     we've done is we've reanalyzed two fire scenarios that have been
     previously analyzed in an existing PRA.  This just highlights what the
     two fires are and what the potential effects of the fires are for this
     particular plant.  One is an oil fire in the auxiliary feed water system
     pump B room.  This is for their classification, a so-called alternate
     fire area, and you can see that if the fire does become significant, the
     effects are quite severe.  Four out of four of the non-safety busses
     become affected and would potentially have to be shut down.  You also
     potentially lose the division B 4160-volt safety bus.  That's the safety
     bus for the various safety loads.
         Of course, you lose, obviously, pump B of auxiliary feed
     water, and it turns out in this particular plant, because of where the
     cabling is located, if you had a severe fire in this room, you would
     also affect the ability to operate and control the turbine pump.  This
     is a three-pump system that has two motor pumps, A and B, as well as a
     turbine pump.  This fire would affect one of the motor pumps as well as
     the turbine pump.
         If this situation got this severe, the expectations,
     according to the procedures, would be that you would leave the main
     control room, and then, you would shut down using limited division A,
     that is, dedicated equipment, from the remote shutdown panel, and there
     is an EOP, so called FP-Y, that governs how this is actually
     implemented.
         The other fire is, as I indicated there, a fire concerning
     certain safety busses, and it turns out these safety busses are located
     in the same area, room, if you will, that the remote shutdown panel is
     located.  So this is a so-called dedicated area fire, and again, if this
     fire got severe, such that the feeling was that the operators were
     losing control of the plant, the expectations, per the EOP, would that
     -- well, first of all, you would lose the division A busses and the
     ability to use that diesel and its various loads, and the expectations
     would be you would shut down using division B equipment or so-called
     alternate equipment.
         In this case, they would stay in the main control room to
     operate that equipment, but they're still going to de-energize
     everything and then only energize the B busses and then use the B
     equipment.  So you're still going into a self-induced loss of power
     situation.
         Lastly on this slide, I wanted to indicate what the current
     PRA insights are about the human reliability performance in these two
     fires.  And if you look at what are the sort of dominant lessons learned
     from the HRA analysis for this existing PRA, those are highlighted there
     on the third slide, that there is a potential for a diagnosis error to
     even enter the right EOP, either EOP-Y if it's an alternate area fire or
     EOP-Z if it's a dedicated area fire, so notice that one of the things
     they have to know is where is the fire in order to know which EOP to
     enter.
         And the reasons why the existing human reliability analysis
     technique says that a diagnosis error might occur are indicated here: 
     either the operator would misread or miscommunicate the cues to enter
     the procedure, or he might just plain skip the step and not enter the
     procedure or might misinterpret the instruction regarding when to enter
     the procedure.  Those were highlighted in the PRA as possible reasons
     for why he might make this diagnostic error.
         The more dominant errors, however, in the HRA, if you
     actually look at the quantified results:  they claim that it's much more
     likely the operators will make mistakes in actually implementing the
     EOPs themselves, just because they're very complex and so on and so
     forth.  There are a lot of steps involved.
         Most of the errors, they claim, will be as a result of
     switch positioning errors or just because of the fact that they may omit
     certain steps because they're in a high stress situation.  So that's
     kind of what you learn from the existing PRA if you look at the human
     reliability analysis for these two fires.
         DR. POWERS:  The regulation is that they're required to be
     able to shut this plant down, so you're going to look at carrying out
     that requirement.
         MR. KOLACZKOWSKI:  That is correct; we don't look at the
     errors associated with still safely shutting down, but look at it now
     from an ATHEANA perspective and say that if we think about the context
     of these fires a little more, what might we learn that might be new,
     more lessons learned that we could apply to ways to make the operators
     better-prepared for dealing with these fires than just simply, well,
     they might skip the step.  Well, what are we supposed to do about that? 
     I guess we could say increased training, maybe, but we want to see if
     ATHEANA can provide some additional insights as to how the operator may
     not bring the plant back to a safe condition.
         Yes?
         DR. APOSTOLAKIS:  Who did the PRA you are referring to?
         MR. KOLACZKOWSKI:  I'm sorry?
         DR. APOSTOLAKIS:  The PRA, the existing PRA.  Is that the
     utility?
         MR. KOLACZKOWSKI:  It is a -- yes, it's an IPEEE from a
     licensee.
         DR. APOSTOLAKIS:  Okay.
         MR. KOLACZKOWSKI:  Now, John indicated that one of the first
     things we do after really defining the issue, which, in this case, is
     how can we learn better how the operators might make mistake given these
     two kinds of fires and, therefore, take from that lessons learned and
     ways to improve operator performance given these kinds of fires, once
     we're able to identify that issue, one of the first things we have to do
     is try to understand how does an operator, how does he think these two
     fires would normally proceed?
         This is that defining the base case scenario step.  This is
     trying to come up with that collective operator mindset as to what his
     expectations would be given that these fires actually occurred, and our
     base case is essentially summarized in this and the next slide, and let
     me just kind of quickly go through this, and then, if you have any
     questions, we can proceed to those.
         Of course, one of the first things that would eventually
     occur most likely is once the fire has happened, let's assume for the
     moment that it happens without a person being in the room at the
     particular time, et cetera; it's going to start to affect some
     equipment, et cetera, but one of the first things that will probably
     occur is that we will eventually get a fire detection alarm.  There are,
     at this plant, multiple alarms for detecting smoke, et cetera, in these
     rooms and so on, so we would expect that fairly early in the scenario
     that one of the first indications would be this fire detection alarm.
         The operators then enter what is called EOP FP-X upon a fire
     detection alarm, which basically provides the initial things that they
     do for dealing with once a fire has been detected in the plant.  One of
     the first steps in that procedure is they ask another operator out in
     the plant to go and visually validate that there actually is a fire,
     that this is not a spurious or false alarm, and the procedure almost
     reads as though the intent is that they don't do too much more until
     that validation comes back.
         Let's assume they do get the validation.  Then, the fire
     brigade is then assembled.  It's called on.  And one of the things they
     do is they unlock the doors to the suspected area to make sure that the
     fire brigade is going to have fairly easy access to that area, et
     cetera, and there's a general notification over the Gaitronic system
     that there is a fire in the plant and those kinds of things.
         Now, during this time, especially if the fire is not yet all
     that severe, the plant is still running.  It's just humming along,
     running along fine, and, in fact, the main control room staff are
     attempting to just maintain the plant online and under proper control
     while the fire brigade is now getting assembled and getting ready to do
     their thing.
         We expect that as time proceeds, and let's say the fire
     brigade is finally getting down there, perhaps entering the room, et
     cetera, but if the fire is getting to the point where it's approaching
     the severities that I talked about in the previous slides, then, we're
     going to start seeing erratic operation of some of the
     normally-operating equipment.  Perhaps we're going to start seeing flow
     acting erratically; maybe if you have current indications on certain
     pumps, like the AFW pump, you might begin to see erratic indications of
     the current or maybe voltages on certain busses, depending, again, on
     which cables are affected and when that occurs.
         DR. POWERS:  Isn't it much more likely that the things that
     are going to be affected are the instrumentation and not the core
     itself?
         MR. KOLACZKOWSKI:  That is true, too.  I mean, it depends
     on, looking at in each individual room, how much control and power
     cables there are versus how much instrumentation cables.  Certainly, the
     AFW pump is instrumented to some degree, but the flow instrument for
     flow going to the steam generator might be in an entirely different
     room, and it's unaffected at all.
         So it's very, very plant-specific, obviously, as to what the
     specific effects are, but we would generally say erratic operation of
     equipment, and certainly, your point is well-taken, Dana, of some
     indications may be possible.  But the point is the plant isn't
     necessarily going to trip right away, and in a lot of small fires, as we
     know, the plant ran through the entire scenario just fine.  They put the
     fire out, and that's it.
         Now let's assume for the --
         DR. POWERS:  There is nothing at this point to indicate to
     trip this plant.
         MR. KOLACZKOWSKI:  I'm sorry?
         DR. POWERS:  There is nothing at this point --
         MR. KOLACZKOWSKI:  No, FP-X does not require them at this
     point yet to trip the plant.  And, in fact, they will try to maintain
     plant operation per their procedure at this plant.
         So we have potential erratic behavior of some of the normal
     operating equipment, perhaps some of the indications.  Notice that
     certain standby equipment may also be affected; for instance, that
     turbine pump, the turbine auxiliary feed water pump, and it may also,
     maybe, have cables associated with that pump's control that are burning,
     and yet, they will have no necessarily idea that that pump has been
     affected, because they haven't asked it to try to work yet.  They're
     still running the plant; feed water plants are still on.  They'd have no
     idea that the AFW turbine pump has now become inoperative.  They won't
     know that until they try to use it.
         So just recognize that there is some missing information
     with their situation assessment as to how bad this fire is, okay?  Now,
     also during this time; let's assume the fire brigade is trying to do its
     job.  There is going to be some diversion of attention as well, because
     there's going to be periodic communication between the fire brigade and
     the main control room staff.  One of the things they do is hand out
     radios, et cetera, and there's going to be talking back and forth:  how
     are you coming?  What's the situation?
         Maybe the brigade is saying, well, we haven't entered the
     room yet; there's an awful lot of smoke, et cetera, et cetera.  There's
     going to be some diversion of attention dealing with the fire brigade as
     well as trying to just make sure that the plant is okay.  That's part of
     the overall situation.
         Let's assume for the moment that the conditions get even
     worse.  Either the fire brigade is having trouble getting out the fire
     or whatever.  At some point, if enough erratic behavior is occurring,
     and we're actually beginning to have a lot of difficulty in actually
     controlling the plant, maintaining pressurizer level, maintaining feed
     water flows, whatever, that's when the judgment occurs for the operators
     to then enter either EOP-FP-Y if the fire is in an alternate zone or
     EOP-FP-Z if the fire is in a dedicated zone, and at that point, one of
     the first steps in that procedure is, yes, trip the plant, okay?
         Secondly, then, what they do after that is they, in the
     procedures, is they basically isolate the steam generators, and then,
     they leave -- if they have to, if they're in EOP-FP-Y, they have to
     actually leave the main control room, and then, they start the
     de-energization process, and that's when they actually are pulling
     fuses, pulling breakers out locally in the plant, et cetera, and
     essentially putting the plant into a self-induced blackout.
         Simultaneously, they are -- and they actually take the crew
     and separate them up into about three or four different areas of the
     plant, so you have to also recognize that the crew is no longer working
     as a unit in one room anymore; they're now located in various areas of
     the plant talking on radios.  One guy is over pulling fuses in a DC
     panel; another person is over pulling breakers in an AC bus, et cetera. 
     So they're acting now certainly still in communication but as separate
     entities.
         They de-energize the various buses in the plant, and then,
     they bring on the appropriate bus, depending on whether the fire is in
     an alternate or dedicated zone, and then begin to bring on manually the
     equipment they're going to use to safely shut down the plant.  Now, in
     the base case scenario, even if the fire got this bad, the expectations
     of the operator would be, okay, we enter the right EOP procedure; we go
     through its implementing steps; we carry it out; we eventually
     restabilize the plant.  Sometime during this time, the fire eventually
     gets extinguished, and the scenario is over.
         So in a general sense, this would be sort of the
     expectations, even if the fire got fairly severe, as to what the
     operators' expectations would be as to how the scenario would proceed,
     and that's going to be our starting point to then build deviations on
     that scenario.
         One of the things we'll also do early on in the process is
     we try to focus on, well, what human failure event or events and what
     particular unsafe acts are we really interested in analyzing for?  And
     this slide is meant to attempt to try to summarize really the specific
     human failure event that we're looking at, which is really failure to
     accomplish heat removal.  Let's say we get to the point where they have
     to trip the plant, and now, they have to bring it back into a
     stabilized, cooled state, recognizing they may have to leave the main
     control room and go through this de-energization process and so on, and
     what if they fail to carry that out correctly for one reason or another?
         Taking that overall human failure event and really breaking
     it down into, as we have here, three separate unsafe acts that we're
     really going to be trying to analyze and determine, if we can, the
     probability of that occurring.  UA-1 is really very much closely
     associated with that diagnostic error I talked about in the original
     PRA; that is, one unsafe act could be the failure to enter the right EOP
     or wait too long to enter that EOP, to the point where, perhaps by that
     point, so much equipment damage has occurred; maybe hot shorts have also
     occurred that they have essentially lost all control of the plant and
     the ability to even bring it back to a cooled and safe and stable safe.
         DR. APOSTOLAKIS:  What's too long?  Who determines the
     length of fire?
         MR. KOLACZKOWSKI:  For purposes of this illustration, we
     haven't tried to necessarily answer that question, George.  It would
     obviously depend on the specific plant; how big the fire grows; how fast
     the equipment gets affected.  You know, you could do that by doing
     various com burn runs for that room and so on and so forth.  It would be
     very plant specific.  I mean, I could try to give you some general
     ideas, I suppose, but we have not tried to address that specifically in
     this illustration.
         DR. APOSTOLAKIS:  Okay; but in terms of the base case
     scenario --
         MR. KOLACZKOWSKI:  Yes?
         DR. APOSTOLAKIS:  -- do you have an idea as to how much time
     they have?  I thought that was one of the premises of defining the base
     scenario.
         MR. SIEBER:  It depends on how big the fire is.
         DR. APOSTOLAKIS:  Well, okay, but they have to have some
     sort of an idea how quickly they have to do it.
         MR. KOLACZKOWSKI:  I agree that as part of the base case
     scenario, you would describe for a specific plant how long do they think
     it would take before this fire would get that large and so on, and
     that's going to be a very plant-specific answer.
         DR. APOSTOLAKIS:  I see Jack is shaking his head here.
         MR. SIEBER:  I don't think you can do it.
         DR. APOSTOLAKIS:  So how will the operators act?
         MR. SIEBER:  You act as quickly as you can without making
     any mistakes.
         [Laughter.]
         DR. POWERS:  What's happening in reality is that you've got
     something, the fire alarm or something.  You've got some people doing
     things.  They're talking to you about what they're finding.  In the mean
     time, you're going to have instruments that are telling you something is
     going on, and the urgency, well, it's urgent to get the fire out, but
     it's not urgent to take the plant, to trip the plant until you get
     something urgent.  Who says that?  It's the instrumentation board or the
     people that are talking about it.  They say the fire is very big, and we
     can't get it out with the people we've got; you're going to trip the
     plant.
         DR. APOSTOLAKIS:  And this is now on the order of minutes?
         DR. POWERS:  Minutes.
         MR. KOLACZKOWSKI:  It could be.
         DR. POWERS:  Yes; I know.  I mean, some of us are more
     incredulous than others, but maybe that's just an area that somebody is
     going to have to work on.  It's in the area of most extreme abuse, I
     think; what's already a very laborious process.
         DR. APOSTOLAKIS:  I think that's related also to the problem
     of screening at the beginning.  In other words, you really have to try
     to make this not to look like it's an open-ended process that only a few
     select people can apply.
         I have another question.  I'm confused there by the second
     paragraph.
         MR. KOLACZKOWSKI:  Okay; I was going to get to that, George.
         DR. APOSTOLAKIS:  I think we have to hurry.
         MR. KOLACZKOWSKI:  Okay; go ahead.
         DR. APOSTOLAKIS:  Triggered error mechanisms include no
     entry to procedures.  And then, it says tends to lead to unsafe acts,
     including taking no action.  I thought the mechanism was something
     different.  I agree with the last statement, but if they delay or they
     take no action, that's an unsafe act.  I just don't see how it is an
     error mechanism.
         MR. KOLACZKOWSKI:  Yes, it looks like maybe that is
     miscategorized and should be down as an error type.
         DR. APOSTOLAKIS:  Okay; so it shouldn't be classified as a
     trigger mechanism.
         MR. KOLACZKOWSKI:  I think I would agree with you, George.
         DR. APOSTOLAKIS:  Okay; I think we've got the flavor of the
     search.
         MR. KOLACZKOWSKI:  Okay.
         DR. APOSTOLAKIS:  Unless the members want to see two, three
     -- do you want to continue on to the deviation scenario development now?
         MR. KOLACZKOWSKI:  That's fine; that's fine.
         DR. APOSTOLAKIS:  Number 30?
         MR. KOLACZKOWSKI:  That's fine.
         So we go through various searches to try to come up with
     credible ways a scenario could be different, such that they trigger
     certain error mechanisms that we think will lead to the error types of
     interests, okay?  Now, we actually -- once we've gone through those
     searches, and we have some idea of credible ways that the scenario might
     deviate from the base that really sets up the potential for the unsafe
     acts that we're interested in, we then summarize those characteristics
     into a troublesome scenario or scenarios; it might be more than one,
     okay?
         In this particular case, based on what we learned on the
     searches in this illustration, we selected the following time line of
     events that would be somewhat different.  Imagine, if you will, that the
     fire detection for whatever reason was delayed, either because of
     perhaps some of the fire detection equipment not working and/or the fire
     develops very slowly, which is getting sort of to the next bullet but
     progressively.
         Also, let's say the fire brigade has trouble putting out the
     fire, although perhaps it reports back to the main control room that it
     is almost under control.  Obviously, with the kinds of things that
     that's going to do, it's going to delay the decision process; allow the
     potential for more equipment to be damaged before, in fact, the
     operational staff take action; and if they're getting reports back by
     the fire brigade saying we've just about got it out, again, the feeling
     is going to be one of almost relief and say well, we're just about out
     of this thing.
         Now, beyond the initial fire conditions, also some other
     later deviations that we're going to include in this "deviation
     scenario" is that suppose that the fire duration and progression is such
     that it gets so severe that it actually has cross-divisional equipment
     effects.  Perhaps it lasts longer than two or three hours, and
     eventually, fire barriers get defeated or whatever, and/or other good
     equipment, that is, the equipment they're going to try to use to safely
     shut down the plant, what if it fails to function, like the diesel
     doesn't start?  Those that we think are credible, realistic deviations
     in the scenario that could make the scenario much more troublesome. 
     Next slide.
         DR. APOSTOLAKIS:  So where are you using the fact that they
     may be reluctant to abandon the control room?
         MR. KOLACZKOWSKI:  Well, again, that's been recognized as
     part of one of the vulnerabilities, and the fact that we have a scenario
     now that is going to develop slowly, and also, they're going to be
     getting good reports from the fire brigade, we're basically saying
     that's going to strengthen that reluctance.  They're going to be less
     willing to leave the main control room given that's the situation,
     because they think the fire is just about out, and they're not sure what
     all the effects of the fire are, in fact, because it's progressed so
     slowly.
         DR. APOSTOLAKIS:  So that's not part of the deviation
     scenario?
         MR. KOLACZKOWSKI:  It is a reason why the deviation scenario
     is what it is.  We're saying that this kind of a scenario, as described,
     is going to strengthen or increase the reluctance factor.  The scenario
     is not the PCF.  The scenario is described in an equipment sense.
         DR. APOSTOLAKIS:  What's the PCF?
         MR. KOLACZKOWSKI:  I'm sorry; I said PCS; PSS.  The scenario
     is going to strengthen certain performance shaping factors.  In one case
     here, one of the performance shaping factors, one of the negative ones,
     is this reluctance.
         DR. APOSTOLAKIS:  So if one asks now what is the error
     forcing context --
         MR. KOLACZKOWSKI:  Yes.
         DR. APOSTOLAKIS:  How many do you have, and which ones are
     they?
         MR. KOLACZKOWSKI:  Okay; in this case, I guess we would say
     we're describing one overall context.  What you have before you on this
     deviation scenario slide, the previous slide, is essentially the plant
     conditions part of it.  The actual performance shaping factors, I don't
     think I have a slide on that, but the performance shaping factors which
     make up the other part of the context would be things like unfamiliarity
     with such a situation; reluctance to want to deenergize the plant and/or
     if necessary leave the main control room and so on and so on.
         And so, you would then describe those performance shaping
     factors, and then, together, if you say given those performance shaping
     factors and this kind of a scenario, we think we have an overall context
     which may lead to higher probabilities of not entering the procedure in
     time or carrying it out incorrectly, et cetera, those three UAs that I
     talked about.
         DR. APOSTOLAKIS:  I mean, I thought that the error forcing
     context is central to all of this.  So I sort of expected the view graph
     that said this is it.
         MR. KOLACZKOWSKI:  Probably should have stressed the
     performance shaping factors; you're right.  We only presented this --
         DR. APOSTOLAKIS:  Is it the performance shaping factors or
     the context?  Or these are part of the context?
         MR. KOLACZKOWSKI:  Yes; if you go back to the framework,
     you'll notice that the error forcing context box has in it two things: 
     the plant conditions --
         DR. APOSTOLAKIS:  Yes.
         MR. KOLACZKOWSKI:  -- and the operator performance shaping
     factors, and what we're saying is suppose the plant conditions are as
     I've described in this deviation scenario.  That's going to trigger a
     lot of those other vulnerabilities that we talked about in the previous
     step, which really become the performance shaping factors; that is, he's
     going to have a reluctance to want to deenergize the plant, et cetera,
     et cetera.
         DR. APOSTOLAKIS:  So you have a number of error forcing
     contexts by selecting from the deviation scenario development.
         MR. KOLACZKOWSKI:  Yes, you could; yes, you could.
         DR. APOSTOLAKIS:  I think that's a critical --
         MR. KOLACZKOWSKI:  You could potentially have numerous
     contexts.
         DR. APOSTOLAKIS:  You need to emphasize it and say these are
     the contexts we're identifying.
         MR. KOLACZKOWSKI:  Okay; okay, good point.
         Okay; given now we think we have a scenario that will, if it
     develops in the way that we described in the deviation scenario, we
     think along with the performance shaping factors provides us a more
     error-prone situation or error forcing context, as we call it.  One of
     the things that we also do before we really enter the quantification
     stage is think about well, what if it really did get this bad?  What are
     the potential recoveries?
         I guess just quickly, for the case where he doesn't enter
     the EOP or enters it way too late, we've assumed that if things got that
     bad, right now for this illustrative analysis, we're not allowing any
     recovery in that situation, and by the way, that's very similar to what
     was done in the existing PRA.  The existing PRA said if things get that
     bad that he never made the decision to even enter the EOP, he's not
     going to get out of this thing if the fire continues.  So we're sort of
     in line with what the existing PRA was in that case.
         If the fire grows, and it affects both the alternate and the
     dedicated equipment, which was one of the aspects of our deviation
     scenario possibilities, well, obviously, now, now, the question becomes
     what's he going to do, given he's got alternate equipment burning as
     well as dedicated equipment burning, and really, there is no procedural
     guidance for that.  He's supposed to enter one or the other case, not
     both.  So if the fire grows and affects both the equipment, or, if when
     he gets to the so-called good equipment, that is, the equipment not
     affected by the fire that randomly fails, that could occur because of --
     this is getting to your point, George -- the operator could be making
     those problems occur, not just that the equipment fails.
         This is sort of the operator inducing an initiator; in this
     case, this is the operator actually causing the reason why the equipment
     doesn't work.  Maybe he doesn't try to start it up in the right sequence
     or something like that, and so, it doesn't work properly.
         Now, we have allowed recovery for that in the analysis, and
     I think maybe the best thing I ought to do is go to the event tree,
     which is the next slide, that will show the interrelationship of the
     recovery with these unsafe acts.  This is obviously very simplistic, but
     what it's meant to do is cover really the key points that we're worried
     about in how the scenario could progress.  Notice we have the fire at
     the beginning.  Suppose the operator does not timely enter into the
     correct EOP?  That was the one that we said we're not going to allow a
     recovery for.  That's unsafe act number one.  If that occurs, we're
     going to assume for event tree purposes that that goes to core damage,
     like the existing PRA did.
         But suppose it does enter the procedure, and suppose the
     fire does not jump to separation barriers; that is, it still remains in
     only the alternate area or only the dedicated area.  And then,
     additionally, if the good equipment that he then tries to operate works,
     well, that's the way out.  That's the okay scenario he's trying to get
     to.  But if there is a problem either with the equipment working or if
     the fire, in fact, jumps over into -- let's say it starts in the
     alternate area and jumps to the dedicated area, maybe because of an
     Appendix R weakness, or maybe there's a fire door inadvertently left
     open, something like that, so the fire could get into the AFW pump A
     room, for instance, as well.
         Then, the operator is going to have to try to deal with this
     situation that he's got fire affecting both alternate and dedicated
     equipment, or he has to deal with the fact that the good equipment has
     randomly failed and is not working, and when allowing a recovery there,
     he has to make a decision as to what sort of recovery action to take,
     and then, obviously, he has to carry out that recovery action.
         That recovery action would probably be something like, well,
     let me go try to use the A equipment again, even though it's the
     equipment that's burning, because the B diesel isn't starting, so I've
     got to go try to use the A diesel.  That's my only out at this point. 
     So in event tree space, this is sort of the relationship between the UAs
     and the equipment and the recovery and how that's sort of all panning
     out.
         DR. APOSTOLAKIS:  Isn't this similar to an operator action?
         MR. KOLACZKOWSKI:  I guess certainly from the concept
     standpoint, yes; in terms of laying out the possible sequences, yes.
         Next slide.  George, I don't know if you want to get into
     the details --
         DR. APOSTOLAKIS:  No.
         MR. KOLACZKOWSKI:  -- of the codification other than to say
     that we used the existing PRA information to try to quantify, well,
     what's the chance this set of plant conditions would actually occur this
     way.  And then, as we said, as far as actually coming up with the
     probabilities of the unsafe acts, at this point, they're still largely
     based on judgment and using other types of techniques like HEART to try
     to get some idea of what those numbers ought to be.
         DR. APOSTOLAKIS:  Why don't you go on to the difference
     between existing --
         MR. KOLACZKOWSKI:  Okay.
         DR. APOSTOLAKIS:  -- PRAs?
         MR. KOLACZKOWSKI:  So that takes me to the last slide in my
     presentation, which is really what we want to stress more than the
     quantitative numbers.  As with PRA, the real value of doing PRA is what
     you get out of doing the process.  The numbers are fine, and they sort
     of set some priorities, but we think the same is true of ATHEANA.  And
     from a qualitative aspect, what we've done here is compare the existing
     PRA human performance observations and sort of what you learned out of
     the existing HRA and what you might learn out of doing an ATHEANA type
     of HRA on these same two fires, and these are meant just to compare the
     types of fixes or lessons learned, if you will, out of the HRA analysis
     that one might gain from the existing PRA versus the ATHEANA results,
     and let me just generally characterize them as I think the existing PRA
     gives you some sort of very high level ideas of some things that you
     might fix, and they generally fit the category of well, let's just train
     them more, or let's make this step bolder in the procedure so he won't
     skip it.
         I think in going through the ATHEANA process and really
     understanding what the vulnerabilities are and how the scenario
     differences might trigger those vulnerabilities to be more prominent, I
     think you learn more specifics as to ways to improve the plant, either
     from a procedural standpoint, a labelling standpoint, et cetera, and
     what the specific needs are, such as like that first one up there on the
     extreme upper right.  Clearly, there is a need for a minimum and
     definitive criteria for when to enter EOP-FP-Y or Z.
         DR. BARTON:  That may be almost impossible to come up with: 
     how many meters; out of whack by how many degrees?  Some of that is
     going to be real hard to put numbers on, numbers or definite criteria
     for getting in there.
         MR. KOLACZKOWSKI:  Granted; I'm not saying that all of them
     can be done or should be done, but these are the types of insights one
     can gain out of doing an ATHEANA type of analysis out of this.  Unless
     you want to go through specific ones, that pretty much ends the
     presentation.  It's trying to be a practical illustration of how the
     actual searches and everything work.
         DR. POWERS:  I guess I'm going back to the question of what
     has been accomplished?  Why do we feel it's necessary to go to such a
     heroic effort on the human reliability analysis?  And if we could
     understand why we want to do that, maybe we could decide whether we've
     accomplished what we set out to do.
         MR. KOLACZKOWSKI:  My short answer to that is go back to one
     of the first slides we had this morning.  If you look at real serious
     accidents, they usually involve operators not quite understanding what
     the situation was; certain tendencies, et cetera, are built into their
     response mechanisms, and therefore, they made mistakes, and PRAs, quite
     frankly, as good as they do to try to determine where the risks of
     nuclear power plant accidents lie, et cetera, still do not deal very
     well with possible errors of commission, places where operators might
     take an action that, in fact, would be unsafe relative to the scenario. 
     So maybe we're missing some of where the real risk lies.
         DR. POWERS:  I think we see this kind of a problem,
     especially when we look at severe accidents, pertaining to accidents
     where the operators disappear.  Something happens to them, because they
     don't affect things very much.
         And you get peculiar findings out of that, like we have
     people swearing that the surge line is going to fail; the four steam
     generators to fail or the vessel fails, because that's where -- the
     operator has apparently taken a powder and gone someplace and don't try
     to put any water into it, and despite what we saw at TMI, the surge line
     fails, and so, accidents become benign that otherwise would be -- and
     understanding the operator is going to take a powder, that will do
     something that seems like a very valuable thing.
         The question you have to ask is is this enough, or should we
     do something much more?
         [Laughter.]
         MR. KOLACZKOWSKI:  I don't know how to respond to that.
         DR. POWERS:  Well, putting it another way, I assume you can
     figure out the inverse to that statement, because that's already too
     much.
         MR. CUNNINGHAM:  Part of the reason we're coming out to talk
     to the committee and other people is just to sort out, okay, what are
     the next steps?  We've taken a set of steps.  We've made an investment
     and made a decision to go down a particular route.
         DR. POWERS:  Well, could you work and research just maybe
     operators might put water in and the surge line not fail first?
         [Laughter.]
         MR. KOLACZKOWSKI:  We'll do that.  We'll try to convince
     them.
         DR. POWERS:  Try to convince them that TMI actually did
     occur.
         [Laughter.]
         MR. CUNNINGHAM:  People forget things.
         DR. POWERS:  But it is possible that it pours down under
     pressure and not had the surge line fail.
         MR. CUNNINGHAM:  Yes.
         DR. APOSTOLAKIS:  Are you going to be here this afternoon?
         MR. CUNNINGHAM:  I don't know about most of us
     but --
         DR. APOSTOLAKIS:  Until about 3:00?
         MR. FORESTER:  I'd have to change my flight.
         MR. CUNNINGHAM:  Some of us will be here.
         DR. APOSTOLAKIS:  Okay; I propose that we recess at this
     time so that Tom and I can go to a meeting, and we will talk about the
     conclusion, followup activities at 12:45.
         MR. CUNNINGHAM:  12:45 is fine by us.
         DR. THOMPSON:  I only have two more slides.
         MR. CUNNINGHAM:  We just have two slides, George, if you can
     just bear with us.
         DR. APOSTOLAKIS:  Yes, but I want to go around the table.
         DR. POWERS:  Unfortunately, he has an hour and a half of
     questions.
         DR. APOSTOLAKIS:  Yes.
         [Laughter.]
         DR. APOSTOLAKIS:  Is the staff requesting a letter?
         MR. CUNNINGHAM:  We are not requesting a letter, no.
         DR. APOSTOLAKIS:  Okay.
         MR. CUNNINGHAM:  If you would like to write one, that's
     fine, but we are not requesting it.
         DR. APOSTOLAKIS:  Okay.
         DR. POWERS:  We could write one on surge line failures.
         [Laughter.]
         DR. APOSTOLAKIS:  So let's reconvene at 12:45.
         MR. CUNNINGHAM:  12:45.
         [Whereupon, at 11:45 a.m., the meeting was recessed, to
     reconvene at 12:43 p.m., this same day.].                   A F T E R N O O N  S E S S I O N
                                                     [12:43 p.m.]
         DR. APOSTOLAKIS:  Okay; we are back in session.
         Mr. Cunningham is going to go over the conclusions,
     Catherine, so then, perhaps, we can go around the table here and get the
     members' views on two questions:  the first one, do we need to write a
     letter, given the error forcing context that the staff is not requesting
     a letter.
         [Laughter.]
         DR. APOSTOLAKIS:  And the second, what do you think, okay? 
     So the staff will have a record of what you think.  So, who is speaking? 
     Catherine?
         DR. THOMPSON:  Okay; just real quickly, I want to go over
     two slides:  the conclusion slide, we talked about all of this in the
     last couple of hours that we think ATHEANA provides a workable approach
     that achieves realistic assessments of risk.  We can get a lot of
     insights into plant safety and performance and have fixes, if you will.
         DR. POWERS:  It boils down to a lot on what you call
     workable.  It looks to me like it's not a workable approach.  If I try
     to apply it unfettered, I have some limitation on where I'm going to
     focus it, but it completely gets out of hand very quickly.
         MR. CUNNINGHAM:  That's also true of event tree and fault
     tree analysis and lots of other parts of PRA.  I think one of the issues
     that was discussed this morning of how do we fetter it, if you will, or
     keep it from becoming unfettered, and I think that's a legitimate issue
     that we perhaps can talk to you about more.
         DR. POWERS:  Yes; you need something that says, okay, you
     need something that's a nice progression, so that you can go from
     zeroeth order, first order, second order and have everybody agree, yes,
     this is a second order application.
         MR. CUNNINGHAM:  Yes, yes, and that, I think, again,
     probably within the team, we have those types of things in our heads.
         DR. POWERS:  Yes.
         MR. CUNNINGHAM:  But it's not very constructive from the
     outside world, yes.
         DR. APOSTOLAKIS:  The same goes to a straightforward.
         MR. CUNNINGHAM:  Of course; it's intuitively obvious,
     perhaps, that it's straightforward or some such things.
         DR. POWERS:  I got the impression that you had a variety of
     search processes that made it comprehensive; they may not have made it
     straightforward but a comprehensive search process.
         MR. CUNNINGHAM:  Okay.
         DR. THOMPSON:  Some of the followup activities.
         DR. APOSTOLAKIS:  Wait a minute, now, Catherine, you were
     too quick to change that.
         DR. THOMPSON:  Good try.
         DR. APOSTOLAKIS:  This comes back to the earlier comment
     regarding objectives.  I don't think your first bullet should refer to
     risk.  Your major contribution now is not risk assessment.  You may have
     laid the foundation; that's different.  But right now, it seems to me
     the insights that one gains by trying to identify the contexts and so on
     is your major contribution, you know, and that can have a variety of
     uses at the plant and so on.
         So I wouldn't start out by saying that you have an approach
     to achieve a realistic assessment of risk.
         MR. CUNNINGHAM:  Okay.
         DR. APOSTOLAKIS:  You don't yet.
         MR. CUNNINGHAM:  Okay.
         DR. APOSTOLAKIS:  I, in fact, would make it very clear that
     there are two objectives here, if you agree, of course.  One is this
     qualitative analysis, which I think I view as been knocked down a little
     bit and then the risk part, okay?
         MR. CUNNINGHAM:  Yes.
         DR. APOSTOLAKIS:  I think you should make it very clear,
     because if I judge this on the basis of risk assessment, then I form a
     certain opinion.  If I judge it from the other perspective, the opinion
     is very different.
         MR. CUNNINGHAM:  Okay; I'll note that.
         DR. APOSTOLAKIS:  Develops insights:  I have associated over
     the years the word insights with failed projects.
         [Laughter.]
         DR. APOSTOLAKIS:  Whenever some project doesn't produce
     anything --
         [Laughter.]
         DR. APOSTOLAKIS:  -- you have useful insights.
         [Laughter.]
         DR. APOSTOLAKIS:  So in my view, you should not use that
     word, even though it may be true.
         MR. CUNNINGHAM:  Okay.
         DR. APOSTOLAKIS:  Supports resolution of regulatory and
     industry issues; you didn't give us any evidence of that, but I take
     your word for it.
         MR. CUNNINGHAM:  Okay.
         DR. APOSTOLAKIS:  Okay.
         MR. CUNNINGHAM:  So insights will be removed from the
     lexicon.
         [Laughter.]
         MR. CUNNINGHAM:  Along with forcing, I guess, is another one
     we have to remove.
         DR. APOSTOLAKIS:  Yes; the thing about unsafe acts and human
     failure events, I really don't understand the difference.
         MR. CUNNINGHAM:  Yes; that's one of the things I was
     thinking about this morning in listening to this is again, within the
     team, I think it's well understood what those different terms means. 
     But to the --
         DR. APOSTOLAKIS:  Yes.
         MR. CUNNINGHAM:  -- the general public, it's not going to be
     real clear.
         DR. APOSTOLAKIS:  But if it's an unsafe act, it should be a
     failure demand?  That's why it's unsafe?
         MR. CUNNINGHAM:  I don't know.
         DR. APOSTOLAKIS:  From the words, from the words; it doesn't
     follow.  And you are saying in the text that they are expected to act
     rationally.  So why are you calling what they did -- anyway.
         MR. CUNNINGHAM:  Anyway, yes, we will try to do a better job
     of mapping those things out.
         DR. THOMPSON:  Okay.
         MR. CUNNINGHAM:  Followup issues?
         DR. THOMPSON:  These are some activities that we'd like to
     get in a little bit more.  Some of them are already planned.
         DR. POWERS:  You don't have any my surge line up there.
         DR. THOMPSON:  Surge line?
         [Laughter.]
         MR. CUNNINGHAM:  There was a typo.  We meant to say surge
     line.
         [Laughter.]
         DR. POWERS:  What you do is you didn't get the steam
     generator tube rupture problems.
         DR. THOMPSON:  Okay; we obviously are pretty much done with
     the fire issue.  We're now working on PTS issue with Mr. Woods and some
     other members of the branch and helping him look at the human aspects of
     that.  We'd like to get into some of the digital INC area, see what that
     could add to the human error when they start working along with digital
     INC.
         DR. UHRIG:  Are you looking at that strictly from the
     operations standpoint, or are you going to get back into the code
     development aspect?
         DR. SEALE:  The software side.
         DR. THOMPSON:  Software; we haven't -- these are things that
     possibly we could get into.  This isn't really planned yet, digital INC
     part.  So I don't know how far we would get into that.
         DR. APOSTOLAKIS:  So when you say digital, what exactly do
     you mean?  I guess it's the same question.  The development of the
     software or the man-machine interaction?
         DR. THOMPSON:  I think the man-machine.
         MR. CUNNINGHAM:  We were thinking not so much the
     development as it's being used in the facilities.
         DR. THOMPSON:  Right.
         DR. UHRIG:  The difference between an analog and a digital
     system is relatively minor when it comes to the interface.  It's the
     guts that's different.  Pushing the wrong button, it doesn't make any
     difference whether it's digital or analog.
         MR. CUNNINGHAM:  Yes; again, this has been suggested as a
     topic that what we're doing here might dovetail well with other things
     that are going on in the office.  It hasn't gone much further than that
     at this point.
         DR. POWERS:  At what point do we get some sort of comparison
     of the leading alternatives to ATHEANA for analyzing human fault so that
     you get some sort of quantitative comparison of why ATHEANA is so much
     better than the leading competitors?
         MR. CUNNINGHAM:  A quantitative comparison or --
         DR. POWERS:  Well, a transparent comparison.  You tried some
     things where you said here's what you get from ATHEANA, and here's what
     you get from something else.  Any other different?  But it's hard for me
     to go away from saying this saying ATHEANA is just infinitely better
     than the existing PRA results.  Quite the contrary; I'm feeling that the
     things in the existing PRA must be pretty good.
         DR. BARTON:  A lot of them are very similar.
         DR. POWERS:  Yes, pretty similar.
         MR. CUNNINGHAM:  Okay; they are similar but --
         DR. BARTON:  The whole process may end up fixed it sooner to
     the fix out of play, the methods I'm using now.
         MR. CUNNINGHAM:  What happens in the context of like the
     fire example is you're identifying new scenarios as you go through the
     trees that seem to have some credible probability.  How, you know, what
     the value or what the probabilities are that will be associated with
     them is still something we're still exploring.  We expect that we will
     find scenarios that will have a substantial probability and will, you
     know, lead to unsafe acts or core damage accidents or whatever.  Again,
     they go back to you look at the history of big accidents in industrial
     facilities, and you see these types of things occurring, so we're trying
     to match the event analysis with the real world, if you will.  In a
     sense, that's one of the key tests, I think, of how well this performs
     is that do we seem to be capturing what shows up as important in serious
     accidents?
         There are a couple of things that aren't on this slide that
     we've talked about this morning.  We discussed for a good while the
     issue of quantification, that that may be -- is that on there?  I can't
     read the thing; okay, improved quantification.
         DR. APOSTOLAKIS:  What is that?
         MR. CUNNINGHAM:  It's one of those bullets.
         DR. APOSTOLAKIS:  Full-scale HRA/PRA?
         MR. CUNNINGHAM:  No, the fourth one down, improved
     quantification tools.
         DR. APOSTOLAKIS:  I would say in degrading quantification.
         MR. CUNNINGHAM:  I'm sorry?  Okay; quantification tools
     comes up as an issue.
         DR. APOSTOLAKIS:  Why does the NRC care about whether
     ATHEANA applies to other industries?
         MR. CUNNINGHAM:  Because it gives us some confidence that
     it's capturing the right types of human performance.  As we've talked
     about many times or several times this morning, big accidents and
     complex technologies, we think, have a similar basis in human
     performance or are exacerbated or caused by similar types of events. 
     Given that we don't have many big accidents in nuclear power plants, I
     think it's important that we go out and --
         DR. APOSTOLAKIS:  Did we ever apply this to other industries
     to gain the same kind of lessons?  Let them use it.
         MR. CUNNINGHAM:  Again, it's not so much the --
         DR. APOSTOLAKIS:  In my years at the Nuclear Regulatory
     Commission, I don't know how much effort you plan to --
         MR. CUNNINGHAM:  Well, part of it, it's not a big effort,
     but it's also something where I think it's important to help establish
     the credibility of the modeling we have.
         DR. APOSTOLAKIS:  Like among pilots or airliners?
         MR. CUNNINGHAM:  Yes, the aircraft industry, over the years,
     we've had some conversations with NTSB and with NASA and places like
     that.  Again, it's complex industries where you have accidents and --
         DR. APOSTOLAKIS:  I think developing quantification tools
     and the team aspects in NNR will keep you busy for another 7 years, so I
     don't know about the other industries.  Again, that's my personal
     opinion.
         MR. CUNNINGHAM:  Well, you can take that in several ways. 
     One of them is do you consider those the highest priority issues on the
     --
         DR. APOSTOLAKIS:  I find them the most difficult, the most
     difficult, applying it to other industries.
         MR. CUNNINGHAM:  I don't think we'd disagree with you.
         DR. APOSTOLAKIS:  I mean, it makes sense to -- adds
     credibility to say, yes, we did it in this context and it's --
         MR. CUNNINGHAM:  Yes.
         DR. APOSTOLAKIS:  But I wouldn't put too much effort into
     it.
         DR. SEALE:  But the preferable thing would be to have
     someone else use ATHEANA, and then --
         DR. APOSTOLAKIS:  Yes.
         DR. SEALE:  -- you could get them to act as an independent
     reviewer of your work and vice versa.
         MR. CUNNINGHAM:  Sure.
         DR. SEALE:  That strikes me as a much more --
         MR. CUNNINGHAM:  In that context, maybe apply is the wrong
     word but interact with other industries --
         DR. SEALE:  Yes.
         MR. CUNNINGHAM:  -- complex industries on the -- for the
     credibility and the application of ATHEANA.
         DR. APOSTOLAKIS:  Well, you also have, it seems to me, a
     nuclear HRA community.  Why are the teams developing whatever processes
     or whatever?  Is it because they're not aware of ATHEANA yet?
         MR. CUNNINGHAM:  You're taking some of the next
     presentation, which is on the international work that we're doing.
         DR. APOSTOLAKIS:  I'm not sure that we're going to have that
     presentation.
         MR. CUNNINGHAM:  Okay.
         DR. APOSTOLAKIS:  I think we should conclude by discussing
     what we've heard, unless you really feel that -- I mean, I look at it. 
     It's not just really useful.
         MR. CUNNINGHAM:  No, no, I'm sorry; there's a separate
     presentation.
         DR. APOSTOLAKIS:  There is?
         MR. CUNNINGHAM:  Yes; remember this morning that we
     discussed -- one of the first things on the agenda was the work we're
     doing internationally.  We put that off until after that.
         DR. APOSTOLAKIS:  How many view graphs do you have on that?
         MR. CUNNINGHAM:  It's about eight or something like that. 
     We can cover it in 5 or 10 minutes.
         DR. APOSTOLAKIS:  I think we should do that right now.
         MR. CUNNINGHAM:  Okay; it's up to you.
         DR. POWERS:  I would hope you would be able to tell me that
     little -- the Halden program plays or could play in the ATHEANA
     methodology.
         MR. CUNNINGHAM:  Do you want to go ahead and go to the
     international?
         DR. POWERS:  Whenever it's appropriate.
         DR. APOSTOLAKIS:  It's up to you, Mark.  I think we're done
     with this.
         MR. CUNNINGHAM:  We're done with this; then, let's go ahead,
     and we'll cover the international thing.
         DR. APOSTOLAKIS:  I want to reserve at least 5 minutes for
     comments from the members.
         MR. CUNNINGHAM:  Okay.
         DR. APOSTOLAKIS:  Before we go on to the Sorenson
     presentation.
         MR. CUNNINGHAM:  Okay.
         Basically, as we've been doing this ATHEANA work and our
     other HRA work, we've had two principal mechanisms for interacting
     internationally with other developers and appliers of HRA methods.  One
     is through the CSNI principal working group five on PRA; in particular,
     there was something called the task group 97-2, which is working on the
     issue of errors of commission.
         DR. APOSTOLAKIS:  Who is our member?
         MR. CUNNINGHAM:  I'm sorry?
         DR. APOSTOLAKIS:  Who represents the NRC there, PWG-5?
         MR. CUNNINGHAM:  We have two or three different
     interactions.  Joe Murphy is the chairman of PWG-5; I'm the U.S.
     representative on 5; the chair of the 97-2 task group was Ann
     Ramey-Smith.  We also have our COOPRA programs.  One of the working
     groups there was established to look at the impact of organizational
     influences on risk.
         DR. APOSTOLAKIS:  Is that what the Spaniards are doing?
         MR. CUNNINGHAM:  Yes, that's where the Spanish come in. 
     It's the international cooperative PRA research program.  It doesn't fit
     the --
         DR. APOSTOLAKIS:  That is one of the Former Chairman
     Jackson's initiative papers.
         MR. CUNNINGHAM:  Correct; she wanted to -- she wanted the
     regulators to work more closely together, and there were a couple of
     research groups established as part of that.
         Anyway, okay, the PWG-5 task 97-2 had three general goals. 
     You want to look at insights, although perhaps that's no longer the
     right word to use; develop perspectives on errors of commission to apply
     some of the available methods which supposedly handle errors of
     commission and for quantitative and non-quantitative, more qualitative
     analysis of errors of commission and to look at what data would be
     needed to support types of analysis.
         DR. POWERS:  Have any of the technical fields -- I can with
     modest amount of effort, have you seen the database that -- is there
     someplace that I would go to find data that are pertinent to human
     reliability analysis?
         MR. CUNNINGHAM:  Do you want to answer that?  I'm going to
     have one of my colleagues come up and answer that a little more
     explicitly.  One of the people over here was shaking her head; I don't
     know.
         MS. RAMEY-SMITH:  No, that's a short answer.
         [Laughter.]
         DR. APOSTOLAKIS:  Would you identify yourself please?
         DR. POWERS:  Before she identifies herself as a major expert
     in the field that I noticed last year our first exposure to ATHEANA was
     on human reliability analysis, brand spanking new, put out by a book
     publisher, and so I immediately acquired a copy of this book; read it
     for an entire airplane flight from Albuquerque to Washington, D.C. and
     found not one data point in the entire book.  But there were 30-some
     papers on various human reliability analyses but not one data point.
         DR. SEALE:  We still need to know who she is.  For the
     record, please?
         MS. RAMEY-SMITH:  Ann Ramey-Smith, NRC.
         If I can recall, the question was is there a database that
     you can turn to, and the short answer from our perspective of the kind
     of analysis that -- and from the perspective that we think you should do
     an analysis, which is within the context of what's going on in the plant
     and performance shaping factors and so on, there is not a database that
     exists that we can turn to and go -- and make inferences based on
     statistical data.
         The fact is that we've developed our own small database that
     has operational data in it that we have analyzed.  There are various and
     sundry databases of various sorts.  The question comes down, and one of
     the questions that this PWG-5 is going to address is the fact that we
     have a lot of databases, none of which may serve the needs of the
     specific methods that people are trying to apply.
         DR. UHRIG:  Would there not be a lot of information
     available through the LERs?
         MS. RAMEY-SMITH:  Oh, if that were true.  Actually, there is
     quite a lot of information available on the LERs.  Unfortunately, it's
     difficult oftentimes in those writeups to understand fully what the
     context was, to understand why the operators did what they did and what
     were the consequences and what were the timing and so on and so forth. 
     One concern that some of the HRA folks have is that possible changes to
     the LER rule will even strip from the reports the little information
     that it had before, so we're concerned about that.
         The better source for information, actually, has been the
     AIT reports and some very excellent reports that were previously done by
     AEOD when they did studies of particular events that maybe didn't rise
     to the level of AITs but were very in-depth analyses, and we were able
     to make use of those, particularly early on when we were doing this
     iterative evaluation of operating experience.  It was quite helpful.
         DR. POWERS:  One of the issues that NRR is having to
     struggle with is these criteria in what actions should be automated as
     opposed to being manual.  How long does it take somebody to diagnose a
     situation and respond to it?  And there are several that they have,
     because they have some good guidelines; they just don't have any data.
         MS. RAMEY-SMITH:  I think this approach would be very
     helpful for understanding -- what is it? -- B-17, the safety-related
     operator actions.  I think that the agency would be wise to evaluate
     that issue within the context of PRA.
         DR. APOSTOLAKIS:  This looks to me like a benchmark
     exercise.  Is that what it is?
         MR. CUNNINGHAM:  No; the sense that I have is that someday,
     we might be able to get to a benchmark exercise, but the principal
     players weren't comfortable at this point in constraining the analysis
     to that degree.
         DR. APOSTOLAKIS:  So, oh, yes, because you're saying they
     apply to events of the --
         MR. CUNNINGHAM:  That's right; we have a variety of
     different methods, and what we were doing was trying to see what these
     methods were giving us, so we didn't try to constrain it to a particular
     method or a particular event.
         DR. APOSTOLAKIS:  Okay; thank you.
         MR. CUNNINGHAM:  As you can see on page 4, we have a number
     of different methods applied.  ATHEANA was applied by the U.S. group,
     the Japanese in people in the Netherlands; also different methods
     applied such as MERMOS, SHARP.  We have the Czech Republic spelled
     correctly today, so that was an advancement over yesterday.
         [Laughter.]
         MR. CUNNINGHAM:  And some other models that, as you can see,
     we go back to the Borsele theory.
         DR. APOSTOLAKIS:  Is SHARP really a model?
         Okay; let's go on.
         MR. CUNNINGHAM:  Okay; slides five and six are a number of
     the conclusions that are coming out of the task 97-2.  I'm not sure I
     want to go into any of the details today, but you can see the types of
     the issues that they're dealing with and what the report will look like. 
     The report has been by and large has been finished; the report of this
     group has been finished.  It's going to go before the full CSNI next
     month, I believe, for approval for publication.  So it's essentially --
     this part is particularly -- is essentially done.
         DR. APOSTOLAKIS:  The words are a little bit important here. 
     The rational identification of errors of commission is difficult.  What
     do you mean by rational?
         MS. RAMEY-SMITH:  That was the word that was chosen in the
     international community that everyone was comfortable with.  But the way
     you can think of it is it's as opposed to experientially, you know, so
     that it's more predicting to sit down and to be able to identify errors
     of commission a priori.
         DR. APOSTOLAKIS:  Do you mean perhaps systematic?
         MS. RAMEY-SMITH:  Yes, that could have -- I guess the point
     is to be able to I guess systematically analyze it, you know, a priori
     be able to identify an error of commission.  Systematic is a perfectly
     good word.  This was just the word -- we used on this slide the words
     that, in the international group that was working on this, they were
     comfortable with.
         DR. APOSTOLAKIS:  And what is cognitive dissonance?
         MS. RAMEY-SMITH:  Okay; perhaps Dr. Thompson would like to
     --
         DR. APOSTOLAKIS:  That was an international term?
         MS. RAMEY-SMITH:  No, cognitive dissonance is from the good
     old field of psychology.
         DR. APOSTOLAKIS:  Oh, okay.
         DR. BARTON:  It's Greek.
         DR. APOSTOLAKIS:  What?
         DR. BARTON:  It's Greek.
         [Laughter.]
         DR. SEALE:  Could I ask if this group of international
     experts had all of these different approaches, presumably, they would
     have a great deal of common interest in making certain things like LERs
     helpful about what's there.  Has anyone put together a sort of a
     standard format for what it would take to get an LER that had the
     information you needed in it be able to generate a database?
         MR. CUNNINGHAM:  Actually, one of the follow-on tasks of
     this work is for the HRA people here to go back and try to lay out what
     data do they need based on their experience with this type of thing.  So
     today, I don't think we have it, but I think over the next year or so,
     CSNI PWG-5 is going to be undertaking an effort to put that in the
     lifestyle.
         DR. SEALE:  It seems to me that should be something you
     could go ahead on, and whatever happens, at least now, you'll be getting
     information that's complete --
         MR. CUNNINGHAM:  Yes.
         DR. SEALE:  -- in some sense.
         MR. CUNNINGHAM:  Yes.
         DR. APOSTOLAKIS:  That would be a very useful result.
         MR. CUNNINGHAM:  And that's one of the things that PWG-5 is
     going to undertake.
         MR. SIEBER:  Does that mean that every LER a plant puts out
     here goes through the ATHEANA program?
         DR. APOSTOLAKIS:  No, no, no, no.  The ATHEANA has developed
     guidance about the LERs.  The guys who write the LERs don't need to know
     about ATHEANA.
         MR. CUNNINGHAM:  Okay.
         DR. SEALE:  Just what it takes to have all of that planning
     data and things like that in it so that you've got a picture.
         MR. CUNNINGHAM:  Just two clarifications.  One was this
     isn't the ATHEANA guys; it's the -- this international group of HRA
     people, so it's the MERMOS guys and all those guys are going to be doing
     it.  It's not an ATHEANA specific issue.
         The second, I was talking about data needs in general.  I
     wasn't trying to suggest that all of the data needs that we had would
     automatically translate into something at LER, a change in the LER
     reporting requirements.  I wasn't suggesting that.
         DR. APOSTOLAKIS:  There has been a continuing set of
     discussions on human liability, and as I remember, former member Jay
     Carroll was raising that issue every chance he had.  How can you
     restructure the LERs so that the information is useful to analysts? 
     Because the LERs were not designed -- they were designed for the PRA
     phase, right?  You don't need another review for that.
         MR. CUNNINGHAM:  The LERs have a particular role, and as
     that role is defined even today, it's not going to provide a lot of the
     detailed information.  Now in parallel, though, with the development of
     all of the LER generation, you have the NPO and NRC and industry work in
     EPIX, which will be collecting information that is much more relevant to
     PRA types of analyses.  So I wouldn't so much focus on LERs as EPIX.
         DR. APOSTOLAKIS:  It would be nice to influence what those
     guys are doing.
         MR. CUNNINGHAM:  Yes.
         DR. APOSTOLAKIS:  Okay; next.
         MR. CUNNINGHAM:  Okay; going on to slide seven on the COOPRA
     working group on risk impact of organizational influences, basically,
     we're trying to -- the goal of the working group is to identify the
     relationships between measurable organizational variables and PRA
     parameters so that you can bring the influence in and explicitly model
     the influence in PRAs.
         DR. APOSTOLAKIS:  Next.
         MR. CUNNINGHAM:  Overall, I don't think I need to go into
     the outcomes as much as -- I think it's understood as to what that is. 
     Right now, it's fairly early in the process.  We're trying to get a
     better understanding of what people are doing in this area.  You alluded
     to the Spanish work in this area.  The Spanish are one of the key
     contributors in here.  How many countries are involved in this?
         MS. RAMEY-SMITH:  It's about six or seven.
         MR. CUNNINGHAM:  Okay; about six or seven countries; the UK,
     France, Spain, Germany, did you say?
         MS. RAMEY-SMITH:  Yes, Germany.
         MR. CUNNINGHAM:  Argentina, Japan?
         MS. RAMEY-SMITH:  Japan.
         MR. CUNNINGHAM:  Japan.  They're trying to work together on
     this issue.  Basically, again, this is fairly early in the work here. 
     There's going to be another meeting early next year to basically take
     the next step forward in the COOPRA work.  That's --
         DR. APOSTOLAKIS:  That's it?
         MR. CUNNINGHAM:  That's the short summary of the
     international work.
         DR. APOSTOLAKIS:  Okay.
         DR. POWERS:  And so, the Halden program has no impact on
     your --
         MR. CUNNINGHAM:  I'm sorry?
         DR. POWERS:  The Halden program has no impact on your --
         MR. CUNNINGHAM:  The Halden program has traditionally -- Jay
     Persensky sitting back here knows far more about it than I -- but has
     traditionally been oriented towards not so much human reliability
     analysis for PRA but for other human factors issues.  There has been
     some ideas that Halden will become more involved in human reliability
     analysis.  That's at least, I guess, in the formative stages.
         MR. PERSENSKY:  Jay Persensky, Office of Research.
         Halden has proposed for their next 3-year program, which
     starts in November, the development of an HRA-related activity based
     primarily on input from PWG-5, because a number of the people that have
     been involved with the Halden human error analysis project also serve on
     that or have served on that task force.  The goal, as I understand it at
     this point, is aimed more towards trying to take the recommendations
     with regard to kinds of data and seeing whether or not they can play a
     role in that.  At this point, it is in the formative stage, but it's
     looking more at that aspect of data since they do collect a lot of data,
     at least simulator data in-house.
         Now, whether it can be used or not is another question.  And
     that's what they're looking at at this point.
         DR. POWERS:  Is cross-cultural data any good?  In other
     words, if I collect data on the Swedish or Norwegian operators on a
     Finnish plant, is that going to be any good for human error analysis,
     modeling or for American operators on American plants?
         MS. RAMEY-SMITH:  It has the same context.
         MR. CUNNINGHAM:  When you say data, it depends.  If you're
     talking about probabilities, I don't know that any of the particular
     probabilities will apply, because again, there's a strong context
     influence.  Can it provide some more qualitative insights?  I suspect it
     could but again --
         DR. POWERS:  Cognitive things?  What does it tell you about
     processing information, things like that?  Are there big enough cultural
     differences that it's not applicable?  I would assume that Japanese data
     would just be useless for us.
         MR. CUNNINGHAM:  I wasn't thinking of the Japanese, but
     there may be some cultures where it would be of real questionable use
     depending on the basic management and organization and how they do
     things and whatever, it could be and not be very applicable.
         DR. APOSTOLAKIS:  Okay; all right, why don't we go quickly
     around the table for the two questions:  Should we write a letter, and
     what is your overall opinion?
         Mr. Barton?
         DR. BARTON:  Yes; I think we need to write a letter.  But
     let me tell you what my opinion is first --
         DR. APOSTOLAKIS:  Okay.
         DR. BARTON:  -- and maybe we can figure out if my opinion is
     similar to others; maybe not.  I fail to see the usefulness of this tool
     for the work that's involved.  Maybe I need to see some more examples. 
     I mean, the fire example doesn't prove to me that ATHEANA is much better
     than existing processes I know when looking at EOPs and how I train
     people and how people use procedures or react to plant transients.
         I think that as I look at this process, I also see where a
     lot of some of these actions depend on safety cultures, conservative
     decision making, et cetera, et cetera, and those two tie into this to
     understand more help and more safety culture and conservative decision
     making also.
         I think the tool -- I don't want to poo poo the tool, but I
     think it's a lot of work, and I don't see that you get a lot of benefit
     out of going through this process to really make it something that
     people are going to have to use in their sites unless this is a
     voluntary thing.  I don't know what the intent of ATHEANA is, but I
     don't see that benefit with the amount of effort I have to put into it.
         DR. APOSTOLAKIS:  And you would recommend the committee to
     write a letter stating this?
         DR. BARTON:  Well I think that if everybody else feels the
     same way, I think we need to tell somebody, you know, maybe that they
     ought to stop the process or change course or whatever.
         DR. POWERS:  I guess I share your concern that what we've
     seen may not reveal the definite capability of this, because there seem
     to be a lot of people here who are very enthusiastic about it.  Based on
     what was presented on the fire, I come away with -- it just didn't help
     me very much.
         DR. BARTON:  It didn't help me either, frankly.
         DR. POWERS:  But putting a good face forward or seeing how
     it's applied I think is something we ought to do more of and more of a
     comparison to why is it so much better than the other, and I agree with
     you, the fire analysis just didn't help me very much at all.
         DR. APOSTOLAKIS:  Mr. Siebert?
         MR. SIEBER:  I will probably reveal how little I know about
     this whole process, but I did read the report, and I came away first of
     all with a nuclear power plant perspective -- it's pretty complex; for
     example and this reviews HRA, PSF, UA, HFE and HEM, all of those were
     used in this discussion.  For a power plant person, I have difficulty
     with all of those acronyms.  I had some difficulty in figuring out
     ordinary things like culture and background and training, and we
     struggled with that.  So it could be -- the writeup could be a little
     simpler as it is.  The only way I could read it was to write the
     definitions of all of these things down, and every time one would come
     up, I would look at what I wrote down.
         The second thing was the actual application.  In a formal
     sense, I think it's pretty good.  And it would be useful to analyze some
     events to try to predict the outcomes of some events from a quantitative
     standpoint.  That was left unreasoned.  It was sort of like you arrive
     at a lot of things without -- and to me, that's not quantification. 
     That's just a numerical opinion, and I'm not sure that that's -- the
     other thing that I was struck by was when I figured the cost to apply it
     would be with NUREG 2600 which was 10 to 15 people to do a level three
     PRA over a period of several months.
         If I add ATHEANA onto that, I basically add 5 people.  I add
     5 people over a period of a year or so.  That's a lot of people. 
     Several of the people are key people, like the SRA.  The training
     manager; the simulator operator; I mean, our simulators are running
     almost 24 hours a day at this point.  So I think that the ability to
     make that investment, they would have to decide who am I going to lay
     off?
         So there would have to be a clear description of why some of
     the somebody other than the NRC would be motivated to do this, and I
     can't find it in the fire scenario.  There would be an awful lot of
     places where it would be very, very difficult to describe, you know,
     where all of this decision making or lack of decision making is.  It is
     understandable and logical; it's complex to read.  It's the state of the
     art.  It would be expensive to apply.  If you could show how this
     benefits safety --
         DR. BARTON:  And improve safety?
         DR. THOMPSON:  And improve safety.
         DR. APOSTOLAKIS:  That's it?
         MR. SIEBER:  That's it.
         DR. APOSTOLAKIS:  Bob?
         DR. SEALE:  Well, I have to apologize first for not being
     here for the presentation on fire.  Mario and I were doing some other
     things on license renewal.  I was impressed with the fact that the
     information that was presented on ATHEANA seemed to be a lot more
     detailed and a lot more thoughtful than what we had heard in the past. 
     It's very clear that the staff has been busy trying to firm up a lot of
     the areas that we had raised questions about in the past.  At the same
     time, I think of the 7 years.  I seem to recall that it had something to
     do with the cycle on some things in the Bible.
         [Laughter.]
         DR. SEALE:  But it seems to me for all of the reasons that
     you've heard from these people here and which I'm sure that you'd hear
     from other people, including plant people out there plant inspectors;
     that is, NRC people at the sites and so forth that you very badly need
     some application to show where this process worked, and I don't know
     enough about it to make a dogmatic judgment on my own as to whether or
     not those applications are there, but I would advise you to look very
     carefully to see if you can find someplace where you'd have a gotcha or
     two, because you clearly need a gotcha.
         The other thing, though, is that in terms of the things that
     are in this international program, I do believe that whatever format the
     human performance problem takes in the future, you can make some
     recommendations as to what it takes to put our experience as we live it
     today in a form which would be more readily retrievable when we do have
     a human factors process that's a little more workable, and so, you know,
     I just think you need to look at examples and an application.  That's
     where you're going to find your advocates if you're going to find any.
         DR. BARTON:  George, they did a fire scenario, and, you
     know, if you find this thing to the Indian Point II or the Wolf Creek
     draindown, what would you learn from that plant?  Because I just left
     the plant yesterday, and one of the agenda items we had was human
     performance at the plant, and it's not improving.  And I look at how
     could ATHEANA really help?  And when you look at the day-to-day human
     performance events, this wouldn't do a thing for those kind of, you
     know, day-to-day errors.
         You know, you're doing control rod manipulation.  This is
     typical kind of stuff.  You're doing control rod manipulation.  You have
     the guy at the controls.  He's briefed; he's trained; he's licensed. 
     You have a peer checker.  You go through the store; you go through all
     of the principles.  You get feedback into your three-way communications;
     the whole nine yards.  You're going to move this rod two notches out,
     and you do everything, and the guy goes two notches in.
         Now, tell me how ATHEANA -- and this is the typical stuff
     that happens in a plant on a day-to-day basis.  Now, tell me how I go
     through the ATHEANA process, and it's going to help me do something
     different other than whack this guy's head off, you know.  And, see, Jay
     agrees with me.
         MR. PERSENSKY:  They didn't get to the part of cutting his
     head off.
         [Laughter.]
         DR. POWERS:  Well, it strikes me that they will find an
     approach that they could tackle exactly that question.  It strikes me
     that I came in here saying ah, they have a new way to do PRA, put human
     reliability analysis in total in this, and I see a nice package.  I
     think they're not.  I think they need to work on the way they tackle
     really tough reliability issues.
         For instance, you pretty much set up one where you could
     apply all of these techniques that we talked about here to that
     particular issue, and I bet you they would come up with a response.  In
     fact, that's the lesson I get.  There is enough horsepower on it that
     you will get something useful on it.  And what they don't have is
     something that allows me to go and do the entire human reliability
     portion of a safety analysis, you know, and just turn the crank.  This
     is more for working on the really tough issues.  It's perfect for my
     surge line issue.  I mean, they could really straighten Tray Tinkler
     out.
         [Laughter.]
         DR. POWERS:  Which would be a start.
         MR. CUNNINGHAM:  We don't want to promise too much.
         MR. SIEBER:  One of the things that's stated early on in the
     NUREG concept is that you don't blame people, and I'm sure you want to
     do that.  On the other hand, when I read that, I thought secretly to
     myself some people just mess up.  You pull records on operators, and you
     find some will make one mistake and some another, and when you move in
     instead of moving out, you know, there may be a lack of attention to
     detail or a lack of safety culture or a lack of attitude or what have
     you that is preventing that person from doing the right thing, and I
     think that you've
     missed --
         DR. POWERS:  The documentation used to be a lot worse.  I
     mean, earlier documentation was really anathema to dare say that
     somebody screwed up.
         [Laughter.]
         DR. APOSTOLAKIS:  Dr. Uhrig?
         DR. POWERS:  I'll take another shot at it.
         DR. APOSTOLAKIS:  Okay; Dr. Uhrig?
         DR. UHRIG:  A couple of things.  One, anytime I've ever been
     involved with a plant with a serious problem, there has always been some
     unexpected turn of events that actually changed the nature of the
     problem, and I don't know how you would approach that.  That's an
     observation.
         The second one is it strikes me that if you need data, a
     modification of the LER procedures is a pretty straightforward process. 
     It's not simple.  I don't think you go to rulemaking to get the
     information that you need.  I don't think so.
         MR. CUNNINGHAM:  It would require rulemaking, absolutely,
     and a major fight.
         DR. UHRIG:  Yes.
         MR. CUNNINGHAM:  And a major fight before that rulemaking
     every got very far.
         DR. POWERS:  I don't think that's the problem.  I really
     don't.
         DR. APOSTOLAKIS:  But if you convince people you have the
     right approach --
         DR. POWERS:  I don't think it's a question of approach.  You
     know, when I first came in, you need a bunch of data to prepare this,
     and I'm not sure.  I think you need a bunch of problems to solve --
         MR. CUNNINGHAM:  Yes.
         DR. POWERS:  -- more than they need data to verify.  I think
     if I were these guys, I'd be out looking for every one of these
     problems, and there's just one on the criteria for when they have to
     automate versus manual action that's been sitting over like a lump, and
     I think you guys could attack that problem and get something very useful
     out of it.
         DR. APOSTOLAKIS:  Anything else?
         DR. UHRIG:  That issue is another one that somehow needs to
     get addressed.  We have literally done what we can do with training.  I
     think we're asymptomatically approaching this problem, well, you can
     train people.  Maybe automation is the next step.  And I don't know
     quite how this would be done.
         DR. POWERS:  They have a very interesting kind of plan that
     would allow for people to accomplish -- you can't do it in that period
     of time, you have to automate.  How long do you have to rely on somebody
     to recognize; they've got to do something to do it, and then, you would
     surely have to -- you need those kinds of numbers, and we've got some,
     you know.  But there's no reason to think that it's real well-founded. 
     The database that they're based on is proprietary.  We can't even get
     it.  And this looks like a methodology that I think attacks that problem
     very well.
         DR. APOSTOLAKIS:  Dr. Bonaca?
         DR. BONACA:  Well, you know, thinking about what's being
     done here, one of the problems I always see is about operators, people
     are always writing about what the operators will do at the most distant
     -- and it's very hard to bring most of this together.  But, again, you
     know, I want to reemphasize the fact that where it is happening in that
     unique fashion was in the thinking-oriented procedure.  Any experience
     that has been in the industry, it was a massive experience.  Only when
     you put thousands of man hours when you have operators thinking together
     with engineers, with people who develop event trees, very specific trees
     with multiple options and so on and so forth; I think there has to be
     some opportunity to benefit by grounding some of the work in ATHEANA on
     comparison to what was done there, maybe just the EPGs, for example,
     taking some example, getting some of the people involved in those.
         I think the products will be people.  You have some model of
     verification.  You have some way to stand on some of the hypotheses of
     ATHEANA.  Everything is speculative.  It's probably correct, but we need
     to have some benchmark.
         And second, that may offer you some simplification process
     and some issues that already have been dealt with in those efforts; take
     a look at procedures that may -- may help you in simplifying the
     process.  But I can't go any further in speaking about it.  But again,
     the point I'm making is that that's the only place that I know operators
     and analysts and development of processes came together for a long time. 
     But I think that there will be a great benefit, actually, in trying to
     anchor ATHEANA on some benchmark, some comparison or some statement.
         DR. APOSTOLAKIS:  Well, I find it a bit disturbing that two
     of the members with hands-on plant experience are so negative.  I would
     like to ask the subcommittee whether we should propose to write a
     letter, whose form will have to be discussed and content.
         DR. POWERS:  I don't think we have to write a letter that's
     critical.  We need to have something that tells you to judge that data,
     and I don't think we need to write a letter on the external safety
     mechanisms.  Cultural data, for example, on an organization.
         DR. APOSTOLAKIS:  The letter may say that.
         DR. POWERS:  If it says that, then fine.
         DR. APOSTOLAKIS:  Express reservations for the present state
     and may urge the further application with the explicit wish that the
     thing become more valuable.
         DR. BARTON:  I would agree with that.
         DR. APOSTOLAKIS:  The letter doesn't have to say stop it. 
     In fact, I wouldn't propose such a letter.
         DR. POWERS:  Maybe we should say that these people should
     spend a year tackling three or four problems, visible, useful problems
     that -- and show the value of this technique, because I think it's not a
     technique that's going to get used.  It would be wrong to hurt this,
     when I think they're just getting to the point where they can actually
     do something.
         DR. APOSTOLAKIS:  The letter, the contents of the letter are
     to be discussed; I think I got a pretty good idea of how you gentlemen
     feel, and certainly, I didn't hear anybody say stop this, although Mr.
     Barton came awfully close.
         Yes, sir?
         MR. SIEBER:  I wouldn't want to be interpreted as negative,
     but I think things --
         DR. APOSTOLAKIS:  But you have been.
         MR. SIEBER:  No, I think things are needed.
         DR. APOSTOLAKIS:  Yes.
         MR. SIEBER:  I think simplification is needed; a good
     objective is needed; what we need to accomplish.
         DR. APOSTOLAKIS:  Does everyone around the table agree that
     a letter along those lines, which, of course will be discussed in
     December will be useful?
         DR. POWERS:  I have reservations about the simplification,
     because I know in the area -- we do have computer codes that are highly
     detailed, very complex things that we use for attacking the heart of
     very complex, tough problems; much more simplified techniques that we
     use for doing broad, scoping analyses, and I think there's room in this
     field, and I think maybe one of the flaws that's existed in the past in
     this human reliability area is that everybody was trying to make the one
     thing that would fit all hard problems, easy problems --
         DR. APOSTOLAKIS:  Right.
         DR. POWERS:  -- long problems, short problems, and maybe we
     do need to have a tiered type of approach in which you say, okay, I've
     got a kind of a scoping tool that --
         DR. APOSTOLAKIS:  No, I think --
         DR. POWERS:  I've got this one that's attacking the really
     tough, really juicy problems that have defied any useful resolution in
     the past.
         DR. APOSTOLAKIS:  I think the issue of screening, scoping
     the analysis, the raised approach that was mentioned earlier, all that
     part, I understand as part of this, and that was that you should have --
     there also is -- but you have to convince me first that this event
     deserves that treatment.
         DR. POWERS:  Yes.
         DR. APOSTOLAKIS:  And that's what's missing right now.  I
     would agree with Dana that you don't have to simplify everything, but
     I'm inclined to say that the majority of the events would deserve it.
         Now, naturally, when you develop a methodology, of course,
     you attack the most difficult part, but I think a clear message here is
     develop maybe a screening approach, a phased approach that would say for
     these kinds of events, do this, which is fairly straightforward and
     simple; for other kinds of events, you do something else until you reach
     the kinds of events and severe accidents that really deserve this
     full-blown approach that may take time, take experts to apply.
         You know, this criticism that plant people should be able to
     apply it, I don't know how far it can go, because if it's very
     difficult, they are known to hire consultants.  So this is the kind of
     thing that they have to think about.  We're not going to tell them how
     to do it, but that's what I understand by your call for simplification. 
     You're not asking for something that says do A, B, C, and you're done.
         Okay; so it seems to me that we have consensus, unless I
     hear otherwise, that a letter along these lines will be appropriate to
     issue, and I'm sure we'll negotiate the words and the sentences in
     December.
         Dana?  Your silence is approval?
         DR. POWERS:  No, my silence is that I'm encouraging at this
     point.
         DR. APOSTOLAKIS:  Yes; yes.
         DR. POWERS:  It's okay to have a methodology at this point
     that only Ph.D.s in human reliability analysis can understand very well.
         DR. APOSTOLAKIS:  I understand the concern about the tone,
     but I also want to make it very clear in the written record that these
     gentlemen have reservations and not random members.  I don't think Mr.
     Bonaca is going to express as extreme views as you, but I'm not sure
     he's far away from your thinking.  So if I have the three utility
     members thinking that way, I think the letter should say something to
     that effect without necessarily discouraging further development or
     refinement.
         DR. POWERS:  Yes.
         DR. APOSTOLAKIS:  But it's only fair; the letter will be
     constructive, but it will clearly state the concerns, and perhaps we
     should meet a year from now or something like that.  We can say
     something like that in the letter.  We look forward to have interactions
     with the staff.
         DR. POWERS:  I think I would really enjoy giving them some
     time to go off and think about some problems to attack and come back and
     say we think we're going to attack these two problems next time or
     something like that.  I think that would be really interesting, because
     I think there are some problems out there that line organizations really
     need some help on solving, and I'm absolutely convinced that the human
     element is going to become of overwhelming importance if we're going to
     have a viable nuclear energy industry in this country.
         The operators are asked to do so much, and it's going to be
     more and more with less and less over time, and we need to have
     something that constrains us saying, yes, the operators will do this,
     because right now, nothing constrains us from saying yes, the operators
     have to be trained on this; they have to know this; they have to worry
     about this and like that, and at some point, where that process has to
     be constrained a little bit.
         But I think I really come in much more enthusiastic about
     this than you thought I would.
         DR. APOSTOLAKIS:  Okay; I think I've heard enough.  I can
     draft a letter.  I'm sure it will be unrecognizable after --
         [Laughter.]
         DR. APOSTOLAKIS:  But at least I have a sense of the
     subcommittee.
         DR. SEALE:  Nobody overhead.
         DR. APOSTOLAKIS:  Yes.
         MR. SIEBER:  Can we see a copy of it before the meeting?
         DR. APOSTOLAKIS:  I'll do my best; I'll do my best, Jack,
     before the meeting.  I urge you to send emails with your concerns; yes,
     and I will do my best to include your thoughts.  I took notes here, but,
     you know, John, if you want to send me a fax or call me.
         DR. BARTON:  Okay.
         DR. APOSTOLAKIS:  Or Jack, because I'm particularly
     interested -- I mean, this is the way this committee has functioned in
     the past.  I mean, if cognizant members express reservations, their
     views carry a lot of weight.
         Is there anything else the members want to say before we
     move on to safety culture?
         [No response.]
         DR. APOSTOLAKIS:  I must say I was pleasantly surprised to
     hear again the same members talk about how they wanted to see safety
     culture addressed.  Miracles never cease, I must say.
         MR. CUNNINGHAM:  Could I ask a question?  I believe we're on
     the agenda for the full committee in December.
         DR. APOSTOLAKIS:  Yes.
         DR. BARTON:  I think it has to be.  I think after this,
     you're going to have to be.
         MR. CUNNINGHAM:  That's the question.  What would you like
     for us --
         DR. BARTON:  To brief the other members.
         DR. APOSTOLAKIS:  How much time do you have?
         MR. PERALTA:  Probably just 45 minutes?
         DR. BARTON:  How much?
         DR. APOSTOLAKIS:  Forty-five minutes.
         Would it be useful to talk about the fire scenario and in
     the context of the scenario explain ATHEANA?  I don't think they can do
     both.
         MR. CUNNINGHAM:  I would agree.  I don't think we can do
     both.
         DR. POWERS:  I think they ought to just explain ATHEANA.  I
     don't think they should try the fire scenario.
         DR. APOSTOLAKIS:  I thought the scenario, the members found
     extremely useful.
         DR. BARTON:  Well, I think yes, it is, because it shows how
     they tried to apply --
         DR. APOSTOLAKIS:  Right.
         DR. BARTON:  -- the principles to an actual situation.  I
     think that does help.  Are you sure we can't squeeze some more time off?
         DR. POWERS:  No.
         DR. BARTON:  No, we can't.
         DR. APOSTOLAKIS:  Let us ask if Mr. Cunningham can structure
     it in such a way that he has the scenario, and on the way, you are
     explaining the method?
         MR. CUNNINGHAM:  Mr. Cunningham will try in 45 minutes.
         DR. APOSTOLAKIS:  We are reminded here -- is the document
     going to be available before the meeting on the fire scenario?
         MR. CUNNINGHAM:  I'm sorry; the --
         DR. APOSTOLAKIS:  We don't have anything in writing on the
     --
         MR. CUNNINGHAM:  On the fire scenario?  Will we have that
     for the full committee?
         MR. KOLACZKOWSKI:  There is certainly a draft available.
         MR. CUNNINGHAM:  Okay.
         MR. KOLACZKOWSKI:  The NRC has not a chance to review it
     yet, so it certainly is subject to revisions.
         DR. THOMPSON:  It's still in development.
         MR. CUNNINGHAM:  Okay.
         DR. APOSTOLAKIS:  So we're not going to have it?
         MR. KOLACZKOWSKI:  I don't think you're going to have it.
         MR. CUNNINGHAM:  No, okay.
         DR. APOSTOLAKIS:  Will that be a factor?  We cannot comment
     on something that we don't have?  But we have a presentation.  We have
     view graphs with a comparison, so we can comment on those, right?  We
     can say that we didn't have a written document, but they have some nice
     statements.
         Mark, again, I don't want to tell you how to structure the
     presentation, but the figure you have -- well, the classic ATHEANA --
         MR. CUNNINGHAM:  Yes.
         DR. APOSTOLAKIS:  -- maybe you can use that one and explain
     the elements of the process and then jump into the scenario.
         MR. CUNNINGHAM:  Okay.
         DR. APOSTOLAKIS:  I don't know.
         MR. CUNNINGHAM:  Okay.
         DR. APOSTOLAKIS:  Okay?  And we will try to refrain from
     repeating the same questions that we have done here, right?  And I see
     some smiles on the faces of some of my colleagues.
         [Laughter.]
         DR. APOSTOLAKIS:  But we will try; we will try.
         I think in fairness to Mr. Sorenson, we should move quickly
     on to his presentation, and I must tell you that I have to disappear at
     3:30, so, Jack -- where is Jack?
         MR. CUNNINGHAM:  Jack is in the back.
         DR. BARTON:  He said 3:30.
         DR. APOSTOLAKIS:  But I want some discussion.
         DR. BARTON:  And you have to leave at when?
         DR. SEALE:  He has to leave at 3:30.
         DR. APOSTOLAKIS:  Who's leaving at 1:00?
         DR. BARTON:  No, I said you have to leave when?
         DR. APOSTOLAKIS:  3:30.  So we have about an hour and a
     half.  I think it should be plenty, yes?
         DR. SEALE:  I have to leave at about 3:30, too.
         DR. APOSTOLAKIS:  Okay; no problem.  3:30, 3:32.
         [Pause.]
         DR. APOSTOLAKIS:  Okay; this is an initiative of the ACRS. 
     We don't know yet how far it will go; for example, our last initiative
     was on defense in depth, and it went all the way to presenting a paper
     at the conference PSA 1999, writing a letter to the commission and so
     on.  That does not mean that every single initiative we start will have
     that evolution.
         This is the first time that members of this committee
     besides myself are being presented with this, and we also plan to have a
     presentation to the full committee at the retreat; then, the decision
     will be up to the committee as to what the wisest course of action will
     be.  We have asked members of the staff to be here, like Mr. Rosenthal,
     who left; he is coming back.  Jay is here, and we asked the ATHEANA
     people to stay.  They kindly agreed to do it.  So we'll get some
     reaction from experts to our initial thoughts here, and again, where
     this is going to go is up to the committee, and we'll see.
         Mr. Sorenson has been working very diligently on this, so I
     think he deserves now some time.
         Jack?
         MR. SORENSON:  Thank you; I am Jack Sorenson.  This
     discussion is based on a paper that George asked me to write earlier
     this year.  There is a draft on his desk for comment.  But getting to
     this stage took a bit longer, I think, than either one of us thought.
         What I've attempted to do is put together a tutorial that
     will help non-practitioners of human factors-related things to
     understand what the state of the art is and what all the pieces are. 
     This morning, you heard -- and early this afternoon -- a great deal of
     discussion on one piece of a picture that I would like to draw in
     somewhat larger terms.  There is no attempt here to advance the state of
     the art in safety culture; just to understand it.  There is no attempt
     to review or critique the NRC human factors program.
         What you will hear is undoubtedly a somewhat naive view, and
     I would encourage those of you who are expert in one or more aspects of
     the subject to offer, I hope, gentle corrections when you feel I have
     misrepresented something.
         DR. APOSTOLAKIS:  I wonder why anyone would ask this
     committee to be gentle?
         [Laughter.]
         DR. APOSTOLAKIS:  Aren't we always?
         MR. SORENSON:  I was not, of course, not referring to the
     committee as being ungentle.
         DR. APOSTOLAKIS:  Oh, I see.
         [Laughter.]
         MR. SORENSON:  The three questions that were posed, I think
     by the planning and procedures subcommittee relative to safety culture
     are what is it?  Why is it important?  And what should the ACRS and NRC
     do about it?  We'll find out that the middle question, why it's
     important, is probably easier to deal with than either what it is or
     what people should do.
         The term safety culture was actually introduced by the
     International Nuclear Safety Analysis Group in their report on the
     Chernobyl accident in 1986.  A couple of years later, they actually
     devoted a publication to safety culture, and in that publication, they
     define it as shown here:  safety culture is that assembly of
     characteristics and attitudes in organizations and individuals which
     establishes that as an overriding priority, nuclear plant safety issues
     receive the attention warranted by their significance.
         There are other definitions that may be useful, and we may
     get to them later if it turns out that they're important, but the main
     thing is that there are -- whatever definitions of safety culture you
     use, there are requirements established essentially at three levels. 
     There are policy level requirements and management level requirements,
     and those two things together create an environment in which individuals
     operate, and it's the interaction between the individuals and the
     environment that is generally understood to be important here.
         The framework is determined by organizational policy and by
     management action and the response of individuals working within that
     framework.  Go on to four, please.
         Just a quick preliminary look at why it's important.  To
     understand its importance, I think you can simply look at what James
     Reason refers to as organizational accidents that have occurred over the
     10 years following TMI.  Of course, within the nuclear industry, it was
     the TMI accident that focused everybody on human factors issues.  In the
     10 years following TMI, there were a number of accidents where
     management and organization factors, safety culture, if you will, you
     know, played an important role.
         The numbers in parentheses following each of these on the
     list are the number of fatalities that occurred.  There was an American
     Airlines accident, plane taking off from Chicago where an engine
     separated from the wing.  It was later traced to faulty maintenance
     procedures.  The Bhopal accident in India, where methylisocyanate was
     released resulting in 2,500 fatalities; the Challenger accident;
     Chernobyl; Herald of Free Enterprise; some of you may be less familiar
     with that.  This was the case of a ferry operating between the
     Netherlands and England that set sail from its Dutch port with the bow
     doors open; capsized with somewhere around 190 fatalities.
         And the last one was Piper Alpha; it was an accident on an
     oil and gas drilling platform where one maintenance crew removed a pump
     from service, removed a relief valve from the system, replaced it with a
     blind flange which was leaking and leaking flammable condensate, and the
     second maintenance crew, the second shift crew, attempted to start the
     pump, and there was an explosion and resulting fire.
         In the nuclear business, other than Chernobyl and TMI, we
     typically end up looking at what are called near misses or significant
     precursors.  Two that come to mind are the Wolf Creek draindown event,
     where the plant was initially in mode four, I believe; 350 pounds per
     square inch; 350 degrees Fahrenheit.  There were a number of activities
     going on; heat removal was by way of the RHR system.  There was a valve
     opened, and 9,200 gallons of water were discharged from the primary
     system to the refueling water storage tank in about a minute.  The cause
     was overlapping activities that allowed that path to be established.
         There were numerous activities.  The work control process
     placed heavy reliance on the control room crew.  There was the
     simultaneous performance of incompatible activities, which were boration
     of one RHR train and strobe testing of an isolation valve in the other
     train.  The potential for draindown was identified but was not acted
     upon.  Probably the most significant item here was that the test was
     originally planned, the strobe testing was originally planned for a
     different time and was deferred, and there was no proper review done of
     the impact of that deferral.
         More recent event, Indian Point II, trip and partial loss of
     AC power.  The plant tripped on a spurious overtemperature delta-T
     signal; off-site power was lost to all the vital 480-volt buses.  One of
     those buses remained deenergized for an extended period and caused
     eventual loss of 125-volt bus and 120-volt AC instrument bus.  All
     diesels started, but the one powering the lost bus tripped.
         This had a number of human factors related to it.  The trip
     was due to noise in the overtemperature delta-T channel that was known
     to be noisy, and the maintenance to fix it had never been completed. 
     The loss of off-site power was due to the fact that the load tap changer
     was in manual rather than automatic, and that resulted in the loss of
     power to the buses.  The diesel trip occurred because there was an
     improper set point in the overcurrent protection and an improper loading
     sequence, and after that, post-trip activities were criticized by the
     NRC for being more focused on normal post-trip activities and not enough
     on the state of risk that the plant was in in attempting to recover from
     that risk.
         One of the things that is worth spending just a minute on is
     the idea of culture as a concept in organizational behavior.  The
     International Nuclear Safety Advisory Group introduces the term safety
     culture pretty much out of the blue.  They make no attempt to tie it
     back to the rather substantial body of literature that exists in either
     anthropology, where culture is a common term, or in organizational
     development, where it has become somewhat more common in the last 20
     years or so.
         The term is not without controversy, if you will,
     particularly among the organizational development people.  The term --
     the idea of ascribing something called culture to an organization
     started to show up in the organizational development literature in the
     very early eighties.  The two best-known books are probably Tom Peters'
     In Search of Excellence and a book by Deal and Kennedy entitled
     Corporate Cultures, and they essentially set out to determine why it was
     that organizations or at least some organizations didn't behave in ways
     that were clearly reflected in their structures; they were looking for
     some other attribute of the organization, and they settled on the term
     culture.
         There are people in the literature who take exception to
     that.  The expectation is if you use the term culture in an
     organizational sense or in the sense of a safety culture that it carries
     with it some of the properties of its original use.  That may or may not
     be true in the case of organizational culture or safety culture, but the
     fact remains that it has found a place in the literature.  It is quite
     widely used, particularly with respect to nuclear technology.  You will
     also find it in other writings in other industries, such as the process
     industries and aviation.
         Having said that, you find that virtually everyone then goes
     on to define it in a way that suits their immediate purpose.  I would
     like to go back to an opening remark which I missed, and that's that I
     knew I was going to have some difficulty with this assignment when I ran
     across an INSAG statement that said safety culture was the human element
     of defense in depth, and having spent a couple of years in defense in
     depth, it just seemed unfair that --
         [Laughter.]
         DR. POWERS:  One thing that you have to remember about the
     origins of the concept is that it came up after the Chernobyl accident. 
     There was a strong effort among parts of some people in the IAEA to
     shelter the RBNK design criticism, and you had to criticize the
     operators, okay?  But criticizing the operators individually was not
     going to fly any better, okay?  Because if you had a bad operator
     individually, why did they allow it?  Why did the system allow this bad
     operator to be this?  You had to go to this safety culture, okay?
         [Laughter.]
         DR. POWERS:  That preserved the RBNK from being attacked,
     and at the same time, it led to protecting the operators individually.
         DR. BARTON:  You have to admit they were a poor example of
     safety culture?
         DR. POWERS:  What did you say?
         DR. BARTON:  Nothing.
         MR. SORENSON:  Well, that makes a good bit of sense,
     obviously.  Although the idea of employee attitude or management and
     worker attitude having a significant impact on safety of operations, you
     know, considerably predates Chernobyl.  You can find references back to
     the early 20th Century when industrial accidents started to become
     significant in some way.
         Okay; I think we can, yes, go on to -- the definition on
     organizational culture, which is a little easier to deal with than
     safety culture, that was offered by a critic of the Peters and Kennedy
     and Deal books is the definition here.  Organizational culture:  the
     shared values; what is important and beliefs, how things work that
     interact with an organization's structure and control systems to produce
     behavioral norms; the way we do things around here.  This one appeared
     in an article by Brill, Utah and Fortune in the mideighties, and you'll
     see it repeated in very much the same form in current literature.
         The last phrase, the way we do things around here, I
     actually tracked back to one of the managing directors of MacKenzie and
     Company.  It seems to be the most concise definition of culture that --
         DR. APOSTOLAKIS:  That's the best one I like.
         MR. SORENSON:  There are competing terms:  safety culture,
     organizational culture, management and organizational factors, safety
     climate, safety attitudes, high reliability organizations, culture of
     reliability, and they all mean more or less the same or slightly
     different things, depending on how they're used and what the
     investigator decides to do with them.
         So I think it's important to keep in mind that there are no
     -- there is no generally agreed upon definition.  We are dealing with
     the way organizations work and the way people within those organizations
     react, and at some point, you choose a definition that fits your use and
     then hopefully apply it consistently thereafter.
         Dr. Powers?
         DR. POWERS:  This one is the sixth sigma?
         DR. APOSTOLAKIS:  I've heard that, to.
         DR. BARTON:  Sick or sixth?
         DR. POWERS:  Sixth.
         MR. SORENSON:  That's one of the zero-defect --
         DR. POWERS:  Yes.
         MR. SORENSON:  -- cults, is it not?
         DR. POWERS:  Do everything right, yes.
         MR. SORENSON:  Yes; I've run across the term within the last
     few weeks and I --
         DR. POWERS:  There was a survey in the Wall Street Journal
     about a month ago.
         DR. APOSTOLAKIS:  Did this agency actually do a
     self-assessment of its safety climate a couple of years ago?
         MR. SORENSON:  There was a survey by the inspector general. 
     I've actually been through the slides that are on the Web on that.  I
     don't think I've ever seen the text of the report.  And they were
     looking for something a little different than I would have called --
     than what I would have termed safety culture.  They were looking for, I
     think, more of the focus of the organization on its mission and assuming
     that if people were focused on the mission of the organization that that
     is factory safety culture.  I may be misrepresenting that but --
         DR. APOSTOLAKIS:  They put climate.
         MR. SORENSON:  They used the word culture.
         DR. APOSTOLAKIS:  No, I remember the word climate, because I
     was impressed.
         MR. SORENSON:  Well, they may have used that also, but I
     think the survey was titled a safety culture survey.
         DR. APOSTOLAKIS:  The French are using climate as well. 
     Climate is supposed to be really culture.  Culture is more permanent,
     presumably.
         MR. SORENSON:  One of the better-known writers in this
     general field, James Reason, in his book on managing organizational
     accidents lists the characteristics of a safety culture as a culture
     that results in -- that encourages the reporting of problems and the
     communicating of those problems to everybody throughout the
     organization; a culture in which or an organizational climate in which
     the members, the workers, feel justice will be done; an organization
     that is flexible in the sense of being able to shift from a hierarchical
     mode of operation under normal circumstances to a different mode of
     operation during a crisis or an emergency and then shift back; and then,
     finally, a learning organization where the information that is made
     available is incorporated into the way things are done.
         DR. SEALE:  That clearly indicates, then, that a safety
     culture is not an evolving set of values but rather a break with the
     past; I mean, I can think of organizations you might characterize as a
     benevolent dictatorship, and that was the way in which safety was
     imposed.  I guess under those circumstances, you would have to say the
     old DuPont organization really didn't have a safety culture, although it
     had a remarkable safety record.
         MR. SORENSON:  Yes; I think that's a fair characterization,
     as a matter of fact.
         DR. APOSTOLAKIS:  And I think a lot of the old-timers in the
     U.S. nuclear Navy also dismiss all of this and say Rickover never needed
     it.
         Now, the question is was the culture of the Navy good
     because of one man?  And do you want that?  Or do you want something
     more than that?  Rickover certainly didn't think much about human
     factors.
         DR. POWERS:  If you don't have enough people to go around --
         DR. APOSTOLAKIS:  I don't know, but Rickover did a good job.
         DR. POWERS:  There are people who would take a different
     view on that.
         [Laughter.]
         DR. POWERS:  And I think you can fairly honestly show that
     there are good and bad aspects of his approach, of his tyranny.
         MR. SORENSON:  There were two boats lost.
         DR. POWERS:  The time it takes to put a boat to sea, the
     mission of those boats and things like that -- you can change your
     approach.
         MR. SIEBER:  We did a survey a number of years ago of the
     idea of safety culture.  About 700 people out of 1,100 responded.  They
     had the same list you have, except they added personal integrity to that
     and caring attitude to that.
         DR. BARTON:  There were other characteristics.
         MR. SIEBER:  And that seemed to really work.  It changed the
     attitude in that facility; it really did, just finding out the
     practices.
         DR. BONACA:  Although the attribute of flexibility, I think,
     goes a long way in the direction.  That's the key item that you
     described there of when you go to technical issues, the ability of
     flattening out organization and not having any more pecking order or a
     fear of bringing up issues.  Flexibility is very important.
         MR. SORENSON:  One can deduce from the literature a few
     common attributes that virtually every -- all of the investigators
     share:  good communications; senior management commitment to safety;
     good organizational learning and some kind of reward system for
     safety-conscious behavior, and the lists expand from that point, if you
     will.
         DR. BARTON:  Conservative decision making.
         MR. SORENSON:  I'd like to take a step back here just a
     little bit and try to put the safety culture issue into the context of
     the larger issue of human factors, and to do that, I think looking at
     the National Research Council report done in 1988 on the NRC human
     factors program is useful.
         The National Research Council identified five areas that
     they thought the NRC, the nuclear regulators, should address in their
     human factors research.  First was the human-system interface; second,
     the personnel subsystem; the third, human performance; the fourth,
     management and organization; and the fifth, the regulatory environment. 
     The first two items, human-system interface and personnel subsystem,
     deal primarily with the man-machine interface, the way the machines are
     designed and the way the personnel are trained.
         Human performance in the context of that report is intended
     to deal with what this morning was referred to as unsafe acts of one
     kind or another, the actions of the system and equipment operators, and
     the management and organization, what they call management and
     organization factors are part of what they called a culture of --
     fostering a culture of reliability.  That was their phrase rather than
     safety culture; and third, the regulatory environment which dealt with
     the issue of how regulatory actions impacted the way the licensees did
     business.
         The safety culture, as I'm attempting to deal with it today,
     is focused on the fourth item, management and organization.  It creates
     the environment that human actions are taken in, and it may contain the
     ingredients to create what James Reason calls latent errors, those
     things which change the outcome of an unsafe act, but the issue of
     safety culture deals with the management and organization factors and
     the climate it creates, the conditions it creates for the human to
     operate in.
         One of the difficulties I had in going through the
     literature was trying to understand what all the pieces were, and so,
     one of the things that I ended up doing that helped me, and I think
     could be generally helpful in putting some of the pieces together is to
     look at all of the things that go into the process of establishing some
     interesting relationship between something called safety culture and
     operational safety or ultimately some measure of risk, and this figure
     shows the first half-dozen steps in that process.
         But the idea here is if safety culture is interesting for me
     from an operational safety standpoint, you need to be able to establish
     something about those relationships.  The process typically starts off
     with defining some kind of an organizational paradigm.  Mintzberg's
     machine bureaucracy is very often used for nuclear power plants, and
     then, as soon as it's used, it's criticized for having several
     shortcomings.  The investigators need to have some idea of how the
     organization works, and they generally should start with some definition
     of safety culture, what it is.
         Having done that, then, they need to define some attributes
     of safety culture, and it might be the ones that I listed a few minutes
     ago:  good organizational learning, good communications and so forth,
     but there are somewhere between a half a dozen and 20 of those
     attributes that can be identified, and having done that, then to
     evaluate organizations, you need to look -- you need to have a way to
     measure those things that you've just identified, and you might put
     together personnel surveys or, you know, technical audits or whatever,
     but you need some kind of evaluation technique that involves looking at
     how the organization, how an organization actually works.
         Having designed the evaluation techniques, you need to
     collect data, and then, you need to have, once you have data, you need
     to have something to -- that tells you how to judge that data, and I've
     indicated that by choosing external safety metrics; if you collect
     cultural data, for example, on an organization, how do you decide that
     that organization is safe or not safe in judging the cultural data?
         In their simplest form, those external metrics might be a
     SALP score.  They might be the performance indicators that we're using
     now.  They might be earlier performance indicators.  But the
     investigator makes some choice of what he's going to compare his
     cultural parameters to.
         And typically, that correlation is done with some sort of
     regression analysis, and as a result of doing the, you find out that
     some number of the safety culture elements you started with, you know,
     correlate with your safety parameters, and some don't.  And the output
     from that first stage, then, is which of these safety culture elements
     turn out to be significant.
         The remainder of the process, then, if you want to carry it,
     you know, all the way to its logical conclusion is you would like to be
     able to use these significant safety culture elements to modify in some
     way your measure of risk, and the next figure -- if you can move that
     one over a bit; pick up the balance of that.  The bottom path there
     identifies, you know, relating the elements that you've decided are
     significant to the PRA parameters or models; box 11 finally modifying
     the PRA parameters and ultimately calculating a new risk metric.
         DR. APOSTOLAKIS:  So I guess ATHEANA, then, because you
     don't necessarily have to go to that, ATHEANA would be somewhere there
     in between 9 and 10, perhaps?
         MR. SORENSON:  I would put -- well, it doesn't work on
     performance indicators, as I understand it.  I would say ATHEANA covers
     8 and 11; is that a fair statement?
         DR. APOSTOLAKIS:  It definitely does, but perhaps to take
     advantage of the qualitative aspects, you need an extra box so you don't
     just make it PRA.  So before eight, you might have the qualitative
     aspects of ATHEANA, and then, at the start of eight, of course, you have
     to do the quantification.
         MR. SORENSON:  I would be delighted to get critiques on
     this, too.
         DR. APOSTOLAKIS:  Don't worry; don't encourage people.
         [Laughter.]
         DR. APOSTOLAKIS:  Susan, you wanted to say something?  You
     have to come to the microphone, please; identify Your Honor.
         MS. COOPER:  Susan Cooper with SAIC.
         I think with respect to interaction with ATHEANA, there are
     certainly two different ways.  Already, we're trying to incorporate some
     symptoms, if you will, of culture and some of the preparation for doing
     ATHEANA.  We'd like the utility people to try to examine what are their
     pre-operational problems as part of identifying what their informal
     rules or maybe some things that are, if you will, symptoms of a culture
     that, when they play it out through a scenario development and
     deviations, it would be organizationally-related, but we don't have what
     we see from some of the events, some of the other things that the
     organization can do that might set up a scenario, so we recognize that
     there may be some pieces missing, and we certainly need some kind of
     input to know not only what -- you know, what from the organization is
     going to cause things but then also, then, what is the impact on the
     plant?  There are a couple of different pieces.
         DR. APOSTOLAKIS:  Now, if just for a couple of historical
     purposes, we go to the previous one, no, yes; box four, collect and
     analyze data, that was essentially the reason why one of the earlier
     projects on organizational factors funded by this agency was killed. 
     The proposed way of collecting data was deemed to be extremely
     elaborate.  They implemented it at Diablo Canyon, and the utility
     complained.
         So, there is this additional practical issue here that you
     have to do these things without really --
         DR. POWERS:  I don't know why.
         DR. APOSTOLAKIS:  Dana's commentary here, I mean, certain
     things, by their very nature, require a detailed investigation.  I mean,
     I don't know where this idea has come from that everything has to be
     very simple and done in half an hour, but I think it's important to bear
     in mind that the utility complained, and the management of the agency
     decided no more of this.  I'm willing to be corrected if anybody knows
     any different, but that was my impression.
         MR. SORENSON:  Well, and we'll touch on that
     one --
         DR. APOSTOLAKIS:  Okay; sorry.
         MR. SORENSON:  -- a little in a couple of slides, as a
     matter of fact, but you're right:  one of the results early on was that
     people did try to look for non-intrusive ways to collect data.  One
     possibility is to look at the way the organization is structured, which
     you can deduce from, you know, organizational documents, if you will.
         DR. APOSTOLAKIS:  Yes, but the attitudes, you would never
     get that.
         MR. SORENSON:  You don't pick them up and --
         DR. APOSTOLAKIS:  These attitudes, you don't pick that up.
         MR. SORENSON:  And interestingly enough, the people that
     started down that path after a few years started to pull in something
     that they called culture, the way an organization worked.
         Yes; I will, time-permitting, go through at least one
     example that sort of traces through those boxes, if you will.  I would
     like to comment on the upward path on slide 16.  The -- what you would
     really like to do is to be able to identify some number of performance
     indicators that were indicative of the safety culture elements and that
     you could translate, in turn, into modifications of the PRA parameters,
     and the idea there is if you can identify those performance indicators,
     then, you don't have to go back and do the intrusive measurements once
     you've validated the method.
         And so, in the best of all possible worlds, you know, one
     would, you know, have processes that follow that upward path.  Now, I
     would hasten to add in summarizing on this figure that there is a lot
     that goes on inside every one of those boxes, and, in fact, when I was
     discussing this with Joe Murphy -- I guess he's not here today -- and at
     one point, we pointed at one box in particular, and I asked him a
     question about it, and he said, well, of course, in that box, miracles
     occur, and that's still --
         DR. APOSTOLAKIS:  Did he also tell you that there's a NUREG
     from 1968 whose number he remembered that addresses it?
         [Laughter.]
         DR. APOSTOLAKIS:  I mean, Joe usually does that.
         [Laughter.]
         DR. APOSTOLAKIS:  PNL published a report in 1968 in March --
         [Laughter.]
         DR. APOSTOLAKIS:  -- that is relevant.
         MR. SORENSON:  So the -- anyway, the summary here is that
     this path is neither short nor simple.
         DR. APOSTOLAKIS:  Yes.
         MR. SORENSON:  There are a lot of pieces that go into
     establishing a relationship between safety culture or other management
     and organizational factors and some risk metric.
         Let me see what we might need to do here.  How much time do
     you want to leave for discussion, George?
         DR. APOSTOLAKIS:  Well, you are doing fine.
         MR. SORENSON:  Okay.
         DR. APOSTOLAKIS:  I think people can interrupt as they see
     fit.
         MR. SORENSON:  Okay.
         DR. APOSTOLAKIS:  So, you're doing fine.
         MR. SORENSON:  What I'd like to do now is go through some of
     the boxes and some examples of some work that has been done referring
     back to figures 15 and 16.  As the figure indicates, the process starts
     out somehow with a model of the organization you're interested in, and
     my conclusion as a layperson was that you can look at essentially the
     way an organization is structured; the way it behaves or its processes
     or some combination of those things.
         If you look at slide 18, this was an attempt to look at
     structure only.  The work actually started at, I believe, Pacific
     Northwest Laboratories and was continued by the same investigators,
     although at different places, over the next several years, and here,
     they attempted to look strictly at what they could deduce from the way
     the organization described itself, if you will.  It does not involve
     culture.  If you look at the literature referenced by these folks versus
     the literature referenced by organizational culture people, it's a
     different body of literature.  There's very little cross-referencing.
         This was designed to be non-intrusive.  It has an obvious
     difficulty right up front, and that is that there are a lot of factors
     to try to correlate.  They made an attempt to correlate with things like
     unplanned scrams, safety system unavailabilities, safety system
     failures, licensee event reports and so forth.  There was other work
     sponsored by the NRC that began at, I believe, at Brookhaven; Sonia
     Haber and Jacobs and others, not all at Brookhaven, I would hasten to
     add, and this was a slightly different perspective on the same thing. 
     They came up with 20 factors that included something they called
     organizational culture and safety culture, and this was the -- where the
     -- one where the data gathering, if you will, did become very intrusive. 
     They made up surveys and went out and talked to a bunch of plant people
     and shadowed managers and so on and so forth, and they probably got
     pretty good data, but it was not an easy process.
         Then, there is another process developed by -- I was going
     to say that eminent social psychologist.
         DR. APOSTOLAKIS:  I would like to add that Mr. et al. is
     here.
         MR. SORENSON:  Yes, good.
         DR. APOSTOLAKIS:  His first name is et; last name is al.
         [Laughter.]
         DR. APOSTOLAKIS:  We call him Al.
         MR. SORENSON:  Anyway, one of the contributions here was to
     reduce the 20 factors to half a dozen, which makes the process more
     tractable, if you will, but it's a little different also in the sense
     that it focuses on the work processes of the organization and how those
     are implemented, and, in fact, the next figure, I believe, is an example
     of their model of a corrective maintenance work process, and the
     analysis includes looking at the steps in the process and identifying
     the -- what they call barriers or defenses that ensure that an activity
     is done correctly, and you can map these activities back onto the
     earlier list of six attributes, if you will, to determine the
     relationship between the organization and the work processes.
         DR. APOSTOLAKIS:  One important observation, though:  these
     six are not equally important to every one of these.  This is a key
     observation.  For example, goal prioritization really is important to
     the first box, prioritization of the work process, whereas technical
     knowledge, for example, means different things for execution and
     different things for prioritization.  So that was a key observation that
     Rick made on the factors that Haber and others proposed to deal with the
     work process.  Then, it meant different things than he proposed.
         And most of the latent errors of some significance were the
     result of wrongful prioritization.  That is, we will fix it at some
     time, when it breaks; unfortunately, it breaks before you could --
         MR. SORENSON:  Okay; moving on to the next box in the
     activity diagram, coming up with some way to measure safety culture or
     whatever organizational factor you are concerned with, there is, you
     know, the obvious candidates:  document reviews, interviews,
     questionnaires, audits, performance indicators.  But I think the thing
     that struck me here is that regardless of what list of safety culture
     attributes you start with, in this process, you're going to end up with
     some questions that you hope represent those attributes in some way, so
     when you get done, you don't have just, you know, a direct measurement
     of organizational learning; you have answers to a set of questions that
     you hope are related in some way to organizational learning.
         DR. POWERS:  The difficulty in drafting the questionnaire
     that gives you the information that you're actually after must be
     overwhelming.  I mean, the problems that they have on these political
     polls, they can get any answer they want depending on how they construct
     the question.  I assume that the same problems affect the
     questionnaires.
         MR. SORENSON:  I would assume so, but this is also to assume
     what psychologists -- the organization --
         DR. APOSTOLAKIS:  Never rely on one measuring instrument.
         MR. SORENSON:  Would anybody like to comment on the
     difficulty that goes on within that box?
         DR. APOSTOLAKIS:  It's hard.
         MR. SORENSON:  It's hard.
         DR. POWERS:  That's a separate field of expertise,
     formulating questionnaires, is it not?  I'm really concerned that you
     asked too much to be able to formulate a questionnaire that allows
     somebody to map an organization accurately when you have this difficulty
     that I can get any answer that I want depending on how I construct the
     questions.
         MR. SORENSON:  Of course, part of the way round that is --
     well, there are ways of designing questionnaires so that the same
     question gets asked six different ways, and you can check for
     consistency and poor wording.
         DR. POWERS:  What do you do when they're inconsistent?  Do
     you throw it out?
         MR. SORENSON:  That's what you pay psychologists for.
         DR. POWERS:  I mean, I don't see that you're out of the game
     here.  I mean, I had enough to do with employee opinion poll taking and
     what not that it's been known that there is a culture or a discipline
     doing these things, and there are well-known principles, like the second
     year of the employee opinion poll, the results are always worse than the
     first year; the people filling out the questionnaires have gotten better
     at filling out questionnaires, so they can be more vicious in their
     evaluations.  I mean, it just strikes me as a flawed process.
         MR. SORENSON:  Well, I think part of the answer to that is
     you try to measure enough things that if your measure is flawed on one
     or two or three of them, you can still get the -- an indication of the
     attribute that you're really trying to measure.
         DR. SEALE:  It's interesting, because so many organizations
     now have been convinced that their organization has to be a
     participatory autocracy, and so, they ask these questions in the
     questionnaires, and as you say, they deteriorate almost invariably, but
     they also systematically ignore the results, so that --
         [Laughter.]
         DR. SEALE:  But, you know, in the name of, as I say,
     participatory autocracy, they do it.
         DR. POWERS:  I am intimately familiar with one organization
     who is absolutely convinced that the fact that they conducted a
     questionnaire on a particular aspect of behavior excuses them from ever
     again having to attend to that.
         [Laughter.]
         DR. APOSTOLAKIS:  Why didn't you include the behaviorally
     anchored rating scales?
         MR. SORENSON:  I didn't intentionally exclude it.  I didn't
     see it as different from -- in a process sense from what's here.  I may
     have misread that.
         DR. APOSTOLAKIS:  Anyway, okay, that's another of the
     instruments that's available.
         But let's go.
         MR. SORENSON:  Okay; selecting external safety metrics:  I
     mentioned that briefly earlier, you know, one can rely on performance
     evaluations, performance indicators, do some sort of expert elicitation
     to evaluate the organization.  In some industries, which we'll touch on
     in particular, process in aviation, actually, I have accident rates that
     you can use as a metric, where there is good statistical data on
     accident rates.  But again, the point I'm trying to make here is that
     the investigator chooses that as part of the evaluation process, and
     sometimes, that is lost sight of.
         In the chemical industry, process industries in particular,
     they tend to use the audit techniques.  They don't have the same
     reluctance to gather field data that seems to exist in the nuclear power
     business.  They tend to use the terminology safety attitudes and safety
     climate versus safety culture, and the studies that I've looked at used
     either self-reported accident rates or what they call loss of
     containment accident rates, you know, covering relatively large numbers
     of facilities.  One study covered, I think, 10 facilities managed by the
     same company, for example; 10 different locations.
         And these studies in the process industries have resulted in
     very strong statistical correlations between the attributes of safety
     culture that we've been talking about here and accident rates, and you
     can show that the low accident rate plants, you know, show strong safety
     culture attributes.  The typical correlation they might start out with,
     you know, 19 or 20 attributes as the Brookhaven people did and find out
     that 14 or 15 of those correlate and five don't for some reason.
         DR. SEALE:  Jack, how much of that, though, is due to the
     fact that the elements of positive numbers on the accident rate are the
     inverse or one minus the numbers on the safety culture?  I mean, they're
     almost -- the way you characterize your safety culture almost certainly
     is painted by the idea that one of the worst things that can happen to
     you is an accident.
         MR. SORENSON:  Well, certainly, you've got to look at how
     the measurement is done.  I don't have a quick answer.
         DR. SEALE:  No, I mean, what if you had just for instance or
     just for the fun of it, let's say we had two plants, and both of them
     didn't have any accidents; one of them had a good safety culture and one
     of them didn't.  I don't know if your questionnaire would actually
     detect or make that distinction.
         MR. SORENSON:  In that case, I think you're absolutely
     right, but precisely the point I'm trying to make here is that in this
     case, we are not looking at plants with zero accident rates.  We're
     looking at plants that have very low accident rates and very high ones.
         DR. SEALE:  Yes.
         MR. SORENSON:  So we've got statistics here that we don't
     have in the nuclear power business.  The ratio of the best performing to
     the worst performing in terms of accident rates is typically about 40,
     the factor is.  And, in fact, I'll come back to that later.  The reason
     that one of these folks makes the point is in aviation --
         DR. APOSTOLAKIS:  PSA is one minus the -- you know, that's
     my problem.
         MR. SORENSON:  The aviation business, which presumably uses
     roughly the same equipment and roughly the same training methods
     worldwide for commercial passenger airlines, there's a difference of
     about a factor of 40 between the best and worst performing airlines.
         DR. SEALE:  Yes.
         MR. SORENSON:  So the point here is precisely that in those
     areas where you've got data, you can correlate these safety culture
     elements, if you will.
         Which brings us to, you know, the areas of weakness or
     discomfort, most of which have been touched on here earlier.  One of
     them is that at this point, nobody pretends to understand the mechanism
     by which the thing we call safety culture affects operational safety. 
     Second area was what you just touched on, Bob.  There is a lack of valid
     field data in the nuclear power business in particular.  First, the
     actual accident rates are low, but there's even a lack of data on the
     safety culture side in general.
         And the third area is there are no good performance
     indicators that have been identified at this point; clearly an area that
     needs additional attention, not only in the nuclear power business.
         DR. BARTON:  I think you're looking at too high a level for
     the field data to be looking at accidents.  I think you don't have to
     look at accidents.  Go look at lower levels of performance in the
     organization; go look at industrial safety events.  Go look at human
     performance or look for operator errors.  Go look at maintenance people
     not following procedures.
         If you go look at a whole bunch of those things and relate
     that, you'll find out that the culture is different at that plant than
     it is at the other plant that hasn't had a major accident either but
     doesn't have the same numbers of those types of --
         DR. SEALE:  You could probably use LERs just as easy of
     that.
         DR. APOSTOLAKIS:  Or any number of attributes --
         DR. BONACA:  The trouble with LERs is there are not enough
     LERs written.  These plants write three or four LERs a year.  I don't
     know if there's enough data there.
         DR. BARTON:  Or whatever the correct level of --
         DR. SEALE:  Yes.
         DR. BONACA:  There are corrective action systems at the
     plants --
         DR. SEALE:  Yes, yes.
         DR. BONACA:  Because there are 20,000 inputs per plant.
         DR. SEALE:  Yes.
         DR. BONACA:  Probably, that's the biggest window that you
     have.
         DR. APOSTOLAKIS:  So you are saying that it would be perhaps
     worthwhile to see if some performance indicators can be formulated using
     this kind of evidence?
         DR. BARTON:  I think so.
         DR. APOSTOLAKIS:  Instead of going to models?  That's a good
     idea.
         DR. BARTON:  Think about it.
         DR. APOSTOLAKIS:  It would be extremely tedious to go
     through those records.
         DR. BARTON:  Oh, yes.
         DR. APOSTOLAKIS:  But it would probably be worthwhile.
         MR. SIEBER:  A lot of plants.
         DR. POWERS:  You can find people within an organization
     oftentimes who know those records surprisingly well.  If you have a lot
     more, then it's a lot easier.
         DR. BONACA:  I mean, an example of performance indicators at
     IAEA and all places, one could ask whether or not they should be nine or
     whatever.  But have they had those elements that were --
         DR. SEALE:  They weren't accidents.
         DR. BONACA:  No, incidents.
         DR. APOSTOLAKIS:  But that is a necessary assumption that
     this really is a good indication of what will happen if there is a need
     for an ATHEANA kind of system, but it may be very good when it comes to
     a major -- when they pay attention.  In fact, we had a guy call
     maintenance people; more than 50 percent, to my surprise, thought that
     the procedure was useless; they never followed them.  They thought they
     were for idiots.
         Now, those guys probably are very good, but if you are
     blind, you say oh, they don't use the procedures; my God, bad, bad boy. 
     Yes; they're probably doing a better job than somebody else who goes
     with --
         DR. BONACA:  Even there, that's another issue.
         DR. APOSTOLAKIS:  So I think there is this presumption,
     although I like the idea, because at least you get something concrete,
     but maybe that's something else to think about:  how much can you
     extrapolate from these fairly minor incidents, because there is this --
     Jack didn't mention, but people also distinguish between the formal
     culture and the informal culture, the way things really get done.  And
     do they take shortcuts?  They do all sorts of things.  And these are
     good people usually.  I mean, they're not -- but I think that's a good
     idea.  It's a good idea.  It's just that, I mean, they have -- you know,
     whenever anybody proposes anything here, you have to say something
     negative about it.  So, there it goes.
         Alan, you have to come to the microphone.
         MR. KOLACZKOWSKI:  Alan Kolaczkowski, SAIC.
         George, that's the very reason why, in the ATHEANA part, I
     think we're looking at both the EOPs and the formal rules, but then, you
     saw we also look at tendencies and informal rules.
         DR. APOSTOLAKIS:  Right.
         MR. KOLACZKOWSKI:  That's where we're trying to capture some
     of those -- part of the culture, if you will:  how do they really do it? 
     What are the ways they really react when this parameter does this?  What
     are their tendencies?  I think we're trying to capture some of that.  We
     use the terminology informal versus formal rules, but I think we're
     talking about the same kind of thing.
         DR. APOSTOLAKIS:  Yes.
         MR. SORENSON:  By the way, though, not all investigators
     agree that let me call them near misses or incidents extrapolate
     properly to accidents.
         DR. APOSTOLAKIS:  Yes, you have to make some assumptions.
         MR. SORENSON:  And also, the people who question that also
     question whether the human performance information or models in the
     nuclear business translate to those in other hazardous industries. 
     That's not a given.
         DR. APOSTOLAKIS:  Go ahead.
         DR. SEALE:  But the point may be, though, that the extent to
     which the organization has the capability of absorbing near misses in
     such a way that they do not propagate to major accidents may be the
     thing that's the measure of safety culture.
         MR. SORENSON:  Well, Reason would agree with that very
     precisely, because his definition of safety culture, you know, is, in
     effect, that culture which leads to a small incidence of latent errors
     that go undiscovered.  And it's the latent errors that translate, you
     know, a single unsafe act into a disaster.
         DR. SEALE:  And then, but the ability to correct for the
     error in other parts of the organization so that it doesn't grow --
         MR. SORENSON:  Right.
         DR. APOSTOLAKIS:  But I think another measure of goodness
     which is really objective is to see whether they actually have work
     processes to do some of these things.  Rick is working with -- Rick Weil
     is trying to develop organizational learning work.  So what you find is
     that yes, everybody says, boy, organizational readiness, sure, yes, we
     do that.  But how do you do it?  And that's where he gets stuck.  We do
     it.  Somehow, we do it.  There is no work process; they have no formal
     way of taking a piece of information, screening it, because that's the
     problem there:  they get too many of those.
         DR. BARTON:  How many do they get a week or about a year?
         DR. APOSTOLAKIS:  About 6,000 items a year; I mean, here,
     you're not going to be producing power just to study 6,000 items.
         [Laughter.]
         DR. BARTON:  I hope not.
         DR. APOSTOLAKIS:  So there is no formal mechanism for
     deciding what is important, which departments should look at it, and I
     think that's an objective measure.
         DR. BARTON:  Yes, it is, because you can prioritize those
     6,000.
         DR. APOSTOLAKIS:  But they don't.
         DR. BARTON:  You can put them in buckets.  Well, I know
     plants that do.
         DR. APOSTOLAKIS:  I'm sure; and those have a better culture.
         DR. BARTON:  I don't necessarily agree with that.
         [Laughter.]
         DR. APOSTOLAKIS:  All right; no, but it is an objective
     measure of the existence of the processes themselves.  It is a measure
     of some attempt to do something.
         DR. BONACA:  But it is also a measure of the way the work is
     getting accomplished or not accomplished that gives you some reflection
     on potential initiators.  For example, a process that is overwhelmed
     that is unable to accomplish work on a daily basis, something is going
     to happen out there, because we're starting an item; you are closing it. 
     You're delaying items, and something is going to start in a new activity
     before you close the other one at some point.
         And so, if you look at that, you have a clear indication,
     and we're trying to begin to correlate that.  So you have some
     indication of really what kind of a story.  Now, the question is are
     they going to affect the unavailability of a system?  See, we don't know
     that.
         DR. APOSTOLAKIS:  It may, but -- but there is something to
     the argument that -- not just nuclear.  But it seems to be consensus of
     organizational learning is a key characteristic of good organizations. 
     Now, if I see that, I really don't need to see real data to prove that. 
     I mean, those guys are not stupid.  They know what they're talking
     about.  And, in fact, I remember there was a figure from a paper in the
     chemical, whatever; it was a British journal, comparisons of good
     organizations, excellent organizations.  The key figure that
     distinguished excellent from everybody else was this feedback loop,
     organizational learning, from your own experience and that of others,
     and it's universal.
         Anyway, let's have Jack continue.  He's almost done, I
     understand.
         MR. SORENSON:  Yes; there are a couple more slides here, and
     I did want to touch on what we've just been discussing, you know, the
     evidence that a safety culture is important to operational safety. 
     There is an overwhelming consensus among the investigators; if there is
     a subculture that thinks an attitude doesn't matter, I didn't find it in
     the literature in any event.
         The accident rate data is pretty convincing.  I confess
     obviously to not being an expert, but the writing, again, supports that. 
     People outside of the field seem to think they have good statistical
     information there.  And the little bit of nuclear power plant field data
     that there is, some of what the Brookhaven people did, Hauber and her
     colleagues and the little bit that was done in the Pacific Northwest
     Laboratory work confirmed a correlation between safety culture elements
     and operational safety as they defined it.  There are not enough data,
     but what's there was positive.
         I'm going to, on the last slide, relate my impressions again
     as a non-practitioner as to what is missing from the literature.  Some
     of this, I've deduced from what other people have written and some just
     from my own feelings on the papers that I review.  There is a lack of
     field data relative to nuclear power plant operations.  There might be
     easy ways to get it, but right now, it's not there.
         One needs to understand the mechanism by which safety
     culture or other management and organizational factors affect safety. 
     We need performance indicators for safety culture or related things.  We
     need to understand the role of the regulator in promoting safety
     culture, and we need to know something about the knowledge, skills and
     abilities of the front line inspectors in a regulatory environment where
     safety culture is important.  One of the things that struck me in doing
     the research on this work is that we are -- we, the NRC -- are right in
     the middle of attempting to change the way we do regulation.  We are
     embarking on and evaluating a new reactor oversight process.  We are
     trying to convert our regulatory basis to something we're calling
     risk-informed and maybe performance-based, and other regulators
     elsewhere in the world, particularly in the UK, are observing that.  If
     one is going to make this kind of a change, then you probably cannot do
     it within the kind of prescriptive regulatory framework that the U.S. is
     using at the moment.
         That being the case, something called safety culture and how
     one fosters it becomes very important relative to the new regulatory
     process that we are expecting to implement.  There is certainly a
     reluctance on the part of the NRC to, you know, venture into anything
     that would smack of regulating management and an even stronger
     reluctance on the part of the industry to, you know, allow any small
     motion in that direction, but it seems to me that in the context of this
     new regulatory regime, that management is terribly important, and at a
     minimum, the agency needs to understand in what ways is it important,
     and how does the agency best foster this ownership of safety amongst its
     licensees, and I don't think we know that right now.
         That's all I have.
         DR. APOSTOLAKIS:  Yes; the big question is really what is it
     that a regulator can do without actually managing the facility.  That's
     really the fear.
         Dennis Bley, please?
         MR. BLEY:  My name is Dennis Bley.  I'm with Buttonwood
     Consulting.  I have to leave in just a minute --
         DR. APOSTOLAKIS:  Sure.
         MR. BLEY:  -- so I thought I would say a couple of words
     quickly.  The last 5 years, I've been on the National Academy committee
     overseeing the Army's destruction of chemical weapons, and the program
     manager for chemical weapons destruction has sponsored a lot of digging
     into this area, and I think maybe they would be willing to share what
     they've found.
         We've had people on our committee from DuPont, and, you
     know, the strong view from DuPont, coming back to what you were talking
     about earlier, is that if you get the little things under control, the
     industrial accident rates, those things, you won't have a bad accident. 
     A lot of people don't believe that.  They do very strongly.  Jim
     Reason's book you were talking about, I think the last chapter, tenth
     chapter, he goes into that in some detail.
         I kind of think from NRC's point of view, it gets difficult,
     because the expertise the Army has brought together to help them look at
     this in many places has all argued strongly that strong regulation and
     compliance don't get you where you want to be with respect to safety; it
     has to be the individual organization taking ownership, and all the way
     through, certain things are unacceptable, certain kinds of behavior are
     unacceptable by anybody, and that has to get buried into the whole
     organization.
         Just an aside on ATHEANA, it would be -- where you pointed
     out where they would fit together, I think that's about right, and we've
     actually got, if you look at some of our examples, a little of that
     coming in but nothing like a real solid process for trying to find all
     of it.  But I think you can -- there has been so much work in this area
     by so many different people, including studies in industrial facilities,
     that it probably doesn't make sense to do it all over again.
         But I'll just leave it with that.  That's the one source
     I've seen where people have -- they've really tried to draw a broad
     range of expertise together to help them with the problem, which they
     haven't solved.
         DR. APOSTOLAKIS:  I believe the fundamental problem that we
     have right now is that people understand different things when the issue
     of culture is raised and so on.  There was a very interesting exchange
     between the commissioners and the Senators.  Senator -- I don't
     remember; Inhofe?
         DR. SEALE:  Inouye?
         DR. APOSTOLAKIS:  No, no, no.
         DR. SEALE:  Inhofe, yes.
         DR. APOSTOLAKIS:  He was told by Former Chairman Jackson
     something about -- it was somebody else; not the chairman about culture
     and organizational factors and boy, he said I've never heard -- he said
     I'm chairing another subcommittee of the Senate where we deal with
     Boeing Corporation and all of those big -- and I've never heard the FAA
     trying to manage the culture at Boeing and this and that, and how dare
     you at the NRC think about that?
         And then, of course, we have our own commission stopping all
     work, you know, overnight a year or so ago, and I think it's this
     misunderstanding; you know, I really don't think it's the role of the
     regulator to go and tell the plant manager or vice president how to run
     his plant.  On the other hand, there are a few things that perhaps a
     regulator should care about.  I don't know what they are, but for
     example, the existence of a minimum set of good work processes, in my
     opinion, is our business, and especially if we want to foster this new
     climate that I believe both Dennis and Jack referred to.  In a
     risk-informed environment, some of the responsibility goes to the
     licensee.  Now, we are deregulating electricity markets and so on, so
     that's going to be even more important.
         But I guess we never really had the opportunity to identify
     the areas where it is legitimate for a regulatory agency to say
     something and the areas where really it is none of our business, and
     it's the business of the plant.  And because of the fear that we are
     going to take over and start running the facility, we have chosen to do
     nothing as an agency.
         DR. SEALE:  Well, that goes to the question of where is it
     we ought to butt out?  Where should we butt out?  What are the things
     that we do that are counterproductive?
         DR. APOSTOLAKIS:  Absolutely right; absolutely right.
         DR. BONACA:  But again, I think if you want to talk about
     culture, management up there, it's very, very hard, and again, we're
     struggling with looking at an indication of an organization that works
     or doesn't work.  At the industrial level, there are indications all
     over the place.  But those indicators have to do with does the work
     process work, for example?  Is the backlog that people perceive they
     have overwhelming them?  What kind of -- absolutely.  And again, there
     is work that is being done inside these utilities to look at those
     indicators there, and they don't even measure management per se; simply
     something is wrong with the organization.  When you have something wrong
     with the organization, you go to the management, and you change it,
     because you expect that you will be able to manage that.
         But I'm saying that it's probably feasible to come down to
     some of these indicators, and I think that the utilities are trying to
     do that.
         MR. SIEBER:  I would sort of like to add:  I've been to some
     regional meetings for clients of mine where the plants have been having
     problems, where the regional administrator or his staff has asked
     questions about performance indicators on productivity, and for example,
     a lot of these processes are just a bunch of in-boxes, you know, like
     your work process.  Which one is the in-box that has big holes in it? 
     Why isn't work getting done?  I've seen the NRC ask those questions.  I
     think they're legitimate questions, and on an individual basis, I think
     that they're appropriate questions, but I have not seen an initiative to
     ask them across the board.
         DR. BARTON:  They all do relate to cultural issues.
         MR. SIEBER:  That's right.
         DR. POWERS:  Yes.
         MR. SIEBER:  Each one of them by itself is an indicator, and
     I think industrial safety is a prime indicator.  You know, if you --
         DR. BARTON:  If it wasn't, they wouldn't spend so much time
     looking at it.
         MR. SIEBER:  Yes, and we actually hired DuPont, who is very
     good, to help us with ours, and our record, our accident rates, went
     down by over 90 percent.  I mean, it actually worked, and that's part of
     the culture.  If you can't make yourself safe, how can you make a power
     plant safe?
         DR. BARTON:  There are things you can look at without really
     getting into the management, so to speak, of the company.  I think you
     have to draw that line, because the industry is going to get nervous as
     heck.  They're just going to say -- they'll start looking at the safety
     culture and management's confidence and all that stuff.  I think there
     is a set of things that you can look at objectively and determine what
     is the culture of this organization.  You just have to figure out how to
     package it.
         DR. APOSTOLAKIS:  That's the problem.
         DR. BARTON:  How to package it.
         DR. APOSTOLAKIS:  That's the problem.
         DR. BARTON:  Expect that if you're looking at a bunch of
     indicators right now that I would tell you would fit into a box called
     culture.  Look at it right now.
         DR. BONACA:  Well, I mean, again, there have been efforts;
     I've been participating in one, and I believe that if you look at other
     people who do it, they're finding out the same points.  Now, again,
     you're going down to opinions for objective readings of certain boxes of
     work being accomplished or not accomplished.
         DR. BARTON:  And that's the problem.  It's what you can do
     when you take this data, and you get it back to the region, and that's
     where people really get nervous now.
         DR. BONACA:  But I was talking about trying to correlate,
     for example, working efficiencies of backlogs, actual outcomes that you
     can measure somewhat for using PRA.  That's -- I mean, that's probably
     something that you can do.
         MR. SIEBER:  One of the problems is that the boxes from
     plant to plant are not standardized.  The thresholds that differ from
     plant to plant.  So interplant comparisons are not very accurate.  On
     the other hand, you know, something is better than nothing.  And that's
     what plant managements use to determine the state of culture and how
     safe they are and how safe they aren't and how well their processes
     work.  That's how you run the plant.
         DR. POWERS:  One of the things that I find most troublesome
     right now is taking the DuPont experience, and this attitude I hear all
     the time, the Mayer approach toward safety; you take care of all the
     little things, and the big things will take care of themselves versus we
     want to focus on the most important things in risk assessment.  We seem
     to be dichotomizing opposite views.  I'm wondering if we really want the
     outcome we're going to get going to risk-informed.
         DR. SEALE:  I'm not so sure.
         DR. POWERS:  It seems like it's worth thinking about,
     because these things have been very successful in another industry.
         DR. SEALE:  The thing, though, is that the things that are
     getting ruled out, if you will, on the basis of not contributing to risk
     are not the little things that show up in the plant performance things. 
     They're truly the -- they're the not even on the radar screen things. 
     At least that's my impression.  It's a good point, but I don't think
     you're talking about the same population when you say risk versus low
     risk on the one hand and little things versus big things on the other
     hand.
         DR. APOSTOLAKIS:  Anyone from the staff or from the audience
     want to say anything?
         [No response.]
         DR. APOSTOLAKIS:  Okay; any other comments?
         [No response.]
         DR. APOSTOLAKIS:  Thank you very much.  We will adjourn. 
     So, this meeting of the subcommittee is adjourned.
         [Whereupon, at 3:10 p.m., the meeting was concluded.]

Page Last Reviewed/Updated Wednesday, February 12, 2014