United States Nuclear Regulatory Commission - Protecting People and the Environment

ACCESSION #:  9412050146

                       LICENSEE EVENT REPORT (LER)

FACILITY NAME:  Palo Verde Unit 2                         PAGE: 1 OF 12

DOCKET NUMBER:  05000529

TITLE:  Class 1E Batteries in a Degraded Condition

EVENT DATE:  10/06/94   LER #:  94-004-     REPORT DATE:

OTHER FACILITIES INVOLVED:                          DOCKET NO:  05000

OPERATING MODE:  5   POWER LEVEL:  0

THIS REPORT IS SUBMITTED PURSUANT TO THE REQUIREMENTS OF 10 CFR
SECTION:
50.73(a)(2)(i), and 50.73(a)(2)(viii)(A)

LICENSEE CONTACT FOR THIS LER:
NAME:  Burton A. Grabo, Supervisor,         TELEPHONE:  (602) 393-6492
       Nuclear Regulatory Affairs

COMPONENT FAILURE DESCRIPTION:
CAUSE:  B   SYSTEM:  EJ   COMPONENT:  BTRY MANUFACTURER:  A626
REPORTABLE NPRDS:  Y

SUPPLEMENTAL REPORT EXPECTED:  YES

EXPECTED SUBMISSION DATE:  12/30/94

ABSTRACT:

At approximately 2200 MST on October 7, 1994, Palo Verde Unit 2 was in
Mode 5 (COLD SHUTDOWN), when Train A (battery banks A and C) was declared
inoperable because a projection of the test results based upon
anticipated degradation indicated that they did not meet the 90% criteria
of Technical Specification Surveillance Requirement (TS SR) 4.8.2.1.e,
rendering both trains of Class 1E batteries inoperable.  Train B (battery
banks B and D) was declared inoperable (on October 1, 1994) because the
measure capacity was slightly less than the required 90% capacity stated
in TS SR 4.8.2.1.e.

Subsequent investigation has determined that Train B had been slightly
below the 90% capacity as required by TS SR 4.8.2.1.e since February 1,
1994.  Since both trains of batteries are required in Mode 1, Unit 2
operated in a condition prohibited by the plant's TS until it shutdown
and reached Mode 5 on September 18, 1994.  This condition is being
reported per 50.73(a)(i)(B).

The Equipment Root Cause of Failure Analysis (ERCFA) has not been
completed to date.  The investigation has narrowed the possibilities for
the degraded capacity to either the manufacturing process or the quality
control process during manufacturing.  This manufacturing problem has
potentially led to a single cause that caused both Class 1E battery
trains to become inoperable.  Therefore, a 50.73(a)(2)(vii) notification
is also being made by this LER.

There have been no previously similar events reported pursuant to 10CFR
50.73.

END OF ABSTRACT

TEXT                                                         PAGE 2 OF 12

I.   DESCRIPTION OF WHAT OCCURRED:

     A.   Initial Conditions:

          At 2200 MST on October 7, 1994, Palo Verde Unit 2 was in Mode 5
          (COLD SHUTDOWN) with Pressurizer pressure of 10 psia and a Main
          Coolant Temperature of 107 degrees F.

     B.   Reportable Event Description:

          Event Classification:    Any operation or condition prohibited
                                   by the plant's Technical
                                   Specifications.

                                   Any event where a single cause or
                                   condition caused at least one
                                   independent train or channel to become
                                   inoperable in multiple systems or two
                                   independent trains or channels to
                                   become inoperable in a single system
                                   designed to: A) Shut down the reactor
                                   and maintain it in a safe shutdown
                                   condition; B) Remove residual heat; C)
                                   Control the release of radioactive
                                   material; or D) Mitigate the
                                   consequences of an accident.

          At approximately 2200 MST on October 7, 1994, Palo Verde Unit 2
          was in Mode 5 (COLD SHUTDOWN), when Train A (battery banks A
          and C) was declared inoperable because a projection of the test
          results based upon anticipated degradation indicated that they
          did not meet the 90% criteria of Technical Specification
          Surveillance Requirement (TS SR) 4.8.2.1.e, rendering both
          trains of Class 1E batteries inoperable.

TEXT                                                         PAGE 3 OF 12

          Train B (battery banks B and D) had been declared inoperable on
          October 1, 1994 because the measured capacity was slightly less
          than the required 90% capacity stated in TS SR 4.8.2.1.e. TS SR
          4.8.2.1.e, states the following in part:

          At least once per 60 months, during shutdown, by verifying that
          the battery capacity is at least 80% (Exide) or 90% (AT&T) of
          the manufacturer's rating when subjected to a performance
          discharge test.  This performance discharge test may be
          performed in lieu of the battery service test required by
          Surveillance Requirement 4.8.2.1d.

          Subsequent investigation has conservatively determined that
          Train B had been slightly below the 90% capacity as required by
          TS SR 4.8.2.1.e since it was last tested on February 1, 1994.
          Since both trains of batteries are required in Mode 1, Unit 2
          operated in a condition prohibited by the plant's TS until it
          shutdown and reached Mode 5 on September 18, 1994.

          During the recent mid-cycle outage, Unit 2 was performing TS SR
          4.8.2.1.e to satisfy the IEEE Standard 450-1980 requirement to
          capacity test new batteries within the first two years of
          service.  On September 23, 1994, the test results for battery
          banks A and C were 91.6% and 90.6% respectively.  While the
          test results met TS SR, the capacity of the battery banks were
          below what was expected.  As a result, the B and D battery
          banks were capacity tested.  On October 1, 1994, the B and D
          battery banks were declared inoperable because the measured
          capacity was slightly less than the required 90% capacity
          stated in TS SR 4.8.2.1.e.

          As a result of this unexpected degradation, Electrical
          Maintenance Engineering (utility, non-licensed) performed an
          individual cell and battery capacity evaluation on previous
          tests of banks A, B, C, and D, factory tests, and additional
          testing on the Unit 2 spare cells.

TEXT                                                         PAGE 4 OF 12

          The evaluation concluded that the failure mechanism causes the
          batteries to degrade during the discharge/recharge cycle,
          therefore, the projections for battery capacity from this
          evaluation indicated that all four battery banks in Unit 2 were
          inoperable.  Subsequently, on October 7, 1994, battery banks A
          and C were also declared inoperable because a projection of the
          test results based upon anticipated degradation indicated that
          they did not meet the 90% criteria of TS SR 4.8.2.1.e.

          Based on this new information, a condition prohibited by the
          plant's TS has been identified.  TS 3.8.2.1 requires that both
          trains of Direct Current (DC) sources be operable in Modes 1
          through 4.  Due to the fact that the degradation mechanism is
          related to discharge/recharge cycling of the batteries, APS has
          concluded that battery banks B and D were conservatively at
          89.0% and 88.3% capacity respectively following the ST
          performed in February 1, 1994.  Therefore, battery banks B and
          D have been slightly below the 90% capacity as required by TS
          SR 4.8.2.1.e since February 1, 1994.  Unit 2 operated in Mode 1
          with this condition until it shutdown for the September
          mid-cycle outage and was in Mode 5 on September 18, 1994.

          On October 9, 1994, APS submitted a proposed TS amendment to
          Specification 3/4.8.2, DC Sources, under emergency
          circumstances.

          APS has concluded that the failure mechanism causes the
          batteries to degrade during the discharge/recharge cycle and
          that the projected capacities of the banks following the last
          capacity discharge test are: 1) Bank A, 78.82%, 2) Bank B,
          82.49%, 3) Bank C, 76.73%, and 4) Bank D, 81.75%.  As such, all
          banks were above the calculated design minimum capacity of
          55.22% at all times.

TEXT                                                         PAGE 5 OF 12

          The amendment asked for approval to suspend the requirements of
          TS 4.0.1 and 4.0.4 until entry into Mode 4 coming out of the
          fifth refueling outage or upon any deep discharge of the
          battery.  On October 13, 1994, TS amendment 71 was approved by
          the NRC.

          Therefore, on October 13, 1994, battery banks A, B, C, and D
          were declared operable.

     C.   Status of structures, systems, or components that were
          inoperable at the start of the event that contributed to the
          event:

          As stated in Section I.B above, Train B (battery banks B and D)
          of the Class 1E batteries were inoperable since February 1,
          1994.  However, based on information at the time Train B was
          believed to be operable because of its capability to pass the
          surveillance test requirements.

     D.   Cause of each component or system failure, if known:

          On October 6, 1994, the three worst cells from Train B, battery
          bank D arrived at C&D Charter Power Services, Inc. (the
          manufacturer) for a root cause determination.  This root cause
          effort is being overseen by APS with the participation of C&D,
          AT&T (the cell designer) and Failure Prevention International
          (FPI, consultant).

          The three cells have undergone preliminary examination to
          exclude sudden loss of capacity as the potential cause of
          failure.  The three sudden failure modes are 1) open or near
          open circuit, 2) low impedance short circuit, and 3) massive
          contamination of electrode active material.  Tear down of three
          cells revealed some signs of poor workmanship, but there was no
          evidence that the diminished capacity is attributable to
          physical damage of the cell internal components.

          The cell examinations also ruled out sudden loss of capacity
          (due to item 1 or 2 as stated above) as a failure mode for
          these particular cells.

TEXT                                                         PAGE 6 OF 12

          Chemical analysis of the electrolyte and the plate's active
          material has ruled out massive contamination (item 3 as stated
          above).  It was, therefore, concluded that for this cell the
          failure mode would be a step decline in capacity due to an
          active material utilization problem during the
          discharge/recharge cycle.  APS, with concurrence from AT&T and
          FPI, has concluded that the discharge behavior is not
          indicative of a sudden discharge type failure.

          Currently, the Equipment Root Cause of Failure Analysis (ERCFA)
          is focusing on the possibility of manufacturing problems or
          quality controls problems during the manufacturing process.

          The ERCFA team has identified that the loss of capacity is a
          result of "PbO sub 2 grain isolation" (a reduction of active
          material utilization due to a loss of electronic contact
          between the PbO sub 2 grains themselves within the plate and/or
          between the grains and the grid).  The most likely cause for
          this isolation can be attributed to improper curing of the
          positive plates during manufacturing.  Chemical and microscopic
          analyses have been performed on samples of plates from both
          failed and "good" cells.  The results of the analyses are still
          being evaluated, but no evidence has been found which would
          invalidate the PbO sub 2 isolation mechanism.

          Degradation of the battery appears to be manifested during the
          second hour of the two hour capacity discharge test, or, in
          other words, the last 530 ampere-hours of the 1060 ampere-hour
          test.  This is evident when reviewing the discharge test curves
          for the batteries.  The actual load profile demand for the A
          battery channel over a two hour period only removes
          approximately 423 ampere-hours.  If a plant demand of only 423
          ampere-hours had occurred on the B and D battery from February
          thru October 1, 1994, Engineering believes the batteries would
          have behaved similar to a 100% capacity battery because only
          423 ampere-hours would have been removed during the two hour
          demand.

TEXT                                                         PAGE 7 OF 12

          As part of the ERCFA Electrical Maintenance Engineering
          evaluated the capacity discharge testing at PVNGS.  The
          following conservatism's were noted:

          During capacity discharge testing at PVNGS, an extra 5 amperes
          were added to the required load current to be conservative for
          any possible tolerances in test equipment.

          These extra 5 amperes results in the battery capacity to
          measure approximately 1% less than it would measure without the
          extra 5 amperes.

          The capacity tests up thru October 1, 1994, were performed
          using the original AT&T two hour rating of 530 amperes.  The
          new two hour AT&T rating for the same battery is now 514
          amperes.  Thus, if the B and D batteries had been discharged
          tested at the new rating of 514 amperes, the battery capacity
          would have measured approximately 4% better.

          In summary, the extra 5 amperes added for test equipment
          tolerances and use of the old 530 ampere rating for two hours
          on the AT&T cells account for the B and D batteries performing
          approximately 5 less in capacity on October 1, 1994, than they
          would have performed otherwise.  Thus, the actual capacity test
          on October 1, 1994, would have expected to measure
          approximately 5% better if they were tested at the new 514
          ampere rating and the extra 5 amperes not added.

          Further examination and testing will be performed to determine
          the cause of failure.  A supplement to the LER will be made
          when a root cause of failure has been determined.

     E.   Failure mode, mechanism, and effect of each failed component,
          if known:

          The cause of failure has not been finalized.

TEXT                                                         PAGE 8 OF 12

          However, APS had determined through testing that the
          degradation mechanism is related to loss of positive plate
          activity which is aggravated by the discharge/recharge cycling
          of the batteries as discussed in Section I.D.

          APS has concluded that the failure mechanism causes the
          batteries to degrade during the discharge/recharge cycle and
          that the projected capacities of the banks following the last
          capacity discharge test are: 1) Bank A, 78.82%, 2) Bank B,
          82.49%, 3) Bank C, 76.73%, and 4) Bank D, 81.75%.  As such, all
          banks were above the calculated design minimum capacity of
          55.22% at all times.

     F.   For failures of components with multiple functions, list of
          systems or secondary functions that were also affected:

          The DC banks A and B provide control power for Alternating
          Current (AC) load groups 1 and 2 respectively.  These banks
          also provide vital instrumentation and control power [JC] for
          channels A and B, respectively, of the reactor protection and
          Emergency Safety Features (ESF) systems and diesel generators
          [DG] A and B respectively.  The DC banks C and D provide vital
          instrumentation and control power [JC] for channels C and D,
          respectively, for the reactor protection and ESF systems, and
          other safety-related loads as referenced in Table 8.3-6, Class
          1E DC System Loads, of the Updated Final Safety Analysis Report
          (UFSAR).  There was no safety significance due to the degraded
          capacity of the batteries.  The batteries had sufficient
          capacity for the safety-related loads following a design basis
          event.  See Section II for details.

TEXT                                                         PAGE 9 OF 12

     G.   For a failure that rendered a train of a safety system
          inoperable, estimated time elapsed from the discovery of the
          failure until the train was returned to service:

          As a result of the battery testing and evaluations since
          September 23, 1994, APS has determined that battery discharge
          testing decreased the capacity of batteries by approximately
          10% each time they are tested.

          Therefore, Train B battery banks B and D have been inoperable
          from February 1, 1994, to October 13, 1994.

          APS has concluded that the failure mechanism causes the
          batteries to degrade during the discharge/recharge cycle and
          that the projected capacities of the banks following the last
          capacity discharge test are: 1) Bank A, 78.82%, 2) Bank B,
          82.49%, 3) Bank C, 76.73%, and 4) Bank D, 81.75%.  As such, all
          banks were above the calculated minimum capacity of 55.22% at
          all times.

     H.   Method of discovery of each component or system failure or
          procedural error:

          As discussed in Section I.B., the degraded battery bank
          capacities were found during the performance of TS SR 4.8.2.1.e
          to satisfy the IEEE Standard 450-1980 requirement to capacity
          test new batteries within the first two years of service.

     I.   Safety System Response:

          Not applicable -- there were no safety system responses and
          none were necessary.

     J.   Failed Component Information:

          The battery banks consist of AT&T LINEAGE 2000 Round Cell
          batteries, model KS-20472 List 1H.  The cells are high specific
          gravity acid with a name plate rating of 1850 Ampere-Hour.

TEXT                                                        PAGE 10 OF 12

II.  ASSESSMENT OF THE SAFETY CONSEQUENCES AND IMPLICATIONS OF THIS
     EVENT:

     Four Class 1E Direct Current (DC) power banks designated as channel
     A, B, C, and D are provided in each unit.  These channels consist of
     125V DC bus [BU], 125V DC battery bank, and a battery charger [BYC].

     Train A consists of channels A and C and Train B consists of
     channels B and D.  Both DC Trains are required to be operable in
     Modes 1 through 4 per TS LCO 3.8.2.1 and one DC Train is required to
     be operable in Modes 5 and 6 per TS LCO 3.8.2.2.  The DC power
     sources are required to ensure that sufficient power is available to
     supply safety-related equipment required for safe plant shutdown and
     the mitigation and control of accident conditions.  Therefore, a
     change in battery capacity requirements does not involve a
     significant increase in the probability of an accident previously
     evaluated.

     APS has determined, through calculation and test, that the most
     highly loaded battery bank can continue to perform its
     safety-related function with its capacity as low as 55.22% of the
     original installed capacity.  Capacity discharge tests run in
     September 1994, indicate capacities of 91.6% for bank A, 89.0% for
     bank B, 90.6% for bank C, and 88.3% for bank D.

     APS has concluded that the failure mechanism causes the batteries to
     degrade during the discharge/recharge cycle and that the projected
     capacities of the banks following the last capacity discharge test
     are: 1) Bank A, 78.82%, 2) Bank B, 82.49%, 3) Bank C, 76.73%, and 4)
     Bank D, 81.75%.  As such, all banks were above the calculated
     minimum capacity of 55.22% at all times.

     An analysis showed that the projected capacities of the battery
     banks will provide at least 15% margin above that required for
     safety-related loads.  To accomplish this 11 cells in Bank A, 4
     cells in Bank B, 12 cells in Bank C, and 4 cells in Bank D were
     replaced.

TEXT                                                        PAGE 11 OF 12

     The projected capacities are expected to be in excess of 85% for
     each bank.  As such, the battery banks have sufficient capacity for
     the safety-related loads following a design basis event.  In
     addition, the majority of the degradation of the battery cells
     occurs during discharge testing of the batteries.

     Therefore, since no discharge testing of the batteries will be
     performed between now and the next refueling outage, the battery
     capacity will remain above that needed to fulfill the required
     safety function.

     The event did not result in any challenges to the fission product
     barriers or result in any releases of radioactive materials.
     Therefore, there were no adverse safety consequences or implications
     as a result of this event.  This event did not adversely affect the
     safe operation of the plant or the health and safety of the public.

III. CORRECTIVE ACTION:

     A.   Immediate:

          o    On October 7, 1994, battery banks A and C were declared
               inoperable based on projection of anticipated degradation.
               TS LCO 3.8.2.2 Action a was entered to prevent any
               positive reactivity additions.

          o    On October 9, 1994, APS submitted a proposed TS amendment
               to Specification 3/4.8.2, DC Sources, under emergency
               circumstances.  The amendment asked for approval to
               suspend the requirements of TS 4.0.1 and 4.0.4 until entry
               into Mode 4 coming out of the fifth refueling outage or
               upon any deep discharge of the battery.  Also, several
               compensatory actions were placed on the Unit 2 batteries
               in accordance with the TS amendment submittal.  On October
               13, 1994, TS amendment 71 was approved by the NRC.
               Therefore, on October 13, 1994, battery banks A, B, C, and
               D were declared operable.

TEXT                                                        PAGE 12 OF 12

     B.   Action to Prevent Recurrence:

          During the fifth refueling outage APS will replace all of the
          cells from the original AT&T lot in the 4 Class 1E batteries
          for Unit 2.  Also, the 23 replacement cells and the 8 spare
          cells used from Units 1 and 3 will be replaced.  The fifth
          refueling outage is scheduled to begin on February 4, 1995.

          A root cause of failure is still under investigation therefore,
          actions to prevent recurrence have not been determined to date.
          A supplement will be made to the LER following a root cause of
          failure determination.  The cause of failure is expected to be
          completed by December 30, 1994.

IV.  PREVIOUS SIMILAR EVENTS:

     There have been no similar events to this type of failure reported
     pursuant to 10CFR50.73.

V.   ADDITIONAL INFORMATION:

     On October 13, 1994, APS received NRC approval to suspend provisions
     4.0.1 and 4.0.4 of TS for battery capacity testing requirements
     until entry into Mode 4 following the fifth refueling outage.  Both
     trains of Class 1E batteries were declared operable on October 13,
     1994, and a Unit 2 restart began.  Unit 2 reached Mode 1 (POWER
     OPERATIONS) on October 17, 1994.

ATTACHMENT TO 9412050146                                      PAGE 1 OF 1

                     Arizona Public Service Company
                 PALO VERDE NUCLEAR GENERATING STATION
              P.O. BOX 52034 o PHOENIX, ARIZONA 85072-2034

                                        192-00912-JML/BAG/BE
JAMES M. LEVINE                         November 21, 1994
VICE PRESIDENT
NUCLEAR PRODUCTION

U.S. Nuclear Regulatory Commission
ATTN: Document Control Desk
Mail Station P1-37
Washington, DC 20555

Dear Sirs:

Subject:  Palo Verde Nuclear Generating Station (PVNGS)
          Unit 2
          Docket No. STN 50-529 (License No. NPF-51)
          License Event Report 94-004-00
          File: 94-020-404

Attached please find Licensee Event Report (LER) 94-004 prepared and
submitted pursuant to 10CFR50.73.  This LER reports the identification of
a condition that caused the Class 1E batteries to become inoperable.
This LER was originally due to be submitted on November 4, 1994.  On
October 25, 1994, an extension until November 21, 1994, was granted by
Howard Wong, Branch Chief, Reactor Projects (Telecon).  In accordance
with 10CFR50.73(d), a copy of this LER is being forwarded to the Regional
Administrator, NRC Region IV.

If you have any questions, please contact Burton A. Grabo, Supervisor,
Nuclear Regulatory Affairs, at (602) 393-6492.

                                   Sincerely,

JML/BAG/BE/pv

Attachment

cc:  W. L. Stewart  (all with attachment)
     L. J. Callan
     K. E. Perkins
     K. E. Johnston
     INPO Records Center

*** END OF DOCUMENT ***


Page Last Reviewed/Updated Thursday, March 29, 2012