Part 21 Report - 1995-083
ACCESSION #: 9412050146
LICENSEE EVENT REPORT (LER)
FACILITY NAME: Palo Verde Unit 2 PAGE: 1 OF 12
DOCKET NUMBER: 05000529
TITLE: Class 1E Batteries in a Degraded Condition
EVENT DATE: 10/06/94 LER #: 94-004- REPORT DATE:
OTHER FACILITIES INVOLVED: DOCKET NO: 05000
OPERATING MODE: 5 POWER LEVEL: 0
THIS REPORT IS SUBMITTED PURSUANT TO THE REQUIREMENTS OF 10 CFR
SECTION:
50.73(a)(2)(i), and 50.73(a)(2)(viii)(A)
LICENSEE CONTACT FOR THIS LER:
NAME: Burton A. Grabo, Supervisor, TELEPHONE: (602) 393-6492
Nuclear Regulatory Affairs
COMPONENT FAILURE DESCRIPTION:
CAUSE: B SYSTEM: EJ COMPONENT: BTRY MANUFACTURER: A626
REPORTABLE NPRDS: Y
SUPPLEMENTAL REPORT EXPECTED: YES
EXPECTED SUBMISSION DATE: 12/30/94
ABSTRACT:
At approximately 2200 MST on October 7, 1994, Palo Verde Unit 2 was in
Mode 5 (COLD SHUTDOWN), when Train A (battery banks A and C) was declared
inoperable because a projection of the test results based upon
anticipated degradation indicated that they did not meet the 90% criteria
of Technical Specification Surveillance Requirement (TS SR) 4.8.2.1.e,
rendering both trains of Class 1E batteries inoperable. Train B (battery
banks B and D) was declared inoperable (on October 1, 1994) because the
measure capacity was slightly less than the required 90% capacity stated
in TS SR 4.8.2.1.e.
Subsequent investigation has determined that Train B had been slightly
below the 90% capacity as required by TS SR 4.8.2.1.e since February 1,
1994. Since both trains of batteries are required in Mode 1, Unit 2
operated in a condition prohibited by the plant's TS until it shutdown
and reached Mode 5 on September 18, 1994. This condition is being
reported per 50.73(a)(i)(B).
The Equipment Root Cause of Failure Analysis (ERCFA) has not been
completed to date. The investigation has narrowed the possibilities for
the degraded capacity to either the manufacturing process or the quality
control process during manufacturing. This manufacturing problem has
potentially led to a single cause that caused both Class 1E battery
trains to become inoperable. Therefore, a 50.73(a)(2)(vii) notification
is also being made by this LER.
There have been no previously similar events reported pursuant to 10CFR
50.73.
END OF ABSTRACT
TEXT PAGE 2 OF 12
I. DESCRIPTION OF WHAT OCCURRED:
A. Initial Conditions:
At 2200 MST on October 7, 1994, Palo Verde Unit 2 was in Mode 5
(COLD SHUTDOWN) with Pressurizer pressure of 10 psia and a Main
Coolant Temperature of 107 degrees F.
B. Reportable Event Description:
Event Classification: Any operation or condition prohibited
by the plant's Technical
Specifications.
Any event where a single cause or
condition caused at least one
independent train or channel to become
inoperable in multiple systems or two
independent trains or channels to
become inoperable in a single system
designed to: A) Shut down the reactor
and maintain it in a safe shutdown
condition; B) Remove residual heat; C)
Control the release of radioactive
material; or D) Mitigate the
consequences of an accident.
At approximately 2200 MST on October 7, 1994, Palo Verde Unit 2
was in Mode 5 (COLD SHUTDOWN), when Train A (battery banks A
and C) was declared inoperable because a projection of the test
results based upon anticipated degradation indicated that they
did not meet the 90% criteria of Technical Specification
Surveillance Requirement (TS SR) 4.8.2.1.e, rendering both
trains of Class 1E batteries inoperable.
TEXT PAGE 3 OF 12
Train B (battery banks B and D) had been declared inoperable on
October 1, 1994 because the measured capacity was slightly less
than the required 90% capacity stated in TS SR 4.8.2.1.e. TS SR
4.8.2.1.e, states the following in part:
At least once per 60 months, during shutdown, by verifying that
the battery capacity is at least 80% (Exide) or 90% (AT&T) of
the manufacturer's rating when subjected to a performance
discharge test. This performance discharge test may be
performed in lieu of the battery service test required by
Surveillance Requirement 4.8.2.1d.
Subsequent investigation has conservatively determined that
Train B had been slightly below the 90% capacity as required by
TS SR 4.8.2.1.e since it was last tested on February 1, 1994.
Since both trains of batteries are required in Mode 1, Unit 2
operated in a condition prohibited by the plant's TS until it
shutdown and reached Mode 5 on September 18, 1994.
During the recent mid-cycle outage, Unit 2 was performing TS SR
4.8.2.1.e to satisfy the IEEE Standard 450-1980 requirement to
capacity test new batteries within the first two years of
service. On September 23, 1994, the test results for battery
banks A and C were 91.6% and 90.6% respectively. While the
test results met TS SR, the capacity of the battery banks were
below what was expected. As a result, the B and D battery
banks were capacity tested. On October 1, 1994, the B and D
battery banks were declared inoperable because the measured
capacity was slightly less than the required 90% capacity
stated in TS SR 4.8.2.1.e.
As a result of this unexpected degradation, Electrical
Maintenance Engineering (utility, non-licensed) performed an
individual cell and battery capacity evaluation on previous
tests of banks A, B, C, and D, factory tests, and additional
testing on the Unit 2 spare cells.
TEXT PAGE 4 OF 12
The evaluation concluded that the failure mechanism causes the
batteries to degrade during the discharge/recharge cycle,
therefore, the projections for battery capacity from this
evaluation indicated that all four battery banks in Unit 2 were
inoperable. Subsequently, on October 7, 1994, battery banks A
and C were also declared inoperable because a projection of the
test results based upon anticipated degradation indicated that
they did not meet the 90% criteria of TS SR 4.8.2.1.e.
Based on this new information, a condition prohibited by the
plant's TS has been identified. TS 3.8.2.1 requires that both
trains of Direct Current (DC) sources be operable in Modes 1
through 4. Due to the fact that the degradation mechanism is
related to discharge/recharge cycling of the batteries, APS has
concluded that battery banks B and D were conservatively at
89.0% and 88.3% capacity respectively following the ST
performed in February 1, 1994. Therefore, battery banks B and
D have been slightly below the 90% capacity as required by TS
SR 4.8.2.1.e since February 1, 1994. Unit 2 operated in Mode 1
with this condition until it shutdown for the September
mid-cycle outage and was in Mode 5 on September 18, 1994.
On October 9, 1994, APS submitted a proposed TS amendment to
Specification 3/4.8.2, DC Sources, under emergency
circumstances.
APS has concluded that the failure mechanism causes the
batteries to degrade during the discharge/recharge cycle and
that the projected capacities of the banks following the last
capacity discharge test are: 1) Bank A, 78.82%, 2) Bank B,
82.49%, 3) Bank C, 76.73%, and 4) Bank D, 81.75%. As such, all
banks were above the calculated design minimum capacity of
55.22% at all times.
TEXT PAGE 5 OF 12
The amendment asked for approval to suspend the requirements of
TS 4.0.1 and 4.0.4 until entry into Mode 4 coming out of the
fifth refueling outage or upon any deep discharge of the
battery. On October 13, 1994, TS amendment 71 was approved by
the NRC.
Therefore, on October 13, 1994, battery banks A, B, C, and D
were declared operable.
C. Status of structures, systems, or components that were
inoperable at the start of the event that contributed to the
event:
As stated in Section I.B above, Train B (battery banks B and D)
of the Class 1E batteries were inoperable since February 1,
1994. However, based on information at the time Train B was
believed to be operable because of its capability to pass the
surveillance test requirements.
D. Cause of each component or system failure, if known:
On October 6, 1994, the three worst cells from Train B, battery
bank D arrived at C&D Charter Power Services, Inc. (the
manufacturer) for a root cause determination. This root cause
effort is being overseen by APS with the participation of C&D,
AT&T (the cell designer) and Failure Prevention International
(FPI, consultant).
The three cells have undergone preliminary examination to
exclude sudden loss of capacity as the potential cause of
failure. The three sudden failure modes are 1) open or near
open circuit, 2) low impedance short circuit, and 3) massive
contamination of electrode active material. Tear down of three
cells revealed some signs of poor workmanship, but there was no
evidence that the diminished capacity is attributable to
physical damage of the cell internal components.
The cell examinations also ruled out sudden loss of capacity
(due to item 1 or 2 as stated above) as a failure mode for
these particular cells.
TEXT PAGE 6 OF 12
Chemical analysis of the electrolyte and the plate's active
material has ruled out massive contamination (item 3 as stated
above). It was, therefore, concluded that for this cell the
failure mode would be a step decline in capacity due to an
active material utilization problem during the
discharge/recharge cycle. APS, with concurrence from AT&T and
FPI, has concluded that the discharge behavior is not
indicative of a sudden discharge type failure.
Currently, the Equipment Root Cause of Failure Analysis (ERCFA)
is focusing on the possibility of manufacturing problems or
quality controls problems during the manufacturing process.
The ERCFA team has identified that the loss of capacity is a
result of "PbO sub 2 grain isolation" (a reduction of active
material utilization due to a loss of electronic contact
between the PbO sub 2 grains themselves within the plate and/or
between the grains and the grid). The most likely cause for
this isolation can be attributed to improper curing of the
positive plates during manufacturing. Chemical and microscopic
analyses have been performed on samples of plates from both
failed and "good" cells. The results of the analyses are still
being evaluated, but no evidence has been found which would
invalidate the PbO sub 2 isolation mechanism.
Degradation of the battery appears to be manifested during the
second hour of the two hour capacity discharge test, or, in
other words, the last 530 ampere-hours of the 1060 ampere-hour
test. This is evident when reviewing the discharge test curves
for the batteries. The actual load profile demand for the A
battery channel over a two hour period only removes
approximately 423 ampere-hours. If a plant demand of only 423
ampere-hours had occurred on the B and D battery from February
thru October 1, 1994, Engineering believes the batteries would
have behaved similar to a 100% capacity battery because only
423 ampere-hours would have been removed during the two hour
demand.
TEXT PAGE 7 OF 12
As part of the ERCFA Electrical Maintenance Engineering
evaluated the capacity discharge testing at PVNGS. The
following conservatism's were noted:
During capacity discharge testing at PVNGS, an extra 5 amperes
were added to the required load current to be conservative for
any possible tolerances in test equipment.
These extra 5 amperes results in the battery capacity to
measure approximately 1% less than it would measure without the
extra 5 amperes.
The capacity tests up thru October 1, 1994, were performed
using the original AT&T two hour rating of 530 amperes. The
new two hour AT&T rating for the same battery is now 514
amperes. Thus, if the B and D batteries had been discharged
tested at the new rating of 514 amperes, the battery capacity
would have measured approximately 4% better.
In summary, the extra 5 amperes added for test equipment
tolerances and use of the old 530 ampere rating for two hours
on the AT&T cells account for the B and D batteries performing
approximately 5 less in capacity on October 1, 1994, than they
would have performed otherwise. Thus, the actual capacity test
on October 1, 1994, would have expected to measure
approximately 5% better if they were tested at the new 514
ampere rating and the extra 5 amperes not added.
Further examination and testing will be performed to determine
the cause of failure. A supplement to the LER will be made
when a root cause of failure has been determined.
E. Failure mode, mechanism, and effect of each failed component,
if known:
The cause of failure has not been finalized.
TEXT PAGE 8 OF 12
However, APS had determined through testing that the
degradation mechanism is related to loss of positive plate
activity which is aggravated by the discharge/recharge cycling
of the batteries as discussed in Section I.D.
APS has concluded that the failure mechanism causes the
batteries to degrade during the discharge/recharge cycle and
that the projected capacities of the banks following the last
capacity discharge test are: 1) Bank A, 78.82%, 2) Bank B,
82.49%, 3) Bank C, 76.73%, and 4) Bank D, 81.75%. As such, all
banks were above the calculated design minimum capacity of
55.22% at all times.
F. For failures of components with multiple functions, list of
systems or secondary functions that were also affected:
The DC banks A and B provide control power for Alternating
Current (AC) load groups 1 and 2 respectively. These banks
also provide vital instrumentation and control power [JC] for
channels A and B, respectively, of the reactor protection and
Emergency Safety Features (ESF) systems and diesel generators
[DG] A and B respectively. The DC banks C and D provide vital
instrumentation and control power [JC] for channels C and D,
respectively, for the reactor protection and ESF systems, and
other safety-related loads as referenced in Table 8.3-6, Class
1E DC System Loads, of the Updated Final Safety Analysis Report
(UFSAR). There was no safety significance due to the degraded
capacity of the batteries. The batteries had sufficient
capacity for the safety-related loads following a design basis
event. See Section II for details.
TEXT PAGE 9 OF 12
G. For a failure that rendered a train of a safety system
inoperable, estimated time elapsed from the discovery of the
failure until the train was returned to service:
As a result of the battery testing and evaluations since
September 23, 1994, APS has determined that battery discharge
testing decreased the capacity of batteries by approximately
10% each time they are tested.
Therefore, Train B battery banks B and D have been inoperable
from February 1, 1994, to October 13, 1994.
APS has concluded that the failure mechanism causes the
batteries to degrade during the discharge/recharge cycle and
that the projected capacities of the banks following the last
capacity discharge test are: 1) Bank A, 78.82%, 2) Bank B,
82.49%, 3) Bank C, 76.73%, and 4) Bank D, 81.75%. As such, all
banks were above the calculated minimum capacity of 55.22% at
all times.
H. Method of discovery of each component or system failure or
procedural error:
As discussed in Section I.B., the degraded battery bank
capacities were found during the performance of TS SR 4.8.2.1.e
to satisfy the IEEE Standard 450-1980 requirement to capacity
test new batteries within the first two years of service.
I. Safety System Response:
Not applicable -- there were no safety system responses and
none were necessary.
J. Failed Component Information:
The battery banks consist of AT&T LINEAGE 2000 Round Cell
batteries, model KS-20472 List 1H. The cells are high specific
gravity acid with a name plate rating of 1850 Ampere-Hour.
TEXT PAGE 10 OF 12
II. ASSESSMENT OF THE SAFETY CONSEQUENCES AND IMPLICATIONS OF THIS
EVENT:
Four Class 1E Direct Current (DC) power banks designated as channel
A, B, C, and D are provided in each unit. These channels consist of
125V DC bus [BU], 125V DC battery bank, and a battery charger [BYC].
Train A consists of channels A and C and Train B consists of
channels B and D. Both DC Trains are required to be operable in
Modes 1 through 4 per TS LCO 3.8.2.1 and one DC Train is required to
be operable in Modes 5 and 6 per TS LCO 3.8.2.2. The DC power
sources are required to ensure that sufficient power is available to
supply safety-related equipment required for safe plant shutdown and
the mitigation and control of accident conditions. Therefore, a
change in battery capacity requirements does not involve a
significant increase in the probability of an accident previously
evaluated.
APS has determined, through calculation and test, that the most
highly loaded battery bank can continue to perform its
safety-related function with its capacity as low as 55.22% of the
original installed capacity. Capacity discharge tests run in
September 1994, indicate capacities of 91.6% for bank A, 89.0% for
bank B, 90.6% for bank C, and 88.3% for bank D.
APS has concluded that the failure mechanism causes the batteries to
degrade during the discharge/recharge cycle and that the projected
capacities of the banks following the last capacity discharge test
are: 1) Bank A, 78.82%, 2) Bank B, 82.49%, 3) Bank C, 76.73%, and 4)
Bank D, 81.75%. As such, all banks were above the calculated
minimum capacity of 55.22% at all times.
An analysis showed that the projected capacities of the battery
banks will provide at least 15% margin above that required for
safety-related loads. To accomplish this 11 cells in Bank A, 4
cells in Bank B, 12 cells in Bank C, and 4 cells in Bank D were
replaced.
TEXT PAGE 11 OF 12
The projected capacities are expected to be in excess of 85% for
each bank. As such, the battery banks have sufficient capacity for
the safety-related loads following a design basis event. In
addition, the majority of the degradation of the battery cells
occurs during discharge testing of the batteries.
Therefore, since no discharge testing of the batteries will be
performed between now and the next refueling outage, the battery
capacity will remain above that needed to fulfill the required
safety function.
The event did not result in any challenges to the fission product
barriers or result in any releases of radioactive materials.
Therefore, there were no adverse safety consequences or implications
as a result of this event. This event did not adversely affect the
safe operation of the plant or the health and safety of the public.
III. CORRECTIVE ACTION:
A. Immediate:
o On October 7, 1994, battery banks A and C were declared
inoperable based on projection of anticipated degradation.
TS LCO 3.8.2.2 Action a was entered to prevent any
positive reactivity additions.
o On October 9, 1994, APS submitted a proposed TS amendment
to Specification 3/4.8.2, DC Sources, under emergency
circumstances. The amendment asked for approval to
suspend the requirements of TS 4.0.1 and 4.0.4 until entry
into Mode 4 coming out of the fifth refueling outage or
upon any deep discharge of the battery. Also, several
compensatory actions were placed on the Unit 2 batteries
in accordance with the TS amendment submittal. On October
13, 1994, TS amendment 71 was approved by the NRC.
Therefore, on October 13, 1994, battery banks A, B, C, and
D were declared operable.
TEXT PAGE 12 OF 12
B. Action to Prevent Recurrence:
During the fifth refueling outage APS will replace all of the
cells from the original AT&T lot in the 4 Class 1E batteries
for Unit 2. Also, the 23 replacement cells and the 8 spare
cells used from Units 1 and 3 will be replaced. The fifth
refueling outage is scheduled to begin on February 4, 1995.
A root cause of failure is still under investigation therefore,
actions to prevent recurrence have not been determined to date.
A supplement will be made to the LER following a root cause of
failure determination. The cause of failure is expected to be
completed by December 30, 1994.
IV. PREVIOUS SIMILAR EVENTS:
There have been no similar events to this type of failure reported
pursuant to 10CFR50.73.
V. ADDITIONAL INFORMATION:
On October 13, 1994, APS received NRC approval to suspend provisions
4.0.1 and 4.0.4 of TS for battery capacity testing requirements
until entry into Mode 4 following the fifth refueling outage. Both
trains of Class 1E batteries were declared operable on October 13,
1994, and a Unit 2 restart began. Unit 2 reached Mode 1 (POWER
OPERATIONS) on October 17, 1994.
ATTACHMENT TO 9412050146 PAGE 1 OF 1
Arizona Public Service Company
PALO VERDE NUCLEAR GENERATING STATION
P.O. BOX 52034 o PHOENIX, ARIZONA 85072-2034
192-00912-JML/BAG/BE
JAMES M. LEVINE November 21, 1994
VICE PRESIDENT
NUCLEAR PRODUCTION
U.S. Nuclear Regulatory Commission
ATTN: Document Control Desk
Mail Station P1-37
Washington, DC 20555
Dear Sirs:
Subject: Palo Verde Nuclear Generating Station (PVNGS)
Unit 2
Docket No. STN 50-529 (License No. NPF-51)
License Event Report 94-004-00
File: 94-020-404
Attached please find Licensee Event Report (LER) 94-004 prepared and
submitted pursuant to 10CFR50.73. This LER reports the identification of
a condition that caused the Class 1E batteries to become inoperable.
This LER was originally due to be submitted on November 4, 1994. On
October 25, 1994, an extension until November 21, 1994, was granted by
Howard Wong, Branch Chief, Reactor Projects (Telecon). In accordance
with 10CFR50.73(d), a copy of this LER is being forwarded to the Regional
Administrator, NRC Region IV.
If you have any questions, please contact Burton A. Grabo, Supervisor,
Nuclear Regulatory Affairs, at (602) 393-6492.
Sincerely,
JML/BAG/BE/pv
Attachment
cc: W. L. Stewart (all with attachment)
L. J. Callan
K. E. Perkins
K. E. Johnston
INPO Records Center
*** END OF DOCUMENT ***
Page Last Reviewed/Updated Wednesday, March 24, 2021
Page Last Reviewed/Updated Wednesday, March 24, 2021