Part 21 Report - 1996-782
ACCESSION #: 9701090072
NOTE: This text document was processed from a scanned
version or an electronic submittal and has been
processed as received. Some tables, figures,
strikeouts, redlines, and enclosures may not have
been included with this submittal, or have been
omitted due to ASCII text conversion limitations. In
order to view this document in its entirety, you may
wish to use the NUDOCS microfiche in addition to the
electronic text.
Root Cause Analysis
of
Struthers-Dunn Operate-Reset Relay
Latch Failures
for Salem Generating Station
Condition Reports
CR 960906195
CR 961101269
CR 961105229
December 6, 1996
Principal Investigator
Craig Bersak
Prepared by:
Craig Bersak, Engineering Assurance
Approved by:
Rajkowski, Manager - Bailey Replacement Project
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
TABLE OF CONTENTS
1.0 EXECUTIVE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . 2
2.0 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . 5
o Background . . . . . . . . . . . . . . . . . . . . . . . . . 5
o Identification of Occurrence . . . . . . . . . . . . . . . . 5
o Conditions Prior to Occurrence . . . . . . . . . . . . . . . 5
o Root Cause Analysis Methodology . . . . . . . . . . . . . . . 5
3.0 DESCRIPTION OF OCCURRENCE/SCAQ . . . . . . . . . . . . . . . . . . 6
o Chronology of Events . . . . . . . . . . . . . . . . . . . . 6
o Troubleshooting, Remedial, and Compensatory Actions
Taken . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7
o Notification of Authorities/Industry . . . . . . . . . . . . 8
o Related Failure Modes . . . . . . . . . . . . . . . . . . . . 8
4.0 ANALYSIS & CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . .10
Issue #1: Seismic adequacy of B255 style relays for
Salem Station . . . . . . . . . . . . . . . . . . . . . . . .10
o Description of the Issue . . . . . . . . . . . . . . . . . .10
Issue #2: Failure of relay to latch into OPERATE upon a
valid demand. . . . . . . . . . . . . . . . . . . . . . . . .10
o Description of the Issue . . . . . . . . . . . . . . . . . .10
o Causal Factor Analysis . . . . . . . . . . . . . . . . . . .11
Issue #3: Spurious unlatching of the relay without a
valid demand. . . . . . . . . . . . . . . . . . . . . . . .14
o Description of the Issue . . . . . . . . . . . . . . . . . .14
o Causal Factor Analysis . . . . . . . . . . . . . . . . . . .14
Issue #4: Unsuitability of new relays . . . . . . . . . . . . . .15
o Description of the Issue . . . . . . . . . . . . . . . . . .15
o Causal Factor Analysis . . . . . . . . . . . . . . . . . . .15
Prior Similar Events and Operating Experience . . . . . . . . . .16
Generic Implications . . . . . . . . . . . . . . . . . . . . . . .17
Safety Significance . . . . . . . . . . . . . . . . . . . . . . .17
5.0 CORRECTIVE ACTIONS . . . . . . . . . . . . . . . . . . . . . . . .19
o Corrective Actions and Accountabilities . . . . . . . . . . .19
o Measures to Evaluate CA Effectiveness and
Accountabilities . . . . . . . . . . . . . . . . . . . . . .20
6.0 OTHER ISSUES AND IMPACTS . . . . . . . . . . . . . . . . . . . . .21
o Issues Beyond the Scope of Investigation . . . . . . . . . .21
o Common Mode Failures, as Appropriate . . . . . . . . . . . .21
o Reportability . . . . . . . . . . . . . . . . . . . . . . . .21
o Industry Information Dissemination . . . . . . . . . . . . .21
7.0 SOURCES OF INFORMATION . . . . . . . . . . . . . . . . . . . . . .22
i
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
o Contacts . . . . . . . . . . . . . . . . . . . . . . . . . .22
o References . . . . . . . . . . . . . . . . . . . . . . . . .22
8.0 INVESTIGATOR COMMENTS . . . . . . . . . . . . . . . . . . . . . . .23
9.0 APPENDICES & ATTACHMENTS . . . . . . . . . . . . . . . . . . . . .23
ii
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
1.0 EXECUTIVE SUMMARY
Description of Event
This root cause evaluation addresses failures of Struthers-Dunn B255
style latching relays to latch on demand and to reset without a valid
demand signal. Subsequent to the identification of failures at Salem
Station, PSE&G was notified by its vendor, Farwell & Hendricks, Inc., of
its submission of a 10CFR21 notification of a potential defect and of the
premature failure of a relay during seismic evaluation. Condition
Reports 961101269 and 961105229 were written to document these concerns.
Causes
The root cause of the relay latching problems was a reduction in the
elongation of the coil spring for the latching mechanism. This reduction
occurred during revisions to the design of the relay frame and reset coil
bracket. As a result of the reduced spring load the impact of the
operate armature upon the latch could result in its movement away from
the latched position, allowing the relay to return to the reset
configuration. Latched relays would require the addition of energy from
some outside source, i.e., vibration or jarring, to become unlatched.
Seismic concerns developed when PSE&G was notified that a verification
relay of similar manufacture dates had become unlatched during seismic
testing above 5G broadband acceleration when mounted in the floor mount
configuration. The Salem applications of the relays were evaluated and
found to have limiting seismic requirements below this level. As a
result their seismic qualification was and is suitable for Salem Station
applications.
Corrective Actions
Coordination with the relay manufacturer and vendor, Magnecraft/
Struthers-Dunn (MSD) and Farwell & Hendricks, Inc.(F&H), has identified a
replacement spring to restor the latching force to levels consistent with
the 1970 vintage relays. Relays purchased from F&H are being returned to
F&H to have the replacement spring installed and become re-dedicated for
PSE&G's use. MSD has incorporated the replacement spring into the
manufacture of this series relay. New folios have been established to
identify refurbished and newly manufactured relays from the suspect
population.
A formal relay bench testing protocol has been established and
proceduralized to identify potential relay failures prior to their
installation into the plant. This testing procedure has been aligned
with the F&H testing procedure to assure consistency.
A Bailey Replacement Project has been initiated to identify and replace
the suspect relays prior to entering a mode where the component impacts
plant safety. Safety
1
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
Evaluations in accordance with 10CFR50.59 are being performed to assure
that the unchanged relays do not impact safety requirements for the mode
to be entered.
Safety Significance
The relays involved with this evaluation are utilized in the control
systems for both safety-related and non-safety related components
throughout both Salem units.
All of the failures identified to date have involved DC powered relays,
predominantly of the 24-28 VDC ratings. In addition to the 24-28 VDC
relays, Salem utilizes 125 VDC relays (two failures) and 110 VAC relays
(0 failures) of the B255 style. All of the recent failures have been
with relays staged for or installed in the Bailey Control Cabinets.
The 24-28 VDC relays typically are used in the interfacing circuitry
between the operators control station pushbuttons and the subsequent
actuation of the actual device. Within these applications the relays
typically serve to establish and maintain the last selected
AUTO/MANUAL mode of operation (83 device)
Breaker demand position (checking relay (3 device))
Alarm acknowledge (74 device)
The 125 VDC relays typically provide breaker controls for the actuated
device.
Per the UFSAR SER Section 7.3.4, "the automatic operation of the ESF
equipment is independent of any action in the 28V circuitry."
Nonetheless, improper operation of the relays imposes additional demands
and distractions on the operators. Additionally, failures, such as the
RH29s spurious shift to MANUAL, can adversely impact the operation of the
controlled equipment. In the case of the RH29s, a spurious shift to
MANUAL can result in operation of the associated RHR pump with less than
minimum pump flow. A complete search of all affected control circuits
was not conducted to determine the extent of this problem.
Plant mode specific safety evaluations in accordance with 10CFR50.59 are
being performed prior to the entry into any defined mode to ascertain the
impact of any component which may have a failure prone relay within its
control circuitry. These safety evaluations will be attached to this
report as they become available.
Generic Issues
None
2
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
Reportability
These occurrences have been reported to the NRC by means of a four hour
report on November 5, 1996 and LER 272/96-031-00.
Part 21 notification was initiated on November 1, 1996 by F&H.
3
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
2.0 INTRODUCTION
o Background
This root cause evaluation addresses failures of Struthers-Dunn B255
style latching relays to latch on demand and to reset without a valid
demand signal. Subsequent to the identification of failures at Salem
Station, PSE&G was notified by its vendor, Farwell & Hendricks, Inc., of
its submission of a 10CFR21 notification of a potential defect and of the
premature failure of a relay during seismic evaluation. Condition
Reports 961101269 and 961105229 were written to document these concerns.
o Identification of Occurrence
During post-maintenance testing of 22 Residual Heat Removal (RHR) pump,
the 22 loop Heat Exchanger Bypass valve, 22RH29, was observed to shift
from Automatic to Manual mode without a demand from the operator. During
the PMT activity, RHR pump flow was being reduced to open 22RH29. When
the low-flow setpoint, 500 gpm, was reached the valve spuriously swapped
to MANUAL, resulting in a failure of the valve to open as required.
Separately, the Slow Speed start for the 22 Boric Acid Transfer Pump
failed to latch in when selected. When the operator released the push-
button, the pump stopped without generating an alarm.
Additional component failures due to latching relay problems are listed
in Attachment 3.
o Conditions Prior to Occurrence
At the time of discovery, both Salem Unit 1 and Unit 2 were defueled. An
extended maintenance/refueling outage was on-going for Unit 2 with outage
activities suspended on Unit 1.
o Root Cause Analysis Methodology
Failure Modes Analysis was utilized in the root cause evaluation of this
event. A Failure Modes Chart can be found in Attachment 1.
Additionally, Change Analysis was performed (by FPI International)
comparing the recently purchased relays to the 1970s vintage relays
installed as original equipment. The problems encountered with these
relays was discussed and reviewed with both the relay manufacturer,
Magnecraft/Struthers-Dunn (MSD), and the qualifying vendor, Farwell &
Hendricks, Inc. (F&H).
4
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
3.0 DESCRIPTION OF OCCURRENCE/SCAQ
o Chronology of Events
Prior to the commencement of the current Salem Unit 1 and 2 outage, the
degradation of relays within the Bailey Relay cabinets was identified.
The primary aspect of the observed degradation was an increase in contact
resistance due to the formation of a cadmium chloride salt within the
relay case. As a result of this adverse trend, a Bailey Relay
Inspection/Replacement Project was initiated with the commitment to
perform a 100% inspection of each unit's Bailey cabinets prior to the
unit's return to service. A work standard (BAILEY 001) was developed
providing the technician with criterion for evaluating the relays via
visual inspection. Relays which were judged by the technician to be
degraded were replaced with recently pruchased relays. During the Post
Maintenance Testing and following return of the component to the
Operations Department, failures of latching style relays were observed at
an unacceptable frequency. Additionally, during bench testing of the
relays, prior to their installation, an approximate 10% latching failure
rate was observed. As a result of the field and bench testing failures,
condition report 960906195 was initiated to identify the root cause of
the relay problem.
In 1996, 15 instances of Struthers-Dunn Operate-Reset Relays failing to
latch into the OPERATE state or spuriously resetting have been identified
on Salem Unit # 2. Representative of these failures are the following
occurrences:
On June 6,1996, the "LOCAL MANUAL" alarm for the 2B Emergency Diesel
Generator did not lock in when acknowledged by the operator.
On June 30, 1996, 22RH19, the 22 Residual Heat Removal Discharge
valve, would return to the closed position after initially
responding to an OPEN demand; i.e., stroke open, reach its open
limit and then promptly close without a close demand from the
operator.
On August 30 1996 Post Maintenance Testing (PMT) was performed on 22
Residual Heat Removal (RHR) pump following completion of corrective
maintenance work order 960828275. This work order was initiated due
to excessive oil leakage. During the PMT, 22 RHR loop flow was
being reduced to cause its heat Exchanger Bypass valve, 22RH29, to
automatically open on the resulting low flow (<500 gpm) signal. As
the low flow setpoint was approached the bypass valve spuriously
swapped from AUTO to MANUAL mode of operation.
There were three separate occurrences of problems with the 22 Boric
Acid Transfer (BAT) Pump. On August 19, 1996, the pump stopped when
its mode was shifted by the operator from AUTO to MANUAL. It
subsequently failed to restart on a manual demand. On September 6,
5
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
1996 during PMT of the 22 BAT Pump Control Circuit, the pump
failed to remain operating in slow speed when the "22 SLOW START"
pushbutton was released. Similarly, during a separate PMT, the 22
BAT pump failed to operate in "BORATE".
All of these problems were diagnosed as a failure of the associated
Operate-Reset relay to latch, or remain latched, in the Operate
condition.
o Troubleshooting, Remedial, and Compensatory Actions Taken
Troubleshooting
On each occurrence troubleshooting of the affected circuitry was
performed by the Salem Maintenance Department. In each case, one or more
Operate-Reset Relays was determined to be operating improperly and
replaced with a like relay which had been successfully bench tested.
A relay testing protocol was established and proceduralized in SC.MD-
PM.ZZ-0204(Q), Bailey Relay Inspection, to establish consistency in relay
testing and provide an arduous standard to eliminate potential relay
failures prior to their installation in the plant. This procedure cycled
each latching relay through a minimum of 10 cycles to verify its
performance. This procedure was issued September 13, 1996.
Communications with the relay manufacturer, MSD, were initiated to
investigate and resolve the problem. A sample of the failed relays was
brought to MSD's Darlington, SC facility for evaluation by MSD's Product
Engineering Manager. A PSE&G Root Cause Engineer accompanied the relays
during their evaluation. Only one of the relays repeated the latching
failure. That relay was found to have its latching coil spring stretched
at an attachment loop. When the spring was replaced the relay functioned
properly.
FPI International was commissioned to perform a Change Analysis of the
relays to determine the cause of failure. Their report is included as
attachment 5.
F&H was contacted regarding the observed failures. Their Systems
Engineering Manager and Sales Manager visited the Salem facility at which
time the failures were discussed in detail. At this visit, F&H
identified that they had not performed qualification testing of the
relays in the "floor mounted" orientation utilized within the Bailey
Relay Cabinets. (Adequacy of the relays in the floor mounted
configuration at Salem had previously been established by PSE&G's
evaluation of the Hope Creek Bailey Cabinet Seismic Report.)
6
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
Remedial Actions
Following identification of the root cause of the relay's unlatching, the
reduction in spring elongation, a design change to the relays was
proposed by MSD replacing the existing coil spring with a shorter spring.
The relay latching mechanism with the replacement spring had been
utilized in other MSD relay designs.
Compensatory Actions
A Bailey Relay Project team was formed by the NBU's Senior Vice President
- Nuclear Engineering to evaluate the relay problem and initiate the
corrective actions. The safety significance of the failure prone relays
is being evaluated by a 10CFR50.59 evaluation for the then existing
condition and for each subsequent mode change until the replacement of
the suspect population. A prioritized listing of components, keyed to
the modes where the component was required to be operable, was
established to focus the replacement of the suspect relays with upgraded
relays. Additional components were added as necessary based
on the 50.59 evaluations. Changes in plant mode were delayed until the
necessary relays could be replaced.
o Notification of Authorities/industry
The Nuclear Regulatory Commission was notified by F&H of the potentially
degraded component on November 1, 1996. Their final report, including
their root cause evaluation, was due to be issued November 30, 1996 and
has been delayed.
PSE&G made a four-hour report upon notification of the seismic testing
failure by F&H. A Licensee Event Report, 272/96-031-00, was issued
discussing PSE&G's response to the seismic, Part 21, and functional
failures.
o Related Failure Modes
Previous relay problems have been identified related to high contact
resistances and the formation of "white deposits" within the relay case
for other styles of Struthers-Dunn relays. As a result of this a 100%
inspection of Bailey Relay Cabinets and replacement of degraded relays
was initiated during the current outage. Subsequent analysis by the
PSE&G Maplewood Test Labs identified the deposits as "similar to a
corrosion product of cadmium containing chlorine such as found in cadmium
chloride."
An INPO Operating Experience Notice, OE 5537, identified degraded
performance (premature failures) of 219 style relays with Polypropylene
relay coils. During the aforementioned inspection, relays having this
type coil are replaced with bench-tested relays.
A sample of 12 relays was sent to Wyle Laboratories for failure analysis
of the high resistance condition. Wyle Laboratories report no. 45053R95,
"Failure Analysis of
7
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
Struthers-Dunn Relays From Public Service Electric and Gas Co.,"
concludes that "the high contact resistance appeared to be the result of
the depositing of residual flux and/or its reaction products on the
contacts."
Operation of the relay coils at higher than 110% of rated voltage can
result in the premature failure of the relay coil. The 28 VDC, 125 VDC,
and 115 Vac systems are typically maintained withing voltage ranges that
approach or slightly exceed the 110% values. This condition can result
in premature coil failure, especially among continuously energized
relays.
These failure mechanisms result in high contact resistances or operating
coil failures. They do
not impair the relay's ability to latch if the operate coil is
sufficiently energized.
The failures identified within this evaluation are not the result of any
of these related failure modes.
8
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
4.0 ANALYSIS & CONCLUSIONS
Issue #1: Seismic adequacy of B255 style relays for Salem Station.
o Description of the Issue
On November 5, 1996, F&H notified PSE&G that one of two verification
relays had failed seismic qualification by becoming unlatched at
acceleration levels above 5G broadband when tested in a floor mounted
configuration.
A walkdown by System Engineering was performed to identify the locations
where the latching relays are used in the floor mounted configuration.
Plant Applications Group evaluated the various locations identified and
determined that none were subject to seismic levels in excess of 5G. As
a result, the relays were determined, as a group, to be qualified for the
seismic spectra at Salem station.
Issue #2: Failure of relay to latch into OPERATE upon a valid demand.
o Description of the Issue
The Struthers-Dunn 255 style Operate-Reset Relay, Figure 1, consists of
two coils, a spring-loaded mechanical latch and a set of contacts
operated off of the armature associated with the OPERATE coil. With the
coil initially in the reset condition and both coils not energized, the
spring force resulting from four of the contact leaves pulls the OPERATE
coil armature via the contact slider to the reset condition. When the
OPERATE coil is energized its armature is drawn to the coil. The
movement of the armature results in the contact slider changing the state
of the relay contacts and allows the latch pawl to descend capturing the
armature in the operate condition. With the operate armature caught by
the latch, power to the OPERATE coil can be removed without changing the
state of the relay contacts. The relay is reset by energizing the RESET
coil. When the RESET coil energizes the latch pawl is lifted allowing
the OPERATE armature to be returned to the reset condition and changing
the state of the relay contacts. Simultaneously energizing both coils
will result in the relay attaining its OPERATE state without latching the
OPERATE armature. If both coils are then simultaneously de-energized,
the final state of the relay will depend upon which coil's magnetic field
collapses first; either latching the OPERATE armature or allowing it to
return to the reset state.
The majority of the failures previously identified resulted when the
relay failed to maintain its latched condition when the OPERATE coil was
de-energized. This allowed the relay's contacts to be returned to their
RESET configuration.
9
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
o Causal Factor Analysis
Three potential areas of failure were evaluated for this issue:
Mechanical failure of the relay, electrical failure and human error.
Mechanical Failure
A mechanical failure of the relay to latch can result from the following
means:
Latching Pawl or Operate Armature prevented from moving freely
Each of the failed relays was examined and operated to
determine if there was any blockage or impairment to the motion
of the moving parts. Each of the relays operated at minimum
voltage with successful latching. The cases of each relay was
examined with no foreign material noted that may have prevented
the latch or armature from traveling freely.
Mis-formed Latch Pawl
The latch pawl on the relay is a square cut piece of solid
nylon. Each of the latches was examined under a 15X magnifier
to determine if the trailing (latching) edge had been mis-
formed or worn in a manner that would allow the OPERATE
armature to return to the reset position. No deformation of
any latch was identified.
Mis-formed Armature "Catch Step"
The portion of the armature that is "captured" by the latch
pawl has a "catch step" pressed into it during manufacture. A
relay returned by Farwell & Hendricks, the qualifying
distributor from whom PSE&G purchases the relays, to MSD was
noted as having a misformed "catch step." The misformed step
allowed the tip of the armature to slip past the latch pawl,
resetting the relay without demand.
The recent PSE&G failed relay were examined for repetition of
this failure. None of the relays was observed to have a
misformed armature "catch step."
Insufficient spring loading of the latch pawl
One failed relay, removed from the 22RH29 control circuitry,
was noticed as having the coil spring which applies the latch
preload deformed. One of the attachment loops of the coil
spring had become extended increasing the overall length of the
spring by 3/64". This elongation effectively removed the
spring force seating the latch when the OPERATE
10
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
coil was energized and allowed the armature to repeatedly
return to the reset condition. The elongated spring was
replaced with a properly formed spring and the relay was
observed to function without subsequent problems.
Change Analysis, performed by FPI International, identified
that the current relay design had reduced the elongation of
this spring and resulted in a reduction of latching force being
applied. Review of FPI's finding with MSD indicated that the
distance between the attachment anchors for the coil spring had
reduced when the reset coil attachment bracket was changed
sometime in 1993. This reduction in spring elongation, coupled
with variations in spring constants with a given set of
springs, resulted in some of the relays having marginal
latching capability.
When the OPERATE coil is energized, its armature is drawn
toward it beyond the amount required to seat the latch. When
the coil deenergizes the contact springs attempt to return to
the reset state. Normally the latch prevents the return to the
reset state. However, in those relays where the spring is
weak, the impact of the operate armature hitting the latch
results in the latch "bouncing" out of position and the relay
returning to the reset condition. It was noticed by the
Technicians that relays which had failed were susceptible to
unlatching when the case was lightly tapped.
Electrical Failure
An electrical failure of the relay to latch can result from the following
means:
Insufficient voltage being applied to the operate coil
The DC and AC control power busses at Salem typically are above
their nominal design voltages. None of the failures occurred
during a degraded voltage condition. Nominal Salem 28 VDC bus
voltage is 29 VDC. Each of the failed relays was tested at
minimum allowable voltage per the relay specification, i.e.,
19.2 VDC (80% of 24 VDC), repeatedly without an observed
latching failure. One of the failed relays was subsequently
tested at a highly degraded voltage of 10 VDC and did not
demonstrate a latching failure.
Energizing both the OPERATE and RESET coils simultaneously.
When both coils are energized the relay's contacts will achieve
the operate condition, but the latch will be raised preventing
it from capturing
11
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
the armature. If power is simultaneously removed, the final
state of the relay will depend upon which coil's magnetic field
collapses first.
The control circuits for the Boric Acid Transfer pump, 2B
Emergency Diesel Generator Alarm, and 21RH19 were evaluated for
a means of inadvertently energizing both coils. Each potential
path was determined to be protected by a surge suppression
diode which would prevent current flow through the RESET coil
when the OPERATE coil was selected.
On October 9, 1996, testing of the 22 BAT pump circuit was
performed (WO# 960924247) to determine whether any unwarranted
energization of a RESET coil existed. Voltage with respect to
the negative bus was obtained across each coil and diode as
each control pushbutton was depressed. Voltage at the Relay
Cabinet was determined to be 27.21 VDC. No improper voltage
was seen the RESET coils when any control console pushbutton
was pressed.
Residual Magnetism of the RESET coil core
During the evaluation of a relay, its RESET coil was energized
continuously at 24 VDC for 5 minutes. When the RESET coil was
de-energized, the latch remained affixed to the coil. This was
sustained for approximately 25 minutes. Momentary reenergizing
the RESET coil would re-establish sufficient residual magnetism
to again hold the latch away from the armature. Nonetheless,
with power removed from the RESET coil and promptly applied to
the OPERATE coil the latch was returned to its proper position
and maintained a positive latch. The OPERATE coil was verified
to have sufficient magnetic flux to restore the latch by
blocking the armature's motion preventing the latch from
becoming disloged by the impact of the armature hitting the
OPERATE coil.
Operator Error
Human error can result in a failure of the relay to latch can result from
the following means:
Simultaneous demand signals to both coils
The OPERATE AND RESET coils are manually actuated off of
separate control pushbuttons. Simultaneous operation of
multiple pushbuttons is contrary to normal plant operating
practices. Additionally, the observation were primarily made
while performing Post-Maintenance Testing which is done in an
intentionally controlled and deliberate manner to observe any
abnormalities with the component.
12
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
Insufficient duration of demand signal
It was observed that when the operate coil of the relay for the
2B EDG LOCAL MANUAL acknowledge function was momentarily
energized a failure to latch condition could be repeated. The
product specification gives 25 ms as the relay operate time.
This cause is unlikely to have occurred due to the manner in
which the operators are trained to operate the controls.
Console pushbuttons are held long enough to observe a direct
effect of the pushbutton, e.g., valve position indication
change, pump start, etc., the inherent delays would result in
the pushbutton being depressed for much greater than minimum
time.
Jarring of the relay
Mechanically agitating the relay was observed to cause the
relay to unlatch. This required directly striking the relay.
The relay is normally protected by the structure of the relay
cabinet, preventing the necessary access.
In February 1996, PSE&G Hudson Generating Station identified to
MSD a problem with 255 style relays becoming unreliable
following a 30 inch drop to a carpeted floor. MSD identified
that this was a result of how the operate coil was mounted to
the relay frame. A drop or severe shock resulted in a change
of alignment between the armature and latch. A change in
construction implemented with date code 9240 (fortieth week of
1992) provided resolution. None of the recent Salem failures
involved relays dated prior to 9501.
Issue #3: Spurious unlatching of the relay without a valid demand.
o Description of the Issue
The Struthers-Dunn 255 style Operate-Reset Relay is designed to maintain
its last demanded condition, OPERATE or RESET, even upon loss of power
and subsequent reenergizing of the relay. On separate occasions, the
Loop 21 and 22 Residual Heat Removal Heat Exchanger Bypass Valves,
21(22)RH29, have spuriously changed from AUTO to MANUAL modes of
operation. Each of these occurrences have been coincident with the flow
in the respective RHR Loop being reduced to the Low Flow setpoint. With
the valve in AUTO and the associated pump in service, a low flow
condition is designed to open the associated RH29 valve to preserve a
minimum flow flowpath for its pump.
o Causal Factor Analysis
13
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
Three potential causes of the spurious reset of the relays were
identified: mutual interference (Magnetic interaction) between adjacent
relays, jarring of the relay, and degradation of the latch pawl.
Degradation of the latch was eliminated due to visual inspection of the
relays and successful bench testing subsequent to the failure. No
visible wear of the latch was observed nor other degradation that would
weaken its performance.
Magnetic Interaction between relays is suspected due to the coincident
actuation of the low flow relay, co-located within the same relay can, at
the time that the change in operating mode was seen. The relay can
containing the AUTO-MANUAL (83) relay, also contains two 115 VAC 219
style relays actuated on High and Low RHR Loop flow respectively. All
three relays are mounted vertically, base down, in a line along their
long axis, each relay base adjacent to the next.
This hypothesis was tested by mounting the 22RH29 255 style relay which
failed in a relay module with two 219 relays. With the 255 relay
latched, the 219 relays were energized without causing the 255 relay to
reset. Additionally, a 255 relay was latched and placed alongside a 219
relay with the 219 relay coil adjacent to each side of the 255 relay.
The 219 relay was energized without being able to cause the 255 relay to
reset.
This potential cause was discussed with the vendor Magnacraft/Struthers-
Dunn (MSD) (Mr. Tom Mahaffey). He commented that the 219 relay design
limited the magnetic flux outside of its case.
FPI International, Inc. confirmed that the magnetic field outside the
case of an energized relay was too small (< 1 Gauss) to have impacted the
operation of an adjacent relay.
Jarring of the relay is the most probable cause of the relay's
unlatching. Due to the reduction of the spring's latching force, only a
small external force is necessary to unlatch a failure prone relay.
While the failures in this mode were observed to be concurrent with the
energization of an adjacent relay, it is unlikely that any significant
force was transmitted to the latching relay. The most probable initiator
was opening or closing a cabinet door or floor borne vibration due to
movement of a heavy load in the vicinity. However, neither of these two
conditions has been positively confirmed to have occurred concurrent with
the relay failure.
Issue #4: Unsuitability of new relays.
o Description of the Issue
As part of the current Salem 1 and 2 outages, a 100% inspection of the
Bailey Cabinets and relays is being performed. In preparation for these
activities a large quantity of
14
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
replacement relays were purchased from Farwell and Hendricks (F&H). F&H,
in turn purchases the relays from MSD and then qualifies the relays for
nuclear applications. Prior to a replacement relay's installation in the
field, it is bench tested to verify its suitability. During the bench
testing, a number of new relays were found to exhibit high contact
resistance and/or crimped leads along the chassis of the relay.
o Causal Factor Analysis
The cause of the wire crimping is due to poor workmanship during the
final assembly steps of the relay as the clear cover is placed over the
relay assembly. This was discussed with MSD during a site visit on
September 12, 1996. They attribute this due to the short lead time
provided in manufacturing the relays for PSE&G.
A relay with a crimped lead to the RESET coil was tested with the wire
shorted to the chassis. No impairment of the relay function was
observed.
High contact resistances is also attributed to poor workmanship controls
during manufacture and subsequent qualification of these relays. F&H
during its qualification of the relays for PSE&G identified a small
number of problems with relay performance to MSD for their resolution.
Five relays were returned by F&H to MSD for evaluation. MSD subsequently
reported that one did not latch reliably and two exhibited problems with
contacts. The remaining two exhibited no problems. Prior to March 1996,
255 series relay contacts were not 100% tested by MSD. As a result of
the F&H identification, 100% contact testing was initiated. MSD noted
that "cleaning the contacts restored satisfactory operation. It a
virtual certainty these relays would have functioned flawlessly under
normal load conditions."
The single relay that did not latch reliably was attributed to a
misformed armature "catch step." The misformed armature was believed to
be one that was used to set up the press that forms the step and should
not have been reused in manufacture of a relay.
The concern with high contact resistance was determined to be a result of
flux residue from the attachment of the contact pads to the contact leaf
spring. "Burning in" the contact by operating the contact through
repeated cycles under a 1 amp load has been shown to eliminate this
residue and restore proper contact resistance. A requirement to "burn
in" each relay has been added to the purchase order specification for the
relays.
Prior Similar Events and Operating Experience
The MMIS, TextSearch and NPRDS Databases were utilized to identify prior
Similar Events and Operating Experience pertinent to this evaluation.
15
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
MMIS work orders in current, history, and archive status were searched
for occasions where folio X 40 0407 or folio X 40 0408 parts were issued
to identify replacements due to similar failures. Attachment 3 is a
tabulation of pertinent occurrences.
TextSearch was queried for keywords "relay" and "Struthers" or "Dunn" or
"MSD", and "relay" and "latch" within 25 words of "failure." INPO OE 5537
was the sole instance of a reported problem with Struthers-Dunn relays.
This OE notice reported that 219 style relays manufactured prior to 1983
exhibited abnormal discoloration resulting from out-gassing of the coil
encapsulant. This condition, as well as any other unusual discoloration
of the relay, is being identified during he current Bailey Relay
Inspection Program. Any relays identified as discolored are being
replaced. This OE does not pertain to the current condition.
NPRDS was searched for industry experience with Struthers-Dunn relays.
This search was initiated by querying for Component = RELAY, Manufacturer
= MSD (formerly Struthers Dunn). Twenty three records were identified
satisfying these criteria. Four of these records involve 255 style
relays. All four of these instances occurred at Salem and were captured
within the MMIS search.
Contact was made with the NPRDS coordinator at any facility which
reported via NPRDS a problem with MSD manufactured relays. None of the
facilities identified any usage of B255 style relays.
As noted previously, F&H identified problems, predominantly with relays
produced early in the PSE&G order, to MSD during their qualification
testing. For their qualification of the relays F&H tests 100% of the
relays prior to shipment to PSE&G.
PSE&G Hudson Generating Station identified a problem with relays easily
coming out of alignment due to dropping. MSD responded by stating that
the problem had been previously identified and corrected in 1992.
Generic Implications
These relays are used throughout the control circuitry at both Salem
units. Similar relays are used at Hope Creek Station. Following
formation of the Bailey Relay Project team, an evaluation of relays used
at Hope Creek was performed. This evaluation determined that no relays
manufactured after September 1992 had been installed at Hope Creek.
Therefore, none of the Hope Creek relays were susceptible to this
latching failure due to the reduction in spring elongation.
Safety Significance
The relays involved with this evaluation are utilized in the control
systems for both safety-related and non-safety related components
throughout both Salem units.
16
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
All of the failures identified-to date have involved DC powered relays,
predominantly of the 24-28 VDC ratings. In addition to the 24-28 VDC
relays, Salem utilizes 125 VDC relays (two failures) and 110 VAC relays
(0 failures) of the B255 style. All of the recent failures have been
with relays staged for or installed in the Bailey Control Cabinets.
The 24-28 VDC relays typically are used in the interfacing circuitry
between the operator's control station pushbuttons and the subsequent
actuation of the actual device. Within these applications the relays
typically serve to establish and maintain the last selected
AUTO/MANUAL mode of operation (83 device)
Breaker demand position (checking relay (3 device))
Alarm acknowledge (74 device)
The 125 VDC relays typically provide breaker controls for the actuated
device.
Per the UFSAR SER Section 7.3.4, "the automatic operation of the ESF
equipment is independent of any action in the 28V circuitry."
Nonetheless, improper operation of the relays imposes additional demands
and distractions on the operators. Additionally, failures, such as the
RH29s spurious shift to MANUAL, can adversely impact the operation of the
controlled equipment. In the case of the RH29s, a spurious shift to
MANUAL can result in operation of the associated RHR pump with less than
minimum pump flow. A complete search of all affected control circuits
was not conducted to determine the extent of this problem.
Plant mode specific safety evaluations in accordance with 10CFR50.59 are
being performed prior to the entry into any defined mode to ascertain the
impact of any component which may have a failure prone relay within its
control circuitry. These safety evaluations will be attached to this
report as they become available.
17
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
5.0 CORRECTIVE ACTIONS
o Corrective Actions and Accountabilities
Compensatory Actions
Upon identification of the failure of a relay in service troubleshooting
and repairs are performed per the Work Control Program (NAP-9).
Identification of the suspect relay, bench testing and installation of a
replacement relay is performed. Following replacement, testing of the
affected circuit is conducted prior to its being returned to service.
Failed relays are turned over to the System Manger for evaluation.
Interim Measures
Bailey Relay Inspection procedure, SC.MD-PM.ZZ-0204(Q) Rev. 0, was issued
on 9/13/96. This relay provides consistent direction for the performance
of the relay inspections and establishes formal acceptance criteria for
the evaluation of the relays as well as a permanent record of the
findings.
Salem System Manager has established a database to compile the details of
relay problems. This database, if maintained, will facilitate trending
for future problems.
Corrective Actions to Prevent Recurrence
Resolution of the latching problem has been developed by the replacement
of the latch coil spring with a shorter spring. This replacement spring
was identified by MSD and has been used with the B255 style latching
motor in other MSD manufactured relays. The replacement of the spring
restores the latching force to levels consistent with the 1970 vintage
relays. A two phase approach has been established to acquire replacement
relays with the spring modification. Initially, post September 1992
manufactured relays have been returned from folio to F&H. F&H has
committed to replace the spring and rededicate the relays for PSE&G's
use. Additional relays are being purchased via F&H from MSD which
incorporate both the different spring and a change in the wire gauge used
in the RESET coil. The larger diameter wire in the RESET coil provides
additional manufacturing margin to assure the relay's performance under
degraded voltage conditions. Both the refurbished relay and the newly
manufactured relay have been assigned new folio numbers to aid in the
tracking of their installation in the plant.
Bailey Relay Inspection procedure, SC.MD-PM.ZZ-0204(Q) has been issued
and upgraded to incorporate the findings of this root cause
investigation. Additionally, alignment of the relay testing protocol
between PSE&G and F&H has been established to improve the acceptability
of relays received by PSE&G.
18
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
Continued 100% augmented receipt inspection of Bailey relays will
continue until adequate quality levels have been demonstrated by the
vendor.
CRCA# 0 Responsibility: Dept.: xxxxx Due Date: 12/31/97
Replacement of B255 Style Latching Relays manufactured after September
1992 is to be performed in accordance with the schedule developed by the
Bailey Relay Project and the requirements of each Mode's Safety
Evaluation.
CRCA # 0 Responsibility: Dept.: SMD Due Date: 2/1/96
Safety Evaluation of the potential impact of the suspect B255 relays upon
the upcoming mode shall be performed prior to mode change until the
suspect relay population has been replaced with upgraded/newly
manufactured relays
CRCA # 0 Responsibility: Dept.: ESSA Due Date: 2/1/96
o Measures to Evaluate CA Effectiveness and Accountabilities
Evaluation of the in-plant performance of the upgraded/newly manufactured
B255 relays will be performed by trending relay data from both the system
manager's relay database and maintenance rule component performance data.
CRVR #01 Responsibility: Dept.: ESSA Due Date: 12/31/97
19
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
6.0 OTHER ISSUES AND IMPACTS
o Issues Beyond the Scope of Investigation
Several procurement issues appear to have contributed to the bailey relay
problem. These are indicated by
The time pressures applied to the manufacturer and subsequent lapses
in quality of the relays received.
Miscommunication of the qualification requirements for the relays to
F&H.
Differences in testing protocols/requirements between PSE&G and F&H.
Nuclear Procurement and Materials Management recently completed a level 1
Root Cause investigation into the "Effectiveness of the NBU Materials
Management Process", CR 960729163. This report addresses similar
procurement concerns identified over the same interval as the purchase of
these relays. As a result, this Bailey Relay investigation did not
pursue the procurement issues.
o Common Mode Failures, as Appropriate
The degradation in the functionality of the relay latching mechanism
established a means of common mode failure within the various systems
that utilize these relays in their control scheme. As a result, mode
changes have been delayed until the potential impact have been evaluated
and necessary equipment for the forth coming mode restored to operability
by the installation of an upgraded/newly manufactured relay.
o Reportability
These occurrences have been reported tot he NRC by means of a four hour
report on November 5, 1996 and LER 272/96-031 -00.
Part 21 notification was initiated on November 1, 1996 by F&H.
o Industry Information Dissemination
Industry dissemination will be provided by the Part 21 notification
initiated by F&H.
20
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
7.0 SOURCES OF INFORMATION
o Contacts
a. Thomas Mahaffey, Product Engineering Mgr., Magnecraft/Struthers-
Dunn, (803) 395-8512
b. Larry Bowman, Regional Sales Mgr., Magnecraft/Struthers-Dunn, (717)
764-698
c. Mike Wooldridge, Systems Engineering Manger, Farwell and Hendricks,
(513) 528-7900
d. Matt Artz, Contract Engineer, Farwell & Hendricks, (513)528-7900
e. Brian Miracle, Test Engineer, Farwell & Hendricks, (513)528-7900
f. Mike Bell, Test Engineer, Farwell & Hendricks, (513)528-7900
g. Len Rajkowski, Bailey Relay Replacement Manager (x-5142)
h. Sandra Jannetty, System Manger (x2704)
i. Mike Panko, System Manger (x-7192)
j. Ken Staring, System Manger (x-2157)
o References
a. Maintenance and Application Guide for Control Relays and Timers,
NMAC Report TRA 02067
b. Protective Relay Maintenance and Application Guide, NMAC Report NP-
7216
c. Engineer's Relay Handbook, revised second edition, National
Association of Relay Manufacturers
d. Struthers-Dunn Commercial/Industrial Relays Catalog
e. Wyle Laboratories Report, Report Number 45053R95, "Failure Analysis
of Struthers-Dunn Relays From Public Service Electric And Gas Co.",
February 19 1996
f. Vendor Telcon Summary, Tom Mahaffey, MSD Magnacraft and Ken Staring,
Salem System Engineering, 6/13/1995
21
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
9. Maplewood Research and Testing Laboratory Report, Report # 77299,
"Energy-Dispersive X-ray Chemical Analysis of Particles Found Inside
Struthers Dunn Type 219 ABAP Relay from 21RH29 Circuit (63LX/RBA),
No. 2 Unit - Salem Generating Station", July 12, 1996
h. Maplewood Research and Testing Laboratory Report, Report # 72668,
"Examination of Relay from Reactor Control System, Salem Generating
Station", June 15, 1989
i. Salem Significant Event Response Team (SERT) 95-02 Report, "Salem
Unit 2 Reactor Trip on Partial Loss of Switchyard June 7,1995", July
18, 1995
j. FPI International, "Struthers-Dunn Model 255XCXP Relay Root Cause
Evaluation for Salem Nuclear Generating Station", October 20, 1996,
FPI 96-829
8.0 INVESTIGATOR COMMENTS
None
9.0 APPENDICES & ATTACHMENTS
1) Attachment 1 Failure Mode Chart
2) Attachment 2 NPRDS Data
3) Attachment 3 MMIS Data
4) Attachment 4 Work Standard BAILEY 001
5) Attachment 5 FPI Root Cause Report
22
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station
Figure 1 "Struthers-Dunn 255 Style Operate Reset Relay" omitted.
23
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station Attachment 1
Figure 2 omitted.
24
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station Attachment 3
Figure omitted.
25
Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195
Relay Latch Failures for Salem Generating Station Attachment 3
Table, 5 pages (26-30) omitted.
*** END OF DOCUMENT ***
Page Last Reviewed/Updated Wednesday, March 24, 2021