Part 21 Report - 1996-782

ACCESSION #: 9701090072 NOTE: This text document was processed from a scanned version or an electronic submittal and has been processed as received. Some tables, figures, strikeouts, redlines, and enclosures may not have been included with this submittal, or have been omitted due to ASCII text conversion limitations. In order to view this document in its entirety, you may wish to use the NUDOCS microfiche in addition to the electronic text. Root Cause Analysis of Struthers-Dunn Operate-Reset Relay Latch Failures for Salem Generating Station Condition Reports CR 960906195 CR 961101269 CR 961105229 December 6, 1996 Principal Investigator Craig Bersak Prepared by: Craig Bersak, Engineering Assurance Approved by: Rajkowski, Manager - Bailey Replacement Project Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station TABLE OF CONTENTS 1.0 EXECUTIVE SUMMARY . . . . . . . . . . . . . . . . . . . . . . . . . 2 2.0 INTRODUCTION . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 o Background . . . . . . . . . . . . . . . . . . . . . . . . . 5 o Identification of Occurrence . . . . . . . . . . . . . . . . 5 o Conditions Prior to Occurrence . . . . . . . . . . . . . . . 5 o Root Cause Analysis Methodology . . . . . . . . . . . . . . . 5 3.0 DESCRIPTION OF OCCURRENCE/SCAQ . . . . . . . . . . . . . . . . . . 6 o Chronology of Events . . . . . . . . . . . . . . . . . . . . 6 o Troubleshooting, Remedial, and Compensatory Actions Taken . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 o Notification of Authorities/Industry . . . . . . . . . . . . 8 o Related Failure Modes . . . . . . . . . . . . . . . . . . . . 8 4.0 ANALYSIS & CONCLUSIONS . . . . . . . . . . . . . . . . . . . . . .10 Issue #1: Seismic adequacy of B255 style relays for Salem Station . . . . . . . . . . . . . . . . . . . . . . . .10 o Description of the Issue . . . . . . . . . . . . . . . . . .10 Issue #2: Failure of relay to latch into OPERATE upon a valid demand. . . . . . . . . . . . . . . . . . . . . . . . .10 o Description of the Issue . . . . . . . . . . . . . . . . . .10 o Causal Factor Analysis . . . . . . . . . . . . . . . . . . .11 Issue #3: Spurious unlatching of the relay without a valid demand. . . . . . . . . . . . . . . . . . . . . . . .14 o Description of the Issue . . . . . . . . . . . . . . . . . .14 o Causal Factor Analysis . . . . . . . . . . . . . . . . . . .14 Issue #4: Unsuitability of new relays . . . . . . . . . . . . . .15 o Description of the Issue . . . . . . . . . . . . . . . . . .15 o Causal Factor Analysis . . . . . . . . . . . . . . . . . . .15 Prior Similar Events and Operating Experience . . . . . . . . . .16 Generic Implications . . . . . . . . . . . . . . . . . . . . . . .17 Safety Significance . . . . . . . . . . . . . . . . . . . . . . .17 5.0 CORRECTIVE ACTIONS . . . . . . . . . . . . . . . . . . . . . . . .19 o Corrective Actions and Accountabilities . . . . . . . . . . .19 o Measures to Evaluate CA Effectiveness and Accountabilities . . . . . . . . . . . . . . . . . . . . . .20 6.0 OTHER ISSUES AND IMPACTS . . . . . . . . . . . . . . . . . . . . .21 o Issues Beyond the Scope of Investigation . . . . . . . . . .21 o Common Mode Failures, as Appropriate . . . . . . . . . . . .21 o Reportability . . . . . . . . . . . . . . . . . . . . . . . .21 o Industry Information Dissemination . . . . . . . . . . . . .21 7.0 SOURCES OF INFORMATION . . . . . . . . . . . . . . . . . . . . . .22 i Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station o Contacts . . . . . . . . . . . . . . . . . . . . . . . . . .22 o References . . . . . . . . . . . . . . . . . . . . . . . . .22 8.0 INVESTIGATOR COMMENTS . . . . . . . . . . . . . . . . . . . . . . .23 9.0 APPENDICES & ATTACHMENTS . . . . . . . . . . . . . . . . . . . . .23 ii Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station 1.0 EXECUTIVE SUMMARY Description of Event This root cause evaluation addresses failures of Struthers-Dunn B255 style latching relays to latch on demand and to reset without a valid demand signal. Subsequent to the identification of failures at Salem Station, PSE&G was notified by its vendor, Farwell & Hendricks, Inc., of its submission of a 10CFR21 notification of a potential defect and of the premature failure of a relay during seismic evaluation. Condition Reports 961101269 and 961105229 were written to document these concerns. Causes The root cause of the relay latching problems was a reduction in the elongation of the coil spring for the latching mechanism. This reduction occurred during revisions to the design of the relay frame and reset coil bracket. As a result of the reduced spring load the impact of the operate armature upon the latch could result in its movement away from the latched position, allowing the relay to return to the reset configuration. Latched relays would require the addition of energy from some outside source, i.e., vibration or jarring, to become unlatched. Seismic concerns developed when PSE&G was notified that a verification relay of similar manufacture dates had become unlatched during seismic testing above 5G broadband acceleration when mounted in the floor mount configuration. The Salem applications of the relays were evaluated and found to have limiting seismic requirements below this level. As a result their seismic qualification was and is suitable for Salem Station applications. Corrective Actions Coordination with the relay manufacturer and vendor, Magnecraft/ Struthers-Dunn (MSD) and Farwell & Hendricks, Inc.(F&H), has identified a replacement spring to restor the latching force to levels consistent with the 1970 vintage relays. Relays purchased from F&H are being returned to F&H to have the replacement spring installed and become re-dedicated for PSE&G's use. MSD has incorporated the replacement spring into the manufacture of this series relay. New folios have been established to identify refurbished and newly manufactured relays from the suspect population. A formal relay bench testing protocol has been established and proceduralized to identify potential relay failures prior to their installation into the plant. This testing procedure has been aligned with the F&H testing procedure to assure consistency. A Bailey Replacement Project has been initiated to identify and replace the suspect relays prior to entering a mode where the component impacts plant safety. Safety 1 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station Evaluations in accordance with 10CFR50.59 are being performed to assure that the unchanged relays do not impact safety requirements for the mode to be entered. Safety Significance The relays involved with this evaluation are utilized in the control systems for both safety-related and non-safety related components throughout both Salem units. All of the failures identified to date have involved DC powered relays, predominantly of the 24-28 VDC ratings. In addition to the 24-28 VDC relays, Salem utilizes 125 VDC relays (two failures) and 110 VAC relays (0 failures) of the B255 style. All of the recent failures have been with relays staged for or installed in the Bailey Control Cabinets. The 24-28 VDC relays typically are used in the interfacing circuitry between the operators control station pushbuttons and the subsequent actuation of the actual device. Within these applications the relays typically serve to establish and maintain the last selected AUTO/MANUAL mode of operation (83 device) Breaker demand position (checking relay (3 device)) Alarm acknowledge (74 device) The 125 VDC relays typically provide breaker controls for the actuated device. Per the UFSAR SER Section 7.3.4, "the automatic operation of the ESF equipment is independent of any action in the 28V circuitry." Nonetheless, improper operation of the relays imposes additional demands and distractions on the operators. Additionally, failures, such as the RH29s spurious shift to MANUAL, can adversely impact the operation of the controlled equipment. In the case of the RH29s, a spurious shift to MANUAL can result in operation of the associated RHR pump with less than minimum pump flow. A complete search of all affected control circuits was not conducted to determine the extent of this problem. Plant mode specific safety evaluations in accordance with 10CFR50.59 are being performed prior to the entry into any defined mode to ascertain the impact of any component which may have a failure prone relay within its control circuitry. These safety evaluations will be attached to this report as they become available. Generic Issues None 2 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station Reportability These occurrences have been reported to the NRC by means of a four hour report on November 5, 1996 and LER 272/96-031-00. Part 21 notification was initiated on November 1, 1996 by F&H. 3 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station 2.0 INTRODUCTION o Background This root cause evaluation addresses failures of Struthers-Dunn B255 style latching relays to latch on demand and to reset without a valid demand signal. Subsequent to the identification of failures at Salem Station, PSE&G was notified by its vendor, Farwell & Hendricks, Inc., of its submission of a 10CFR21 notification of a potential defect and of the premature failure of a relay during seismic evaluation. Condition Reports 961101269 and 961105229 were written to document these concerns. o Identification of Occurrence During post-maintenance testing of 22 Residual Heat Removal (RHR) pump, the 22 loop Heat Exchanger Bypass valve, 22RH29, was observed to shift from Automatic to Manual mode without a demand from the operator. During the PMT activity, RHR pump flow was being reduced to open 22RH29. When the low-flow setpoint, 500 gpm, was reached the valve spuriously swapped to MANUAL, resulting in a failure of the valve to open as required. Separately, the Slow Speed start for the 22 Boric Acid Transfer Pump failed to latch in when selected. When the operator released the push- button, the pump stopped without generating an alarm. Additional component failures due to latching relay problems are listed in Attachment 3. o Conditions Prior to Occurrence At the time of discovery, both Salem Unit 1 and Unit 2 were defueled. An extended maintenance/refueling outage was on-going for Unit 2 with outage activities suspended on Unit 1. o Root Cause Analysis Methodology Failure Modes Analysis was utilized in the root cause evaluation of this event. A Failure Modes Chart can be found in Attachment 1. Additionally, Change Analysis was performed (by FPI International) comparing the recently purchased relays to the 1970s vintage relays installed as original equipment. The problems encountered with these relays was discussed and reviewed with both the relay manufacturer, Magnecraft/Struthers-Dunn (MSD), and the qualifying vendor, Farwell & Hendricks, Inc. (F&H). 4 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station 3.0 DESCRIPTION OF OCCURRENCE/SCAQ o Chronology of Events Prior to the commencement of the current Salem Unit 1 and 2 outage, the degradation of relays within the Bailey Relay cabinets was identified. The primary aspect of the observed degradation was an increase in contact resistance due to the formation of a cadmium chloride salt within the relay case. As a result of this adverse trend, a Bailey Relay Inspection/Replacement Project was initiated with the commitment to perform a 100% inspection of each unit's Bailey cabinets prior to the unit's return to service. A work standard (BAILEY 001) was developed providing the technician with criterion for evaluating the relays via visual inspection. Relays which were judged by the technician to be degraded were replaced with recently pruchased relays. During the Post Maintenance Testing and following return of the component to the Operations Department, failures of latching style relays were observed at an unacceptable frequency. Additionally, during bench testing of the relays, prior to their installation, an approximate 10% latching failure rate was observed. As a result of the field and bench testing failures, condition report 960906195 was initiated to identify the root cause of the relay problem. In 1996, 15 instances of Struthers-Dunn Operate-Reset Relays failing to latch into the OPERATE state or spuriously resetting have been identified on Salem Unit # 2. Representative of these failures are the following occurrences: On June 6,1996, the "LOCAL MANUAL" alarm for the 2B Emergency Diesel Generator did not lock in when acknowledged by the operator. On June 30, 1996, 22RH19, the 22 Residual Heat Removal Discharge valve, would return to the closed position after initially responding to an OPEN demand; i.e., stroke open, reach its open limit and then promptly close without a close demand from the operator. On August 30 1996 Post Maintenance Testing (PMT) was performed on 22 Residual Heat Removal (RHR) pump following completion of corrective maintenance work order 960828275. This work order was initiated due to excessive oil leakage. During the PMT, 22 RHR loop flow was being reduced to cause its heat Exchanger Bypass valve, 22RH29, to automatically open on the resulting low flow (<500 gpm) signal. As the low flow setpoint was approached the bypass valve spuriously swapped from AUTO to MANUAL mode of operation. There were three separate occurrences of problems with the 22 Boric Acid Transfer (BAT) Pump. On August 19, 1996, the pump stopped when its mode was shifted by the operator from AUTO to MANUAL. It subsequently failed to restart on a manual demand. On September 6, 5 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station 1996 during PMT of the 22 BAT Pump Control Circuit, the pump failed to remain operating in slow speed when the "22 SLOW START" pushbutton was released. Similarly, during a separate PMT, the 22 BAT pump failed to operate in "BORATE". All of these problems were diagnosed as a failure of the associated Operate-Reset relay to latch, or remain latched, in the Operate condition. o Troubleshooting, Remedial, and Compensatory Actions Taken Troubleshooting On each occurrence troubleshooting of the affected circuitry was performed by the Salem Maintenance Department. In each case, one or more Operate-Reset Relays was determined to be operating improperly and replaced with a like relay which had been successfully bench tested. A relay testing protocol was established and proceduralized in SC.MD- PM.ZZ-0204(Q), Bailey Relay Inspection, to establish consistency in relay testing and provide an arduous standard to eliminate potential relay failures prior to their installation in the plant. This procedure cycled each latching relay through a minimum of 10 cycles to verify its performance. This procedure was issued September 13, 1996. Communications with the relay manufacturer, MSD, were initiated to investigate and resolve the problem. A sample of the failed relays was brought to MSD's Darlington, SC facility for evaluation by MSD's Product Engineering Manager. A PSE&G Root Cause Engineer accompanied the relays during their evaluation. Only one of the relays repeated the latching failure. That relay was found to have its latching coil spring stretched at an attachment loop. When the spring was replaced the relay functioned properly. FPI International was commissioned to perform a Change Analysis of the relays to determine the cause of failure. Their report is included as attachment 5. F&H was contacted regarding the observed failures. Their Systems Engineering Manager and Sales Manager visited the Salem facility at which time the failures were discussed in detail. At this visit, F&H identified that they had not performed qualification testing of the relays in the "floor mounted" orientation utilized within the Bailey Relay Cabinets. (Adequacy of the relays in the floor mounted configuration at Salem had previously been established by PSE&G's evaluation of the Hope Creek Bailey Cabinet Seismic Report.) 6 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station Remedial Actions Following identification of the root cause of the relay's unlatching, the reduction in spring elongation, a design change to the relays was proposed by MSD replacing the existing coil spring with a shorter spring. The relay latching mechanism with the replacement spring had been utilized in other MSD relay designs. Compensatory Actions A Bailey Relay Project team was formed by the NBU's Senior Vice President - Nuclear Engineering to evaluate the relay problem and initiate the corrective actions. The safety significance of the failure prone relays is being evaluated by a 10CFR50.59 evaluation for the then existing condition and for each subsequent mode change until the replacement of the suspect population. A prioritized listing of components, keyed to the modes where the component was required to be operable, was established to focus the replacement of the suspect relays with upgraded relays. Additional components were added as necessary based on the 50.59 evaluations. Changes in plant mode were delayed until the necessary relays could be replaced. o Notification of Authorities/industry The Nuclear Regulatory Commission was notified by F&H of the potentially degraded component on November 1, 1996. Their final report, including their root cause evaluation, was due to be issued November 30, 1996 and has been delayed. PSE&G made a four-hour report upon notification of the seismic testing failure by F&H. A Licensee Event Report, 272/96-031-00, was issued discussing PSE&G's response to the seismic, Part 21, and functional failures. o Related Failure Modes Previous relay problems have been identified related to high contact resistances and the formation of "white deposits" within the relay case for other styles of Struthers-Dunn relays. As a result of this a 100% inspection of Bailey Relay Cabinets and replacement of degraded relays was initiated during the current outage. Subsequent analysis by the PSE&G Maplewood Test Labs identified the deposits as "similar to a corrosion product of cadmium containing chlorine such as found in cadmium chloride." An INPO Operating Experience Notice, OE 5537, identified degraded performance (premature failures) of 219 style relays with Polypropylene relay coils. During the aforementioned inspection, relays having this type coil are replaced with bench-tested relays. A sample of 12 relays was sent to Wyle Laboratories for failure analysis of the high resistance condition. Wyle Laboratories report no. 45053R95, "Failure Analysis of 7 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station Struthers-Dunn Relays From Public Service Electric and Gas Co.," concludes that "the high contact resistance appeared to be the result of the depositing of residual flux and/or its reaction products on the contacts." Operation of the relay coils at higher than 110% of rated voltage can result in the premature failure of the relay coil. The 28 VDC, 125 VDC, and 115 Vac systems are typically maintained withing voltage ranges that approach or slightly exceed the 110% values. This condition can result in premature coil failure, especially among continuously energized relays. These failure mechanisms result in high contact resistances or operating coil failures. They do not impair the relay's ability to latch if the operate coil is sufficiently energized. The failures identified within this evaluation are not the result of any of these related failure modes. 8 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station 4.0 ANALYSIS & CONCLUSIONS Issue #1: Seismic adequacy of B255 style relays for Salem Station. o Description of the Issue On November 5, 1996, F&H notified PSE&G that one of two verification relays had failed seismic qualification by becoming unlatched at acceleration levels above 5G broadband when tested in a floor mounted configuration. A walkdown by System Engineering was performed to identify the locations where the latching relays are used in the floor mounted configuration. Plant Applications Group evaluated the various locations identified and determined that none were subject to seismic levels in excess of 5G. As a result, the relays were determined, as a group, to be qualified for the seismic spectra at Salem station. Issue #2: Failure of relay to latch into OPERATE upon a valid demand. o Description of the Issue The Struthers-Dunn 255 style Operate-Reset Relay, Figure 1, consists of two coils, a spring-loaded mechanical latch and a set of contacts operated off of the armature associated with the OPERATE coil. With the coil initially in the reset condition and both coils not energized, the spring force resulting from four of the contact leaves pulls the OPERATE coil armature via the contact slider to the reset condition. When the OPERATE coil is energized its armature is drawn to the coil. The movement of the armature results in the contact slider changing the state of the relay contacts and allows the latch pawl to descend capturing the armature in the operate condition. With the operate armature caught by the latch, power to the OPERATE coil can be removed without changing the state of the relay contacts. The relay is reset by energizing the RESET coil. When the RESET coil energizes the latch pawl is lifted allowing the OPERATE armature to be returned to the reset condition and changing the state of the relay contacts. Simultaneously energizing both coils will result in the relay attaining its OPERATE state without latching the OPERATE armature. If both coils are then simultaneously de-energized, the final state of the relay will depend upon which coil's magnetic field collapses first; either latching the OPERATE armature or allowing it to return to the reset state. The majority of the failures previously identified resulted when the relay failed to maintain its latched condition when the OPERATE coil was de-energized. This allowed the relay's contacts to be returned to their RESET configuration. 9 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station o Causal Factor Analysis Three potential areas of failure were evaluated for this issue: Mechanical failure of the relay, electrical failure and human error. Mechanical Failure A mechanical failure of the relay to latch can result from the following means: Latching Pawl or Operate Armature prevented from moving freely Each of the failed relays was examined and operated to determine if there was any blockage or impairment to the motion of the moving parts. Each of the relays operated at minimum voltage with successful latching. The cases of each relay was examined with no foreign material noted that may have prevented the latch or armature from traveling freely. Mis-formed Latch Pawl The latch pawl on the relay is a square cut piece of solid nylon. Each of the latches was examined under a 15X magnifier to determine if the trailing (latching) edge had been mis- formed or worn in a manner that would allow the OPERATE armature to return to the reset position. No deformation of any latch was identified. Mis-formed Armature "Catch Step" The portion of the armature that is "captured" by the latch pawl has a "catch step" pressed into it during manufacture. A relay returned by Farwell & Hendricks, the qualifying distributor from whom PSE&G purchases the relays, to MSD was noted as having a misformed "catch step." The misformed step allowed the tip of the armature to slip past the latch pawl, resetting the relay without demand. The recent PSE&G failed relay were examined for repetition of this failure. None of the relays was observed to have a misformed armature "catch step." Insufficient spring loading of the latch pawl One failed relay, removed from the 22RH29 control circuitry, was noticed as having the coil spring which applies the latch preload deformed. One of the attachment loops of the coil spring had become extended increasing the overall length of the spring by 3/64". This elongation effectively removed the spring force seating the latch when the OPERATE 10 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station coil was energized and allowed the armature to repeatedly return to the reset condition. The elongated spring was replaced with a properly formed spring and the relay was observed to function without subsequent problems. Change Analysis, performed by FPI International, identified that the current relay design had reduced the elongation of this spring and resulted in a reduction of latching force being applied. Review of FPI's finding with MSD indicated that the distance between the attachment anchors for the coil spring had reduced when the reset coil attachment bracket was changed sometime in 1993. This reduction in spring elongation, coupled with variations in spring constants with a given set of springs, resulted in some of the relays having marginal latching capability. When the OPERATE coil is energized, its armature is drawn toward it beyond the amount required to seat the latch. When the coil deenergizes the contact springs attempt to return to the reset state. Normally the latch prevents the return to the reset state. However, in those relays where the spring is weak, the impact of the operate armature hitting the latch results in the latch "bouncing" out of position and the relay returning to the reset condition. It was noticed by the Technicians that relays which had failed were susceptible to unlatching when the case was lightly tapped. Electrical Failure An electrical failure of the relay to latch can result from the following means: Insufficient voltage being applied to the operate coil The DC and AC control power busses at Salem typically are above their nominal design voltages. None of the failures occurred during a degraded voltage condition. Nominal Salem 28 VDC bus voltage is 29 VDC. Each of the failed relays was tested at minimum allowable voltage per the relay specification, i.e., 19.2 VDC (80% of 24 VDC), repeatedly without an observed latching failure. One of the failed relays was subsequently tested at a highly degraded voltage of 10 VDC and did not demonstrate a latching failure. Energizing both the OPERATE and RESET coils simultaneously. When both coils are energized the relay's contacts will achieve the operate condition, but the latch will be raised preventing it from capturing 11 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station the armature. If power is simultaneously removed, the final state of the relay will depend upon which coil's magnetic field collapses first. The control circuits for the Boric Acid Transfer pump, 2B Emergency Diesel Generator Alarm, and 21RH19 were evaluated for a means of inadvertently energizing both coils. Each potential path was determined to be protected by a surge suppression diode which would prevent current flow through the RESET coil when the OPERATE coil was selected. On October 9, 1996, testing of the 22 BAT pump circuit was performed (WO# 960924247) to determine whether any unwarranted energization of a RESET coil existed. Voltage with respect to the negative bus was obtained across each coil and diode as each control pushbutton was depressed. Voltage at the Relay Cabinet was determined to be 27.21 VDC. No improper voltage was seen the RESET coils when any control console pushbutton was pressed. Residual Magnetism of the RESET coil core During the evaluation of a relay, its RESET coil was energized continuously at 24 VDC for 5 minutes. When the RESET coil was de-energized, the latch remained affixed to the coil. This was sustained for approximately 25 minutes. Momentary reenergizing the RESET coil would re-establish sufficient residual magnetism to again hold the latch away from the armature. Nonetheless, with power removed from the RESET coil and promptly applied to the OPERATE coil the latch was returned to its proper position and maintained a positive latch. The OPERATE coil was verified to have sufficient magnetic flux to restore the latch by blocking the armature's motion preventing the latch from becoming disloged by the impact of the armature hitting the OPERATE coil. Operator Error Human error can result in a failure of the relay to latch can result from the following means: Simultaneous demand signals to both coils The OPERATE AND RESET coils are manually actuated off of separate control pushbuttons. Simultaneous operation of multiple pushbuttons is contrary to normal plant operating practices. Additionally, the observation were primarily made while performing Post-Maintenance Testing which is done in an intentionally controlled and deliberate manner to observe any abnormalities with the component. 12 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station Insufficient duration of demand signal It was observed that when the operate coil of the relay for the 2B EDG LOCAL MANUAL acknowledge function was momentarily energized a failure to latch condition could be repeated. The product specification gives 25 ms as the relay operate time. This cause is unlikely to have occurred due to the manner in which the operators are trained to operate the controls. Console pushbuttons are held long enough to observe a direct effect of the pushbutton, e.g., valve position indication change, pump start, etc., the inherent delays would result in the pushbutton being depressed for much greater than minimum time. Jarring of the relay Mechanically agitating the relay was observed to cause the relay to unlatch. This required directly striking the relay. The relay is normally protected by the structure of the relay cabinet, preventing the necessary access. In February 1996, PSE&G Hudson Generating Station identified to MSD a problem with 255 style relays becoming unreliable following a 30 inch drop to a carpeted floor. MSD identified that this was a result of how the operate coil was mounted to the relay frame. A drop or severe shock resulted in a change of alignment between the armature and latch. A change in construction implemented with date code 9240 (fortieth week of 1992) provided resolution. None of the recent Salem failures involved relays dated prior to 9501. Issue #3: Spurious unlatching of the relay without a valid demand. o Description of the Issue The Struthers-Dunn 255 style Operate-Reset Relay is designed to maintain its last demanded condition, OPERATE or RESET, even upon loss of power and subsequent reenergizing of the relay. On separate occasions, the Loop 21 and 22 Residual Heat Removal Heat Exchanger Bypass Valves, 21(22)RH29, have spuriously changed from AUTO to MANUAL modes of operation. Each of these occurrences have been coincident with the flow in the respective RHR Loop being reduced to the Low Flow setpoint. With the valve in AUTO and the associated pump in service, a low flow condition is designed to open the associated RH29 valve to preserve a minimum flow flowpath for its pump. o Causal Factor Analysis 13 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station Three potential causes of the spurious reset of the relays were identified: mutual interference (Magnetic interaction) between adjacent relays, jarring of the relay, and degradation of the latch pawl. Degradation of the latch was eliminated due to visual inspection of the relays and successful bench testing subsequent to the failure. No visible wear of the latch was observed nor other degradation that would weaken its performance. Magnetic Interaction between relays is suspected due to the coincident actuation of the low flow relay, co-located within the same relay can, at the time that the change in operating mode was seen. The relay can containing the AUTO-MANUAL (83) relay, also contains two 115 VAC 219 style relays actuated on High and Low RHR Loop flow respectively. All three relays are mounted vertically, base down, in a line along their long axis, each relay base adjacent to the next. This hypothesis was tested by mounting the 22RH29 255 style relay which failed in a relay module with two 219 relays. With the 255 relay latched, the 219 relays were energized without causing the 255 relay to reset. Additionally, a 255 relay was latched and placed alongside a 219 relay with the 219 relay coil adjacent to each side of the 255 relay. The 219 relay was energized without being able to cause the 255 relay to reset. This potential cause was discussed with the vendor Magnacraft/Struthers- Dunn (MSD) (Mr. Tom Mahaffey). He commented that the 219 relay design limited the magnetic flux outside of its case. FPI International, Inc. confirmed that the magnetic field outside the case of an energized relay was too small (< 1 Gauss) to have impacted the operation of an adjacent relay. Jarring of the relay is the most probable cause of the relay's unlatching. Due to the reduction of the spring's latching force, only a small external force is necessary to unlatch a failure prone relay. While the failures in this mode were observed to be concurrent with the energization of an adjacent relay, it is unlikely that any significant force was transmitted to the latching relay. The most probable initiator was opening or closing a cabinet door or floor borne vibration due to movement of a heavy load in the vicinity. However, neither of these two conditions has been positively confirmed to have occurred concurrent with the relay failure. Issue #4: Unsuitability of new relays. o Description of the Issue As part of the current Salem 1 and 2 outages, a 100% inspection of the Bailey Cabinets and relays is being performed. In preparation for these activities a large quantity of 14 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station replacement relays were purchased from Farwell and Hendricks (F&H). F&H, in turn purchases the relays from MSD and then qualifies the relays for nuclear applications. Prior to a replacement relay's installation in the field, it is bench tested to verify its suitability. During the bench testing, a number of new relays were found to exhibit high contact resistance and/or crimped leads along the chassis of the relay. o Causal Factor Analysis The cause of the wire crimping is due to poor workmanship during the final assembly steps of the relay as the clear cover is placed over the relay assembly. This was discussed with MSD during a site visit on September 12, 1996. They attribute this due to the short lead time provided in manufacturing the relays for PSE&G. A relay with a crimped lead to the RESET coil was tested with the wire shorted to the chassis. No impairment of the relay function was observed. High contact resistances is also attributed to poor workmanship controls during manufacture and subsequent qualification of these relays. F&H during its qualification of the relays for PSE&G identified a small number of problems with relay performance to MSD for their resolution. Five relays were returned by F&H to MSD for evaluation. MSD subsequently reported that one did not latch reliably and two exhibited problems with contacts. The remaining two exhibited no problems. Prior to March 1996, 255 series relay contacts were not 100% tested by MSD. As a result of the F&H identification, 100% contact testing was initiated. MSD noted that "cleaning the contacts restored satisfactory operation. It a virtual certainty these relays would have functioned flawlessly under normal load conditions." The single relay that did not latch reliably was attributed to a misformed armature "catch step." The misformed armature was believed to be one that was used to set up the press that forms the step and should not have been reused in manufacture of a relay. The concern with high contact resistance was determined to be a result of flux residue from the attachment of the contact pads to the contact leaf spring. "Burning in" the contact by operating the contact through repeated cycles under a 1 amp load has been shown to eliminate this residue and restore proper contact resistance. A requirement to "burn in" each relay has been added to the purchase order specification for the relays. Prior Similar Events and Operating Experience The MMIS, TextSearch and NPRDS Databases were utilized to identify prior Similar Events and Operating Experience pertinent to this evaluation. 15 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station MMIS work orders in current, history, and archive status were searched for occasions where folio X 40 0407 or folio X 40 0408 parts were issued to identify replacements due to similar failures. Attachment 3 is a tabulation of pertinent occurrences. TextSearch was queried for keywords "relay" and "Struthers" or "Dunn" or "MSD", and "relay" and "latch" within 25 words of "failure." INPO OE 5537 was the sole instance of a reported problem with Struthers-Dunn relays. This OE notice reported that 219 style relays manufactured prior to 1983 exhibited abnormal discoloration resulting from out-gassing of the coil encapsulant. This condition, as well as any other unusual discoloration of the relay, is being identified during he current Bailey Relay Inspection Program. Any relays identified as discolored are being replaced. This OE does not pertain to the current condition. NPRDS was searched for industry experience with Struthers-Dunn relays. This search was initiated by querying for Component = RELAY, Manufacturer = MSD (formerly Struthers Dunn). Twenty three records were identified satisfying these criteria. Four of these records involve 255 style relays. All four of these instances occurred at Salem and were captured within the MMIS search. Contact was made with the NPRDS coordinator at any facility which reported via NPRDS a problem with MSD manufactured relays. None of the facilities identified any usage of B255 style relays. As noted previously, F&H identified problems, predominantly with relays produced early in the PSE&G order, to MSD during their qualification testing. For their qualification of the relays F&H tests 100% of the relays prior to shipment to PSE&G. PSE&G Hudson Generating Station identified a problem with relays easily coming out of alignment due to dropping. MSD responded by stating that the problem had been previously identified and corrected in 1992. Generic Implications These relays are used throughout the control circuitry at both Salem units. Similar relays are used at Hope Creek Station. Following formation of the Bailey Relay Project team, an evaluation of relays used at Hope Creek was performed. This evaluation determined that no relays manufactured after September 1992 had been installed at Hope Creek. Therefore, none of the Hope Creek relays were susceptible to this latching failure due to the reduction in spring elongation. Safety Significance The relays involved with this evaluation are utilized in the control systems for both safety-related and non-safety related components throughout both Salem units. 16 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station All of the failures identified-to date have involved DC powered relays, predominantly of the 24-28 VDC ratings. In addition to the 24-28 VDC relays, Salem utilizes 125 VDC relays (two failures) and 110 VAC relays (0 failures) of the B255 style. All of the recent failures have been with relays staged for or installed in the Bailey Control Cabinets. The 24-28 VDC relays typically are used in the interfacing circuitry between the operator's control station pushbuttons and the subsequent actuation of the actual device. Within these applications the relays typically serve to establish and maintain the last selected AUTO/MANUAL mode of operation (83 device) Breaker demand position (checking relay (3 device)) Alarm acknowledge (74 device) The 125 VDC relays typically provide breaker controls for the actuated device. Per the UFSAR SER Section 7.3.4, "the automatic operation of the ESF equipment is independent of any action in the 28V circuitry." Nonetheless, improper operation of the relays imposes additional demands and distractions on the operators. Additionally, failures, such as the RH29s spurious shift to MANUAL, can adversely impact the operation of the controlled equipment. In the case of the RH29s, a spurious shift to MANUAL can result in operation of the associated RHR pump with less than minimum pump flow. A complete search of all affected control circuits was not conducted to determine the extent of this problem. Plant mode specific safety evaluations in accordance with 10CFR50.59 are being performed prior to the entry into any defined mode to ascertain the impact of any component which may have a failure prone relay within its control circuitry. These safety evaluations will be attached to this report as they become available. 17 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station 5.0 CORRECTIVE ACTIONS o Corrective Actions and Accountabilities Compensatory Actions Upon identification of the failure of a relay in service troubleshooting and repairs are performed per the Work Control Program (NAP-9). Identification of the suspect relay, bench testing and installation of a replacement relay is performed. Following replacement, testing of the affected circuit is conducted prior to its being returned to service. Failed relays are turned over to the System Manger for evaluation. Interim Measures Bailey Relay Inspection procedure, SC.MD-PM.ZZ-0204(Q) Rev. 0, was issued on 9/13/96. This relay provides consistent direction for the performance of the relay inspections and establishes formal acceptance criteria for the evaluation of the relays as well as a permanent record of the findings. Salem System Manager has established a database to compile the details of relay problems. This database, if maintained, will facilitate trending for future problems. Corrective Actions to Prevent Recurrence Resolution of the latching problem has been developed by the replacement of the latch coil spring with a shorter spring. This replacement spring was identified by MSD and has been used with the B255 style latching motor in other MSD manufactured relays. The replacement of the spring restores the latching force to levels consistent with the 1970 vintage relays. A two phase approach has been established to acquire replacement relays with the spring modification. Initially, post September 1992 manufactured relays have been returned from folio to F&H. F&H has committed to replace the spring and rededicate the relays for PSE&G's use. Additional relays are being purchased via F&H from MSD which incorporate both the different spring and a change in the wire gauge used in the RESET coil. The larger diameter wire in the RESET coil provides additional manufacturing margin to assure the relay's performance under degraded voltage conditions. Both the refurbished relay and the newly manufactured relay have been assigned new folio numbers to aid in the tracking of their installation in the plant. Bailey Relay Inspection procedure, SC.MD-PM.ZZ-0204(Q) has been issued and upgraded to incorporate the findings of this root cause investigation. Additionally, alignment of the relay testing protocol between PSE&G and F&H has been established to improve the acceptability of relays received by PSE&G. 18 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station Continued 100% augmented receipt inspection of Bailey relays will continue until adequate quality levels have been demonstrated by the vendor. CRCA# 0 Responsibility: Dept.: xxxxx Due Date: 12/31/97 Replacement of B255 Style Latching Relays manufactured after September 1992 is to be performed in accordance with the schedule developed by the Bailey Relay Project and the requirements of each Mode's Safety Evaluation. CRCA # 0 Responsibility: Dept.: SMD Due Date: 2/1/96 Safety Evaluation of the potential impact of the suspect B255 relays upon the upcoming mode shall be performed prior to mode change until the suspect relay population has been replaced with upgraded/newly manufactured relays CRCA # 0 Responsibility: Dept.: ESSA Due Date: 2/1/96 o Measures to Evaluate CA Effectiveness and Accountabilities Evaluation of the in-plant performance of the upgraded/newly manufactured B255 relays will be performed by trending relay data from both the system manager's relay database and maintenance rule component performance data. CRVR #01 Responsibility: Dept.: ESSA Due Date: 12/31/97 19 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station 6.0 OTHER ISSUES AND IMPACTS o Issues Beyond the Scope of Investigation Several procurement issues appear to have contributed to the bailey relay problem. These are indicated by The time pressures applied to the manufacturer and subsequent lapses in quality of the relays received. Miscommunication of the qualification requirements for the relays to F&H. Differences in testing protocols/requirements between PSE&G and F&H. Nuclear Procurement and Materials Management recently completed a level 1 Root Cause investigation into the "Effectiveness of the NBU Materials Management Process", CR 960729163. This report addresses similar procurement concerns identified over the same interval as the purchase of these relays. As a result, this Bailey Relay investigation did not pursue the procurement issues. o Common Mode Failures, as Appropriate The degradation in the functionality of the relay latching mechanism established a means of common mode failure within the various systems that utilize these relays in their control scheme. As a result, mode changes have been delayed until the potential impact have been evaluated and necessary equipment for the forth coming mode restored to operability by the installation of an upgraded/newly manufactured relay. o Reportability These occurrences have been reported tot he NRC by means of a four hour report on November 5, 1996 and LER 272/96-031 -00. Part 21 notification was initiated on November 1, 1996 by F&H. o Industry Information Dissemination Industry dissemination will be provided by the Part 21 notification initiated by F&H. 20 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station 7.0 SOURCES OF INFORMATION o Contacts a. Thomas Mahaffey, Product Engineering Mgr., Magnecraft/Struthers- Dunn, (803) 395-8512 b. Larry Bowman, Regional Sales Mgr., Magnecraft/Struthers-Dunn, (717) 764-698 c. Mike Wooldridge, Systems Engineering Manger, Farwell and Hendricks, (513) 528-7900 d. Matt Artz, Contract Engineer, Farwell & Hendricks, (513)528-7900 e. Brian Miracle, Test Engineer, Farwell & Hendricks, (513)528-7900 f. Mike Bell, Test Engineer, Farwell & Hendricks, (513)528-7900 g. Len Rajkowski, Bailey Relay Replacement Manager (x-5142) h. Sandra Jannetty, System Manger (x2704) i. Mike Panko, System Manger (x-7192) j. Ken Staring, System Manger (x-2157) o References a. Maintenance and Application Guide for Control Relays and Timers, NMAC Report TRA 02067 b. Protective Relay Maintenance and Application Guide, NMAC Report NP- 7216 c. Engineer's Relay Handbook, revised second edition, National Association of Relay Manufacturers d. Struthers-Dunn Commercial/Industrial Relays Catalog e. Wyle Laboratories Report, Report Number 45053R95, "Failure Analysis of Struthers-Dunn Relays From Public Service Electric And Gas Co.", February 19 1996 f. Vendor Telcon Summary, Tom Mahaffey, MSD Magnacraft and Ken Staring, Salem System Engineering, 6/13/1995 21 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station 9. Maplewood Research and Testing Laboratory Report, Report # 77299, "Energy-Dispersive X-ray Chemical Analysis of Particles Found Inside Struthers Dunn Type 219 ABAP Relay from 21RH29 Circuit (63LX/RBA), No. 2 Unit - Salem Generating Station", July 12, 1996 h. Maplewood Research and Testing Laboratory Report, Report # 72668, "Examination of Relay from Reactor Control System, Salem Generating Station", June 15, 1989 i. Salem Significant Event Response Team (SERT) 95-02 Report, "Salem Unit 2 Reactor Trip on Partial Loss of Switchyard June 7,1995", July 18, 1995 j. FPI International, "Struthers-Dunn Model 255XCXP Relay Root Cause Evaluation for Salem Nuclear Generating Station", October 20, 1996, FPI 96-829 8.0 INVESTIGATOR COMMENTS None 9.0 APPENDICES & ATTACHMENTS 1) Attachment 1 Failure Mode Chart 2) Attachment 2 NPRDS Data 3) Attachment 3 MMIS Data 4) Attachment 4 Work Standard BAILEY 001 5) Attachment 5 FPI Root Cause Report 22 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station Figure 1 "Struthers-Dunn 255 Style Operate Reset Relay" omitted. 23 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station Attachment 1 Figure 2 omitted. 24 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station Attachment 3 Figure omitted. 25 Root Cause Analysis of Struthers-Dunn Operate-Reset CR# 960906195 Relay Latch Failures for Salem Generating Station Attachment 3 Table, 5 pages (26-30) omitted. *** END OF DOCUMENT ***

Page Last Reviewed/Updated Friday, January 31, 2020