Redundancy as a Validated Requirement
Network resilience on a pharma automation project is not just an engineering best practice — it is a validated requirement driven by the risk assessment. When the risk assessment identifies a network failure as a hazard that could cause process data loss or production impact, the HDS must specify a redundancy mechanism that mitigates that risk. The URS references the risk, the HDS specifies the solution, and the OQ verifies the behaviour. A redundancy mechanism that is implemented but not specified in the HDS is outside the validated state — and a mechanism that is specified but never tested produces a validation gap.
The HDS Section 1.2 (Redundancy Strategy) table is the governing document for every resilience decision on the project. It lists every component or function with a single-point-of-failure risk, the redundancy provision for each, and the failure consequence with redundancy in place. This table is reviewed during design approval, and every row that specifies an automatic failover generates a corresponding OQ test case.
Annex 11 §11 requires that data is available and accessible throughout the required retention period. A network architecture with unmitigated single points of failure that could cause data loss does not meet this requirement. The redundancy strategy does not need to guarantee 100% uptime — it needs to demonstrate that reasonably foreseeable failures have been analysed, mitigated proportionately to their risk, and that the residual failure consequences are documented and accepted. A SCADA server that is non-redundant is acceptable if: the PLC buffers data during server downtime, the RTO is defined and achievable, and the failure consequence (temporary loss of HMI) is documented and risk-accepted.
PROFINET Ring with MRP — The Standard for Field Networks
The PROFINET field network — connecting the PLC to remote I/O stations in junction boxes — is the most common point of single-cable failure on a large pharma automation project. A linear PROFINET topology has no resilience: cut the cable between two nodes and all I/O downstream of the break is lost. A ring topology with MRP (Media Redundancy Protocol) eliminates this vulnerability: the ring provides two physical paths between every device, and MRP detects a cable break and reconfigures the ring to restore all communications within 200ms — far faster than the PLC watchdog timeout.
The HDS must specify: that the PROFINET topology is a ring, the MRP manager device (always a managed switch, not the PLC CPU), the MRP ring ports on the switch, and the recovery time specification (≤200ms for IEC 62439-2 Class 1). The IQ verifies the ring topology is cabled as documented. The OQ tests the ring recovery by pulling a cable between two nodes while the PLC is running and confirming: no scan cycle violation, no I/O data loss, and the MRP recovery alarm activates within the specified time.
The HDS Redundancy Strategy Table
HDS Section 1.2 documents the complete redundancy strategy for the system in a table format. Every component or function at risk of single-point failure must appear in this table. The table drives both the OQ test case list and the alarm configuration — every failover event must generate an alarm so operations is aware a redundancy element has been consumed and maintenance is required.
| Component / Function | Redundancy Provision | Failover Mechanism | Failure Consequence (with redundancy) | Alarm Generated |
|---|---|---|---|---|
| PROFINET field ring | Dual-ring topology with MRP | Automatic — MRP manager reconfigures ring within ≤200ms | No production impact — continued operation on alternate ring path | MRP-FAULT — High alarm, maintenance required |
| PLC 24VDC power supply | Dual redundant PSU in hot-standby parallel (Siemens SITOP PSU8600 + SFT redundancy module) | Automatic — SFT module switches to standby PSU on primary failure, <5ms switchover | No production impact — system continues on standby PSU | PSU-FAULT — High alarm, maintenance required |
| SCADA server | Single server (GAMP Cat 4). Daily backup to separate storage. RTO ≤4 hours. | Manual restoration from backup per RBP-SYS-001 | Loss of HMI operator interface. PLC continues autonomous control. Historian data buffered in PLC during outage, synchronised on restoration. | SCADA-COMM-LOSS — High alarm on PLC |
| Historian data continuity | PLC non-volatile data buffer (capacity: 48 hours at 60s scan interval) | Automatic — PLC stores timestamped data during historian outage; synchronises on reconnection in chronological order | No data loss for outages ≤48 hours. Data gap alarm after 5 minutes. | HIST-GAP — High alarm after 5-min gap (ALM-DATA-001) |
| Server storage (SCADA + Historian) | RAID 5 internal storage array; hot-spare drive | Automatic RAID rebuild on drive failure | No production impact — continued operation on degraded RAID. Server alert generated. | IT infrastructure alert — raised to site IT |
| UPS / mains power | APC Smart-UPS — minimum 15 minutes full-load runtime | Automatic switchover to battery on mains failure; orderly shutdown sequence initiated at 5-minute remaining battery | System continues for duration of UPS runtime; graceful shutdown if mains not restored | MAINS-FAIL — Critical alarm; BATTERY-LOW — Critical alarm |
Historian Data Buffering — The Critical OQ Test
The most important resilience test in the OQ for a pharma SCADA system is the historian data buffer test. This test confirms that data recorded during a historian or network outage is not lost — it is buffered in the PLC's non-volatile memory and synchronised to the historian when connectivity is restored. Without this test, there is no validated evidence that GMP records are not lost during an infrastructure failure.
The OQ test procedure for historian data buffering mirrors the test sequence in the QLean OQ template: disconnect the historian at time T1, allow the system to run for at least 6 minutes (to trigger the gap detection alarm at 5 minutes), reconnect at T2, and then query the historian for the T1-T2 period. The historian must contain continuous records for the full outage period — no gap greater than the recording interval (typically 60 seconds). The PLC disaster recovery and backup article covers the buffer mechanism and how to configure the PLC data buffer block for compliant gap-free recovery.
CPU Redundancy — The Design Decision
PLC CPU redundancy — where a hot-standby CPU automatically takes over if the primary fails — is available on most major PLC platforms and is specified on high-availability process lines. For a standard pharma SCADA/PLC project (solution preparation, environmental monitoring, CIP/SIP, HVAC), CPU redundancy is typically not specified, and this is a deliberate, documented decision rather than an oversight. The PLC redundancy article covers the risk justification for this decision in detail.
The key point for the HDS redundancy strategy table is that the decision not to implement CPU redundancy must be explicitly documented — with the consequence in the failure column (loss of control, safe state activation) and the risk acceptance rationale. A blank row or missing row for the PLC CPU is not acceptable. The table must show that the risk was considered and accepted, not that it was ignored.
Every redundancy failover event must generate an alarm. This is the most frequently missed design requirement in the redundancy strategy. When the system fails over from PSU-A to PSU-B, the operators and maintenance team must know — because they now have zero redundancy on power supply and a second failure will be catastrophic. A system that silently fails over, without alarming, is not providing real redundancy from an operational perspective. Every row in the HDS redundancy strategy table that has an automatic failover mechanism must have a corresponding alarm ID in the alarm configuration and a corresponding alarm management test in the OQ.
HDS-SYS-001 Section 1.2 contains the complete Redundancy Strategy table in the format shown above — component, redundancy provision, failover mechanism, failure consequence, and alarm generated. The OQ protocol includes redundancy failover tests for every automatic mechanism in Section 1.2: PROFINET ring cable pull test, PSU single-unit failure test, historian disconnection data buffer test, and UPS mains failure test. Each test references the HDS Section 1.2 row it validates. The alarm generated in each failover is cross-referenced to the alarm database in FDS Appendix B.