Process Control Ethernet
Obsolescent Process Control Network Replacement & Migration
Our customer is in the nuclear supply chain, there is a considerable responsibility for them to maintain a level of production that will continue to service the industry. It is therefore critical that they limit vulnerabilities in their processes at the plant to ensure steady production within a protected environment. Like all manufacturing businesses with such a long, successful history, they need to replace and update elements of their infrastructure to keep their operations running smoothly.
The obsolete system provided connectivity with automatic failover in the case of communication path failure. The legacy failover process, whilst fast, was un-managed; the control room was not aware of either backup availability or the current status of which path was active. This visibility was one of many opportunities for potential network enhancement.
The plant’s fail-safe systems are such that there was no way that an interruption in communication would ever become dangerous. However, the resultant loss of production as the processes were restarted and brought back into operation, would be enormously expensive. Installing a new network would have to be done without affecting the manufacturing process.
Upgrading due to obsolescence has many challenges. It is often the case that the skills that implemented and originally supported a legacy system are no longer in the business or even accessible. This can result in systems being ‘wrapped in cotton wool’ with only essential maintenance being performed. The meaning of ‘obsolescence’ also varies; in some cases it is when a product is deemed End of Life by a manufacturer and in other cases it’s when spare can no longer be found on well-known Internet Auction sites.
An often-unforeseen consequence of living with a ‘generally reliable’ legacy network can be poor documentation and record keeping. When issues arise, the focus is often to implement a workaround as soon as possible to regain service; the tidy up exercise is often overlooked. After a couple of decades of such reactive maintenance and organic expansion makes any documentation untrusted and therefore unreliable.
Key challenges included:
- Refurbishment of a process-wide fibre infrastructure and related facilities.
- Integration of legacy equipment with modern network switching equipment supporting line speeds of up to 10Gbps.
- Creation of a resilient, high-speed, managed network backbone able to support multiple applications, legacy and new, without compromise.
- Providing an enabling platform for future extension, expansion and change.
- Implementation of controls that would limit the impact and alert on any accidental, or other, miss use and a Network Management system that would, among other thing, display them.
- Education of our customer's on-site engineering and support teams
With production outages measured in £M’s, the twenty + year old network supporting our customer's first Enrichment facility was probably the highest value asset IT4A have been asked to replace. Whilst technical challenges were presented and overcame along the journey, it was how the project team came together that ultimately brought the level of success to the project. To migrate a live Nuclear Process control system demands competence and strict controls, but ultimately it is the trust, earned across many months and many activities, that allows the first step of a live migration to be taken.
A critical part of the network upgrade was the preparation of the physical topology and environment within which the new network systems would exist. Whilst not originally considered within scope, significant efforts were made to ensure the environment was optimised and the Infrastructure validated. This is discussed in IT4A Case Study “Physical infrastructure, preparation for network migration”.
Live network migration
The approach to implement the new network in parallel to the existing paid significant dividends when planning the live migration. The first step of the migration was to connect the new network to the old, checking all servers and control elements were visible from the new network – they were. With ‘certified’ evidence that all backup paths were operational, the node-by-node migration plan involved the following additional steps:
Step 2: Fail the primary path across to the backup and confirm PLC/Server reachability
Step 3: Clean and re-certify the primary fibre path
Step 4: Re-interface the primary path to the new network and confirm reachability Note: the legacy, fully functional, transceivers performed the fibre path switching instantaneously and with no noticeable loss of data.
Step 5: Check the new node appears as expected on the NMS.
Network Monitoring and Technical Support
A fundamental component of a network security plan is the support plan that is implemented to ensure measures, implemented to control risks, are actually doing the job.
The migration went incredibly smoothly, with only minimal interruptions to data flow with which the system was well able to cope. Our migration approach was not time critical, mid-migration we were able to stop if operational needs demanded. The legacy network remained intact, available for regression, until full confidence in the new was gained.
From certification of all fibre paths to the implementation of a fully configured GPS linked network management system, IT4A’s approach to challenge every element of design was fully justified. Our customers now have the technically strong solution they need and, as important, the guidance, assurances, education and support that ensures the new network solution is supportable and maintainable for many years to come.
The new, IT4A engineered, managed network has brought the oldest part of the facility up to date, using modern standards and good practise. Significant improvements to the efficiency and reliability of operations at the company have been the result. Equally important has been the peace of mind that the infrastructure supporting production is a scalable, engineered solution that is understood, documented and above all trusted.