One of our valued clients requested Sustainable ICT to undertake research into industry trends and technological advances with respect to industrial control networks and management services. The client was particularly interested in opportunities for team consolidation, service optimisation, and software defined networking within an industrial context.
This work would inform preparation for IT and OT convergence and identification of opportunities to optimise ICT asset expenditure. Whilst full details cannot be shared, the following is a subset of this research which has been approved for sharing for the benefit of others in the industry.
As part of a review of the current operation and elicitation of a future state architecture, it is important to explore industry direction and research findings. The following outlines current trends and insights with respect to industry leaders, researchers and technology direction which will influence the future state of a converged IT and OT asset operation.
IT and OT Convergence
In 2011 Gartner first espoused the opportunity for CIO’s to make optimal use of ICT assets and management practices across both Information Technology (IT) and Operational Technology (OT) functions [1]. Evaluation of convergence, alignment and integration opportunities between IT and OT was promoted as a means through which industrial enterprises could maximise the value of their investment in common services (business and technical) and exploit the value of information flows throughout the enterprise [2].
In recent years the pace at which large enterprises undertook such convergence has increased, with the likes of BHP Billiton, Rio Tinto, Roy Hill and Gold Fields Australia undertaking such effort through to 2017 [3]. Such endeavours sought to make best use of operational and information technology whilst establishing a platform through which to drive business innovation [4]. The strategy is not limited to well-known Australian companies however. The convergence of OT and IT has been evidenced globally as an imperative for creating a connected enterprise and delivering contextually meaningful information to workers regardless of location [5]. Looking to the future, through convergence of operations and technologies, research suggests enterprises will be more capable of exploiting opportunities envisioned in Industry 4.0 such as Internet-Of-Things (IoT) Sensor networks, Machine Learning (ML), and Cloud services into traditionally rigid industrial environments [6] [7].
In review of vendor and supplier information there are many internationally recognised industrial control vendors promoting the move to new ways of working and convergence. Rockwell Automation, for example, released its Connected Enterprise Execution Model, including associated maturity model, for manufacturing and industrial operators. The model aims to assist industry in understanding the technical and cultural changes necessary to take up the opportunities present in IoT, data analytics, scalable computing and mobility [8] [9] [10].
“We often find that less effective legacy processes are in place, and that those processes and work flows have not been designed to take advantage of the OT/IT convergence and the significant benefits available from the connected enterprise.”
Keith Nosbusch, CEO, Rockwell Automation
Such insights are quite poignant given the recent organisational restructuring and redundancies, and may go some way towards answering many of the questions that remain amongst the “old-guard”. Moreover, when viewed through the lens of the [the clients] Strategy of 2018 [6], the significance of successful convergence should not be understated. For the organisation to realise its cost reduction goals and take full advantage of innovations such as IoT, Automation and Smart Metering in the market (amongst many others), the successful convergence of teams, processes and technology should be considered essential.
It is widely acknowledged that the cultural and specialisation differences between IT and OT teams, and the lack of a central governing organisation for OT strategy, contributes to the formation of organisational silo’s [11]. Notwithstanding the need to up-skill IT resources/teams, there is a need for the converged organisation to understand that OT is the core business of the Oil and Gas sector [11] and therefore cost is not a primary driver as is evident in typical IT cost centres. Furthermore, given this fact, the mature organisation would manage the technology footprint as an asset on equal, or near equal, footing as a plant, however this is often not the case. Failure to service, replace and manage the electronic asset risks failure to the safe operation of the physical asset in today’s OT world.
Whilst the expected benefits associated with convergence are reinforced throughout available literature, there is less information espousing the opportunities with respect to operational management processes. However, global digital transformation research body Atos highlights that harmonised IT/OT strategies, governance models and business processes, enabled through a central team of multi-disciplinary team members, are essential components for successful IT/OT convergence [12].
Whilst industry advocates convergence to adequately position an organisation to take advantage of cost reduction opportunities, market innovation and information value [13], there is a need to consider how the underlying technologies are also evolving to become a foundation for such opportunities rather than a constraint.
Technology Convergence and Consolidation
Historically, industrial network designs have been comprised of a sizeable switching infrastructure footprint, rigidly defined network segments and security controls which reflect the control system function, physical location and an industry legacy. In such environments it has been the responsibility of network managers to translate organisational policy into low level network configuration commands and rules. As a result, adapting to change whilst managing the network can be challenging and error-prone [14]. Similarly, the design and deployment of server infrastructure for DCS or SCADA environments, until recent years, had been confined to the deployment of physical server hardware with redundant physical nodes to meet reliability and availability requirements. Overall, these aspects have impacted industrial operator’s ability to adapt to changing economic conditions and constrained the businesses ability to develop and/or maintain a competitive advantage [12].
It is important to acknowledge however, that the nature of the technology environment in industrial operations reflects the criticality of technology to production and the real threat to personal safety in the event of loss of control. These aspects have impacted the pace of change within operational technologies as compared with IT where the consequences of failure are less extreme. In recent years however, a number of technological improvements in IT networking and hosting have shown growing usage of traditionally IT solutions in the OT environment.
Virtualisation technology has become a mainstream approach to running compute workloads in data centres and is proven as a means of reducing maintenance, power and cooling of hardware for compute workloads. Industrial control vendors have also been progressively moving towards new generation systems which embrace both virtualisation and “software defined” technical capabilities. This has been driven through reduction in processor overheads and improvements in I/O performance [14]. Industrial control vendors have also recognised that virtualisation and software defined solutions will reduce the amount of production downtime and deliver reduced expenditure over the life of the asset [15].
Along with compute virtualisation, the virtualisation of the data storage layer for compute has matured in its scale and performance, and has equally gained recognition in the production control field [16] [17] [18]. There are three core types of storage connectivity; Direct Attached Storage (DAS), Network Attached Storage (NAS) and Storage Area Network (SAN). Whilst the objective here is not to elaborate on the difference between the technologies, it is important to highlight that SAN solutions carry block level data and have evolved to use Ethernet as the underlying transport for data through Fibre Channel over Ethernet (FCoE) and Internet Small Computer System Interface (iSCSI). These protocols, along with advancements in object based storage, have enabled the adoption of virtualisation technologies for storage solutions [18]. Virtual storage enables more effective addressing of storage across physical devices, which can span multiple locations, to significantly increase data security and integrity without impacting performance. Through the adoption of a storage cluster spanning two physical sites, in an active-active configuration, a single storage solution can provide automated fail-over, with zero data loss and near-instantaneous recovery [19] for a significant volume of virtualised hosts.
At the network layer, advances in virtualisation of network functions is driving considerable change across industries. The separation of network control from forwarding function through network virtualisation, called software defined networks (SDN), has gained significant interest as a means of simplifying network management and facilitating network evolution. The Open Networking Foundation, a non-for-profit organisation promoting the adoption of open standards based software defined networking, describes software defined networks as offering [20];
- Directly programmable due to its abstraction from underlying functions
- Agile through its ability to meet changing requirements
- Centrally managed through a global view of the network
- Programmatically configured to allow managers the ability to leverage automation
- Open standards based to simplify network design and operation
The history of programmable networks goes as far back as 1995 when the Open Signalling working group initiated the first investigation into separation of hardware and control software [14]. Today, the promise of SDN, particularly in support of industrial automation systems, is in its ability to enable more effective control system distribution, quality of service, resilience, compliance and to gain a more detailed view of the network and traffic under management [21] [22]. The separation of functions which control how one device interacts with its neighbour (control plane) from the functions which control how packets are parsed, forwarded and quality of service is met (forwarding plane) provides operators with improved capabilities for testing and resource management, whilst enabling isolation of traffic flows which introduce security and operational benefits [23].
SDN is not a silver bullet however, as it comes with some challenges; it changes the security model and tooling required of the management team [20], the SDN controller represents another source of failure [24], and there is a lack of familiarity with dynamic infrastructure across the workforce [25]. Of the challenges articulated in the available literature, the threat of security attacks against the centralised SDN controller is expressed as a primary risk to SDN within industrial control [21] [22]. This reality requires consideration of evolving network security controls.
Cybersecurity for Industrial control
Traditionally, control system vendors and OT networks have laggard IT when it comes to advancing cyber-security controls since such environments have typically been physically separated and relied on specialist hardware and software [26]. The commoditisation and reliability of standardised IP based network communications, combined with an increased ferocity and sophistication of business information requirements has driven change to industrial control system design and capabilities.
Multi-tiered security solutions through perimeter firewalls, access control lists (ACL), network zoning (to name a few) were typical controls for the mainstay of industrial control network designs. Today however, contemporary industrial control system designs are evolving to consider a more ubiquitous security approach enabling such capabilities as policy based control with dynamic response and reconfiguration [21]. Such capabilities are arising through the advent of software defined networking, and the potential for a significant reduction in business security risk in control systems is recognised throughout all available literature.
Where traditional networks are made up of a complex mesh of physical appliances, rules, and permissions which are manually implemented and maintained [14], industry research on Software Defined Networking is positing multi-level protection strategies and a reduction in physical appliances through centralisation, application flow whitelisting, inbuilt anomaly detection and the introduction of human-assisted automation solutions for security monitoring and supervision [21]. A noteworthy inclusion is application flow whitelisting, due to its similarity with Application Whitelisting referred to in ASD guidelines and IEC-62443. In the context of contemporary control network design, flow whitelisting represents the definition of allowable “streams” between end-points and/or networks. This presents several management and resource advantages over current designs which utilise numerous ACLs, deep packet inspection rules or host intrusion protection agents. Through the definition and monitoring of application flows, the SDN capability can also handle proprietary protocol payloads and does not require host level intervention [22].
Traditional network security design approaches such as Defence-in-Depth, promoted in the ASD information security principles [27], and industry security standards such as ISA-99 or its successor IEC-62443 are still very much relevant. However, industry research and global trends point to virtualisation across compute, storage and network domains as providing a better foundation for reducing the complexity and cost for delivering, maintaining and supporting such secure networks. From a security standpoint, when the old is married with the new, the ability to secure centralised controllers, espoused as a primary risk to SDN’s, becomes a possibility in which the benefits of SDN and convergence seem to outweigh the negatives.
Conclusion
The body of work surrounding IT/OT convergence, virtualisation and software defined networks suggest considerable advantages to organisations with the courage to undertake such an endeavour. Technological and cultural changes are required to seize the commercial opportunities present in IoT, data analytics and mobility whilst also delivering simplified management, automation and improved security control. It is clear that organisations seeking to converge IT and OT organisational functions without also addressing technological underpinnings are likely to realise minimal management and cost savings.
There is no denying that organisations engaged in such convergence will face considerable challenges with the transition. Strong organisational change management will be necessary to navigate the initial team changes and beyond, with team members struggling to accept change to team dynamics and adjusted boundaries of responsibility. Stalwarts of production control separation will continue to push back on such changes and challenge the maturity of IT solutions which in most cases can meet, or exceed, OT requirements. Organisations will require strong leadership to challenge the status quo, and relegate fear based opposition from traditionalists struggling to retain relevance in a new converged world.
Technologically, vendors and suppliers across most domains are embracing convergence through “software defined” capabilities and virtualisation to reduce management overheads and optimise for change. IT Networking and virtualisation companies recognise the opportunity in industrial control markets and have increased their focus on real-time performance, security and reliability. All research reviewed suggests that the primary challenges for organisations seeking convergence will be in cultural and personal domains, not technological ones. This however requires strong leadership commitment and a transformation structure with the authority to traverse individual opposition and deliver to business outcomes.
References
[1] Gartner Inc., “IT and Operational Technology: Convergence, Alignment and Integration,” Gartner inc, 2011.
[2] CIO.com, “CIOs key to bridging the IT/OT divide,” 1 June 2017. [Online]. Available: https://www.cio.com.au/article/620112/cios-key-bridging-it-ot-divide/.
[3] R. Crozier, “Roy Hill to back IT-OT integration,” ITNews, Mar 2017. [Online]. Available: https://www.itnews.com.au/news/roy-hill-to-back-it-ot-integration-456400. [Accessed Feb 2019].
[4] R. Crozier, “BHP Billiton to bring IT-OT to east coast mines,” ITNews, Jan 2017. [Online]. Available: https://www.itnews.com.au/news/bhp-billiton-to-bring-it-ot-to-east-coast-mines-447788. [Accessed Feb 2019].
[5] S. Wekare, “IT and OT Convergence: The case of implementing the connected enterprise,” University of Dublin, 2016.
[6] [client author], “IT Strategy,” [the client], 2018.
[7] B. Marr, “What is Industry 4.0? Here’s A Super Easy Explanation For Anyone,” Forbes.com, Sept 2018. [Online]. Available: https://www.forbes.com/sites/bernardmarr/2018/09/02/what-is-industry-4-0-heres-a-super-easy-explanation-for-anyone/#a1b26319788a. [Accessed Feb 2019].
[8] Rockwell Automation, “The Connected Enterprise – Bringing people, processes and technology together,” Rockwell Automation.
[9] Rockwell Automation, “The Connected Enterprise Maturity Model,” Rockwell Automation.
[10] Meet the boss TV, “Standardized and Connected – Manufacturing Success,” Meet the boss TV in association with Rockwell Automation.
[11] R. Kranendonk, “The Convergence and Integration of Operational Technology and Information Technology Systems,” Delft University of Technology, Delft, Netherlands.
[12] ATOS, “The convergence of IT and Operational Technology,” ATOS, 2012.
[13] OSIsoft, “Challenges, Opportunities and Strategies for Integrating OT and IT with the Modern PI System,” OSIsoft LLC, San Leandro, CA, 2015.
[14] B. A. A. Nunes, “A Survey of Software-Defined Networking: Past, Present, and Future of Programmable Networks,” IEEE Communications Society, 2014.
[15] H. Forbes, “Virtualization and Industrial Control,” ARC Advisory Group, Jan 2018. [Online]. Available: https://www.arcweb.com/blog/virtualization-industrial-control. [Accessed Feb 2019].
[16] J. Kempf, “Best Practices for Virtualization in Process Automation,” Process-worldwide.com, 2014. [Online]. Available: https://www.process-worldwide.com/best-practices-for-virtualization-in-process-automation-a-426557/. [Accessed Feb 2019].
[17] Wind, “Requirements for Virutalisation in Next-Generation Industrial Control Systems,” Mar 2018. [Online]. Available: https://storyscape.industryweek.com/wp-content/uploads/2018/03/requirements-virtualization-next-gen-industrial-cs-white-paper.pdf. [Accessed Feb 2019].
[18] M. Salmenperä, “Virtualisation Applicability to Industrial Automation,” Jan 2019. [Online]. Available: https://dspace.cc.tut.fi/dpub/bitstream/handle/123456789/23723/Lappalainen.pdf?sequence=1. [Accessed Feb 2019].
[19] M. Ištok, “VMWare vSAN: Your Modern Storage,” [Online]. Available: https://www.delltechnologies.com/content/dam/delltechnologies/images/forum/emea/presentations/sk-sk/itt-09.pdf. [Accessed Feb 2019].
[20] Centre for Cyber and Information Security, Norwegian University of Science and Technology, “Prospects of Software-Defined Networking in Industrial Operations,” International Journal on Advances in Security, pp. 101-110, 2016.
[21] H. Sandor, “Software Defined Response and Network Reconfiguration for Industrial Control Systems,” in Critical Infrastructure Protection XI: 11th IFIP AICT, 2017.
[22] A. Aydeger, “Software Defined Networking for Smart Grid Communications,” Florida International University, 2016.
[23] G. Kalman, “Prospects of Software-Defined Networking in Industrial Operations,” International Journal on Advances in Security, vol. 9, pp. 101-111, 2016.
[24] R. Pujar, “Path protection and Failover strategies in SDN networks,” in Open Networking Summit, Santa Clara, CA, 2016.
[25] D. Geer, “Five reasons IT pros are not ready for SDN investment,” TechTarget, 10 2014. [Online]. Available: https://searchnetworking.techtarget.com/feature/Five-reasons-IT-pros-are-not-ready-for-SDN-investment. [Accessed Feb 2019].
[26] R. R. R. Barbosa, “Flow Whitelisting in SCADA Networks,” Aalborg University, Denmark.
[27] Department of Defense, “Australian Government Information Security Manual – Principles,” Australian Government, 2016.
[28] Network Computing, “SDN: Time to Move On, Gartner Says,” 3 Oct 2017. [Online]. Available: https://www.networkcomputing.com/networking/sdn-time-move-gartner-says.
[29] R. Chua, “From SDN-Washing to SDN-Hiding,” 18 Nov 2018. [Online]. Available: https://www.sdxcentral.com/articles/analysis/from-sdn-washing-to-sdn-hiding/2018/11/.
[30] N. Regola, “Recommendations for Virtualization Technologies in High Performance Computing,” in 2nd IEEE International Conference on Cloud Computing Technology and Science, 2010.
[31] K. Steenstrup, “2016 Strategic Roadmap for IT/OT Alignment,” Gartner Inc, 2015.
[32] Honeywall International, “Virtualization support for process control systems,” Control Engineering, 7 Sept 2011. [Online]. Available: https://www.controleng.com/articles/virtualization-support-for-process-control-systems/. [Accessed Feb 2019].
[33] P. Hubbard, “The case against SDN implementation,” TechTarget, Aug 2015. [Online]. Available: https://searchnetworking.techtarget.com/opinion/The-case-against-SDN-implementation. [Accessed Feb 2019].
[34] K. Ahmed, “Software Defined Networks in Industrial Automation,” 2018.
[35] M. Sainz, “Software Defined Networking opportunities for intelligent security enhancement of Industrial Control Systems,” Jan 2018. [Online]. Available: https://www.researchgate.net/publication/319251133_Software_Defined_Networking_Opportunities_for_Intelligent_Security_Enhancement_of_Industrial_Control_Systems. [Accessed Feb 2019].