Utility Cybersecurity: Shut Back Doors to Critical Operational Systems

The best defense against any cybersecurity attack starts with the front line of an organization and extends to business partners. Employees, contractors, and third-party suppliers must be trained to be vigilant regarding system and application use, maintenance, and physical access. These individuals are the ones most likely to be targeted and are also the people who, when properly trained, will be first to notice an attempted intrusion. Training should be performed systematically and tailored to align with business roles to ensure best practices for cybersecurity are always top of mind. The objective for training should be to instill a culture of cybersecurity, one where individuals are vigilant about recognizing potential threats and equipped with processes and procedures to fend them off.

On October 18, 2018, the Federal Energy Regulatory Commission (FERC) approved the supply chain risk management reliability standards, CIP-013-1, submitted by the North American Electric Reliability Corporation (NERC) in response to the commission’s directives from Order No.829.⁴ The purpose of CIP-013-1 is to mitigate cybersecurity risks in a utility’s supply chain, including communications technology and ICS. Compliance with CIP-013-1 requires the development of one or more plans to address four objectives for high- and medium-impact Bulk Electric System (BES) cybersystems:⁵

1. Software integrity and authenticity.
2. Vendor remote access.
3. Information system planning.
4. Vendor risk management and procurement controls.

Before procuring any mission-critical system, including a communications technology, energy companies should closely examine their vendors’ security protocols, including how often measures are updated to counter evolving threats. Vendors must be held accountable to ensure that software, hardware, and other components have not been tampered with or maliciously infected before arrival onsite. Specific security requirements, expectations, and controls should be included in a Statement of Work (SOW), contracts, and Requests for Proposals (RFPs). Another alternative is to tie payments to the validation of implemented security controls and features. This linkage will motivate vendors to be vigorous in their compliance, tightening their own security and achieving higher standards through creative, enhanced solutions.

For hardware and software designed and manufactured overseas, vendors should be required to utilize tamper tapes to secure boxes and track all shipments end-to-end using a certified signature method. The goal is to create an audit trail and ensure the shipment never deviates from its safe route to its destination. Even with strict controls, it is a challenge for the average vendor to accomplish the objectives because of inadequate processes or controls. Unfortunately, non- compliant vendors may be the only option available. Nonetheless, utilities should push back and demand vendors do more to reduce the likelihood of breaches and exploitations.

Before granting system access to a vendor, complete a thorough screening and contract process. Ensure vendor employees have gone through background checks. Also, use only secure connections from the vendor’s network. The vendor must be able to adhere to the utility’s corporate security policy. A review of the vendor’s security policy and controls may be necessary to find out how well the vendor is going to be able to secure the utility’s data and the interconnections between both systems. Only vetted and authorized personnel should be allowed onto the utility’s network.

This vendor-focused activity leads to Vendor Risk Management and Supply Chain Risk Management efforts at a corporate level, which is now recommended by the National Institute of Standards and Technology (NIST) through its published guidance document, SP 800-161 “Supply Chain Risk Management Practices.”⁶

After a successful implementation of security awareness training and supply chain risk management, the next step is to test the mission-critical systems and applications in a controlled environment before deploying into the production network. Mission-critical communication systems should be set up and tested using various automated and manual tools to validate that security requirements, expectations, and controls are met. Scenarios such as misuse testing— acting like a user—are employed to provide some confidence that the application will behave correctly under stress-based conditions. These tests are sometimes performed by external organizations under the term, “red team.” Vulnerability testing, looking for common security weaknesses, penetration testing, and acting like a hacker should be considered at this phase. Any discovered vulnerabilities should be noted and communicated to the appropriate vendor. If the vendor cannot immediately resolve the issue, then request the vendor create a Plan of Action and Milestones (POA&M) item, including a secure workaround until the vulnerabilities are fixed. The ultimate objective is to introduce “clean” systems and applications to the production environment to establish a clean baseline.

Utilities should require vendors to provide architecture and networking design of a mission-critical communications system. It should include all hardware and software connections and state the necessary source and destination ports, including port ranges, and services and processes tied to each port required for business operations. Only the required logical network ports and services deemed necessary should be utilized. It is essential to identify approved ports and services to help network defenders manage network traffic through firewalls and intrusion-detection and prevention systems. The best practice is to logically disable/uninstall unnecessary ports and services on all devices within the production environment to mitigate unauthorized access. In addition, a packet-filtering firewall should be leveraged to look at destination and source addresses, ports, and services requested. At the network layer, only the approved whitelisted ports and services should be accepted. All unauthorized incoming and outgoing traffic should be disabled or blocked.

Every applicable major and minor release of security patches and firmware updates from third-party vendors should be tracked and evaluated expeditiously. Test security patches and firmware updates in a controlled environment prior to full production deployment. Confirm that each new device is fully patched before deploying to the production environment. Utilize an application and system scanning tool to validate that security patches and firmware updates are up-to-date.

There is an intricate balance with patch management—security verses availability. There are times when a security patch could impair the behavior of the system or cause downtime. A secured system that is not fully functioning or offline is useless. Meanwhile, an available system with security holes may be subject to various threats. Utilities and their suppliers should be able to assess risk versus reward as well as potential compensating measures. Not all vulnerabilities have related patches, so system administrators must not only be aware of applicable vulnerabilities and available patches, but also of other methods of remediation (e.g., device or network configuration changes, employee training) that limit the exposure of systems to vulnerabilities.

Malware attacks come in many insidious forms, from viruses, worms, and Trojan horses, to hybrids and exotic programs. There are various malware solutions on the market. Choosing the right solution for the appropriate environment can be an overwhelming task. The right solution should at least provide standard and embedded system and application protection and must be updated to receive and distribute the latest definitions. Additionally, the solution should have the option of either agent or agentless deployment. For embedded devices that do not support malware solutions, it is essential to have a layered approach—firewall and intrusion detection and prevention systems. These compensating controls will help cover and reduce potential exposure. Some mission- critical communication systems may not be able to run a malware solution due to an adverse impact on the system and application. In this case, the best option is to exclude the identified and approved directories and executables to maintain a host-based malware solution.

Configuration management should be instituted to reduce unauthorized changes and record implemented changes. Utilities should establish a Configuration Control Board (CCB), which is typically comprised of business unit and information technology managers. The CCB is the organizational group responsible for overseeing all configuration changes to active systems, including approving, disapproving, or deferring a request, managing costs, and minimizing downtime. When a change is presented to the CCB for approval, the system and application owners should be notified before authorization. This allows for review and evaluation of the proposed change to be conducted. After deployment, all parties involved in any update—including vendors, users, and application owners—should be notified to allow time to provide information and training to the operators and support staff affected by the change. Whenever unscheduled changes must be implemented, and time does not allow for a prescribed protocol to be followed, those changes should still be managed and controlled. A solid change-management process that includes proper vetting will help minimize changes that could have an adverse impact on the production environment.

A mission-critical communications system should not only go through change control, but a baseline profile should be established for each device and application. It is important to utilize an automated tool to have an established baseline structure. If any change deviates from the baseline without an approved change control, then the tool should flag such incidents and a specialized team should carefully investigate. If it’s a false positive, accept those changes as the new baseline. If not, remove the change and perform testing to ensure the system and application have not been adversely altered or compromised. Validate that both the system and application are in a secure state and working appropriately.

Access control begins and ends with an organization’s internal policy. The appropriate policy should support local domain or Active Directory authentication. The appropriate data and asset owners should identify approved users and determine access, permission, and restrictions for user roles assigned to a given asset. User access should be restricted based on roles and responsibilities. Role- based access helps prevent unauthorized access to critical and important applications and systems. Further, implementing strong password complexity settings, secure connection, and two-factor authentication will help safeguard the confidentiality and integrity of system and application access. An important aspect of system and application access that is often overlooked is the removal or adjustment of access rights and default credentials. When an employee has been transferred or terminated, or the status of a vendor has changed, the access to electronic systems, applications, and physical facilities should be reviewed, adjusted, and disabled/removed in a timely manner. Also, remove or change default usernames and passwords tied to systems and applications to eliminate the possibility of exploitation.

A Simple Network Management Protocol (SNMP) Manager and SNMP Agent should be installed in the appropriate environment to query, collect, and send system and application information. Whenever a notification trap is triggered (disk capacity, hardware failure, system offline, successful and unsuccessful authentications, password threshold, error messages, etc.), an alert should be sent to the appropriate groups and/or personnel. The alert notification allows system and application custodians to be proactive and help defend the physical and logical security boundaries.