When organizations realize that their poor to non-existent SSH user key management has created a significant hole in their network security, denial is often their first response. But when it comes to security, denial is dangerous. SSH user keys are a means of authentication, similar in some ways to a password, and SSH and SSH user key-based access is pervasively used. 

These keys connect our processes for application-to-application data transfers, and they are the means by which our administrators and developers can conveniently access their environments without having to retype their passwords each time. SSH user keys have several unique properties that provide cause for concern:

1. SSH user keys aren’t necessarily correlated to an identity, which means we can’t easily assess which person or what process the key pair is associated with.
2. Any person who has an SSH client in your organization and access to a server can generate an SSH key pair.
3.SSH user keys don’t expire; they continue to provide access unless they are removed from the locations where they reside.

Those who are in denial will ask, “If SSH user key-based access is such a big problem, why don’t I hear more about breaches caused by their use?”

"​When organizations realize that their poor to non-existent SSH user key management has created a significant hole in their network security, denial is often their first response"

Denial Wreaks Havoc

This scenario is happening right now. We are seeing breaches in which the misuse of SSH has played a significant role in exfiltration of critical data. In the Snowden and Sony breaches, there is significant evidence that SSH user keys were extensively used to gain access to critical servers and data, move laterally to additional assets within the network and exfiltrate data. Poor to non-existent SSH key management and encrypted access control were large contributing factors to both of these headlines.

Another problem is that there is often little to no visibility into organizations’ encrypted SSH and SFTP traffic, thereby rendering our intrusion detection systems, data loss prevention systems, anti-virus and SIEM tools ineffective to identify threats in real time where SSH is potentially being used maliciously.

Resistance is Futile

After organizations overcome outright denial, they often experience resistance. This usually manifests as one of the following statements.

1. But I have so many other more important initiatives:

How can we continue to ignore hundreds of thousands to millions of access credential to our most critical infrastructure? What is more important than securing the crown jewels of our enterprises? What could be a higher priority than securing the primary back-down in terms of access to our most critical infrastructure? 

2. Please don’t do a scan of my environment to show me the size of the problem, because then I will have to fix it:

A scan of the environment is an eye-opening and humbling experience, but it is necessary. It will provide insight into risk around segregation of duty access controls from non-production to production environment. It will demonstrate gaps in how we manage third-party access to our environments. It will give us insight into access that has not been recertified in two or more years and demonstrate where we have key-based access with weak encryption. It will highlight how deeply a malicious actor could penetrate into our environment with a single stolen private key.

3. I already use Kerberos tickets or another privileged access management solution, so we don’t have a problem:

When it comes to managing interactive access and role-based access to infrastructure assets, Kerberos tickets and traditional PAM solutions are excellent. However, what these solutions do not cover well is the management of machine-to-machine and application-to-application-based access. More than 80 percent of the SSH user key-based access in our environments is application-to-application in nature.

4. We are moving to the cloud, so this won’t be a problem:

SSH will continue to play a critical role when it comes to access and the movement of data. Many large enterprises engage in what is known as a “lift and shift” when it comes to the first step of cloud migration. With SSH user key-based access, this means that we have simply migrated the access challenges we faced on-premises into our cloud.

Adapting for Survival

Adaptation is the next step toward network security – gaining visibility, control and ongoing governance of our SSH user key-based access. First, because in most cases SSH user key-based access has not been part of the overall provisioning processes of organizations’ identity management frameworks, we have to first solve the visibility challenge. Visibility consists of developing a mapping of private and public keys to their corresponding interactive or application-to-application connection. This can be achieved through scripted approaches, some of the privileged access management solutions and those dedicated to managing SSH user keys.

It can also be achieved via other, easily available tools that can be used to gain an understanding of how frequently keys are authenticating and from where. Known as verbose logging, it can be activated in the SSH daemon. This information is essential because it provides us the needed intelligence about what key-based access in our environment is valid and what access may no longer be needed. It also can help us identify if private keys are authenticating from IPs that our network does not recognize.

Through a process called “lockdown,” where authorized keys are relocated from users’ home directories to a central root-owned location, we can create an effect where now only users with root-level access are able to generate new SSH user keys going forward. But be careful here; if you lock down access without having a provisioning process in place, you may frustrate many administrators, developers and application owners in your organization.

Automation is the ultimate goal. On average, if a skilled user sets up an SSH user key-based trust and validates that the connection is working, it may take them five minutes. An unskilled SSH user, on the other hand, may take 30 minutes or more. If we do the math, we are generating thousands, tens of thousands or hundreds of thousands SSH user keys a year, and the costs really begin to add up. Automation can be achieved through scripting, solutions dedicated to SSH user key management and even orchestration tools. 

It’s normal to deny or resist something that we don’t want to be true. However, the discovery point is not the time to run from the problem but rather toward a solution. It involves a process of adaptation that enables us to gain visibility of SSH user key-based trusts and manage them consistently and well. Network security today demands no less.