Setting Up SSH Certificates¶

This year, I completed SANS Institute and Counter Hack's Holiday Hack Challenge. One of the speakers, Thomas Bouve, provided an excellent talk about SSH Certificates. Below are my step-by-step notes that I documented as a reference. I strongly encourage anyone interested in the topic to listen to the presentation as Thomas provides more foundational knowledge in the beginning of the video and provides more context and explanations than what I am providing here. But if you're looking for a quick copy/paste/edit of commands to get the job done, this might be a helpful reference.

secure

Notes & Assumptions

In the example, 10.10.10.10 is the address of the server where we want to be able to SSH with a signed certificate.
The username, jesinia will be used to SSH into the server.
It is assumed that the server is rootless and that jesinia has sudo permissions on the server. If you have root access, then feel free to ignore the "sudo" references.
Commands for restarting ssh service are for Fedora or RedHat distributions. If you use a different flavor of Linux, your commands may be different for that portion.
I use vim, but you can use whichever text editor you prefer.
Notes and level of detail is purposely for a broader audience with less experience.

1. Log into Server Using Password¶

In Server Terminal:¶

ssh jesinia@10.10.10.10

What is happening here?

This command is simply logging in and if there is a password required, you will be prompted to enter it.

2. Create a CA Key Pair on Client Side¶

In Client Terminal:¶

cd ~/.ssh && ssh-keygen -C 'SSH certificate authority' -f ca

Use a Passphrase! And protect these keys very well.

What is happening here?

This command moves you into the home directory where a user's SSH keys are stored and then generates a Certificate Authority key pair that will be used to sign users' and hosts' public keys. Signed certificates provide a layer of protection since it means that merely having public key on a host to which you want to SSH; the host will want to see that your public key is recognized by their certificate authority. Additionally, you can remove the option to allow passwords, making this the only method for SSH authentication into the server.

3. Copy Host Keys from Server to Client¶

In Client Terminal:¶

scp jesinia@10.10.10.10:/etc/ssh/ssh_host_\*.pub ~/.ssh/host_keys && ls -la ~/.ssh/host_keys

Note: Using && rather than || or ; because the execution of the subsequent commands depends on successful execution of the previous commands.

If you experience any issues, you may want to run the commands serially rather than as a one-liner:

scp jesinia@10.10.10.10:/etc/ssh/ssh_host_\*.pub ~/.ssh/host_keys

ls -la ~/.ssh/host_keys

What is happening here?

This command authenticates into the server using jesinia's account and pulls back specific server files so that we have them on the client. Specifically, we are pulling back are the SSH host public keys and we are putting them into a directory within our client's .ssh directory called host_keys. Then we're listing the contents of the host_keys directory to confirm.

4. Use the CA to Sign Each Host Key¶

In Client Terminal:¶

ssh-keygen -h -s ca -z 1 -I sshserver -V -5m:+1w -n 10.10.10.10 host_keys/ssh_host_ecdsa_key.pub && ssh-keygen -h -s ca -z 2 -I sshserver -V -5m:+1w -n 10.10.10.10 host_keys/ssh_host_ed25519_key.pub && ssh-keygen -h -s ca -z 3 -I sshserver -V -5m:+1w -n 10.10.10.10 host_keys/ssh_host_rsa_key.pub

You will be prompted for the CA's passphrase to sign for each certificate.

To view any of the certificates:

ssh-keygen -L -f ~/.ssh/host_keys/ssh_host_ed25519_key-cert.pub

What is happening here?

In the first script block, we are sending three commands, one for each of the server's host public keys. The command uses the certificate authority key we created to sign the host public key and provides a validity time period of 5 minutes ago to 1 week from now.

5. Transfer Signed Certificates and Public CA Key from Client to Server¶

In Client Terminal:¶

scp host_keys/* ca.pub jesinia@10.10.10.10:

What is happening here?

Now that we have signed certificates, we need to send them to the server along with the Certificate Authority's public key.

6. Move the Certs and Key to Proper Folder & Adjust Ownership¶

In Server Terminal:¶

sudo mv ssh_host_*cert.pub ca.pub /etc/ssh && sudo chown root:root /etc/ssh/*cert.pub /etc/ssh/ca.pub

Confirm the change:

ls -la /etc/ssh

What is happening here?

In the first script block for this step, we're moving the certificates and the Certificate Authority's public key to the correct location. We did not specify this location in the last command simply because it needs to go into a privileged directory. Then we are changing the owner of the files to all reflect an owner and group of root.

7. Adjust Config File¶

In Server Terminal:¶

sudo vim /etc/ssh/sshd_config.d/sshcerts.conf

In vim (or nano) editor:

HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub
TrustedUserCAKeys /etc/ssh/ca.pub
PasswordAuthentication no
AuthorizedPrincipalsFile /etc/ssh/auth_principals/%u

Note: The last line is only necessary if you are creating an auth_principals folder so that a user who has different authorized principals can use them.

What is happening here?

Creating this configuration file is what actually implements everything we have set up thus far. It tells the system to use the private host keys and their signed certificates and references the ca.pub to verify the certification. It also specifies that users should not be able to authenticate using passwords, thus requiring a signed certificate.

8. Create Auth Principals if Necessary¶

In Server Terminal:¶

sudo mkdir /etc/ssh/auth_principals

9. Restart SSH Service¶

In Server Terminal:¶

sudo systemctl restart sshd

What is happening here?

After saving the configuration file on the server, we now need to restart teh sshd service so that the system will recognize the changes we just made.

10. Add Public CA Key to Client's known_hosts File¶

In Client Terminal:¶

cat ~/.ssh/ca.pub

Copy the output.

vim ~/.ssh/known_hosts

Delete other entries so that the following is the only entry

In vim editor:

@cert-authority 10.10.10.10 <paste output>

What is happening here?

In this step, we are essentially putting the contents of the CA public key into the known_hosts directory and deleting any other references to that IP's keys in the directory. This helps the server recognize that the client is a known entity to the CA.

11. Create SSH Key Pair for User on Client and Sign with CA¶

In Client Terminal:¶

ssh-keygen -C 'jesinia' -f jesinia_ssh_key

Passphrases

If you create a passphrase, you will be prompted for it each time you use the certificate to authenticate. For security, this should be different than your password.

ssh-keygen -s ca -z 4 -I 'jesinia' -V -5m:+52w -n jesinia ~/.ssh/jesinia_ssh_key.pub

What is happening here?

Now that we have the structure all set up, we need to get our user a key pair and get public key signed by the certificate authority so that she can log in with a signed certificate. The first script block is creating the keys, the second is signing the public key.

12. SSH into Server Using the Cert Rather than Password¶

In Client Terminal:¶

ssh jesinia@10.10.10.10 -i jesinia_ssh_key

What is happening here?

This last command uses jesinia's private key on the client machine to ssh into the server. There is no prompt for a password, but if jesinia's key was generated with a passphrase, the user would be prompted to enter the passphrase.

Post-Post¶

After writing this post, I endedup creating a script to automate the process so that it would be just a bit faster. While this process isn't one I do often, I am learning to write scripts and this provided an opportunity to generate one. And yes, I'm aware that storing a password into a script is a bad idea; it is intended to be used for the automated process and then immediately changed.

Automated¶

#!/bin/bash

# Prompt for server login details and passphrase
read -p "Enter the username: " username
read -p "Enter the server IP address: " server_ip
read -p "Enter SSH Port: " SSH_PORT
read -s -p "Enter the SSH password for the user on the server: " ssh_password
echo ""
read -s -p "Enter your desired passphrase for the CA and User's key generation (leave empty for no passphrase): " key_passphrase
echo ""

# Function to execute expect scripts
run_expect() {
  expect -c "$1"
}

# Check and generate User's SSH key pair on the client side if not exists
if [ ! -f ~/.ssh/${username}_ssh_key ]; then
  ssh-keygen -t rsa -b 4096 -C "${username}" -f ~/.ssh/${username}_ssh_key -N "$key_passphrase"
else
  echo "User's SSH key already exists, skipping generation."
fi

# Check and create a CA Key Pair on Client Side with passphrase if not exists
if [ ! -f ~/.ssh/ca ]; then
  ssh-keygen -C 'SSH certificate authority' -f ~/.ssh/ca -N "$key_passphrase"
else
  echo "CA Key Pair already exists, skipping generation."
fi

# Copy Host Keys from Server to Client
scp -P ${SSH_PORT} ${username}@${server_ip}:/etc/ssh/ssh_host_*.pub ~/.ssh/host_keys

# Initialize serial number for signed keys
serial_number=1

# Use the CA to Sign Each Host Key
for key in ~/.ssh/host_keys/*; do
  # Check if the certificate for this key already exists
  if [ ! -f "${key}-cert.pub" ]; then
    # Handle passphrase for signing
    SIGN_KEY="
    spawn ssh-keygen -h -s ~/.ssh/ca -I key_signing -n ${server_ip} -V +52w -z $serial_number ${key}
    expect \"Enter passphrase:\"
    send \"${key_passphrase}\r\"
    expect eof
    "
    run_expect "$SIGN_KEY"
    ((serial_number++))
  else
    echo "Certificate for ${key} already exists, skipping."
  fi
done

# Transfer Signed Certificates and Public CA Key from Client to Server
scp -P ${SSH_PORT} ~/.ssh/host_keys/*cert.pub ~/.ssh/ca.pub ${username}@${server_ip}:

# Sign User's public key with the CA if not already signed
if [ ! -f ~/.ssh/${username}_ssh_key-cert.pub ]; then
  ssh-keygen -s ~/.ssh/ca -I user_${username} -n ${username} -V +52w -z $serial_number ~/.ssh/${username}_ssh_key.pub -N "$key_passphrase"
else
  echo "User's public key already signed, skipping."
fi

# Execute adjustments on the server, including moving files and SSHD config modification
ADJUST_AND_MOVE="
mv /home/${username}/ssh_host_*cert.pub /home/${username}/ca.pub /etc/ssh/ && chown root:root /etc/ssh/*cert.pub /etc/ssh/ca.pub
cat <<'EOF' | sudo tee /etc/ssh/sshd_config.d/sshcerts.conf > /dev/null
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_rsa_key
HostCertificate /etc/ssh/ssh_host_ecdsa_key-cert.pub
HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub
HostCertificate /etc/ssh/ssh_host_rsa_key-cert.pub
TrustedUserCAKeys /etc/ssh/ca.pub
PasswordAuthentication no
EOF
sudo systemctl restart sshd
"

run_expect "spawn ssh -p ${SSH_PORT} ${username}@${server_ip}
expect \"password:\"
send \"${ssh_password}\r\"
expect \"$\"
send \"sudo -s\r\"
expect \"password for ${username}:\"
send \"${ssh_password}\r\"
expect \"#\"
send \"$ADJUST_AND_MOVE\r\"
expect \"#\"
send \"exit\r\"
expect \"$\"
send \"exit\r\"
"

# Add the CA public key to known_hosts with the specified server IP
echo "@cert-authority ${server_ip} $(cat ~/.ssh/ca.pub)" >> ~/.ssh/known_hosts


echo "Setup is complete. To securely log into the server, use the following command:"
echo "ssh -i ~/.ssh/${username}_ssh_key -p ${SSH_PORT} ${username}@${server_ip}"
echo "Once logged in, change your password immediately by executing: passwd"