Jump to contentJump to page navigation: previous page [access key p]/next page [access key n]
documentation.suse.com / SUSE Linux Enterprise Server Documentation / Security and Hardening Guide / Network security / Securing network operations with OpenSSH
Applies to SUSE Linux Enterprise Server 15 SP6

22 Securing network operations with OpenSSH

OpenSSH is the SSH (secure shell) implementation that ships with SUSE Linux Enterprise Server, for securing network operations such as remote administration, file transfers, and tunneling insecure protocols. SSH encrypts all traffic between two hosts, including authentication, to protect against eavesdropping and connection hijacking. This chapter covers basic operations, plus host key rotation and certificate authentication, which are useful for managing larger SSH deployments.

22.1 OpenSSH overview

SSH is a network protocol that provides end-to-end protection for communications between the computers on your network, or between computers on your network and systems outside your network. You may open an SSH session to any other computer, if you have a login and the correct authentication methods for the remote computer.

SSH is a client-server protocol. Any host running the sshd daemon can accept SSH connections from any other host. Every host running sshd can have their own custom configurations, such as limiting who can have access, and which authentication methods are allowed.

Authentication and encryption are provided by encryption key pairs. Each key pair has a public key and a private key. Public keys encrypt, and private keys decrypt. Public keys are meant to be freely shared, while private keys must be protected and not shared. When a private key is compromised, anyone who has possession of it can masquerade as the original key owner.

SSH provides strong protection, because the server and client must both authenticate to each other. When a client first attempts to open an SSH session, the server presents its public host key. If the client already possesses a copy of this key (stored in ~/.ssh/known_hosts on the client machine), the client knows to trust the server. If the client does not have the appropriate host key, it is asked whether it should trust the server:

The authenticity of host '192.168.22.219 (192.168.22.219)'
   can't be established. ECDSA key fingerprint is
   SHA256:yXf6pjV26N0fegvEYIt3HgG95s3Q1X6WYRhtHLF99pUo.
   Are you sure you want to continue connecting (yes/no/[fingerprint])?

The user can type yes or no, or paste their copy of the host key fingerprint for comparison.

Note
Note: Matching host key fingerprints

Distributing copies of your host key fingerprints to your users enables them to verify that they are receiving the correct host keys. When they paste their copy of a host key fingerprint, ssh compares the fingerprints, and accepts the offered host key when the fingerprints match. This ensures a more accurate match than a visual comparison.

You cannot rely on users to use correct verification methods. If the fingerprints do not match, the user can still type yes, or copy the fingerprint in the message, and complete the connection. A stronger alternative is to use certificate authentication, which provides a global authentication mechanism, and does not require perfect behavior from users (see Section 22.8, “OpenSSH certificate authentication”).

If the public keys of a host have changed, the connection is denied with an alarming warning:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the ECDSA key sent by the remote host is
SHA256:keNu/rJFWmpQu9B0SjIuo8NLjbeDY/x3Tktpl7oDJqo.
Please contact your system administrator.
Add correct host key in /home/geeko/.ssh/known_hosts to get rid of this message.
Offending ECDSA key in /home/geeko/.ssh/known_hosts:210
You can use following command to remove the offending key:
ssh-keygen -R 192.168.121.219 -f /home/geeko/.ssh/known_hosts
ECDSA host key for 192.168.121.219 has changed and you have requested strict
checking.
Host key verification failed.

The remedy is to delete the offending key from ~/.ssh/known_hosts with the command given in the warning, then reconnect and accept the new host key.

The openssh package installs the server, client, file transfer commands, and certain utilities.

OpenSSH supports several different types of authentication:

Password authentication

Uses any system login and password on the remote machine. This is the simplest and most flexible authentication because you can open an SSH session from anywhere, on any machine. It is also the least secure, because it is vulnerable to password-cracking and keystroke logging.

Public key authentication

Authenticates with your personal SSH keys, rather than a login and password. This is less flexible than password authentication, because you can open SSH sessions only from a machine that holds your private identity key. It is much stronger because it is not vulnerable to password cracking or keystroke logging; an attacker must possess your private key and know its passphrase.

See Section 22.9, “Automated public key logins with gnome-keyring” to learn how to use gnome-keyring for automated public key authentication in GNOME sessions.

See Section 22.10, “Automated public key logins with ssh-agent” to learn how to use ssh-agent for automated public key authentication in console sessions.

Passphrase-less public key authentication

Public key authentication, paired with private identity keys that do not have passphrases. This is useful for automated services, like scripts and cron jobs. You must protect private keys, because anyone who gains access to them can easily masquerade as the key owner.

Certificate authentication

OpenSSH supports certification authentication, for easier key management, stronger authentication, and large-scale SSH deployments.

SUSE Linux Enterprise Server installs the OpenSSH package by default, providing the following commands:

ssh

The client command for initiating an SSH connection to a remote host.

scp

Secure file copy from or to a remote host.

sftp

Secure file transfer between a client and an SFTP server. (The SFTP protocol (SSH FTP) is not related to FTPS or FTPES (FTP over SSL/TLS), but was written independently.)

ssh-add

Add private key identities to the authentication agent, ssh-agent.

ssh-agent

Manages a user's private identity keys and their passphrases, for public key authentication. ssh-agent holds the passphrases in memory and applies them as needed, so that users do not have to re-type their passphrases to authenticate.

ssh-copy-id

Securely transfer a public key to a remote host, to set up public key authentication.

22.2 Server hardening

OpenSSH ships with a usable default server configuration, but there are additional steps you can take to secure your server.

Important
Important: Maintaining access to a remote SSH server

When you make changes to any SSH server, either have physical access to the machine, or keep an active root SSH session open until you have tested your changes, and everything works correctly. Then you can revert or correct your changes if something goes wrong.

The default server configuration file, /etc/ssh/sshd_config, contains the default configuration, and all the defaults are commented out. Override any default item by entering your own configuration item, uncommented, like the following example that sets a different listening port, and specifies the listening IPv4 address on a multi-homed host:

#Port 22
Port 2022

#ListenAddress 0.0.0.0
ListenAddress 192.168.10.100
Important
Important: Update /etc/services

When you use non-standard listening ports, first check the /etc/services file for unused ports. Select any unused port above 1024. Then document the ports you are using in /etc/services.

It is a best practice to disallow root logins. Instead, log into the remote machine as an unprivileged user, then use sudo to run commands as root. If you really want to allow root logins, the following server configuration example shows how to configure the server to accept only public-key authentication (Section 22.6, “Public key authentication”) for the root user with the PermitRootLogin prohibit-password and PasswordAuthentication options.

The following settings for /etc/ssh/sshd_config strengthen access controls:

Example 22.1: Example sshd_config
# Check if the file modes and ownership of the user’s files and
# home directory are correct before allowing them to login
StrictModes yes

# If your machine has more than one IP address, define which address or
# addresses it listens on
ListenAddress 192.168.10.100

# Allow only members of the listed groups to log in
AllowGroups ldapadmins backupadmins

# Or, deny certain groups. If you use both, DenyGroups is read first
DenyGroups users

# Allow or deny certain users. If you use both, DenyUsers is read first
AllowUsers user1 user2@example.com user3
DenyUsers user4 user5@192.168.10.10

# Allow root logins only with public key authentication
PermitRootLogin prohibit-password

# Disable password authentication and allow only public key authentication
# for all users
PasswordAuthentication no

# Length of time the server waits for a user to log in and complete the
# connection. The default is 120 seconds:
LoginGraceTime 60

# Limit the number of failed connection attempts. The default is 6
MaxAuthTries 4

After changing /etc/ssh/sshd_config, run the syntax checker:

> sudo sshd -t

The syntax checker only checks for correct syntax, and does not find configuration errors. When you are finished, reload the configuration:

> sudo systemctl reload sshd.service

Check the server's key directories for correct permissions.

/etc/ssh should be mode 0755/drwxr-xr-x, owned by root:root.

Private keys should be 0600/-rw-------, owned by root:root.

Public keys should be 0644/-rw-r--r--, owned by root:root.

22.3 Password authentication

With password authentication, all you need is the login and password of a user on the remote machine, and sshd set up and running on the remote machine. You do not need any personal SSH keys. In the following example, user suzanne opens an SSH session to the host sun:

> ssh suzanne@sun

suzanne is prompted to enter the remote password. Type exit and press Enter to close an SSH session.

If the user name is the same on both machines, you can omit it, and then using ssh HOST_NAME is sufficient. After a successful authentication, you can work on the command line or use interactive applications, such as YaST in text mode.

You may also run non-interactive commands (log in, run the command, then the session closes all in one command) on remote systems using the ssh USER_NAME HOST COMMAND syntax. COMMAND must be properly quoted. Multiple commands can be concatenated as on a local shell:

> ssh suzanne@sun "df -h && du -sh  /home"
> ssh suzanne@sun "sudo nano /etc/ssh/sshd_config"

When you run sudo on the remote machine, you are prompted for the sudo password.

22.4 Managing user and host encryption keys

There are several key types to choose from: DSA, RSA, ECDSA, ECDSA-SK, Ed25519, and Ed25519-SK. DSA was deprecated several years ago, and was disabled in OpenSSH 7.0 and should not be used. RSA is the most universal as it is older, and more widely used. (As of OpenSSH 8.2, RSA is deprecated for host keys. Use ECDSA or Ed25519 for host keys.)

Ed25519 and ECDSA are stronger and faster. Ed25519 is considered to be the strongest. If you must support older clients that do not support Ed25519 or ECDSA, create host keys in all three formats.

Note
Note: Older clients are unsafe

Some older SSH clients do not support ECDSA and ED25519. ECDSA and ED25519 were released with OpenSSH 6.5 in 2014. It is important to keep security services updated, and, if possible, to not allow unsafe old clients.

SSH keys serve two purposes: authenticating servers to clients, and authenticating clients to servers (see Section 22.6, “Public key authentication”). Server host keys are stored in /etc/ssh. Users' personal keys are stored in /home/user/.ssh.

/home/user/.ssh is created when the user creates a new SSH key.

Host keys must not have passphrases.

In most cases, private user keys should have strong passphrases.

22.4.1 Creating user SSH key pairs

The following procedure shows how to create user OpenSSH encryption keys.

Procedure 22.1: Creating default and customized keys
  1. To generate a user key pair with the default parameters (RSA, 3072 bits), use the ssh-keygen command with no options. Protect your private key with a strong passphrase:

    > ssh-keygen
    Generating public/private rsa key pair.
    Enter file in which to save the key (/home/tux/.ssh/id_rsa):
    Enter passphrase (empty for no passphrase):
    Enter same passphrase again:
    Your identification has been saved in id_rsa
    Your public key has been saved in id_rsa.pub
    The key fingerprint is:
    SHA256:z0uJIuc7Doy07bFTe1ppZHLVrkD/bWWlBAF/PcHjblU user@host2
    The key's randomart image is:
    +---[RSA 3072]----+
    |          ..o... |
    |           o . +E|
    |        . . o +.=|
    |       . o . o o+|
    |  .   . S . . o +|
    | . =  .= * + . = |
    |  o *.o.= * . +  |
    |   ..Bo+.. . .   |
    |    oo==  .      |
    +----[SHA256]-----+
  2. Create an RSA key pair with a longer bit length:

    > ssh-keygen -b 4096

    OpenSSH RSA keys can be a maximum of 16,384 bits. However, longer bit lengths are not necessarily more desirable. See the GnuPG FAQ for more information, https://www.gnupg.org/faq/gnupg-faq.html#no_default_of_rsa4096.

  3. You may create as many user keys as you want, for accessing different servers. Each key pair must have a unique name, and optionally, a comment. These help you remember what each key pair is for. Create an RSA key pair with a custom name and a comment:

    > ssh-keygen -f backup-server-key -C "infrastructure backup server"
  4. Create an Ed25519 key pair with a custom name and a comment:

    > ssh-keygen -t ed25519 -f ldap-server-key -C "Internal LDAP server"

    Ed25519 keys are fixed at 256 bits, which is equivalent in cryptographic strength to RSA 4096.

22.4.2 Creating SSH server host keys

Host keys are managed a little differently. A host key must not have a passphrase, and the key pairs are stored in /etc/ssh. OpenSSH automatically generates a set of host keys when it is installed, like the following example:

> ls -l /etc/ssh
total 608
-rw------- 1 root root 577834 2021-05-06 04:48 moduli
-rw-r--r-- 1 root root   2403 2021-05-06 04:48 ssh_config
-rw-r----- 1 root root   3420 2021-05-06 04:48 sshd_config
-rw------- 1 root root   1381 2022-02-10 06:55 ssh_host_dsa_key
-rw-r--r-- 1 root root    604 2022-02-10 06:55 ssh_host_dsa_key.pub
-rw------- 1 root root    505 2022-02-10 06:55 ssh_host_ecdsa_key
-rw-r--r-- 1 root root    176 2022-02-10 06:55 ssh_host_ecdsa_key.pub
-rw------- 1 root root    411 2022-02-10 06:55 ssh_host_ed25519_key
-rw-r--r-- 1 root root     96 2022-02-10 06:55 ssh_host_ed25519_key.pub
-rw------- 1 root root   2602 2022-02-10 06:55 ssh_host_rsa_key
-rw-r--r-- 1 root root    568 2022-02-10 06:55 ssh_host_rsa_key.pub

ssh-keygen has a special option, -A, for creating new host keys. This creates new keys for each of the key types for which host keys do not exist, with the default key file path, an empty passphrase, default bit size for the key type, and an empty comment. The following example creates a complete new set of host keys by first deleting the existing keys, then creating a new set:

> sudo rm /etc/ssh/ssh_host*
> sudo ssh-keygen -A

You can replace selected key pairs by first deleting only the keys you want to replace, because ssh-keygen -A does not replace existing keys.

Important
Important: Do not use DSA keys

ssh-keygen -A creates DSA keys, even though they have been deprecated as unsafe for several years. In OpenSSH 7.0 they are still created, but disabled by not being listed in sshd_config. You may safely delete DSA keys.

When you want to rotate host keys (see Section 22.5, “Rotating host keys”), you must create the new keys individually, because they must exist at the same time as your old host keys. Your users authenticate with the old keys, and then receive the list of new keys. They need unique names, to not conflict with the old keys. The following example creates new RSA and Ed25519 host keys, labeled with the year and month they were created. Remember, the new host keys must not have passphrases:

> cd /etc/ssh
> sudo ssh-keygen -b 4096 -f "SSH_HOST_RSA_2022_02"
> sudo ssh-keygen -t ed25519 -f "SSH_HOST_ED25519_2022_02"

You may name your new keys whatever you want.

22.5 Rotating host keys

As of version 6.8, OpenSSH includes a protocol extension that supports host key rotation. SSH server admins must periodically retire old host keys and create new keys, for example if a key has been compromised, or it is time to upgrade to stronger keys. Before OpenSSH 6.8, if StrictHostKeyChecking was set to yes in ssh_config on user machines, users would see a warning that the host key had changed, and were not allowed to connect. Then the users would have to manually delete the server's public key from their known_hosts file, reconnect and manually accept the new key. Any automated SSH connections, such as scheduled backups, would fail.

The new host key rotation scheme provides a method to distribute new keys without service interruptions. When clients connect, the server sends them a list of new keys. Then the next time they log in they are asked if they wish to accept the changes. Give users a few days to connect and receive the new keys, and then you can remove the old keys. The users' known_hosts files are automatically updated, with new keys added and the old keys removed.

Setting up host key rotations requires creating new keys on the server, certain changes to /etc/ssh/sshd_config on the server, and to /etc/ssh/ssh_config on the clients.

First, create your new key or keys. The following example creates a new RSA key and a new Ed25519 key, with unique names and comments. A useful convention is to name them with the creation date. Remember, a host key must not have a passphrase:

# ssh-keygen -t rsa -f ssh_host_rsa_2022-01 -C "main server"
Generating public/private rsa key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ssh_host_rsa_2022-01
Your public key has been saved in ssh_host_rsa_2022-01.pub
The key fingerprint is:
SHA256:F1FIF2aqOz7D3mGdsjzHpH/kjUWZehBN3uG7FM4taAQ main server
The key's randomart image is:
+---[RSA 3072]----+
|         .Eo*.oo |
|          .B .o.o|
|          o . .++|
|         . o ooo=|
|        S . o +*.|
|         o o.oooo|
|       .o ++oo.= |
|       .+=o+o + .|
|       .oo++..   |
+----[SHA256]-----+

# ssh-keygen -t ed25519 -f ssh_host_ed25519_2022-01 -C "main server"
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ssh_host_ed25519_2022-01
Your public key has been saved in ssh_host_ed25519_2022-01.pub
The key fingerprint is:
SHA256:2p9K0giXv7WsRnLjwjs4hJ8EFcoX1FWR4nQz6fxnjxg main server
The key's randomart image is:
+--[ED25519 256]--+
|   .+o ...o+     |
| . .... o *      |
|  o..  o = o     |
|  ..   .. o      |
|   o. o S  .     |
|  . oo.*+   E o  |
|   + ++==..  = o |
|    = +oo= o. . .|
|     ..=+o=      |
+----[SHA256]-----+

Record the fingerprints, for users to verify the new keys.

Add the new key names to /etc/ssh/sshd_config, and uncomment any existing keys that are in use:

## Old keys
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
HostKey /etc/ssh/ssh_host_ecdsa_key

## New replacement keys
HostKey /etc/ssh/ssh_host_rsa_2022-01
HostKey /etc/ssh/ssh_host_ed25519_2022-01

Save your changes, then restart sshd:

# systemctl restart sshd.service

The /etc/ssh/ssh_config file on user machines must include the following settings:

UpdateHostKeys ask
StrictHostKeyChecking yes

Test connecting from a client by opening an SSH session to the server to receive the new keys list. Log out, then log back in. When you log back in you should see something like the following message:

The server has updated its host keys.
These changes were verified by the server's existing trusted key.
Deprecating obsolete hostkey: ED25519
SHA256:V28d3VpHgjsCoV04RBCZpLo5c0kEslCZDVdIUnCvqPI
Deprecating obsolete hostkey:
RSA SHA256:+NR4DVdbsUNsqJPIhISzx+eqD4x/awCCwijZ4a9eP8I
Accept updated hostkeys? (yes/no):yes

You may set UpdateHostKeys ask to UpdateHostKeys yes to apply the changes automatically, and avoid asking users to approve the changes.

For more information:

22.6 Public key authentication

Public key authentication uses your own personal identity key to authenticate, rather than a user account password.

The following example shows how to create a new personal RSA key pair with a comment, so you know what it is for. First change to your ~/.ssh directory (or create it if it does not exist), then create the new key pair. Give it a strong passphrase, and write the passphrase in a safe place:

> cd ~/.ssh
> ssh-keygen -C "web server1" -f id-web1 -t rsa -b 4096

Next, copy your new public key to the machine you want access to. You must already have a user account on this machine, and SSH access to copy it over the network:

> ssh-copy-id -i id-web1 user@web1

Then try logging in with your new key:

> ssh -i id-web1 user@web1
Enter passphrase for key 'id-web1':
Last login: Sat Jul 11 11:09:53 2022 from 192.168.10.122
Have a lot of fun...

You should be asked for your private key passphrase, and not the password for your user account.

To be effective, public key authentication should be enforced on the remote machine, and password authentication not allowed (see Example 22.1, “Example sshd_config”). If you do not have public key authentication access on the remote machine already, you cannot copy your new public key with ssh-copy-id, and must use other means, such as manually copying it from a USB stick to the ~/.ssh/authorized_keys file of the remote user account.

22.7 Passphrase-less public key authentication

This is public key authentication without a passphrase. Create your new private identity keys without a passphrase, and then use them the same way as passphrase-protected keys. This is useful for automated services, such as scripts and cron jobs. However, anyone who succeeds in stealing the private key can easily masquerade as you, so you need to be protective of a passphrase-less private key.

An alternative to using keys without passphrases is gnome-keyring, which remembers and applies your private keys and passphrases for you. gnome-keyring is for GNOME desktop sessions (Section 22.9, “Automated public key logins with gnome-keyring”).

For console sessions, use ssh-agent (Section 22.10, “Automated public key logins with ssh-agent”).

22.8 OpenSSH certificate authentication

OpenSSH introduced certificate authentication in OpenSSH 5.4. Certificate authentication is similar to public key authentication, except hosts and users authenticate to each other with digitally signed encryption certificates instead of encryption keys. Certificate authentication provides central management for server and user certificates, eliminating the need to manually copy user public keys to multiple hosts. It increases security by giving more control to administrators, and less to users.

Certificates consist of a public encryption key, a user-defined identity string, zero or more user names or host names, and other options. User and host public keys are signed by a Certificate Authority (CA) private signing key to create an encryption certificate. Users and hosts trust the public CA key, rather than trusting individual user and host public encryption keys.

Traditional OpenSSH public key authentication requires copying user public keys to every SSH server they need access to (to the appropriate ~/.ssh/authorized_keys files), and relying on users to verify new SSH server host keys before accepting them (stored in ~/.ssh/known_hosts). This is error-prone and complicated to manage. Another disadvantage is OpenSSH keys never expire. When you need to revoke a particular public key, you have to find and remove all its copies on your network.

Automating the whole process (for example, with Ansible) is virtually a necessity. Large organizations, such as Meta (see https://engineering.fb.com/2016/09/12/security/scalable-and-secure-access-with-ssh/), automate the process completely, so they can revoke and replace certificates, and even certificate authorities, as often as they want without disrupting operations.

A prerequisite is the ability to open SSH sessions to all hosts on your network, and perform tasks like editing configuration files and restarting sshd.

Setting up an OpenSSH certificate authority involves the following steps:

  • Set up a secure trusted server to host your certificate authority, for signing host and user keys. Create a new key pair for signing keys. The private key signs user and host keys, and the public key is copied to all users who are allowed access to the server.

  • Receive and sign host public keys, then distribute the new host certificates to their respective hosts. Host certificates go in /etc/ssh, just like host keys.

  • Receive and sign user public keys, then distribute the new user certificates to their owners. User certificates go in ~/.ssh, just like user keys.

  • Edit configuration files on servers and users' machines, and stop and start sshd on hosts as necessary.

  • Revoke certificates as needed, for example when you suspect a certificate has been compromised, a user has left your organization, or a server has been retired. Revoking a certificate is considerably simpler than finding and removing all relevant public key copies.

Users and server admins create and protect their own OpenSSH keys. It is safe to freely share public keys. It is safe to transfer the new certificates by insecure methods, such as email, because they require their private keys to validate.

SSH certificates follow the OpenPGP standard, rather than SSL/TLS, and that the certificate format is OpenPGP, not X.509.

22.8.1 Setting up a new certificate authority

This section describes how to set up a new certificate authority (CA). Give careful consideration to organizing your CA, to keep it manageable and efficient.

Important
Important: Protect your certificate authority

It is important to protect the machine that hosts your certificate authority. Your CA is literally the key to your entire network. Anyone who gains access to your CA can create their own certificates and freely access your network resources, or even compromise your servers and the CA itself. A common practice is to use a dedicated machine that is started only when you need to sign keys.

It is a best practice to create one signing key for servers, and another signing key for clients. If you have a large number of certificates to manage, it can be helpful to create your CAs for hosts and clients on separate machines. If you prefer using a single machine, give each CA its own directory. The examples in this section use /ca-ssh-hosts and /ca-ssh-users. The example machine is ca.example.com.

If your security policy requires keeping copies of users' and host's public keys, store them in their own subdirectories, for easier tracking and avoiding key name collisions.

Important
Important: RSA signing keys deprecated

OpenSSH 8.2, released February 2020, deprecates RSA signing keys. Use Ed25519 or ECDSA.

The following examples create two signing keys, one for signing host keys, and one for user keys. Give them strong passphrases:

> sudo ssh-keygen -t ed25519 -f /ca-ssh-hosts/ca-host-sign-key -C "signing key for host certificates"
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ca-host-sign-key
Your public key has been saved in ca-host-sign-key.pub
The key fingerprint is:
SHA256:STuQ7HgDrPcEa7ybNIW0n6kPbj28X5HN8GgwllBbAt0
 signing key for host certificates
The key's randomart image is:
+--[ED25519 256]--+
|      o+o..      |
|   . . o.=E      |
|    = + B .      |
|   + O + = B     |
|  . O * S = +    |
|   o B + o .     |
|    =o=   .      |
|   o.*+  .       |
|   .=.o+.        |
+----[SHA256]-----+
> sudo ssh-keygen -t ed25519 -f /ca-ssh-users/ca-user-sign-key -C "signing key for user certificates"
Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in ca-user-sign-key
Your public key has been saved in ca-user-sign-key.pub
The key fingerprint is:
SHA256:taYj8tTnjkzgfHRvQ6HTj8a37PY6rwv96V1x+GHRjIk signing key for user certificates
The key's randomart image is:
+--[ED25519 256]--+
|                 |
|             . +.|
|          . E o.o|
|         . + . ..|
|      . S * o .+.|
|     o + + = +..+|
|    . = * . O + o|
|     + = = o =oo+|
|      . o.o  oOX=|
+----[SHA256]-----+

Copy your public user signing key (be sure you are copying the PUBLIC key) to /etc/ssh on all hosts running SSH servers. Then enter the full path of the public user signing key to /etc/ssh/sshd_config on the hosts:

TrustedUserCAKeys /etc/ssh/ca-user-sign-key.pub

Then restart sshd.

22.8.2 Creating host certificates

The following example signs a host public key to create a host certificate for a database server:

> sudo ssh-keygen -s /ca-ssh-hosts/ca-host-sign-key \
   -n venus,venus.example.com -I "db-server host cert" \
   -h -V +4w /etc/ssh/ssh_host_ed25519_key.pub
Enter passphrase:
Signed host key /etc/ssh/ssh_host_ed25519_key-cert.pub: id
"db-server host cert" serial 0 for venus,venus.example.com
valid from 2022-08-08T14:20:00 to 2022-09-05T15:21:19

If there is more than one host key on a server, sign them all.

  • -s is your private signing key.

  • -n is your list of principals. For host certificates, the principals are the machine's host name and fully qualified domain name.

  • -I is the identity string. This is any comment or description you want. The string is logged, to help you quickly find relevant log entries.

  • -h creates a host certificate.

  • -V sets the expiration date for the certificate. In the example, the certificate expires in four weeks. (See the "-V validity_interval" section of man 1 ssh-keygen for allowed time formats.)

Verify that your new certificate is built the way you want:

> ssh-keygen -Lf /etc/ssh/ssh_host_ed25519_key-cert.pub
 /etc/ssh/ssh_host_ed25519_key-cert.pub:
        Type: ssh-ed25519-cert-v01@openssh.com host certificate
        Public key: ED25519-CERT SHA256:/
         U7C+qABXYyuvueUuhFKzzVINq3d7IULRLwBstvVC+Q
        Signing CA: ED25519 SHA256:
         STuQ7HgDrPcEa7ybNIW0n6kPbj28X5HN8GgwllBbAt0 (using ssh-ed25519)
        Key ID: "db-server host cert"
        Serial: 0
        Valid: from 2022-08-08T14:20:00 to 2022-09-05T15:21:19
        Principals:
                venus
                venus.example.com
        Critical Options: (none)
        Extensions: (none)

Add the full path of the new host certificate to /etc/ssh/sshd_config, to make it available to clients:

HostCertificate /etc/ssh/ssh_host_ed25519_key-cert.pub

Restart sshd to load your changes:

> sudo systemctl restart sshd.service

See Section 22.8.3, “CA configuration for users” to learn how to configure clients to accept host certificates.

22.8.3 CA configuration for users

The following example shows how to configure clients to trust your CA rather than individual keys. The example grants access to a single server. This entry must be on a single unbroken line in your users' ~/.ssh/known_hosts files. Move the original ~/.ssh/known_hosts file, and create a new file that contains only the CA configuration. Or, create a global configuration in /etc/ssh/ssh_known_hosts, which has the advantage of being un-editable by unprivileged users:

@cert-authority db,db.example.com ssh-ed25519
 AAAAC3NzaC1lZDI1NTE5AAAAIH1pF6DN4BdsfUKWuyiGt/leCvuZ/fPu
 YxY7+4V68Fz0 signing key for user certificates

List each server the user is allowed to access in a comma-delimited list, for example venus,venus.example.com,saturn,saturn.example.com. You may also grant access to all servers in domains with wildcards, for example *.example.com,*.example2.com.

Try connecting to the server. You should be prompted for the password of the remote account, without being prompted to verify the host certificate.

22.8.4 Creating user certificates

Sign the user's public key:

> sudo ssh-keygen /ca-ssh-hosts/ca-user-sign-key -I "suzanne's cert" -n suzanne -V +52w user-key.pub
 Signed user key .ssh/ed25519key-cert.pub: id "suzanne's cert" serial 0
 for suzanne valid from 2022-09-14T12:57:00 to 2023-09-13T12:58:21

The principal on user certificates is always the username. Store the user's certificate in ~/.ssh on the user's machine.

User certificates replace the ~/.ssh/authorized_keys files. Remove this file from a user account on the remote machine, then try opening an SSH session to that account. You should be able to log in without being prompted for a password. (Remember, the server should have a TrustedUserCAKeys /etc/ssh/ca-user-sign-key.pub line in its /etc/ssh/sshd_config file, so that the server knows to trust your certificate authority.)

Additionally, look for Accepted publickey for suzanne messages in your log files.

22.8.5 Revoking host keys

When you need to revoke a certificate because a server has been compromised or retired, add the certificate's corresponding public key to a file on every client, for example /etc/ssh/revoked_keys:

ssh-ed25519-cert-v01@openssh.com
    AAAAIHNzaC1lZDI1NTE5LWNlcnQtdjAxQG9wZW5zc2guY29tAAAAIK6hyvFAhFI+0hkKehF/
    506fD1VdcW29ykfFJn1CPK9lAAAAIAawaXbbEFiQOAe5LGclrCHSLWbEeUauK5+CAuhTJyz0
    AAAAAAAAAAAAAAACAAAAE2RiLXNlcnZlciBob3N0IGNlcnQAAAAeAAAABXZlbnVzAAAAEXZl
    bnVzLmV4YW1wbGUuY29tAAAAAGMabhQAAAAAYz9YgQAAAAAAAAAAAAAAAAAAADMAAAALc3No
    LWVkMjU1MTkAAAAgfWkXoM3gF2x9Qpa7KIa3+V4K+5n98+5jFjv7hXrwXPQAAABTAAAAC3Nz
    aC1lZDI1NTE5AAAAQI+mbJsQjt/9bLiURse8DF3yTa6Yk3HpoE2uf9FW/
    KeLsw2wPeDv0d6jv49Wgr5T3xHYPf+VPJQW35ntFiHTlQg= root@db

This file must be named in /etc/ssh/sshd_config:

RevokedKeys /etc/ssh/revoked_keys

22.9 Automated public key logins with gnome-keyring

The gnome-keyring package is installed and enabled by default when the GNOME desktop environment is installed. gnome-keyring is integrated with your system login, automatically unlocking your secrets storage at login. When you change your login password, gnome-keyring automatically updates itself with your new password.

gnome-keyring automatically loads all key pairs in ~/.ssh, for each pair that has a *.pub file. You may manually load other keys with the ssh-add command, for example:

> ssh-add ~/.otherkeys/my_key

List all loaded keys:

> ssh-add -L

When you start up your system and then open an SSH session, you are prompted for your private key passphrase.

gnome-keyring remembers the passphrase for the rest of your session. You do not have to re-enter the passphrase until after a system restart.

22.10 Automated public key logins with ssh-agent

The openssh package provides the ssh-agent utility, which retains your private keys and passphrases, and automatically applies your passphrases for you during the current session.

Configure ssh-agent to start automatically and load your keys by entering the following lines in your ~/.profile file:

eval "$(ssh-agent)"
ssh-add

The first line starts ssh-agent, and the second line loads all the keys in your ~/.ssh folder. When you open an SSH session that requires public key authentication, you are prompted for the passphrase. After the passphrase has been provided once, you do not have to enter it again, until after you restart your system.

You may configure ~/.profile to load only specific keys, like the following example that loads id_rsa and id_ed25519:

> ssh-add id_rsa id_ed25519

22.10.1 Using ssh-agent in an X session

On SUSE Linux Enterprise Server, ssh-agent is automatically started by the GNOME display manager. To also invoke ssh-add to add your keys to the agent at the beginning of an X session, do the following:

  1. Log in as the desired user and check whether the file ~/.xinitrc exists.

  2. If it does not exist, use an existing template or copy it from /etc/skel:

    if [ -f ~/.xinitrc.template ]; then mv ~/.xinitrc.template ~/.xinitrc; \
    else cp /etc/skel/.xinitrc.template ~/.xinitrc; fi
  3. If you have copied the template, search for the following lines and uncomment them. If ~/.xinitrc already existed, add the following lines (without comment signs).

    # if test -S "$SSH_AUTH_SOCK" -a -x "$SSH_ASKPASS"; then
    #       ssh-add < /dev/null
    # fi
  4. When starting a new X session, you are prompted for your SSH passphrase.

22.11 Changing an SSH private key passphrase

You may change or remove the passphrase from a private key with ssh-keygen:

> ssh-keygen -pf ~/.ssh/server1
Enter old passphrase:
Key has comment 'shared videos server1'
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.

22.12 Retrieving a key fingerprint

Use the ssh-keygen to display the public key fingerprint. The following example prints the SHA256 hash for a ED25519 key:

> ssh-keygen -lf ldap-server
256 SHA256:W45lbmj24ZoASbrqW0q9+NhF04muvfKZ+FkRa2cCiqo comment (ED25519)

Add the -v flag to display the ASCII art representation of the key:

> ssh-keygen -lvf ldap-server
256 SHA256:W45lbmj24ZoASbrqW0q9+NhF04muvfKZ+FkRa2cCiqo comment (ED25519)
+--[ED25519 256]--+
|                 |
|                 |
|    .. .         |
|  .o..+ +        |
| ...o+ BSo+      |
|. ..o.o =X       |
|...o o..* =      |
|o.*.* =+ = .     |
|E*o*+O. o.o      |
+----[SHA256]-----+

22.13 Starting X11 applications on a remote host

You can run graphical applications that are installed on a remote machine on your local computer. X11Forwarding Yes must be set in the /etc/ssh/sshd_config file on the remote machine. Then, when you run ssh with the -X option, the DISPLAY variable is automatically set on the remote machine, and all X output is exported to the local machine over the SSH connection. At the same time, X applications started remotely cannot be intercepted by unauthorized users.

A quick test is to run a simple game from the remote machine, such as GNOME Mines:

> ssh wilber@sun
Password:
Last login: Tue May 10 11:29:06 2022 from 192.168.163.13
Have a lot of fun...

wilber@sun>  gnome-mines

The remote application should appear on your local machine just as though it were installed locally. (Network lag affects performance.) Close the remote application in the usual way, such as clicking the close button. This closes only the application, and your SSH session remains open.

Important
Important: X11 forwarding does not work on Wayland

X11 forwarding requires the X Window System running on the remote host. The X Window System has built-in networking, while Wayland does not. Therefore Wayland does not support X11 forwarding.

Use the following command to learn if your system runs X or Wayland:

> echo $XDG_SESSION_TYPE
x11

If Wayland is in use, it looks like the following example:

> echo $XDG_SESSION_TYPE
wayland

The systemd way is to query with loginctl:

> loginctl show-session "$XDG_SESSION_ID" -p Type
Type=x11

> loginctl show-session "$XDG_SESSION_ID" -p Type
Type=wayland

22.14 Agent forwarding

By adding the -A option, the ssh-agent authentication mechanism is carried over to the next machine. This way, you can work from different machines without having to enter a password, but only if you have distributed your public key to the destination hosts and properly saved it there. (Refer to Section 22.6, “Public key authentication” to learn how to copy your public keys to other hosts.)

AllowAgentForwarding yes is the default in /etc/ssh/sshd_config. Change it to No to disable it.

22.15 scp—secure copy

scp copies files to or from a remote machine. If the user name on jupiter is different than the user name on sun, specify the latter using the USER_NAME&host format. If the file should be copied into a directory other than the remote user's home directory, specify it as sun:DIRECTORY. The following examples show how to copy a file from a local to a remote machine and vice versa.

> scp ~/MyLetter.tex tux@sun:/tmp 1
> scp tux@sun:/tmp/MyLetter.tex ~ 2

1

local to remote

2

remote to local

Tip
Tip: The -l option

With the ssh command, the option -l can be used to specify a remote user (as an alternative to the USER_NAME&host format). With scp the option -l is used to limit the bandwidth consumed by scp.

After the correct password is entered, scp starts the data transfer. It displays a progress bar and the time remaining for each file that is copied. Suppress all output with the -q option.

scp also provides a recursive copying feature for entire directories. The command

> scp -r src/ sun:backup/

copies the entire contents of the directory src including all subdirectories to the ~/backup directory on the host sun. If this subdirectory does not exist, it is created automatically.

The -p option tells scp to leave the time stamp of files unchanged. -C compresses the data transfer. This minimizes the data volume to transfer, but creates a heavier burden on the processors of both machines.

22.16 sftp—secure file transfer

22.16.1 Using sftp

To copy several files from or to different locations, sftp is a convenient alternative to scp. It opens a shell with a set of commands similar to a regular FTP shell. Type help at the sftp-prompt to get a list of available commands. More details are available from the sftp man page.

> sftp sun
Enter passphrase for key '/home/tux/.ssh/id_rsa':
Connected to sun.
sftp> help
Available commands:
bye                                Quit sftp
cd path                            Change remote directory to 'path'
[...]

22.16.2 Setting permissions for file uploads

As with a regular FTP server, a user can download and upload files to a remote machine running an SFTP server by using the put command. By default the files are uploaded to the remote host with the same permissions as on the local host. There are two options to automatically alter these permissions:

Setting a umask

A umask works as a filter against the permissions of the original file on the local host. It can only withdraw permissions:

permissions original

umask

permissions uploaded

0666

0002

0664

0600

0002

0600

0775

0025

0750

To apply a umask on an SFTP server, edit the file /etc/ssh/sshd_configuration. Search for the line beginning with Subsystem sftp and add the -u parameter with the desired setting, for example:

Subsystem sftp /usr/lib/ssh/sftp-server -u 0002
Explicitly setting the permissions

Explicitly setting the permissions sets the same permissions for all files uploaded via SFTP. Specify a three-digit pattern such as 600, 644, or 755 with -u. When both -m and -u are specified, -u is ignored.

To apply explicit permissions for uploaded files on an SFTP server, edit the file /etc/ssh/sshd_config. Search for the line beginning with Subsystem sftp and add the -m parameter with the desired setting, for example:

Subsystem sftp /usr/lib/ssh/sftp-server -m 600
Tip
Tip: Viewing the SSH daemon log file

To watch the log entries from the sshd use the following command:

> sudo journalctl -u sshd

22.17 Port forwarding (SSH tunneling)

ssh can also be used to redirect TCP/IP connections. This feature, also called SSH tunneling, redirects TCP connections to a certain port to another machine via an encrypted channel.

With the following command, any connection directed to jupiter port 25 (SMTP) is redirected to the SMTP port on sun. This is especially useful for those using SMTP servers without SMTP-AUTH or POP-before-SMTP features. From any arbitrary location connected to a network, e-mail can be transferred to the home mail server for delivery.

# ssh -L 25:sun:25 jupiter

Similarly, all POP3 requests (port 110) on jupiter can be forwarded to the POP3 port of sun with this command:

# ssh -L 110:sun:110 jupiter

Both commands must be executed as root, because the connection is made to privileged local ports. E-mail is sent and retrieved by normal users in an existing SSH connection. The SMTP and POP3 host must be set to localhost for this to work. Additional information can be found in the manual pages for each of the programs described above and in the OpenSSH package documentation under /usr/share/doc/packages/openssh.

22.18 More information

https://www.openssh.com

The home page of OpenSSH

https://en.wikibooks.org/wiki/OpenSSH

The OpenSSH Wikibook

man sshd

The man page of the OpenSSH daemon

man ssh_config

The man page of the OpenSSH SSH client configuration files

man scp , man sftp , man ssh , man ssh-add , man ssh-copy-id , man ssh-keygen

Man pages of several binary files to securely copy files (scp, sftp), to log in (slogin, ssh), and to manage keys.

/usr/share/doc/packages/openssh-common/README.SUSE , /usr/share/doc/packages/openssh-common/README.FIPS

SUSE package specific documentation; changes in defaults with respect to upstream, notes on FIPS mode etc.

22.19 Stopping SSH brute force attacks with Fail2Ban

An SSH brute force attack involves repeated trials of user name and password combinations until the attacker gains access to the remote server. The attacker uses automated tools that test various user name and password combinations to compromise a server.

You can use Fail2Ban software to limit intrusion attempts. Fail2Ban scans the system logs to detect attacks and trigger an action, such as blocking the IPs via the firewall. Fail2Ban is used only to protect services that require a user name and password authentication.

22.19.1 What is Fail2Ban?

Fail2Ban scans the log files in /var/log/apache/error_log and bans IPs that indicate malicious signs, such as too many password attempts, etc. You can then use Fail2Ban to update firewall rules to reject the IP addresses for a specified amount of time.

Fail2Ban comes with filters for various services, such as Apache, SSH, Courier, etc. You can use Fail2Ban to minimize the rate of incorrect authentication attempts.

22.19.2 Using Fail2Ban to stop an SSH brute force attack

You can install and configure Fail2Ban to protect the server against SSH brute force attacks.

Procedure 22.2: Using Fail2Ban to stop an SSH brute force attack
  1. To install Fail2Ban, execute:

    # sudo zypper -n in fail2ban firewalld
  2. When you install Fail2Ban, a default configuration file jail.conf is also installed. This file gets overwritten when you upgrade Fail2Ban. To retain any customizations you make to the file, you can add the modifications to a file called jail.local. Fail2Ban automatically reads both files.

    # cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local
  3. Open the file in your editor.

    vi /etc/fail2ban/jail.local
  4. Take a look at the four settings that you need to know about.

    Fail2Ban settings
    Figure 22.1: jail.local file settings
    ignoreip

    A list of whitelisted IPs that are never banned. The local host IP address 127.0.0.1 is in the list by default, along with the IPv6 equivalent ::1. You can use this setting to add a list of IP addresses that you know should not be banned.

    bantime

    The duration in minutes for which an IP address is banned. When you do not enter a value ending in m or h, it is treated as seconds. If you enter a value of -1, an IP address is permanently banned.

    findtime

    The period of time within which too many failed connection attempts result in an IP being banned.

    maxretry

    The limit for the number of failed attempts.

    Note
    Note

    If a connection from the same IP address makes maxretry failed connection attempts within the findtime period, they are banned for the duration of the bantime. The only exceptions are the IP addresses in the ignoreip list.

    Fail2Ban supports different types of jails, and each jail has different settings for the connection types.

  5. You can use the Fail2Ban service with the following:

    To enable:

    #  systemctl enable fail2ban

    To start:

    #  systemctl start fail2ban

    To check the service status:

    #  systemctl status fail2ban.service
    Important
    Important

    You must restart Fail2Ban each time you make a configuration change.