Github protocol changes and Puppet code deployment

GitHub is in the process of improving Git protocol security by removing insecure DSA keys and legacy ciphers, updating requirements on RSA keys, and adding more secure key signature algorithms ECDSA and ED25519. On January 11, GitHub ran a planned brownout event day to shake out affected clients. Our team at Puppet discovered that many users of Code Manager and r10k were affected by this change. Code Manager and r10k both use libgit2 version 1.9.0, which doesn't yet support the upgraded security of RSA with SHA-2. GitHub plans to make these changes permanent on March 15, which means that you may need to update your authentication methods before then to ensure proper code deployment.

Recognizing and resolving the issue

Given that Puppet previously advised in managing environments with a control repository, it is likely many users will have already created RSA 256 keys as recommended. Any affected public keys which start sha-rsa will stop working from March 15, at which point running puppet code deploy or `r10k deploy environment will result in failures to deploy and an error like this:

$ Found 1 environments. $[ $ { $ "environment": "production", $ "error": { $ "details": { $ "corrected-name": "production" $ }, $ "kind": "puppetlabs.code-manager/deploy-failure", $ "msg": "Errors while deploying environment 'production' (exit code: 1):\nERROR\t -> Unable to determine current branches for Git source 'puppet' (/etc/puppetlabs/code-staging/environments)\nOriginal exception:\nERROR: You're using an RSA key with SHA-1, which is no longer allowed. Please use a newer client or a different key type.\nPlease see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.\n\n at /opt/puppetlabs/server/data/code-manager/worker-caches/deploy-pool-2/[email protected]\n" $ }, $ "id": 15, $ "status": "failed" $ } $]

You will need to recreate the affected key and add it as a deployment key in GitHub.

For Open Source Puppet (OSP) r10k, you will create new keys:

ssh-keygen -t ed25519 -P '' -f /root/.ssh/id_ed25519

Then you will update the private key setting at /etc/puppetlabs/r10k/r10k.yaml

If you are using Puppet Enterprise on the primary Puppet server with Code Manager, then you should create the keys by running:

# ssh-keygen -t ed25519 -P '' -f /etc/puppetlabs/puppetserver/ssh/id-control_repo.ed25519

Then as per Configure Code Manager, you should update the private key setting either in the web console or your Hiera data.

In order to ensure correct permissions are applied to the keys if you are using Puppet Enterprise, run:

# puppet infrastructure configure

If you are using a Replica server on Puppet Enterprise, you should copy to your Replica server both the private key /etc/puppetlabs/puppetserver/ssh/id-control_repo.ed25519 and the public key /etc/puppetlabs/puppetserver/ssh/id-control_repo.ed25519.pub.

Follow the process on GitHub (Managing deploy keys - GitHub Docs) to add the content as a read-only deployment key to your control repo on the public key /etc/puppetlabs/puppetserver/ssh/id-control_repo.ed25519.pub or /root/.ssh/id_ed25519.pub.

Future fix and support for RSA

Libssh2 support for RSA with SHA-2 was added by the Libssh2 project to the Libssh2 library on January 6 and this update will be included in a future release of Puppet Enterprise in order to allow use of RSA 2 SSH keys again.

We know it can be frustrating to face security updates. The good news is that GitHub’s moves to stronger ciphers and keys will ensure the safety and security of code and deployment, and is an excellent opportunity to increase the security of your code and infrastructure.

David Sandilands is a senior solutions architect at Puppet.

Learn more