From 4482aa95a3412098aeba6966a17f473ad55941e9 Mon Sep 17 00:00:00 2001 From: Michael Harp Date: Tue, 11 Aug 2026 06:55:47 -0400 Subject: [PATCH] Add certificate renewal and regeneration how-to Add a how-to page to the openvox-server collection covering expired host, agent, and CA certificates: - Diagnosing which certificate expired with openssl, including the ca_ttl naming trap: the puppet.conf setting (default 5y) governs certs the running CA signs, while certs created by the puppetserver ca CLI (setup, generate --ca-client) get its undocumented 15-year fallback - Regenerating the primary server's host cert offline with puppetserver ca generate --ca-client, including the dropped subject-alt-names caveat and OpenVoxDB refresh via puppetdb ssl-setup -f - Regenerating an expired agent or compiler cert with puppet ssl clean / bootstrap, with a non-blocking manual-signing sequence - Enabling automatic renewal (allow-auto-renewal; the packaged ca.conf ships auto-renewal-cert-ttl 60d, built-in default 90d) and renewing on demand with puppet ssl renew_cert, including its silent no-op when the CA has renewal disabled - Extending an expired CA cert with the puppetlabs/ca_extend Bolt module, or manually by re-signing the existing key (adapted from a community walkthrough by bastelfreak) Also document the auto-renewal and ca-ttl settings in the ca.conf page and add the nav entry. Every command was verified on Ubuntu 24.04 containers running openvox-server 8.15.2, openvox-agent 8.28.1, and openvoxdb, including the CA extension procedure on a backdated expired CA. Closes #446 Co-authored-by: Claude Signed-off-by: Michael Harp --- _data/nav/openvox-server_8x.yml | 2 + .../certificate_renewal.markdown | 216 ++++++++++++++++++ .../config_file_ca.markdown | 8 + docs/_openvox_8x/getting_started.markdown | 2 + 4 files changed, 228 insertions(+) create mode 100644 docs/_openvox-server_8x/certificate_renewal.markdown diff --git a/_data/nav/openvox-server_8x.yml b/_data/nav/openvox-server_8x.yml index 3e3a19244..c43f8f0e0 100644 --- a/_data/nav/openvox-server_8x.yml +++ b/_data/nav/openvox-server_8x.yml @@ -51,6 +51,8 @@ link: infrastructure_crl.html - text: Intermediate CA link: intermediate_ca.html + - text: Renewing and regenerating certificates + link: certificate_renewal.html - text: External SSL termination link: external_ssl_termination.html - text: Restarting OpenVox Server diff --git a/docs/_openvox-server_8x/certificate_renewal.markdown b/docs/_openvox-server_8x/certificate_renewal.markdown new file mode 100644 index 000000000..3179584c0 --- /dev/null +++ b/docs/_openvox-server_8x/certificate_renewal.markdown @@ -0,0 +1,216 @@ +--- +layout: default +title: "Renewing and regenerating certificates" +--- + +Every certificate in an OpenVox deployment has a fixed lifetime. By default, certificates that the running CA signs for agents and compilers are valid for 5 years, while the CA certificate itself is valid for 15 years. When a certificate expires, TLS connections start failing with errors such as `certificate verify failed` or `certificate has expired`, and agent runs stop working. + +This page explains how to find out which certificate expired, how to replace an expired host or agent certificate, how to turn on automatic renewal to prevent future expirations, and how to extend an expired CA certificate without reissuing every agent certificate. + +> **Note:** Both lifetimes are controlled by the same setting, whose name is misleading. The [`ca_ttl`](/openvox/latest/configuration.html#ca_ttl) setting in `puppet.conf` (default `5y`) controls the lifetime of certificates the CA *signs*, not the CA certificate itself. +> The CA certificate's 15-year lifetime is a built-in fallback that `puppetserver ca setup` uses when `ca_ttl` is not set in `puppet.conf`; if you set `ca_ttl` before running setup, the CA certificate uses that value instead. +> The same fallback applies to any certificate the `puppetserver ca` CLI creates while the server is stopped, including the primary server's own certificate, so it can outlive the 5-year agent certificates. + +## Find out which certificate expired + +Check the expiration dates with `openssl`. On the primary server: + +```console +openssl x509 -enddate -noout -in "$(puppet config print hostcert)" +openssl x509 -enddate -noout -in /etc/puppetlabs/puppetserver/ca/ca_crt.pem +``` + +On an agent, check its own certificate and its copy of the CA certificate: + +```console +openssl x509 -enddate -noout -in "$(puppet config print hostcert)" +openssl x509 -enddate -noout -in "$(puppet config print localcacert)" +``` + +To inspect a certificate in full (subject, issuer, alt names, and validity), use `openssl x509 -in .pem -text -noout`. If `ca_crt.pem` contains a bundle of several certificates (the default layout uses an intermediate signing cert plus a root cert), `openssl x509` only shows the first one; use `openssl storeutl -noout -text ca_crt.pem` to print them all. + +With default settings, a certificate that expired 5 years after it was issued is a host certificate, and the CA certificate is still valid: the CA is not due to expire until 15 years after it was created. + +## Regenerate the primary server's certificate + +Use this procedure when the expired certificate is the primary server's own host certificate and the server is also the CA. The new certificate is signed by the existing CA, so agents are unaffected and nothing needs to change on other nodes. + +1. Stop OpenVox Server and back up the SSL directory: + + ```console + systemctl stop puppetserver + cp -a /etc/puppetlabs/puppet/ssl /etc/puppetlabs/puppet/ssl.bak + ``` + +1. Remove the expired certificate and its keys, and the CA's copy of the signed certificate: + + ```console + CERTNAME="$(puppet config print certname)" + rm /etc/puppetlabs/puppet/ssl/certs/"$CERTNAME".pem + rm /etc/puppetlabs/puppet/ssl/private_keys/"$CERTNAME".pem + rm /etc/puppetlabs/puppet/ssl/public_keys/"$CERTNAME".pem + rm /etc/puppetlabs/puppetserver/ca/signed/"$CERTNAME".pem + ``` + +1. Generate a new certificate. The `--ca-client` flag makes this work offline, signing directly with the CA's key while the server is stopped: + + ```console + puppetserver ca generate --certname "$CERTNAME" --ca-client + ``` + + The original certificate usually contains subject alternative names, and `puppetserver ca generate` does not carry them over. Check the backed-up certificate with `openssl x509 -text -noout` and pass every name agents use, such as a load balancer name, a CNAME, or a `puppet` DNS alias, with `--subject-alt-names`. + A certificate created at setup time includes `DNS:puppet` by default, so regenerating without the flag drops that name and agents that connect to the server as `puppet` reject the new certificate. + Don't rely on `puppet config print dns_alt_names` here: the setting is usually empty even when the certificate has alternative names. + + If the command reports that it could not determine whether Puppet Server is online (for example, when the `server` hostname does not resolve), confirm that the service is stopped and rerun with `--force`. + +1. Start OpenVox Server and verify the new certificate: + + ```console + systemctl start puppetserver + openssl x509 -enddate -noout -in "$(puppet config print hostcert)" + ``` + + Then run `puppet agent -t` on an agent to confirm that agents can connect. + +If OpenVoxDB runs on the same host, it keeps its own copies of the certificate, key, and CA certificate in `/etc/puppetlabs/puppetdb/ssl/`. Refresh them and restart OpenVoxDB: + +```console +puppetdb ssl-setup -f +systemctl restart puppetdb +``` + +## Regenerate an agent certificate + +Use this procedure when an agent's certificate has expired, or when a compiler's certificate has expired and the CA runs on a different server. An expired client certificate can't authenticate, so the agent needs a new one signed by the CA. + +1. On the CA server, revoke and remove the old certificate: + + ```console + puppetserver ca clean --certname + ``` + +1. On the agent, delete the expired certificate and key: + + ```console + puppet ssl clean + ``` + +1. On the agent, request a new certificate: + + ```console + puppet ssl bootstrap + ``` + + This submits a new certificate signing request, then waits and retries every 2 minutes until the certificate is signed. Leave it running while you sign the request in the next step; it downloads the certificate on its next retry. + To submit the request without waiting instead, run `puppet ssl bootstrap --waitforcert 0`; it submits the request and then exits with an error noting the certificate has not been signed yet, which is expected. Sign the request, then run the command again to download the certificate. A regular `puppet agent -t` run submits the same request. + +1. If you don't use autosigning, sign the request from a session on the CA server: + + ```console + puppetserver ca sign --certname + ``` + +If the certificate needs subject alternative names, the CA must have `allow-subject-alt-names: true` in the `certificate-authority` section of [`ca.conf`](config_file_ca.html), and the agent must request them, for example with `puppet ssl bootstrap --dns_alt_names ,`. + +For a compiler, stop OpenVox Server on the compiler before cleaning its certificate and start it again after the new certificate is in place. If OpenVoxDB shares the host, refresh its certificate copies as shown in the previous section. + +## Turn on automatic renewal + +OpenVox Server supports automatic certificate renewal, which prevents host certificates from ever reaching their expiration date. It is off by default. +The packaged [`ca.conf`](config_file_ca.html) already contains the settings in its `certificate-authority` section; set `allow-auto-renewal` to `true` and restart OpenVox Server: + +```text +certificate-authority: { + allow-auto-renewal: true + auto-renewal-cert-ttl: "60d" +} +``` + +With auto-renewal enabled, the CA issues certificates to renewal-capable agents with the shorter `auto-renewal-cert-ttl` lifetime instead of the `ca_ttl` value. +The packaged configuration sets `auto-renewal-cert-ttl` to 60 days; when the setting is absent, the built-in default is 90 days. +All OpenVox agents support renewal: when an agent's certificate is within `hostcert_renewal_interval` (30 days by default) of expiring, the agent requests a renewed certificate during its regular run and switches to it transparently, using the [certificate renewal endpoint](ca-api/v1/http_certificate_renewal.html). + +You can also renew a certificate on demand by running `puppet ssl renew_cert` on the agent. It uses the same renewal endpoint and works on any still-valid certificate once renewal is enabled, including certificates issued before you enabled it, and replaces the certificate with one that has the `auto-renewal-cert-ttl` lifetime. +Pass `--if-expiring-in ` (for example `30d`) to renew only when the certificate is close to expiry. +If `allow-auto-renewal` is not enabled on the CA, the command does nothing and still exits successfully, so check the certificate's expiration date afterward to confirm the renewal happened. + +Renewal authenticates with the agent's current certificate, so it only works while that certificate is still valid. It does not work for a certificate that has already expired: use the regeneration procedures above first, then enable auto-renewal. + +## Extend an expired CA certificate + +An expired CA certificate does not require rebuilding the CA and reissuing every certificate in the deployment. Because every host certificate was signed by the CA's private key, re-signing the CA certificate with the same key and subject gives it a new validity period while existing host certificates remain valid. +Only the CA certificate file changes, and you then distribute it to the rest of the deployment. + +The [`puppetlabs/ca_extend`](https://forge.puppet.com/modules/puppetlabs/ca_extend) module automates this procedure and works with open source deployments. +Its `extend_ca_cert` plan re-signs the CA certificate on the primary server (pass `regen_primary_cert=true` if the server's own host certificate has also expired) and repairs an expired CRL along the way, and its `upload_ca_cert` plan distributes the refreshed certificate to agents. +If you can run OpenBolt against your infrastructure, use the module. The manual procedure below performs the same steps. + +> **Warning:** A mistake here can break authentication for the whole deployment. Back up the CA directory (`/etc/puppetlabs/puppetserver/ca`) before you start, and test the procedure in a test environment first if you can. +> If your CA uses the default intermediate layout, `ca_crt.pem` is a bundle of the signing certificate and the root certificate. The same re-signing technique applies to each certificate in the bundle, but you must re-sign the root with the root's key (`root_key.pem`) and the intermediate with the root as issuer, then rebuild the bundle in the same order. + +To extend a CA with a single self-signed certificate manually, on the CA server: + +1. Confirm the CA key matches the CA certificate; the two digests must be identical: + + ```console + cd /etc/puppetlabs/puppetserver/ca + openssl rsa -noout -modulus -in ca_key.pem | openssl md5 + openssl x509 -noout -modulus -in ca_crt.pem | openssl md5 + ``` + +1. Back up the certificate, then turn it into a new CSR signed by the same key: + + ```console + cp -p ca_crt.pem ca_crt.pem.bak + openssl x509 -x509toreq -in ca_crt.pem -signkey ca_key.pem -out ca_csr.pem + ``` + +1. Re-sign it with the CA extensions and a new validity period (15 years here): + + ```console + cat > extension.cnf < ``` +> **Tip:** Certificates issued this way are valid for 5 years and expire silently. Consider [turning on automatic renewal](/openvox-server/latest/certificate_renewal.html#turn-on-automatic-renewal) on the server now, so agent certificates renew themselves instead of expiring. + --- ## Step 3: Set up a control repository