Create Let's Encrypt certificates where the HTTP Challenge reaches its limits

The majority of Let's Encrypt certificates can be issued via HTTP validation (HTTP-01 Challenge), i.e., simply installing certificates on a single web server. However, this method reaches its limits when certificates are to be issued for load-balanced websites or redundant mail servers with different IP addresses and hostnames. The same applies to wildcard certificates; therefore, it cannot be used for wildcard certificates.

DNS validation (DNS-01 Challenge)

With DNS validation (DNS-01 Challenge), certificate requests can be verified using DNS records instead of serving content over HTTP. This means that certificates can be issued simultaneously for multiple web servers, for example, behind a load balancer—or for systems that aren't directly accessible via the internet. Even wildcard certificates are supported by DNS validation.

The acme-dns-certbot tool instructs Certbot to connect to an external DNS server, where validation entries can be automatically set via API as soon as a certificate is requested. The advantage is that Certbot doesn't need to be integrated directly with the DNS provider, nor does it need to be granted full access to your own DNS configuration—which improves security.

Delegated DNS zones are used to forward DNS queries for certificate verification to the third-party DNS service. After the initial setup, any number of certificates can be requested without having to perform manual validation.

A major advantage of the acme-dns-certbot script is that it can be used to issue certificates for individual servers that are located behind a load balancer or are not directly accessible via HTTP. The classic HTTP-01 challenge no longer works in such cases. Alternatively, the validation files would have to be manually stored on each individual server. The tool is also useful when a certificate needs to be issued for a system that is not publicly accessible—just think of internal environments.

Installing Certbot and acme-dns-certbot

As already described in detail here on Let’s Encrypt mit Certbot, Certbot must first be installed. Then, the acme-dns-certbot script is required. The script

~# wget -O /etc/letsencrypt/acme-dns-auth.py github.com/joohoi/acme-dns-certbot-joohoi/raw/master/acme-dns-auth.py

is downloaded directly to the appropriate directory and made executable.

~# chmod a+x /etc/letsencrypt/acme-dns-auth.py

It's important that the script runs under Python 3. Therefore, the first line of the script must be changed

~# vi /etc/letsencrypt/acme-dns-auth.py

to

#!/usr/bin/env python3

if it isn't already.

For security reasons, you should review the GitHub repository and the script code before executing it. Alternatively, you can fork the repository to ensure the code remains under your control.

Configuring acme-dns-certbot

In the next step, the initial setup process is initiated by requesting a certificate via DNS validation.

~# certbot certonly --manual \
     --manual-auth-hook /etc/letsencrypt/acme-dns-auth.py \
     --preferred-challenges dns \
     --debug-challenges -d \*.your-domain \
     -d your-domain

The --manual option is used to disable all automated integration features of Certbot. In this case, a certificate is simply issued instead of automatically installing it on a service. Certbot is configured to use the acme-dns-certbot hook via the --manual-auth-hook option. The --preferred-challenges option instructs Certbot to prefer DNS validation.
Certbot must also be instructed to pause before attempting certificate validation. This is controlled by the --debug-challenges argument. This buys time to set the necessary DNS CNAME entries in the respective zone file required by acme-dns-certbot. The --debug-challenges argument pauses the Certbot process so that the necessary DNS changes can be made.
Any of the desired domain names can be specified with the -d option. When issuing a wildcard certificate, it's important to use escape characters. The asterisk * must be escaped with a backslash "\".

Certbot will then prompt you to interact with a message like this.

...
Output from acme-dns-auth.py:
Please add the following CNAME record to your main DNS zone:
_acme-challenge.your-domain CNAME 47eddf55-38fa-4d22-b43a-882b2366c9bb.auth.acme-dns.io.
Waiting for verification
...

You will then be prompted to add the required DNS CNAME entry to the domain's DNS zone file. This will then delegate control of the _acme-challenge subdomain to the ACME DNS service. This allows acme-dns-certbot to ultimately set the required DNS entries to validate the certificate request.
It is recommended to set the TTL (time-to-live) to approximately 300 seconds to ensure that changes to the entry propagate quickly.

But be careful!
The output "_acme-challenge.your-domain CNAME" is not entirely correct! This is because, for the DNS server, this is not a fully qualified domain name (FQDN)! For example, BIND9 appends "example.tld" as the domain name. The result would then be “_acme-challenge.example.tld.example.tld”, but this subdomain doesn't exist. Validation would fail.

So you should have the information in the background to make this entry with a final period like this:

_acme-challenge.your-domain. CNAME 47eddf55-38fa-4d22-b43a-882b2366c9bb.auth.acme-dns.io.

Or shortened without a final period:

_acme-challenge CNAME 47eddf55-38fa-4d22-b43a-882b2366c9bb.auth.acme-dns.io.

After this entry has been added to the DNS zone file and the Bind9 service has been reloaded on the DNS server, the Certbot process continues with an ENTER key. This validates the certificate request and completes the issuance process. After a few seconds, the certificate is successfully issued, with a similar message from Certbot.

...
Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/your-domain/fullchain.pem
Your key file has beensaved at:
/etc/letsencrypt/live/your-domain/privkey.pem
...

This executed the acme-dns-certbot script for the first time to ensure certificate issuance with the change to the DNS zone file.

Working with acme-dns-certbot

The state has now been reached where at least one Let's Encrypt certificate has been successfully issued with this script via the DNS-01 challenge. As with the original certbot command, certificates can now be added, revoked, and withdrawn for this domain. However, validation against an existing virtual host is no longer required. This makes it possible to issue another independent wildcard certificate:

~# certbot certonly --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth.py --preferred-challenges dns --debug-challenges -d \*.your-domain
 

This means that adding additional wildcard certificates to the already validated DNS domain is much easier. Adding a subdomain with

~# certbot certonly --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth.py --preferred-challenges dns --debug-challenges -d subdomain.your-domain

requires adding another CNAME entry to the corresponding zone file. This will then return the same Certbot output as it did during the initial issuance.

Renewing certificates that are about to expire is done in the same way as described in  “Let’s Encrypt mit Certbot” .

Conclusion

This method of certificate creation is particularly suitable for web servers behind load balancers. Or, in this case, the scenario involves redundant mail servers, which should have consistent SSL/TSL certificates for users if, via DNS failover, the domain name is to be redirected to the IP address of the failover mail server in the event of a mail server failure.