Garage object storage, Caddy and dynamically provisioned TLS certificates
I recently started serving this website internally (as a “staging environment”)
and discovered that Garage + Caddy can automagically serve sites & issue
certificates per S3 bucket, as subdomains under the Garage domain name
(*.garage.example.com). This entry will describe how I setup Garage, Caddy
and dnsmasq to work together in my homelab setup.
Table of Contents
Overview
In this setup,
- Garage acts as backend and hosts a S3 compatible bucket and serves it as static site using built-in functionality,
- Caddy acts as reverse-proxy and queries the Garage admin API before dynamically issuing a certificate per Garage bucket, and
- dnsmasq resolves all subdomains under the Garage base domain to the same
address as its parent (
garage.example.com => 192.168.0.50, for bucket foofoo.garage.example.com => 192.168.0.50, et c)
Caddy/Garage flow
Garage
Configuration
In this configuration we expose several services, but for this entry we are
mainly interested in s3_api and s3_web and their respective configurations.
s3_apiis using port 3900 androot_domain = garage.example.com, and is the endpoint we interface with when we pretend this is a normal S3 service (like when uploading files to the bucket)s3_webis using port 3902 androot_domain = .garage.example.com(notice the dot!), and is from where Garage serves the bucket as a website, one subdomain per bucket. So bucketfoois expected to be served from URLfoo.garage.example.comin this example.
metadata_dir = "path/to/metadata"
data_dir = "path/to/data"
db_engine = "sqlite"
replication_factor = 1
rpc_bind_addr = "[::]:3901"
rpc_public_addr = "[::]:3901"
[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
root_domain = "garage.example.com"
[s3_web]
bind_addr = "[::]:3902"
root_domain = ".garage.example.com"
index = "index.html"
[k2v_api]
api_bind_addr = "[::]:3904"
[admin]
api_bind_addr = "[::]:3903"
Operations
These commands will create a bucket, allow it to be served as a static site,
and create the access keys we can use for accessing said bucket. Notice that
Garage expects a RPC secret for making changes via it’s API, which you can
provide using something like export GARAGE_RPC_SECRET_FILE=/path/to/rpc_secret
in your shell.
# Create a bucket
garage bucket create example-site
# This is what enables serving the site from the bucket
garage bucket website --allow example-site
# Create a key and give it read/write to the bucket
garage key create example-site
garage bucket allow example-site --key example-site --write --read
To upload a static site to the bucket you can use any S3 compatible tool, like minio mc:
mc alias set garage --insecure https://garage.example.com $ACCESS_KEY $SECRET_KEY
mc mirror --insecure --overwrite --remove --quiet public/ garage/example-site
Then, once Caddy is configured you will be able to just visit the site:
curl https://example-site.garage.example.com
…and this will be served with a valid TLS certificate.
Caddy
This is the part I really like with this setup – I can just create a new bucket, enable serving it as a static site, and then visit the URL & Caddy will automagically serve the site with a valid TLS certificate!
Before issuing the certificate, Caddy will validate that the bucket exists using the Garage admin API. I’m just using this for my internal environment so I’m fine with the potential security implications, but I think this should be reasonably safe even for usage out on the Internet with LetsEncrypt certificates.
The setup below is more or less just a copy of the documentation but with my (anonymized) internal ACME endpoints included. The documentation also includes examples for nginx and Traefik, but they don’t support the on-demand TLS certificates like Caddy.
{
on_demand_tls {
ask http://localhost:3903/check
}
acme_ca https://ca.example.com/acme/acme/directory
}
*.garage.example.com {
log {
output file /var/log/caddy/access-*.garage.example.com.log
}
tls oscar@example.com {
ca https://ca.example.com/acme/acme/directory
on_demand
}
reverse_proxy localhost:3902
}
garage.example.com {
log {
output file /var/log/caddy/access-garage.example.com.log
}
tls oscar@example.com {
ca https://ca.example.com/acme/acme/directory
}
reverse_proxy localhost:3900
}
dnsmasq
We will use dnsmasq’s address for this, which will resolve both
garage.example.com and any subdomain to the same IP address. Normally I use
host-record to get A- and PTR-records for a computer, and cname for
setting up alternate names for a computer, but that would require me to
manually setup a CNAME per bucket which is a chore!
Given IP 192.168.0.50 for your Garage service and a base domain of
garage.example.com this is what you need to add to your dnsmasq.conf:
address=/garage.example.com/192.168.0.50
This will obviously only work for internal machines using this dnsmasq as a resolver! For use on the internet with real ACME certificates and DNS zones, consult the relevant documentation.
Conclusion
Now I can just create a new bucket and have it served with a valid certificate internally, how awesome is that?!
Next up: managing Garage using IaC (probably terraform/opentofu)