monotux.tech

Garage object storage, Caddy and dynamically provisioned TLS certificates

dnsmasq Caddy Garage

I recently started serving this website internally (as a “staging environment”) and discovered that Garage + Caddy can automagically serve sites & issue certificates per S3 bucket, as subdomains under the Garage domain name (*.garage.example.com). This entry will describe how I setup Garage, Caddy and dnsmasq to work together in my homelab setup.

Table of Contents

Overview

In this setup,

Caddy/Garage flow

Caddy/Garage flow

Garage

Configuration

In this configuration we expose several services, but for this entry we are mainly interested in s3_api and s3_web and their respective configurations.

metadata_dir = "path/to/metadata"
data_dir = "path/to/data"
db_engine = "sqlite"

replication_factor = 1

rpc_bind_addr = "[::]:3901"
rpc_public_addr = "[::]:3901"

[s3_api]
s3_region = "garage"
api_bind_addr = "[::]:3900"
root_domain = "garage.example.com"

[s3_web]
bind_addr = "[::]:3902"
root_domain = ".garage.example.com"
index = "index.html"

[k2v_api]
api_bind_addr = "[::]:3904"

[admin]
api_bind_addr = "[::]:3903"

Operations

These commands will create a bucket, allow it to be served as a static site, and create the access keys we can use for accessing said bucket. Notice that Garage expects a RPC secret for making changes via it’s API, which you can provide using something like export GARAGE_RPC_SECRET_FILE=/path/to/rpc_secret in your shell.

# Create a bucket
garage bucket create example-site

# This is what enables serving the site from the bucket
garage bucket website --allow example-site

# Create a key and give it read/write to the bucket
garage key create example-site
garage bucket allow example-site --key example-site --write --read

To upload a static site to the bucket you can use any S3 compatible tool, like minio mc:

mc alias set garage --insecure https://garage.example.com $ACCESS_KEY $SECRET_KEY
mc mirror --insecure --overwrite --remove --quiet public/ garage/example-site

Then, once Caddy is configured you will be able to just visit the site:

curl https://example-site.garage.example.com

…and this will be served with a valid TLS certificate.

Caddy

This is the part I really like with this setup – I can just create a new bucket, enable serving it as a static site, and then visit the URL & Caddy will automagically serve the site with a valid TLS certificate!

Before issuing the certificate, Caddy will validate that the bucket exists using the Garage admin API. I’m just using this for my internal environment so I’m fine with the potential security implications, but I think this should be reasonably safe even for usage out on the Internet with LetsEncrypt certificates.

The setup below is more or less just a copy of the documentation but with my (anonymized) internal ACME endpoints included. The documentation also includes examples for nginx and Traefik, but they don’t support the on-demand TLS certificates like Caddy.

{
        on_demand_tls {
                ask http://localhost:3903/check
        }

        acme_ca https://ca.example.com/acme/acme/directory
}

*.garage.example.com {
        log {
                output file /var/log/caddy/access-*.garage.example.com.log
        }

        tls oscar@example.com {
                ca https://ca.example.com/acme/acme/directory
                on_demand
        }

        reverse_proxy localhost:3902
}

garage.example.com {
        log {
                output file /var/log/caddy/access-garage.example.com.log
        }

        tls oscar@example.com {
                ca https://ca.example.com/acme/acme/directory
        }

        reverse_proxy localhost:3900
}

dnsmasq

We will use dnsmasq’s address for this, which will resolve both garage.example.com and any subdomain to the same IP address. Normally I use host-record to get A- and PTR-records for a computer, and cname for setting up alternate names for a computer, but that would require me to manually setup a CNAME per bucket which is a chore!

Given IP 192.168.0.50 for your Garage service and a base domain of garage.example.com this is what you need to add to your dnsmasq.conf:

address=/garage.example.com/192.168.0.50

This will obviously only work for internal machines using this dnsmasq as a resolver! For use on the internet with real ACME certificates and DNS zones, consult the relevant documentation.

Conclusion

Now I can just create a new bucket and have it served with a valid certificate internally, how awesome is that?!

Next up: managing Garage using IaC (probably terraform/opentofu)