This post is part of the Adding MermaidJS Without Any JS series.
Woodpecker CI, Hugo & MermaidJSAdding icons to Mermaid chartsLight/dark theme for MermaidJSGenerate MermaidJS graphs faster
Fiddling around with Woodpecker CI was a lot of fun, so I finally implemented javascript-less MermaidJS graphs! It’s not perfect by any means but I think it is Good Enough(tm) and it involves some terrible regexps so here we go!
Table of Contents
Overview
An overview of the process, it will/might be the last graph in this entry:
╭───╮
│ ● │
╰─┬─╯
│
▼Commit
╭───────────────╮
│ MermaidRender │
╰───────┬───────╯
│
▼
╭───────────╮
│ HugoBuild │
╰─────┬─────╯
│
▼
╭──────────╮
│ Compress │
╰─────┬────╯
┌────────┴─────────┐
▼ ▼
╭─────────────╮ ╭────────────────╮
│ FlyioDeploy │ │ InternalDeploy │
╰──────┬──────╯ ╰────────┬───────╯
└────────┬─────────┘
▼
╭───╮
│ ● │
╰───╯
Mermaid rendering
I got inspired by this amazingly over-engineered argo-workflow based Hugo
build pipeline and stole got inspired.
In short, we will use the mermaid cli to locate and replace mermaid markup with SVGs. I’m also using a regexp to transform the default figure output to a ‘proper’ hugo figure, due to reasons.
In the example below I had to escape all {} or hugo would break their
rendering, fixing the regexp is left as an exercise for the reader.
steps:
mermaidcharts:
image: ghcr.io/mermaid-js/mermaid-cli/mermaid-cli:11.12.0
commands:
- |
#!/bin/sh
set -o errexit # abort on nonzero exitstatus
set -o nounset # abort on unbound variable
set -o pipefail # don't hide errors within pipes
for md_file in $(grep -l -r mermaid content|grep -E "md$"); do
echo "Converting $md_file"
dir=$(dirname $md_file)
bas=$(basename $md_file)
cd $dir
/home/mermaidcli/node_modules/.bin/mmdc \
--puppeteerConfigFile /puppeteer-config.json \
-i $bas \
-o $bas \
--theme $THEME \
--backgroundColor $BACKGROUND \
--outputFormat $OUTPUT_FORMAT
sed -i -E 's|!\[([^]]*)\]\(\./([^ ]+) "([^"]+)"\)|\{\{< figure src="\2" alt="\3" caption="\1" >\}\}|g;s|!\[([^]]*)\]\(\./([^)]*)\)|\{\{< figure src="\2" caption="\1" >\}\}|g' $bas
cd -
done
environment:
THEME: 'default'
BACKGROUND: 'transparent'
OUTPUT_FORMAT: 'svg'
Site building
Nothing fancy, just building the site with the new figures in place:
build:
image: hugomods/hugo:go-git-0.150.1
depends_on:
- mermaidcharts
commands:
- hugo --minify
For my internal site (“staging environment”), the build step looks like this:
build:
image: hugomods/hugo:go-git-0.150.1
depends_on:
- mermaidcharts
commands:
- hugo --baseURL=$INTERNAL_URL --minify --buildDrafts --buildFuture
environment:
INTERNAL_URL:
from_secret: internal-url
Compression
This is probably contra-productive for small files, but I haven’t benchmarked it yet so I’m just leaving this in. This page recommends compressing files larger than MTU which I guess makes sense?
compress:
image: alpine:latest
depends_on:
- build
commands:
- apk add -q --no-cache findutils brotli gzip zstd
- |
find ./public -type f -regex ".*\.\(css\|html\|js\|json\|svg\|xml\)$" \
-exec brotli --best {} \+ \
-exec gzip --best -k {} \+ \
-exec zstd -q {} \+
I’m skipping this step for my internal site as I don’t think my S3 compatible server will even try to use these compressed assets.
Publishing
I’m publishing this site both to fly.io (as documented before) and to an internal S3 compatible storage bucket. For PRs I only publish internally, and for production only to fly.io.
Initially I pushed the production build both internally and externally, but that made all links on the internal site go to the external site and that did not lead to hours of troubleshooting why my internal site didn’t update…
┌─────────────────┐
│ Event triggered │
└────────┬────────┘
│
▼
╭─────────────╮
│ What event? │
╰──────┬──────╯
┌────────┴────────┐
▼push ▼pull_request
┌───────────────┐ ┌───────────┐
│ Master branch │ │ PR branch │
└───────┬───────┘ └─────┬─────┘
│ │
▼ ▼
┌────────┐ ┌────┐
│ fly.io │ │ S3 │
└────────┘ └────┘
fly.io
I used to have a multi-stage Dockerfile to compile this site using the fly.io
infrastructure, but with this CI setup I can just copy the results from the
pipeline into the container instead…using the fly.io infrastructure, by
copying over the pipeline results before building the container, which in turn
will be converted to a firecracker VM. It feels like I can optimize this a
bit but that is a hack for another day!
FROM caddy:2
COPY ./public/ /usr/share/caddy/
COPY ./Caddyfile /etc/caddy/Caddyfile
Publishing is still done using flyctl:
flyio:
image: alpine:latest
depends_on:
- compress
commands:
- apk add --no-cache -q curl
- curl -L https://fly.io/install.sh | sh
- /root/.fly/bin/flyctl deploy
environment:
FLY_ACCESS_TOKEN:
from_secret: flyio-access-token
FLY_APP:
from_secret: flyio-app-name
NO_COLOR: 1
Internal staging
I honestly feel a bit bad since I’m using a minio product to copy the site to my internal storage bucket, but atleast I’m not using the minio server anymore!
publish:
image: minio/mc
depends_on:
- build
environment:
ACCESS_KEY:
from_secret: garage-access
SECRET_KEY:
from_secret: garage-secret
commands:
- mc alias set garage --insecure https://garage.example.com $ACCESS_KEY $SECRET_KEY
- mc mirror --insecure --overwrite --remove --quiet public/ garage/monotux-tech-stage
Conclusion
Woodpecker CI feels like a hammer in hand, and building this blog looks like a nail if you squint your eyes hard enough.
While I still can build this blog on my laptop, it will then be built without
charts. I’m not sure how I feel about complicating things like this, but I
enjoyed creating this Rube Goldberg machine CI pipeline so
that’s that.
And for completeness, here’s my final .woodpecker/publish.yaml:
when:
- event: [push, manual]
branch: master
steps:
mermaidcharts:
image: ghcr.io/mermaid-js/mermaid-cli/mermaid-cli:11.12.0
commands:
- |
#!/bin/sh
set -o errexit # abort on nonzero exitstatus
set -o nounset # abort on unbound variable
set -o pipefail # don't hide errors within pipes
for md_file in $(grep -l -r mermaid content|grep -E "md$"); do
echo "Converting $md_file"
dir=$(dirname $md_file)
bas=$(basename $md_file)
cd $dir
/home/mermaidcli/node_modules/.bin/mmdc --puppeteerConfigFile /puppeteer-config.json -i $bas -o $bas --theme $THEME --backgroundColor $BACKGROUND --outputFormat $OUTPUT_FORMAT
sed -i -E 's|!\[([^]]*)\]\(\./([^ ]+) "([^"]+)"\)|\{\{< figure src="\2" alt="\3" caption="\1" >\}\}|g;s|!\[([^]]*)\]\(\./([^)]*)\)|\{\{< figure src="\2" caption="\1" >\}\}|g' $bas
cd -
done
environment:
THEME: 'default'
BACKGROUND: 'transparent'
OUTPUT_FORMAT: 'svg'
build:
image: hugomods/hugo:go-git-0.150.1
depends_on:
- mermaidcharts
commands:
- hugo --minify
environment:
INTERNAL_URL:
from_secret: internal-url
HUGO_ENV: 'production'
compress:
image: alpine:latest
depends_on:
- build
commands:
- apk add -q --no-cache findutils brotli gzip zstd
- |
find ./public -type f -regex ".*\.\(css\|html\|js\|json\|svg\|xml\)$" \
-exec brotli --best {} \+ \
-exec gzip --best -k {} \+ \
-exec zstd -q {} \+
flyio:
image: alpine:latest
depends_on:
- compress
commands:
- apk add --no-cache -q curl
- curl -L https://fly.io/install.sh | sh
- /root/.fly/bin/flyctl deploy
environment:
FLY_ACCESS_TOKEN:
from_secret: flyio-access-token
FLY_APP:
from_secret: flyio-app-name
NO_COLOR: 1
And my .woodpecker/build.yaml for building PRs:
when:
- event: pull_request
steps:
mermaidcharts:
image: ghcr.io/mermaid-js/mermaid-cli/mermaid-cli:11.12.0
commands:
- |
#!/bin/sh
set -o errexit # abort on nonzero exitstatus
set -o nounset # abort on unbound variable
set -o pipefail # don't hide errors within pipes
for md_file in $(grep -l -r mermaid content|grep -E "md$"); do
echo "Converting $md_file"
dir=$(dirname $md_file)
bas=$(basename $md_file)
cd $dir
/home/mermaidcli/node_modules/.bin/mmdc --puppeteerConfigFile /puppeteer-config.json -i $bas -o $bas --theme $THEME --backgroundColor $BACKGROUND --outputFormat $OUTPUT_FORMAT
sed -i -E 's|!\[([^]]*)\]\(\./([^ ]+) "([^"]+)"\)|\{\{< figure src="\2" alt="\3" caption="\1" >\}\}|g;s|!\[([^]]*)\]\(\./([^)]*)\)|\{\{< figure src="\2" caption="\1" >\}\}|g' $bas
cd -
done
environment:
THEME: 'default'
BACKGROUND: 'transparent'
OUTPUT_FORMAT: 'svg'
build:
image: hugomods/hugo:go-git-0.150.1
depends_on:
- mermaidcharts
commands:
- hugo --baseURL=$INTERNAL_URL --minify --buildDrafts --buildFuture
environment:
INTERNAL_URL:
from_secret: internal-url
publish:
image: minio/mc
depends_on:
- build
environment:
ACCESS_KEY:
from_secret: garage-access
SECRET_KEY:
from_secret: garage-secret
commands:
- mc alias set garage --insecure https://garage.example.com $ACCESS_KEY $SECRET_KEY
- mc mirror --insecure --overwrite --remove --quiet public/ garage/monotux-tech-stage