monotux.tech

Woodpecker CI, Hugo & MermaidJS

CI/CD Woodpecker Hugo

Fiddling around with Woodpecker CI was a lot of fun, so I finally implemented javascript-less MermaidJS graphs! It’s not perfect by any means but I think it is Good Enough(tm) and it involves some terrible regexps so here we go!

Table of Contents

Well, first of all, here is an example chart:

kanban
  Todo
    [Fix light/dark theme]
  [In progress]
    [Write entry about the new diagram functionality]
  [Done]
    [Steal ideas for pipeline]
    [Rewrite flyio deployment pipeline and Dockerfile]
    [Implement bulk-compression]
    [Setup pipeline for compiling mermaid charts to static SVGs]
  [Won't do]
    [Fix linebreaks in regexp]

And it’s final form:

diagram

diagram

Pretty neat! Using the kanban type also highlights that this isn’t perfect – the caption is just a placeholder which I can only change for a few graph types using the accDescr field. I also haven’t figured out how to properly deal with light/dark themes for mermaid yet, atleast not without JavaScript.

Overview

An overview of the process, it will/might be the last graph in this entry:

Overall flow

Overall flow

Mermaid rendering

I got inspired by this amazingly over-engineered argo-workflow based Hugo build pipeline and stole got inspired.

In short, we will use the mermaid cli to locate and replace mermaid markup with SVGs. I’m also using a regexp to transform the default figure output to a ‘proper’ hugo figure, due to reasons.

In the example below I had to escape all {} or hugo would break their rendering, fixing the regexp is left as an exercise for the reader.

steps:
  mermaidcharts:
    image: ghcr.io/mermaid-js/mermaid-cli/mermaid-cli:11.12.0
    commands:
      - |
          #!/bin/sh
          set -o errexit   # abort on nonzero exitstatus
          set -o nounset   # abort on unbound variable
          set -o pipefail  # don't hide errors within pipes

          for md_file in $(grep -l -r mermaid content|grep -E "md$"); do
            echo "Converting $md_file"
            dir=$(dirname $md_file)
            bas=$(basename $md_file)
            cd $dir
            /home/mermaidcli/node_modules/.bin/mmdc \
            --puppeteerConfigFile /puppeteer-config.json \
            -i $bas \
            -o $bas \
            --theme $THEME \
            --backgroundColor $BACKGROUND \
            --outputFormat $OUTPUT_FORMAT
            sed -i -E 's|!\[([^]]*)\]\(\./([^ ]+) "([^"]+)"\)|\{\{< figure src="\2" alt="\3" caption="\1" >\}\}|g;s|!\[([^]]*)\]\(\./([^)]*)\)|\{\{< figure src="\2" caption="\1" >\}\}|g' $bas
            cd -
          done

    environment:
      THEME: 'default'
      BACKGROUND: 'transparent'
      OUTPUT_FORMAT: 'svg'

Site building

Nothing fancy, just building the site with the new figures in place:

  build:
    image: hugomods/hugo:go-git-0.150.1
    depends_on:
      - mermaidcharts
    commands:
      - hugo --minify

For my internal site (“staging environment”), the build step looks like this:

  build:
    image: hugomods/hugo:go-git-0.150.1
    depends_on:
      - mermaidcharts
    commands:
      - hugo --baseURL=$INTERNAL_URL --minify --buildDrafts --buildFuture
    environment:
      INTERNAL_URL:
        from_secret: internal-url

Compression

This is probably contra-productive for small files, but I haven’t benchmarked it yet so I’m just leaving this in. This page recommends compressing files larger than MTU which I guess makes sense?

  compress:
    image: alpine:latest
    depends_on:
      - build
    commands:
      - apk add -q --no-cache findutils brotli gzip zstd
      - |
          find ./public -type f -regex ".*\.\(css\|html\|js\|json\|svg\|xml\)$" \
            -exec brotli --best {} \+ \
            -exec gzip --best -k {} \+ \
            -exec zstd -q {} \+

I’m skipping this step for my internal site as I don’t think my S3 compatible server will even try to use these compressed assets.

Publishing

I’m publishing this site both to fly.io (as documented before) and to an internal S3 compatible storage bucket. For PRs I only publish internally, and for production only to fly.io.

Initially I pushed the production build both internally and externally, but that made all links on the internal site go to the external site and that did not lead to hours of troubleshooting why my internal site didn’t update…

Another (worthless) flowchart describing the obvious

Another (worthless) flowchart describing the obvious

fly.io

I used to have a multi-stage Dockerfile to compile this site using the fly.io infrastructure, but with this CI setup I can just copy the results from the pipeline into the container instead…using the fly.io infrastructure, by copying over the pipeline results before building the container, which in turn will be converted to a firecracker VM. It feels like I can optimize this a bit but that is a hack for another day!

FROM caddy:2
COPY ./public/ /usr/share/caddy/
COPY ./Caddyfile /etc/caddy/Caddyfile

Publishing is still done using flyctl:

  flyio:
    image: alpine:latest
    depends_on:
      - compress
    commands:
      - apk add --no-cache -q curl
      - curl -L https://fly.io/install.sh | sh
      - /root/.fly/bin/flyctl deploy
    environment:
      FLY_ACCESS_TOKEN:
        from_secret: flyio-access-token
      FLY_APP:
        from_secret: flyio-app-name
      NO_COLOR: 1

Internal staging

I honestly feel a bit bad since I’m using a minio product to copy the site to my internal storage bucket, but atleast I’m not using the minio server anymore!

  publish:
    image: minio/mc
    depends_on:
      - build
    environment:
      ACCESS_KEY:
        from_secret: garage-access
      SECRET_KEY:
        from_secret: garage-secret
    commands:
      - mc alias set garage --insecure https://garage.example.com $ACCESS_KEY $SECRET_KEY
      - mc mirror --insecure --overwrite --remove --quiet public/ garage/monotux-tech-stage

Conclusion

Woodpecker CI feels like a hammer in hand, and building this blog looks like a nail if you squint your eyes hard enough.

While I still can build this blog on my laptop, it will then be built without charts. I’m not sure how I feel about complicating things like this, but I enjoyed creating this Rube Goldberg machine CI pipeline so that’s that.

And for completeness, here’s my final .woodpecker/publish.yaml:

when:
  - event: [push, manual]
    branch: master

steps:
  mermaidcharts:
    image: ghcr.io/mermaid-js/mermaid-cli/mermaid-cli:11.12.0
    commands:
      - |
          #!/bin/sh
          set -o errexit   # abort on nonzero exitstatus
          set -o nounset   # abort on unbound variable
          set -o pipefail  # don't hide errors within pipes

          for md_file in $(grep -l -r mermaid content|grep -E "md$"); do
            echo "Converting $md_file"
            dir=$(dirname $md_file)
            bas=$(basename $md_file)
            cd $dir
            /home/mermaidcli/node_modules/.bin/mmdc --puppeteerConfigFile /puppeteer-config.json -i $bas -o $bas --theme $THEME --backgroundColor $BACKGROUND --outputFormat $OUTPUT_FORMAT
            sed -i -E 's|!\[([^]]*)\]\(\./([^ ]+) "([^"]+)"\)|\{\{< figure src="\2" alt="\3" caption="\1" >\}\}|g;s|!\[([^]]*)\]\(\./([^)]*)\)|\{\{< figure src="\2" caption="\1" >\}\}|g' $bas
            cd -
          done

    environment:
      THEME: 'default'
      BACKGROUND: 'transparent'
      OUTPUT_FORMAT: 'svg'

  build:
    image: hugomods/hugo:go-git-0.150.1
    depends_on:
      - mermaidcharts
    commands:
      - hugo --minify
    environment:
      INTERNAL_URL:
        from_secret: internal-url
      HUGO_ENV: 'production'

  compress:
    image: alpine:latest
    depends_on:
      - build
    commands:
      - apk add -q --no-cache findutils brotli gzip zstd
      - |
          find ./public -type f -regex ".*\.\(css\|html\|js\|json\|svg\|xml\)$" \
            -exec brotli --best {} \+ \
            -exec gzip --best -k {} \+ \
            -exec zstd -q {} \+

  flyio:
    image: alpine:latest
    depends_on:
      - compress
    commands:
      - apk add --no-cache -q curl
      - curl -L https://fly.io/install.sh | sh
      - /root/.fly/bin/flyctl deploy
    environment:
      FLY_ACCESS_TOKEN:
        from_secret: flyio-access-token
      FLY_APP:
        from_secret: flyio-app-name
      NO_COLOR: 1

And my .woodpecker/build.yaml for building PRs:

when:
  - event: pull_request

steps:
  mermaidcharts:
    image: ghcr.io/mermaid-js/mermaid-cli/mermaid-cli:11.12.0
    commands:
      - |
          #!/bin/sh
          set -o errexit   # abort on nonzero exitstatus
          set -o nounset   # abort on unbound variable
          set -o pipefail  # don't hide errors within pipes

          for md_file in $(grep -l -r mermaid content|grep -E "md$"); do
            echo "Converting $md_file"
            dir=$(dirname $md_file)
            bas=$(basename $md_file)
            cd $dir
            /home/mermaidcli/node_modules/.bin/mmdc --puppeteerConfigFile /puppeteer-config.json -i $bas -o $bas --theme $THEME --backgroundColor $BACKGROUND --outputFormat $OUTPUT_FORMAT
            sed -i -E 's|!\[([^]]*)\]\(\./([^ ]+) "([^"]+)"\)|\{\{< figure src="\2" alt="\3" caption="\1" >\}\}|g;s|!\[([^]]*)\]\(\./([^)]*)\)|\{\{< figure src="\2" caption="\1" >\}\}|g' $bas
            cd -
          done

    environment:
      THEME: 'default'
      BACKGROUND: 'transparent'
      OUTPUT_FORMAT: 'svg'

  build:
    image: hugomods/hugo:go-git-0.150.1
    depends_on:
      - mermaidcharts
    commands:
      - hugo --baseURL=$INTERNAL_URL --minify --buildDrafts --buildFuture
    environment:
      INTERNAL_URL:
        from_secret: internal-url

  publish:
    image: minio/mc
    depends_on:
      - build
    environment:
      ACCESS_KEY:
        from_secret: garage-access
      SECRET_KEY:
        from_secret: garage-secret
    commands:
      - mc alias set garage --insecure https://garage.example.com $ACCESS_KEY $SECRET_KEY
      - mc mirror --insecure --overwrite --remove --quiet public/ garage/monotux-tech-stage