SQLite, Kubernetes & Litestream

I’ve been learning Kubernetes at home recently, and I’ve found a nice (but slightly dangerous!) pattern for hosting applications backed by SQLite in my homelab.

Table of Contents

Overview

The pattern is relatively simple:

• No persistent volumes, just ephemeral ones
• No deployments, we use a statefulset instead
• No external database, we use litestream to backup/restore automagically

If this sounds familiar it’s because I’ve used this pattern for running SQLite-backed applications on fly.io before.

The obvious issue with this is that we risk losing any data written between Litestream replications if our cluster dies. Since I’m the only user and I have backups through litestream I’m fine with this risk level¹.

Example manifest

In this example we’ll deploy Kanboard, a basic Trello-like application. I’m using ArgoCD & Kustomize to deploy this, but you can just use kubectl apply -f example.yaml as well. I will describe the prerequisites after the yaml.

As we don’t use a persistent volume, any attachments used in Kanboard will disappear on restart, rescheduling of pod et c. Kanboard is not an ideal fit for this pattern! Linkding was much better :-)

The example goes the following:

• Create a namespace to run everything in
• Create a ConfigMap for litestream, which describes what database to backup, and where to backup/restore from
• Create the StatefulSet, which includes an init container (which restores from backup if necessary) and Kanboard + Litestream in a pod. This also creates a ephemeral volume which the database is stored in.
• Create a service for the STS

---
apiVersion: v1
kind: Namespace
metadata:
  name: kanboard-sts
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: litestream
  namespace: kanboard-sts
data:
  litestream.yml: |
	dbs:
	  - path: '/var/www/app/data/db.sqlite'
		replicas:
		  - type: s3
			bucket: '$S3_BUCKET'
			path: '$S3_PATH'
			endpoint: '$S3_ENDPOINT'
			force-path-style: true
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: kanboard
  namespace: kanboard-sts
  labels:
	app: kanboard
spec:
  replicas: 1
  selector:
	matchLabels:
	  app: kanboard
  template:
	metadata:
	  labels:
		app: kanboard
	spec:
	  volumes:
		- name: data
		  emptyDir: {}
		- name: litestream-config
		  configMap:
			name: litestream
			items:
			- key: litestream.yml
			  path: litestream.yml
	  initContainers:
		- name: init-litestream
		  image: litestream/litestream:0.3
		  args:
		   - 'restore'
		   - '-if-db-not-exists'
		   - '-if-replica-exists'
		   - '/var/www/app/data/db.sqlite'
		  envFrom:
		  - secretRef:
			  name: litestream-s3
		  volumeMounts:
			- name: data
			  mountPath: /var/www/app/data
			- name: litestream-config
			  mountPath: /etc/litestream.yml
			  subPath: litestream.yml
	  containers:
		- name: litestream
		  image: litestream/litestream:0.3
		  args: ['replicate']
		  envFrom:
		  - secretRef:
			  name: litestream-s3
		  volumeMounts:
			- name: data
			  mountPath: /var/www/app/data
			- name: litestream-config
			  mountPath: /etc/litestream.yml
			  subPath: litestream.yml
		- name: kanboard
		  image: "docker.io/kanboard/kanboard:v1.2.45"
		  ports:
			- containerPort: 80
			  name: http
		  volumeMounts:
			- name: data
			  mountPath: /var/www/app/data
---
apiVersion: v1
kind: Service
metadata:
  name: kanboard-sts
  namespace: kanboard-sts
spec:
  selector:
	app: kanboard
  ports:
	- name: http
	  port: 80
	  targetPort: http

In order for this to work, you have to:

• Create a bucket to store the LiteStream backups in (you can use backblaze and their free tier for this)
• Create a secret to store this configuration in (we inject this secret into the LiteStream container in the manifest)

The secret needs to contain the following keys:

---
apiVersion: v1
kind: Secret
metadata:
  name: litestream-s3
  namespace: kanboard-sts
stringData:
  # Bucket name
  S3_BUCKET: ""

  # Might look like: https://s3.us-west-000.backblazeb2.com
  S3_ENDPOINT: ""

  # Might look like this
  S3_PATH: "kanboard_replica.sqlite3"

  LITESTREAM_ACCESS_KEY_ID: ""
  LITESTREAM_SECRET_ACCESS_KEY: ""
type: Opaque

Just apply the secret to your namespace.

Caveats / bootstrapping

Update: Before I discovered -if-replica-exists (available since v0.5.0-beta1 from 2021!) for the restore command, this section contained a convoluted method to boostrap an application relying on LiteStream. Now I just include said arguments to my restore command / initContainer and call it a day.

Conclusion

If you can live with the quirky bootstrapping process, it’s This is a fairly simple deployment pattern for SQLite backed applications, and I like to rely on it for my homelab stuff!