The cluster

What is actually there

A seven-node k3s cluster with a highly available control plane on embedded etcd, spread across dedicated machines, a GPU worker for training and backfills, and virtual machines on Proxmox. The second site's cluster is coming up now and being wired for cross-site compute.

The control plane nodes are tainted so nothing schedules on them. I added that after a runaway job took the API server down with it.

Managed as code

ArgoCD runs the cluster in an app-of-apps pattern. The root application syncs manually, everything below it syncs automatically and self-heals, so the repository is the source of truth. Anything I change by hand gets reverted on the next reconcile, which is correct and still took some adjusting to.

My own GitLab sits underneath it, holding the repositories, the runners, the container registry and the CI that builds and deploys the rest.

The supporting cast

Kafka for streaming market data, an Iceberg lakehouse behind a REST catalog, object storage, and a multi-tenant Dagster deployment hosting four code locations. Observability is Prometheus with long-term storage, log aggregation, dashboards and uptime monitoring across both sites.

Then the parts nobody demos: scheduled backups, sealed secrets so nothing sensitive sits in Git in the clear, and a load-balanced ingress with automatic certificates.

Why do it this way

Cost is part of it. Running this on a managed cloud at the scale I use would be expensive, and the GPU work especially so.

The larger reason is that I like the hardware. I enjoy speccing a machine, building it and finding out what gives way under load. Owning the whole stack is also how I found most of the failure modes in the trading platform, from a node freezing under an overcommitted scheduler to an ingest that had quietly gone stale.

It runs the household too, so an outage is not theoretical. People notice.