1. Initial ideas


Functional Requirements

I wanna build an applciation that allows teams to visualise some productivity metrics in their system. This should be pretty flexible and allow the teams to decide how they want to build these metrics. Essentially we should ingest events from the teams service(s) and let the team describe how to reduce these events down to a relevant metric.

That’s pretty vague, so to begin with I wanted to focus on the DORA metrics (do take a look at dora.dev, but we’ll include the definitions below) and pull request approval time. The way this is built should allow a team to configure it themselves, but using these as an aim for a given workflow gives something more concrete to aim for.

Metrics

Change lead time: The amount of time it takes for a change to go from committed to version control to deployed in production.

Deployment frequency: The number of deployments over a given period or the time between deployments.

Failed deployment recovery time: The time it takes to recover from a deployment that fails and requires immediate intervention.

Change fail rate: The ratio of deployments that require immediate intervention following a deployment. Likely resulting in a rollback of the changes or a “hotfix” to quickly remediate any issues.

Deployment rework rate: The ratio of deployments that are unplanned but happen as a result of an incident in production.

Pull Request Approval Time: The time it takes from a Pull (or merge) request being made ready to review, and being approved.

Non-functional requirements

There were a few non-functional requirements I had. This starts to feed into how I’m thinking of designing and building the system, which is subject to change as I get through some of this.

Low-Cost

Given I’m just building this as a side-project and don’t particularly intend to sell it, I wanna keep costs as low as possible. I’m making use of as many services as I can that have a free forever tier, and where I can’t find a free-forever service I will be leaning on pretty aggressive auto-scaling.

Scalable

It would be easy to throw this on a single server, but part of the fun here is trying to build it as cheaply as possible, whilst still retaining a reasonable system design that allows it to scale later.

Easy to run

My intention is that I can actually try and make this available for anyone else to use, within their own ecosystem. For this reason I wanna keep the system as simple as possible so that anyone can run it without having to manage a long list of tools. This could prove particularly challenging given the above constraints for being scalable. Essentially this will mean keeping the amount of software sprawl to a minimum.

Overview

At a high-level I’m gonna aim for an architecture something like:

graph LR
    subgraph Sources
        A1[Github]
        A2[Incident Management]
    end

    subgraph Platform
        subgraph Cloudflare
            B[App UI]
        end
        subgraph Kubernetes
            C[API]
            D[Queue Workers]
        end
        subgraph Neon
            subgraph Postgres
                E[TimescaleDB]
                F[River Queues]
            end
        end
    end

    A1 -->|events| C
    A2 -->|events| C
    B --> C
    C -->|Oauth|A1
    C --> E
    C --> F
    D --> E
    D <--> F

Kubernetes

I’m choosing kubernetes as the base to support a couple of the non-functional requirements outlined above. Primarily, if I wanna make a piece of software easy to install, helm is probably the best way. For my own instance of this application, I can run a kubernetes cluster relatively cheaply in Digital Ocean DOKS, and scalingn the application should be possible almost out of the box with Kubenertes primitives.

Helm

All of this can be easily packaged in a helm chart that would allow for a quick installation running all of these components inside a cluster, or the flexibility to switch out components with better options. I can include a very basic setup of all these components deployed as container images, but allow flags to turn them off when they’re not required. For example, timescaledb in a container is a very quick and easy way to get this running, but given how volatile a kubernetes cluster can be, isn’t the way I’d choose to run a database like Postgres typically.

Neon

Neon provides a free plan for Postgres, admittedly, at a pretty small scale. This should do for the early days, and can scale as far as i’d need, but I may wanna look at other options as scale grows.

App UI

I’m aiming to keep the application UI as a static application. I can’t see any reason that we’d need SSR atm, and this allows me to keep the deployment quite a bit simpler (albeit with a couple of limitations).

I will run this in Cloudflare Pages as it is completely free, and pretty easy to set up. Though I will also setup a container image in a registry that will enable this to run elsewhere pretty easily.

Queues

It does generally go against my instincts to use a relational DB as a message broker for queues. But River Queue looks pretty solid, and a big benefit is that I can implement the outbox pattern “natively” in the code. If I were to use Redis, RabbitMQ or anything else, I’d still need this outbox pattern.

This also means that the app only requires a single database at this stage. I can easily add different drivers for queuing later, I just need to bare that in mind as I write the queuing mechanisms in the app.

API

The API will be a golang CLI, that runs an HTTP server. I will use Huma to help build this API with support for OpenApi specs from day one. I’ve always preferred generating the specs from my code rather than the other way around, and Huma makes this pretty straight forward.

Ingestion

I will likely wanna split out kthe ingestion function into it’s own service at some point, and the API will be designed withn this in mind. The ingestion part of this application (receiving events from various systems) is likely to be the part of the system that needs to scale more dynamically and be the most reliable. I will ensure that the ingestion endpoints are built to be eventually consistent, they will receieve requests, check authenticate, runs some basic validation, and then push data onto a queue for further processing. This should mean I can easily split this out into a separate service when needed.