Leonard Surac

Back to the start page

In production · since 2026

Pi Dev Center

I run around twenty services at home and never knew for sure whether my backups would work when it mattered. So I built myself an operations centre that proves it.

ASP.NET Core 10 · EF Core · Angular 22 · PostgreSQL · PowerShell · GitHub Actions · Tailscale

Problem

My homelab runs around twenty services on two machines, from photos and documents to my own apps. Backups ran by cron script. Whether a backup had run and whether it could really be restored was scattered across log files, and I would only have found out in an emergency.

My role

I designed, built and run the solution, including 74 host scripts that report to it.

Case study

My solution

I did not want to know whether a backup script ran, but whether I get my data back. So the scripts report their result to a central app, and the app regularly restores every backup into a throwaway environment and counts.

  • Backup centre for local and encrypted off-site backups
  • Scheduled restore probes for every data set, including documents with real files
  • Service catalogue against the state both machines report every 15 minutes
  • Operations journal and an inbox for everything that needs my attention

Implementation

The app is built with ASP.NET Core, EF Core, PostgreSQL and Angular. Because it can trigger restores, there is no login but operator keys with an expiry and separate permissions. I store only the hash and compare in constant time:

PiDevCenter.Api/Shared/Infrastructure/Security/OperatorKeyAuthenticationHandler.cs
var key = settings.Keys.SingleOrDefault(item => item.Id == parts[0]);
if (key is null || !key.Enabled || key.ExpiresAt <= TimeProvider.GetUtcNow()) return Invalid();
var hash = SHA256.HashData(Encoding.UTF8.GetBytes(parts[1]));
if (!CryptographicOperations.FixedTimeEquals(hash, Convert.FromHexString(key.SecretSha256))) return Invalid();

Every push rolls out via GitHub Actions with a rehearsal, a rollback probe and an automated acceptance afterwards. Everything is reachable only over HTTPS on my private network.

Decisions

Keys instead of a login
I chose operator keys with a mandatory expiry and four separate permissions (read, test, restore, manage). They are managed in a file instead of a user interface, but a read key can never trigger a restore.
I only delete what I provably created
The recovery lab records every resource before creating it and only cleans those up. Leftovers I remove by hand, but nothing foreign ever disappears.
The verified dump is the export
Instead of building an export feature, the nightly dump that a restore probe restores every week is the complete export. Less code, at the cost of a technical format.

Outcome

Today I see at a glance what runs and what does not. In the first 30 days every daily backup succeeded on every day, and every backup is restored as a probe on a schedule; the control database of 3.2 GiB is back in 35 seconds.