Self-hosted CI/CD test lab for validating NoMercy MediaServer releases across Linux and Windows
  • Shell 78.5%
  • PowerShell 16.2%
  • Python 5.3%
Find a file
Repository files (latest commit first)
Filename Latest commit message Latest commit date
Stoney_Eagle 38219307f9
Make the Mac's runners survive an unattended reboot (#1)
* feat(ffmpeg-verify): make the Mac's runners survive an unattended reboot

actions-runner's svc.sh only produces a user LaunchAgent, which loads at login
rather than at boot. An unattended reboot therefore left every runner on the Mac
down until somebody logged in, silently removing darwin-arm64, darwin-x86_64 and
linux-aarch64 from the fleet with nothing to point at the cause.

A LaunchDaemon would start at boot but lose the keychain, signing identities and
simulators the Xcode runner depends on, and the runner wants runsvc.sh as its
entry point. So this makes the login happen instead, and adds a watchdog for
what login alone does not cover: the Lima guest, which nothing else starts, and
any runner service that failed to load. The watchdog only ever starts things, so
it is safe beside runners owned by someone else.

sysadminctl -autologin is broken on macOS 26 — it writes the user preference and
then fails the credential with SACSetAutoLoginPassword error:22, which reads as
success while auto-login stays non-functional. The installer verifies
/etc/kcpassword rather than trusting the exit status, and writes it directly
when that happens. It refuses to run at all when FileVault is on.

Verified by an actual reboot: auto-login took, the watchdog started the Lima
guest, and both runners were back online roughly 80 seconds after boot.

* docs(ffmpeg-verify): correct two stale claims in the README

The Linux NVENC paragraph blamed WSL2 passthrough, which the later investigation
disproved: passthrough works and the statically linked build is what cannot use
the libcuda shim. Leaving it as written would send the next person to re-debug
the wrong layer.

The provisioning snippet also still described the Windows runner as a logon task
after it was moved to a service.

* fix(ffmpeg-verify): keep the login password out of argv and off /tmp

The password was documented as a --password argument, which any local user
can read from ps for as long as the script runs and which lands in shell
history. It now prompts, reads --password-stdin, and keeps --password only
for automation that can do neither.

The kcpassword temp file was created by a plain redirect, so it sat at the
default umask holding a fixed-XOR obfuscation of that password until install
moved it, and stayed there if the script died in between. mktemp, umask 077
and a trap close both.

Also adds tv.nomercy.autologin-health, a LaunchDaemon covering the one
failure the watchdog structurally cannot: the watchdog is a LaunchAgent, so
it needs the session auto-login exists to create. If auto-login breaks, the
watchdog is not running to notice. The daemon runs at boot outside any
session and reports; it never repairs, because repairing needs the password.

fdesetup status is localised, so the sentence match refused to run on a
non-English system. fdesetup isactive is not.

* fix(ffmpeg-verify): make the health daemon a heartbeat, not a silence

It logged the ok line at most once a day, so the run at boot — the one that
matters — usually wrote nothing. A silent log then means either "everything
is fine" or "the check stopped running", which are the two things it exists
to tell apart. It now writes on every run and rotates its own log.
2026-07-28 04:57:35 +02:00
ffmpeg-verify Make the Mac's runners survive an unattended reboot (#1) 2026-07-28 04:57:35 +02:00
lib fix(ci): pvesh needs --output-format json for jq parsing 2026-03-26 01:33:16 +01:00
platforms fix(install): use systemctl --user for Linux installs 2026-05-02 07:05:36 +02:00
runners feat(ci): configurable runner name prefix via RUNNER_NAME_PREFIX 2026-03-26 08:07:39 +01:00
setup fix: set executable permission on all shell scripts 2026-03-25 23:41:00 +01:00
webhook Initial commit: self-hosted CI/CD test lab for NoMercy MediaServer 2026-03-01 17:00:12 +01:00
.env.example feat(ci): configurable runner name prefix via RUNNER_NAME_PREFIX 2026-03-26 08:07:39 +01:00
.gitattributes chore: add .gitattributes for consistent line endings 2026-03-16 05:50:44 +01:00
.gitignore feat(ci): add .env support for secrets + .gitignore 2026-03-25 23:28:55 +01:00
config.sh feat(ci): configurable runner name prefix via RUNNER_NAME_PREFIX 2026-03-26 08:07:39 +01:00
README.md fix(ci): audit fixes — OpenCore on SATA, .env.example complete, README accurate 2026-03-26 05:35:27 +01:00
run_matrix.sh fix: set executable permission on all shell scripts 2026-03-25 23:41:00 +01:00

nomercy-ci

A self-hosted, zero-cost CI/CD test lab and GitHub Actions runner farm, running entirely on a single Proxmox VE host using LXC containers and KVM virtual machines.

Runners

Ephemeral self-hosted GitHub Actions runners. Fresh environment every job — identical to GitHub-hosted runners.

Setup

# 1. Configure
cp .env.example .env
# Edit .env — fill in RUNNER_GH_TOKEN and review all settings

# 2. Create templates
./runners/setup_runner_templates.sh linux     # LXC — fully automated (~15 min)
./runners/setup_runner_templates.sh macos     # VM — downloads from Apple, manual install from console
./runners/setup_runner_templates.sh windows   # VM — unattended install (~30 min)

# 3. Start the runner pool (maintains N runners, recycles after each job)
./runners/runner_pool.sh all >> /var/log/nomercy-runners/pool.log 2>&1 & disown

# Or install as a service
cp runners/nomercy-runner-pool.service /etc/systemd/system/
systemctl daemon-reload
systemctl enable --now nomercy-runner-pool

How it works

Pool manager starts
  ├── Slot 1: Clone LXC → Boot → Register (--ephemeral) → Run job → Destroy → Repeat
  ├── Slot 2: Clone LXC → Boot → Register (--ephemeral) → Run job → Destroy → Repeat
  ├── ...
  └── Slot N: Same cycle

Each runner picks up one job, runs it, auto-deregisters, gets destroyed, and the pool spawns a replacement. Clean state every job.

Runner types

OS Type Template setup Install
Linux LXC container Fully automated (Ubuntu 24.04 LXC + all CI tools via SSH) ~15 min
macOS KVM VM Downloads recovery from Apple CDN + OpenCore bootloader. Manual OS install from Proxmox console ~30 min
Windows KVM VM Unattended install via autounattend XML + sysprep ~30 min

What's installed on Linux runners

Mirrors GitHub's ubuntu-24.04 hosted runner:

Category Tools
Languages Node 20+22, PHP 8.3+8.4, Java 8/11/17/21/25, .NET 8/9/10, Go 1.24, Ruby, Rust, Python 3
Build tools CMake 3.31, Ninja, Gradle 8.14, Maven 3.9, Ant
Android SDK platforms 34-36, build-tools 34-36.1, NDK 27/28/29
Containers Docker CE + Compose + Buildx
CLIs gh, aws v2, azure, kubectl, helm, packer, fastlane
Browsers Chrome + ChromeDriver, Firefox + Geckodriver
Databases PostgreSQL client, MySQL client, SQLite3

Commands

# Create templates (one-time)
./runners/setup_runner_templates.sh linux
./runners/setup_runner_templates.sh macos
./runners/setup_runner_templates.sh windows
./runners/setup_runner_templates.sh all

# Start runner pool (ephemeral — fresh clone every job)
./runners/runner_pool.sh all                  # all OS types, reads counts from .env
./runners/runner_pool.sh linux 5              # 5 Linux runners only

# Create persistent runners (non-ephemeral)
./runners/create_runner.sh linux 5
./runners/create_runner.sh macos 1
./runners/create_runner.sh windows 1
./runners/create_runner.sh all 2

# Destroy runners + deregister from GitHub
./runners/destroy_runners.sh all
./runners/destroy_runners.sh linux
./runners/destroy_runners.sh --vm 5103

# Resource usage overview
./runners/proxmox-usage.sh

macOS setup notes

macOS in a VM requires OpenCore as the bootloader. The script handles this automatically:

  1. Downloads OpenCore v21 from thenickdude/KVM-Opencore
  2. Downloads macOS recovery from Apple CDN (version configurable via RUNNER_MACOS_VERSION)
  3. Imports both as disk images — OpenCore on sata1, recovery on sata0, install target on sata2
  4. Configures Apple SMC passthrough, SMBIOS, CPU flags (GenuineIntel + invtsc)
  5. All disks on SATA (macOS has no VirtIO SCSI drivers, no IDE)

The script creates the VM, starts it, then waits for SSH. Complete the install from the Proxmox console while the script is running:

  1. OpenCore boot picker appears — select the macOS installer
  2. Open Disk Utility — erase the ~50 GB SATA disk as APFS
  3. Install macOS to that disk
  4. VM reboots — select "macOS Installer" in OpenCore (continues install)
  5. After final reboot — select "Macintosh HD" in OpenCore
  6. Create user ci during initial setup
  7. System Settings > General > Sharing > enable Remote Login (SSH)
  8. Open Terminal: mkdir -p ~/.ssh && echo '<your-key>' >> ~/.ssh/authorized_keys

Once SSH is available, the script automatically:

  • Installs all CI tools via install_macos_runner.sh
  • Shuts down the VM
  • Converts to template

Same flow as Linux and Windows — you don't need to manually run anything after enabling SSH.


Test Matrix

Validates NoMercy MediaServer release binaries across Linux distros and Windows.

Setup

# 1. Bootstrap Proxmox host
TRUENAS_IP=<your-truenas-ip> ./setup/setup_proxmox_host.sh

# 2. Create test templates
./setup/setup_templates.sh                # Linux LXC templates
./setup/setup_windows_template.sh         # Windows VM templates

# 3. Run tests
./run_matrix.sh                           # test latest release
./run_matrix.sh v1.2.3                    # test specific tag

# 4. Webhook (optional)
cp webhook/nomercy-ci.service /etc/systemd/system/
systemctl enable --now nomercy-ci
# Point GitHub webhook at http://<proxmox-ip>:9000/webhook (release event)

Supported platforms

Platform Type Package
Ubuntu 24.04 LXC .deb
Debian 13 LXC .deb
Fedora 43 LXC .rpm
Arch Linux LXC .pkg.tar.zst
Windows 10 KVM VM .exe

Linux targets run in parallel. Windows runs sequentially.


Project Structure

├── config.sh                      # Central config (loads .env)
├── .env.example                   # All settings with defaults
├── run_matrix.sh                  # Test matrix orchestrator
│
├── runners/                       # GitHub Actions runner management
│   ├── setup_runner_templates.sh  # Create Proxmox templates per OS
│   ├── runner_pool.sh             # Ephemeral runner pool manager
│   ├── create_runner.sh           # Create persistent runners
│   ├── destroy_runners.sh         # Destroy runners + deregister
│   ├── proxmox-usage.sh           # Resource usage overview
│   ├── nomercy-runner-pool.service # systemd service for pool
│   ├── install_linux_runner.sh    # Ubuntu 24.04 tool install
│   ├── install_macos_runner.sh    # macOS tool install (Homebrew)
│   └── install_windows_runner.ps1 # Windows tool install (Chocolatey)
│
├── lib/
│   ├── util.sh                    # Logging, SSH wait, VMID allocation
│   ├── lxc.sh                     # LXC lifecycle
│   ├── vm.sh                      # KVM lifecycle
│   ├── verify.sh                  # HTTP health checks
│   └── logs.sh                    # Artifact collection
│
├── platforms/                     # MediaServer install scripts per distro
│   ├── install_ubuntu.sh
│   ├── install_debian.sh
│   ├── install_fedora.sh
│   ├── install_arch.sh
│   └── install_windows.ps1
│
├── setup/                         # One-time Proxmox setup
│   ├── setup_proxmox_host.sh
│   ├── setup_templates.sh
│   ├── setup_windows_template.sh
│   ├── setup_windows_postinstall.ps1
│   ├── autounattend_win10.xml
│   └── autounattend_win11.xml
│
└── webhook/
    ├── webhook_server.py
    ├── webhook_server.env
    └── nomercy-ci.service

Configuration

All settings in .env (loaded by config.sh).

Runner settings

Variable Default Description
RUNNER_GH_TOKEN GitHub PAT with admin:org scope
RUNNER_ORG GitHub organization
RUNNER_GROUP Default Runner group in GitHub
RUNNER_VERSION GitHub Actions runner version
POOL_LINUX 5 Ephemeral Linux runner count
POOL_MACOS 1 Ephemeral macOS runner count
POOL_WINDOWS 1 Ephemeral Windows runner count
RUNNER_LINUX_LABELS Comma-separated labels
RUNNER_MACOS_LABELS Comma-separated labels
RUNNER_WINDOWS_LABELS Comma-separated labels
RUNNER_LINUX_CORES / _MEM 4 / 8192 Resources per Linux runner
RUNNER_MACOS_CORES / _MEM 4 / 8192 Resources per macOS runner
RUNNER_WINDOWS_CORES / _MEM 4 / 8192 Resources per Windows runner
RUNNER_OPENCORE_ISO OpenCore ISO on Proxmox storage
RUNNER_MACOS_VERSION sonoma macOS version to download from Apple
RUNNER_WINDOWS_ISO Windows 10 ISO on Proxmox storage
STORAGE local-lvm Proxmox storage for VM/LXC disks
BRIDGE vmbr0 Network bridge
ISO_STORAGE nas Proxmox storage for ISO files
VIRTIO_ISO nas:iso/virtio-win.iso VirtIO drivers ISO for Windows

Test matrix settings

Variable Default Description
PVE_NODE pve Proxmox node name
WEB_PORT 7626 NoMercy web UI port
LXC_CORES / LXC_MEM 2 / 2048 Linux test container resources
WIN_CORES / WIN_MEM 12 / 16384 Windows test VM resources
WEBHOOK_PORT 9000 Webhook listener port
ARTIFACT_ROOT /mnt/vault/nomercy-artifacts NFS artifact path

Design Principles

  • Ephemeral everything — runners and test clones are created, used once, and destroyed
  • Self-cleaningtrap cleanup EXIT purges resources on failure
  • Zero cost — Proxmox Community Edition, no paid CI services
  • Security-conscious — dedicated ci user, SSH key-only auth, no root SSH
  • No Docker for runners — real LXC containers and VMs, full OS isolation

License

Copyright NoMercy Entertainment. All rights reserved.