A Daily Training Loop That Runs Itself

Shipping one fine-tuned model is a project. Keeping five specialized agents continuously improving is a systems problem.

The answer was a daemon that runs every night at a fixed time, collects new training examples, retrains each agent's adapter, and validates the result before anything touches production. No manual retraining cycles, no one babysitting a training run late at night.

The unglamorous but critical part was the failure handling — a lock file to prevent overlapping runs, and detection of stale locks if a previous run's process died without cleaning up after itself. Automation that silently breaks is worse than no automation at all.