Daemons

A daemon is several workers of the same logic under supervision. A crashed one is restarted with a growing back-off, a stuck one is killed and replaced, and the number of working hands can change on the fly — by hand or by load.

What a daemon is and why

A daemon is a component of two parts: a supervising master and the fleet of identical workers it manages.

The problem. A process solves “keep working continuously” but remains one. The moment there is more work than one worker can handle, questions appear that a lone process never had:

• run five copies — how? Five twin classes, or five systemd units?
• one crashed — who notices and brings it back? And if it crashes in a loop, how do you avoid taking the system down with endless restarts?
• one hung on a network call forever — formally it is alive, ps shows it;
• at night there is half the work — how do you remove the extras without cutting off the busy ones?

Each answer alone is not hard, but together they are a supervisor you would rather not write.

The solution. Winter provides one. You describe the body of one worker and say how many are needed; the master watches the fleet: forks them, counts crashes, waits before restarting, kills stuck ones, and adds or removes workers on your signal.

Quick start

Extend Daemon, describe the work in workerRun() and state the replica count:

<?php

namespace Main\Process;

use Flytachi\Winter\DI\Attribute\Autowired;
use Flytachi\Winter\Kernel\Process\Stereotype\Daemon;

class Emails extends Daemon
{
  #[Autowired] private OutboxRepository $outbox;
  #[Autowired] private Mailer $mailer;

  protected int $replicas = 4;        // four workers

  protected function workerRun(): void
  {
      while ($this->isRunning()) {
          $letter = $this->outbox->takeNext();

          if ($letter === null) {
              $this->sleep(1);
              continue;
          }

          $this->markBusy();
          try {
              $this->mailer->send($letter);
              $this->outbox->markSent($letter->id);
          } finally {
              $this->markIdle();
          }
      }
  }
}

Declare it in the manifest:

#[EnableDaemon(\Main\Process\Emails::class)]
final class Application extends WinterApplication { /* ... */ }

Or start it on its own:

php call daemon main.process.Emails start -d
php call daemon main.process.Emails status

Daemon ● RUNNING
PID        51204
State      RUNNING
Activity   busy
Uptime     6h 12m
Workers    4
Restarts   1

SLOT  PID      STATE       ACT    UPTIME    RESTARTS
#0    51210    running     busy   6h 12m    0
#1    51211    running     idle   6h 12m    0
#2    51212    running     busy   6h 12m    0
#3    52890    running     idle   41m       1

The worker as a separate class

If the worker body is already described by a standalone process, workerRun() can be omitted — name the class instead:

class Emails extends Daemon
{
  protected int $replicas = 4;
  protected ?string $workerClass = \Main\Process\SendWorker::class;
}

That is convenient when the same worker must be runnable both alone and as a fleet. The body must be given by one of the two means: with neither workerRun() nor $workerClass the daemon goes into an endless crash loop.

Settings

The minimum replica count is one: $replicas = 0 is raised to 1. Collapsing the fleet to zero is only possible dynamically, through desiredReplicas().

Restarting crashed workers

What to do with a dead worker is decided by the restart policy.

use Flytachi\Winter\Kernel\Process\Daemon\{RestartPolicy, RestartMode};

protected function restart(): RestartPolicy
{
  return new RestartPolicy(
      mode:        RestartMode::ON_FAILURE,
      maxRestarts: 0,
      backoff:     1.0,
  );
}

Three modes, named by the Kubernetes and systemd convention — if you have written a restart policy there, the semantics are the same:

The growing pause

A restart is not immediate: the pause doubles with each crash of that worker and caps at 30 seconds. With a one-second base that gives

1 → 2 → 4 → 8 → 16 → 30 → 30 → …

The point is that a worker crashing because the database is unreachable does not become an endless fork loop and does not stop the system from recovering.

The liveness watchdog

A crashed worker is visible; a stuck one is not: the process is alive, ps shows it, the work has stopped. There is a separate mechanism for that:

protected float $livenessTimeout = 120.0;

Every worker regularly signals that it is alive. When no signal arrives for longer than the given time, the master kills the worker outright, and from there it follows the ordinary crash path — a pause and a restart.

Scaling

The worker count can change on the fly. Override desiredReplicas() — it is polled regularly and its answer becomes the target:

protected function desiredReplicas(): int
{
  $depth = $this->outbox->pendingCount();

  return match (true) {
      $depth > 10_000 => 20,
      $depth > 1_000  => 10,
      $depth > 0      => 4,
      default         => 1,
  };
}

Zero is allowed too — the fleet collapses entirely and unfolds when work appears.

Why the fleet does not change at once

The answer from desiredReplicas() is a signal, not a command. The supervisor damps it so a random spike does not jerk the fleet back and forth:

use Flytachi\Winter\Kernel\Process\Daemon\ScalingPolicy;

protected function scaling(): ScalingPolicy
{
  return new ScalingPolicy(
      scaleInterval:          1.0,
      scaleUpDelay:           0.0,
      scaleDownStabilization: 60.0,
      cooldown:               3.0,
      scaleStep:              0,
  );
}

The model is deliberately asymmetric: up quickly, down cautiously. A load spike has to be served immediately, whereas removing workers over a minute of quiet is a bad idea — a minute later they would have to come back.

Shrinking respects business: all else being equal the idle workers go first, and a busy one is given until $grace to finish.

Sharing work across workers

The daily difference between a daemon and a process is that there are several workers, and they reach for work at the same time. The framework starts and watches them but does not coordinate them: how to divide the work is your decision.

A task must be claimed atomically: marked as yours and returned in one action, so nothing slips in between “selected” and “took”.

The most portable technique is to mark the row with a conditional update and then read back exactly what you marked:

-- 1. try to claim: mark exactly one free row with our marker
UPDATE outbox
 SET claimed_by = :worker, claimed_at = NOW()
WHERE id = (SELECT id FROM outbox WHERE claimed_by IS NULL ORDER BY id LIMIT 1);

-- 2. take what we, specifically, claimed
SELECT * FROM outbox WHERE claimed_by = :worker AND claimed_at IS NOT NULL LIMIT 1;

In PostgreSQL and MySQL 8 the same thing is shorter — a select that skips locked rows:

SELECT * FROM outbox
WHERE claimed_by IS NULL
ORDER BY id LIMIT 1
 FOR UPDATE SKIP LOCKED;

The slot number or the worker’s PID makes a convenient marker: then stuck tasks show who claimed them.

A worker does not know its number

The obvious idea is to divide work up front — “worker #0 takes even records, #1 odd ones”. That will not work: the slot number is known only to the supervisor, it is not passed to the worker, and it cannot be obtained inside workerRun().

That is deliberate. Numbers live exactly until the first crash: a dead worker is replaced, the fleet may grow or shrink, and a partition tied to the replica count becomes wrong after every such event — part of the work goes to two workers, part to none.

So work must be divided through shared state, not by position: an atomic claim from a queue as above, or a claim over a whole chunk when the unit of work is large.

-- the worker claims not a task but a whole partition of work
UPDATE shards
 SET owner = :pid, taken_at = NOW()
WHERE id = (SELECT id FROM shards
            WHERE owner IS NULL OR taken_at < NOW() - INTERVAL '5 minutes'
            ORDER BY id LIMIT 1);

The second condition — about expiry — is mandatory here: if a worker dies, its partition has to return to the pool rather than stay claimed forever.

Lifecycle hooks

All four are optional and are called in the master.

protected function onWorkerExit(int $slot, int $pid, bool $crashed): void
{
  if ($crashed) {
      $this->metrics->increment('emails.worker_crashed');
  }
}

protected function tick(): void
{
  $this->metrics->gauge('emails.queue_depth', $this->outbox->pendingCount());
}

An exception inside a hook is caught and logged — a broken hook will not bring down the supervisor.

Worker states

In the status output every slot has its own state. It helps you see what the fleet is doing right now.

Slot numbers are stable: a restart returns the worker to the same slot, so #2 in the logs always means the same member of the fleet.

Control from code

A daemon inherits the same static control surface as a process, so the fleet can be started, stopped and inspected straight from the application.

use Main\Process\Emails;

$pid = Emails::dispatch();     // the supervisor went into the background
Emails::stop();                // stop the whole fleet

As with a process, start() blocks its caller — from the web use dispatch().

Fleet status

status() returns not merely the master’s state but a snapshot of the whole fleet: two fields are added to the usual process ones.

For each worker:

That lets you build your own checks and dashboards without parsing console output:

use Flytachi\Winter\Kernel\Process\Activity;

$status = Emails::status();

if ($status === null) {
  Emails::dispatch();                         // the fleet is down — raise it
  return;
}

$busy = array_filter(
  $status->workers,
  fn($w) => $w->activity === Activity::BUSY,
);

// every worker busy for over a minute — worth a look
if (count($busy) === count($status->workers)) {
  $this->alerts->fire('Emails: the whole fleet is busy');
}

Control from the console

php call daemon list                        # every daemon and its fleet size
php call daemon main.process.Emails         # the supervisor in the current terminal
php call daemon main.process.Emails start -d    # in the background
php call daemon main.process.Emails stop        # graceful stop of the whole fleet
php call daemon main.process.Emails status      # state + the worker table
php call daemon main.process.Emails status -v   # plus the master's resource usage

The command alias is call dmn.

Examples

A fixed-size fleet

The commonest case: there is steadily plenty of work and autoscaling is not needed.

<?php

namespace Main\Process;

use Flytachi\Winter\DI\Attribute\Autowired;
use Flytachi\Winter\Kernel\Process\Stereotype\Daemon;

class Webhooks extends Daemon
{
  #[Autowired] private WebhookRepository $hooks;
  #[Autowired] private HttpClient $http;

  protected int $replicas = 8;
  protected float $grace = 20.0;
  protected float $livenessTimeout = 90.0;

  protected function workerRun(): void
  {
      while ($this->isRunning()) {
          $hook = $this->hooks->takeNext();

          if ($hook === null) {
              $this->sleep(1);
              continue;
          }

          $this->markBusy();

          try {
              $this->http->post($hook->url, $hook->payload);
              $this->hooks->markDelivered($hook->id);
          } catch (\Throwable $e) {
              $this->hooks->retryLater($hook->id);
              $this->logger->warning('webhook failed', ['id' => $hook->id]);
          } finally {
              $this->markIdle();
          }
      }
  }
}

Three things without which the fleet works worse: markBusy() stops a worker being retired mid-delivery and resets the state between tasks; retryLater() in the catch returns the task to the queue; $livenessTimeout catches hung HTTP calls.

Autoscaling by queue depth

<?php

namespace Main\Process;

use Flytachi\Winter\DI\Attribute\Autowired;
use Flytachi\Winter\Kernel\Process\Daemon\ScalingPolicy;
use Flytachi\Winter\Kernel\Process\Stereotype\Daemon;

class Imports extends Daemon
{
  #[Autowired] private ImportQueue $queue;

  protected int $replicas = 2;          // the base size
  protected float $grace = 120.0;       // an import must not be cut in half

  protected function desiredReplicas(): int
  {
      return min(16, max(2, (int) ceil($this->queue->pendingCount() / 50)));
  }

  protected function scaling(): ScalingPolicy
  {
      return new ScalingPolicy(
          scaleUpDelay:           5.0,    // a spike must hold for 5 s
          scaleDownStabilization: 300.0,  // shrink only after 5 minutes of quiet
          scaleStep:              4,      // no more than 4 workers at a time
      );
  }
}

The settings here suit the particular job: an import is long, so shrinking should be unhurried, and growth is capped by a step so the database is not handed sixteen connections at once.

Note that there is no workerRun() here — so the body comes from $workerClass. If that is not set either, the daemon crashes on every fork.

One worker, but supervised

A daemon makes sense with a single replica too: you get restarts after crashes and the liveness watchdog, neither of which an ordinary process has.

class SnmpPoller extends Daemon
{
  protected int $replicas = 1;
  protected float $livenessTimeout = 60.0;

  protected function restart(): RestartPolicy
  {
      return new RestartPolicy(mode: RestartMode::ALWAYS, backoff: 5.0);
  }

  protected function workerRun(): void { /* ... */ }
}

ALWAYS fits better than the default here: a poller should never exit cleanly at all, so if it did, something is wrong and it must come back regardless.

Next

• Processes — the worker body primitives: loop, pauses, units of work
• Components — where #[EnableDaemon] is declared
• Scheduler — when the work is tied to the clock
• Logging — where the supervisor writes
• Actuator / Health — watching the application from outside