Skip to content

Configure restart policy

By default a service is not restarted: if its WithStart returns an error, the controller records it and moves on. To make a service self-healing, attach a RestartPolicy with WithRestartPolicy.

Enable restarts

c.Register("worker",
    controls.WithStart(runWorker),
    controls.WithRestartPolicy(controls.RestartPolicy{
        MaxRestarts:    5,
        InitialBackoff: time.Second,
        MaxBackoff:     30 * time.Second,
    }),
)

Now, if runWorker returns a genuine error while the controller context is still live, the supervisor waits a backoff interval and starts it again — up to MaxRestarts consecutive failures.

The policy fields

type RestartPolicy struct {
    MaxRestarts            int           // cap on consecutive failures (0 = unlimited)
    InitialBackoff         time.Duration // first wait before restart (default 1s)
    MaxBackoff             time.Duration // backoff ceiling (default 30s)
    HealthFailureThreshold int           // consecutive Status failures before restart
    HealthCheckInterval    time.Duration // how often Status is polled (default 10s)
    RestartResetInterval   time.Duration // healthy run length that resets the counter (default 30s)
}
  • InitialBackoff / MaxBackoff. The wait before each restart doubles from InitialBackoff, capped at MaxBackoff. A zero field selects its default (1s / 30s).
  • MaxRestarts. Caps consecutive failures (see below), not lifetime restarts. When exceeded, the supervisor gives up on the service and forwards a max restarts exceeded error. 0 means unlimited.

Health-driven restarts

A service whose WithStart returns nil immediately (it serves in a background goroutine) never "exits", so there is no error to restart on. To supervise such a service, combine a WithStatus probe with a HealthFailureThreshold:

c.Register("streamer",
    controls.WithStart(startStreamerInBackground), // returns nil once serving
    controls.WithStatus(func() error {
        return streamer.Ping() // non-nil == unhealthy
    }),
    controls.WithRestartPolicy(controls.RestartPolicy{
        HealthFailureThreshold: 3,               // 3 consecutive bad pings...
        HealthCheckInterval:    5 * time.Second, // ...polled every 5s
        MaxRestarts:            10,
    }),
)

The supervisor polls Status every HealthCheckInterval. Each failure increments a counter; a single success resets it. When the counter reaches HealthFailureThreshold, the service is stopped and restarted (subject to MaxRestarts and backoff).

HealthFailureThreshold requires a Status probe. Health monitoring only runs when both HealthFailureThreshold > 0 and a WithStatus func are present. Without them, a clean-start service simply runs until shutdown.

Consecutive failures and the reset window

MaxRestarts counts consecutive failures, not the lifetime total. A service that fails, restarts, and then runs healthily for RestartResetInterval has its counter reset to zero — so a service that flakes once an hour is never permanently killed by an old failure.

The reset window defaults to 30s (DefaultRestartResetInterval). Tune it with the policy field, or with WithRestartResetInterval:

c.Register("flaky",
    controls.WithStart(run),
    controls.WithRestartPolicy(controls.RestartPolicy{MaxRestarts: 3}),
    controls.WithRestartResetInterval(2*time.Minute), // must run 2m clean to reset
)

WithRestartResetInterval implies a policy. If the service has no RestartPolicy yet, this option creates a default one so the interval takes effect. Order does not matter — apply it before or after WithRestartPolicy.

Exempt expected terminal errors

Some errors are a normal end of run, not a failure — the canonical example is http.ErrServerClosed, returned by ListenAndServe after a graceful Shutdown. Register a controller-wide predicate with WithValidError so the supervisor treats a matching error as a clean stop: it is neither counted toward the restart total nor forwarded on the error channel.

c := controls.NewController(ctx,
    controls.WithValidError(func(err error) bool {
        return errors.Is(err, http.ErrServerClosed) ||
            errors.Is(err, context.Canceled)
    }),
)

WithValidError is a controller option (passed to NewController), and the predicate applies to every service. Contrast it with the per-service restart options above, which are ServiceOptions passed to Register.

Observe restart counts

info, _ := c.GetServiceInfo("worker")
fmt.Printf("restarts=%d lastErr=%v\n", info.RestartCount, info.Error)