Console administration

Monitor scheduled jobs with heartbeats

Create a heartbeat monitor, install the WarnFire client, and report scheduled job results.

Use a heartbeat to detect a scheduled job that fails, runs too long, or never starts. Each monitor belongs to a service and uses its own key. A missing check-in opens an incident through that service’s normal escalation policy.

For every client command and option, see the WarnFire CLI reference .

Before you begin

Create the owning service and integration path and confirm its escalation policy reaches the intended responders. You need tenant-owner or tenant-administrator access to create a heartbeat monitor.

Create a monitor

Automate through the public API

Use a tenant API key with config:read for reads and config:manage for writes, or an authorized tenant session. Send the credential as Authorization: Bearer <credential> to https://api.warnfire.com. All management operations below are scoped to that credential’s workspace.

Creation returns the monitor’s ping key once. Updating a monitor preserves its service and key. Revocation disables that key. Resume does not replay skipped notifications. For exact request and response schemas, use the API Explorer or public OpenAPI document . Reusable exit-code sets and monitor groups are also documented there.

Job check-ins use the separate secret ping URL rather than the management bearer credential: /v1/ping/tenants/{tenant}/{key}/{slug}. GET and POST support success check-ins, /start?rid=<UUID>, /success?rid=<UUID>, /fail?rid=<UUID>, and numeric exit-code actions from 0 through 255. Reuse the run UUID for its completion. Optional diagnostic check-in bodies are limited to 16384 UTF-8 bytes.

POST diagnostic notes to /log?rid=<run UUID>&nid=<note UUID> using UTF-8 plain text, a JSON string, or a nonempty JSON object, limited to 4096 bytes. Reuse the note UUID only for an identical retry. Existing key-only URLs remain supported. Treat ping URLs as secrets; sending a check-in never creates a monitor.

Create in the console

1. Open Configuration → Heartbeats → Create and select the service that owns the job.
2. Give the monitor a name and slug. Choose an interval, cron, or Schedule expression input. Set the timezone, grace period, run-time limit, and failure tolerance for the actual job. Select its Exit-code set, or keep the standard interpretation: 0 succeeds and other codes fail.
3. Select Create monitor. Copy the ping URL immediately; its key appears only once. Store the URL as a secret. Do not put it in a crontab, a shell command argument, or a repository.
4. Open Monitors to see the current state, last check-in, and next deadline. History shows completed runs and bounded diagnostic notes.

Validate or change the schedule

Choose Schedule expression to enter any of these forms:

Intervals accept whole numbers of seconds, minutes, hours, days, or weeks, from one minute through 365 days. They start when the monitor is created or its schedule changes; successful check-ins advance the next deadline by that interval. every 1 day is 24 elapsed hours, so use at HH:MM for a daily local time that follows daylight-saving changes. Times use the 24-hour HH:MM format. The form shows the interpreted schedule and rejects invalid input before saving. Existing monitors retain their interval or cron settings when reopened for edit; choose Schedule expression to change them with the shorthand input.

Calendar schedules require the scheduler’s timezone. If a spring clock jump skips the chosen time, the deadline moves to the first valid instant after the jump. During the fall clock change, a repeated local time counts once. After saving, Monitors → Next deadline shows the server’s actual expected time. Grace applies after that deadline as before. Changing these settings does not install or change the job on its host; the CLI scheduler uses its own syntax.

Configure reusable exit-code sets

Exit codes describe a job’s result, and their meanings depend on the job. Create a named Exit-code set in your workspace and assign it to each monitor that uses those meanings. Several monitors can share one set. Without an assigned set, 0 is successful and every nonzero code is failed.

Create and assign a set

1. Open Configuration → Heartbeats → Exit-code sets and create a set. Give it a name that identifies the job or tool it describes.
2. Add rules for individual codes or inclusive ranges. For a single code, use the same start and end value. Codes must be whole numbers from 0 to 255; ranges must not overlap. Each rule needs a Success, Warning, Error, or Critical severity and a description explaining that result.
3. Write a fallback description and choose Error or Critical for codes with no matching rule. An unmapped code is always Failed, including 0 if you omit it from your custom set. Add explicit rules for all expected successful results.
4. Save the set. In Create or Monitors → Edit, select it in Exit-code set and save the monitor.

For example, a job could use these meanings:

Use your job’s documented meanings. Another tool can interpret the same number differently. A successful nonzero result needs an explicit rule; changing a description alone does not change the outcome.

Edit, copy, or delete a set

Before saving a shared set, review the monitors assigned to it: the change affects future completions for all of them. Copy a set and assign the copy when one job needs different rules without changing its peers.

WarnFire uses the monitor’s assigned set and current saved revision when the completion arrives, including a job that started before the edit. Each run keeps its raw exit code and a snapshot of the set name, revision, outcome, severity, and description used at completion. Editing, renaming, or reassigning a set does not reinterpret earlier runs. A configuration edit alone does not resolve an existing incident.

Before deleting a set, reassign its monitors or return them to the standard interpretation. Deletion is refused while a monitor still uses it. Workspace owners and administrators manage sets; configuration viewers can inspect them without changing them.

How mappings affect paging and recovery

Warnings are accepted completions: they reset consecutive failure counts and can resolve an existing heartbeat incident. Critical does not bypass failure tolerance. The service’s escalation policy and applicable routing rules still control recipients and channels; choosing Critical does not automatically create a different paging policy. The severity that first opens an incident controls its original routing; editing the set does not reroute an open incident.

The Standard preset accepts only 0. The editable Robocopy preset accepts 0 and 1, records 2–7 as non-paging warnings, and treats 8–255 as errors. Microsoft documents values at least 8 as indicating a copy failure; see the Robocopy exit-code reference . Review presets against the actual job before saving or assigning them.

A mapped Failed completion counts toward Reported failures before paging. A mapped Success completion resets that failure count and can resolve the heartbeat incident, even when its raw code is nonzero. The description explains the outcome; the selected severity is recorded separately.

Mappings apply to completions with a numeric exit code. A state-only success or failure check-in keeps its reported outcome. Mappings do not change missed check-ins, unfinished-run timeouts, maintenance, or notification surge protection. A mapped success does not resume alerts paused by the surge limit: Resume alerts remains a manual action.

The client still returns the job’s original exit code to the shell or scheduler. A mapped success does not turn that process exit into zero or change scheduler retries. No client reinstallation is needed to change a set; review the scheduler’s success criteria separately.

Protect responders from notification surges

Set Pause alerts after notifications when creating or editing a monitor. Choose a limit from 1 to 100; 0 disables the limit and is the default for new and existing monitors. This controls incident escalation notifications, independently of Missed check-ins before paging and Reported failures before paging, which control when an incident opens.

What counts toward the limit

The counter counts distinct incident escalation dispatches for this monitor across incidents, escalation steps, recipients, and channels. One email to Alice and one push to Bob count as two. Two channels to Alice also count as two. A retry of the same notification counts once. A dispatch counts when WarnFire permits the provider call, even if the provider later fails or its result is uncertain; the count does not prove that a person received the notification. Notifications stopped before the provider boundary do not consume this limit. While the limit is disabled, dispatches do not increase the counter.

For a limit of 3, WarnFire permits the third dispatch and immediately marks the monitor Alerts paused. Subsequent new notifications are skipped. Concurrent sends share the same counter; notifications already permitted, including their permitted retries, may finish after the pause appears.

What continues while alerts are paused

Check-ins, deadline evaluation, incident creation, recovery, and run history continue. A successful check-in can resolve an incident while Alerts paused remains visible. The normal, non-interrupting resolution notice can still tell responders who were already paged that the incident is over; those notices and service integration messages are outside this escalation limit. Skipped notifications are recorded on the incident timeline with the reason heartbeat alerts paused. This is different from monitor maintenance, which suppresses deadline evaluation during its configured window.

There is no automatic timeout or recovery-based reset. The counter persists through quiet periods, successful runs, incident resolution, and schedule edits. Raising, lowering, or disabling the limit preserves both the counter and any existing pause. Lowering a limit below the counter pauses new notifications at the next dispatch check.

Resume alerts manually

1. Investigate the recurring failures or escalation configuration before restoring notifications.
2. Open Configuration → Heartbeats → Monitors. Check Alerts paused and the dispatch count on the affected monitor.
3. Select Resume alerts. Tenant owners and administrators can resume; operators with read-only configuration access cannot.

Resume clears the pause and resets the counter to zero. It enables future notifications without changing the incident, schedule, ping URL, or history. Skipped notifications are not replayed, and resume does not restart an exhausted escalation. If the limit remains enabled, a new set of dispatches can reach it and pause alerts again. Disabling the limit alone does not resume an already paused monitor; explicitly select Resume alerts as well.

For API clients, set notification_surge_limit in the heartbeat create or update request. Omitted create values default to 0; omitted update values preserve the setting. Monitor responses include notification_surge_limit, consecutive_notifications, and optional alerts_paused_at. Use POST /v1/tenant/config/heartbeats/{id}/resume-alerts to resume (configuration management permission required). A successful request returns 204. Resuming a monitor that is already active is a no-op and does not reset its current counter.

Separate paging thresholds

Missed check-ins before paging counts consecutive scheduled occurrences whose deadline and grace period both pass without a completed check-in. With 3, the first two misses are shown as late and the third opens an incident. 1 preserves the previous behavior. Evaluator retries do not count the same occurrence again; an evaluator returning after an outage counts elapsed occurrences up to the threshold. Interval occurrences remain anchored to the previous deadline while waiting for a check-in; cron uses the selected timezone and the daylight-saving rules above.

After an incident opens, the next deadline continues advancing on that schedule. The missed count remains capped at the paging threshold, and the original missed deadline stays in the incident evidence. Later deadlines do not open duplicate incidents or imply recovery. The Monitors list refreshes every ten seconds while visible.

Reported failures before paging counts consecutive explicit failed completions after applying the assigned exit-code set. Without a set, a nonzero CLI exit status counts as failed. With 3, an incident opens on the third reported failure. The monitor shows failed from the first failure, even below the paging threshold.

A completed check-in resets the missed count and starts the next expectation. A reported failure counts toward the explicit-failure threshold, rather than also being treated as a missing check-in. Only a successful completion resets the explicit-failure count and resolves an open heartbeat incident. An unfinished run still uses its maximum runtime; missed tolerance does not delay a runtime alert. Starting a run alone does not reset the missed count.

Changing a missed threshold keeps the existing count and deadline. Lowering it to the already accumulated count causes paging on the next evaluator pass. Changing the schedule starts a fresh missed expectation when no incident is open; changing settings never resolves an existing incident. Maintenance does not count misses and starts a fresh expectation after its end.

Cron schedules use five fields: minute, hour, day of month, month, and day of week. For example, 0 2 * * * expects a daily run at 02:00 in the selected IANA timezone. Choose the same timezone as the actual scheduler. The console checks syntax and timezone before saving; the server also checks calendar feasibility. Changing the monitor does not install or change the job on its host.

Use Monitors → Edit to change a schedule. A schedule change computes a fresh next deadline and clears an unalerted late state caused by the old deadline. It preserves failed or unfinished runs and open incidents. A successful check-in, rather than a configuration edit, supplies recovery evidence.

Organize monitors into groups

Open Configuration → Heartbeats → Groups to create a named group, then check each heartbeat that belongs in it. A heartbeat can belong to several groups, including groups that contain monitors owned by different services. Renaming a group keeps its identity and memberships. Deleting a group removes only that grouping; the heartbeat monitors and their paging services remain.

The service selected on each monitor still controls incident routing and escalation. Group membership alone does not publish a monitor on a public status page. Status pages need an explicit mapping from the monitor’s owning service to a customer component. Group membership does not create that mapping.

Install the client

The installers and SHA-256 checksums are linked on the Create tab. Download the installer for the machine running the job. Each package includes the client; you do not need Go or a separate binary download.

macOS

The macOS download appears only after the client for the current release has been signed and notarized. If the Clients page says it is temporarily unavailable, wait for that build before installing.

Run bash ~/Downloads/warnfire-macos-installer.sh in Terminal. The installer includes the Developer ID signed and notarized universal client for Apple Silicon and Intel Macs and checks its notarization before installing.

Linux

Run bash ~/Downloads/warnfire-linux-installer.sh. The installer selects the x86-64 or ARM64 client automatically.

Windows

Open WarnFire-Setup.exe. Setup installs for the current user and adds the installation directory to the user PATH. Open a new terminal, then run warnfire version. Use Installed apps → WarnFire Client → Uninstall to remove it. Automated installs and uninstalls accept /S for silent operation.

Install locations and upgrades

Both scripts verify the embedded package, set executable permissions and install warnfire in /usr/local/bin. They request an administrator password if needed. To use a different location without administrator access, append --bin-dir "$HOME/.local/bin"; that directory must be on your PATH, or use the full installed path. If the browser adds a suffix to a downloaded filename, use that filename, quoted when it contains spaces. Run warnfire version afterward.

Run a newer installer to upgrade. Stop running WarnFire jobs before upgrading on Windows. Installers preserve job configuration and heartbeat secrets. For cron and Windows Task Scheduler, use the full executable path because their environment may differ from your terminal.

Install and maintain a scheduled job

The client can install a cron entry or a systemd user timer for the account that runs the job. It writes the ping URL to a private 0600 file, creates a runner, and updates the scheduler. The URL does not appear in scheduler entries or command arguments. Install the client on the job host first, then copy the monitor’s ping URL from Create monitor.

Run warnfire job --help to see every option. Use a short local identifier for --name; rerun job install with the same name to update its schedule, command, or URL. --schedule uses the scheduler’s own syntax. The job command must have an absolute executable path and follows --.

cron on macOS or Linux

For a job that runs every day at 02:00 in the cron host’s local timezone:

pbpaste | warnfire job install --name nightly-billing --scheduler cron \
  --schedule '0 2 * * *' --url-stdin -- /usr/local/bin/my-job

pbpaste reads the copied URL on macOS. On Linux, use your clipboard utility or pipe a private URL file into stdin. You can also supply --url-file PATH; the CLI copies its contents into the managed private file and rejects a source file that other users can read. Do not put the URL itself in a command line.

The CLI edits only its marked entry in the current account’s crontab. It stores job files under ~/.config/warnfire/jobs/nightly-billing/ (or $XDG_CONFIG_HOME/warnfire/jobs/). The job runs as the account that invoked job install, so install it while logged in as the intended job account. Cron uses a small environment and its own timezone; give the job executable and any file paths as absolute paths. Cron may email uncaptured output according to the host’s mail setup; the WarnFire client also sends the bounded diagnostic tail described below.

systemd user timer on Linux

For a user timer with an explicit local calendar time:

warnfire job install --name nightly-billing --scheduler systemd \
  --schedule '*-*-* 02:00:00' --url-file /path/to/private-ping-url \
  -- /usr/local/bin/my-job

The CLI writes warnfire-nightly-billing.service and .timer in the current account’s ~/.config/systemd/user/, reloads the user manager, and enables the timer immediately. The timer starts the job on its next scheduled run; it does not run the job during installation. systemd Persistent=true catches up a missed calendar event when the timer next starts. To run after the account logs out, the host administrator may need to enable lingering for that account. Use warnfire job status nightly-billing to check the installation, and consult the account’s systemd user journal for job logs.

Inspect, change, or remove the installation

warnfire job list
warnfire job status nightly-billing
warnfire job remove nightly-billing

job list shows the local name, scheduler, schedule, and command, without the ping URL. job status checks that the cron entry exists or that the systemd timer is enabled and active; it does not prove that the last job run succeeded. Reinstall under the same name to change a job. To switch from cron to systemd, remove it first, then install it with the other scheduler. Remove stops future scheduled runs and deletes the CLI’s managed URL and runner. It does not delete the monitor or its history in WarnFire. Rotate or revoke the monitor’s key in the console if the old URL may have been exposed.

Scheduled Docker container

The CLI can install a cron entry or systemd user timer that runs an attached docker run and reports its exit status and bounded output. See the Docker heartbeat job guide for the complete setup and verification steps.

Kubernetes CronJob

Use the complete Kubernetes guide for EKS, AKS, GKE, access, image pulls, installation, verification, and troubleshooting. WarnFire publishes the CLI container separately; your job image needs only the job executable. Each CronJob needs its own heartbeat monitor and ping URL.

warnfire job install --name nightly-billing --scheduler kubernetes \
  --context production-cluster --namespace billing \
  --client-image docker.io/craigs/warnfire-cli:<release-sha> \
  --image registry.example.com/billing-jobs:2026-10-01 \
  --schedule '0 2 * * *' --timezone UTC \
  --url-file /path/to/private-ping-url -- /app/my-job
warnfire job status nightly-billing

The CLI applies a managed Secret and CronJob in the explicit context and namespace. An init container supplies the maintained WarnFire client image; the job container receives its ping URL from the Secret, and the client removes it from the child process environment. The URL is never a command argument. The default timezone is UTC; overlapping runs are forbidden and failed jobs are not retried automatically.

warnfire job list shows the locally recorded target. Reinstall the same name to update it; warnfire job remove nightly-billing removes its managed CronJob and Secret from the recorded context and namespace. job status verifies the resources exist and the CronJob is not suspended. It does not prove the last run succeeded; check Heartbeats → History and kubectl --context production-cluster -n billing get jobs for that. Keep the administrator workstation’s managed URL file private, and rotate the monitor key if it is lost or exposed.

Run a job once

To test without adding a schedule, supply the private URL file to run:

WARNFIRE_HEARTBEAT_CONFIG=/etc/warnfire/heartbeat-url \
  warnfire run -- /usr/local/bin/my-job

The client reports when the job starts and finishes, and exits with the job’s exit status. WarnFire interprets the numeric code using the monitor’s assigned exit-code set . Without a set, 0 succeeds and nonzero codes fail with Error severity. Writing to stderr alone does not mark a job as failed.

Both stdout and stderr pass through to the terminal or scheduler as usual. The client also captures their combined last 16 KiB and sends that diagnostic output with the completion check-in, whether the job succeeds or fails. This is a bounded tail, not a complete log or a live stream. Stdin passes through to the job without being recorded. Avoid printing secrets: captured output is sent to WarnFire.

The client does not pass the ping URL to the child process. A failed check-in is printed as a safe error, and the job still runs. The client accepts WARNFIRE_HEARTBEAT_URL instead of a file when a scheduler injects secrets securely; set exactly one of these variables. warnfire run --help documents the direct wrapper options.

Record event messages

Add text and structured JSON messages to the run independently of terminal output:

These options require client 1.2.0 or later.

warnfire run --message='Starting export' --json='{"stage":"export","rows":3}' -- /usr/local/bin/my-job

The client submits these messages after the start check-in, before launching the command. Each message allows 4096 UTF-8 bytes; a run allows 20 messages. JSON accepts a nonempty object or a text string. Printing JSON to stdout records terminal output; it does not create a structured event message.

To append a later message, copy the UUID from Heartbeats → History → Details:

warnfire log --run-id RUN_UUID --text 'Export completed'
warnfire log --run-id RUN_UUID --json '{"rows":150,"stage":"complete"}'

Use the same monitor URL configuration as the wrapper. Messages do not change run outcome, deadlines or incident state. A refused message returns an error.

Every run has a Details action, including runs with no messages. It shows the raw exit status, timestamps, output and event messages. For a mapped completion, check its saved outcome and explanation together with the set name and revision to understand the result. Standalone pings have no measured duration. An unfinished run that exceeds its ceiling is hung; its elapsed timer continues until completion is received. A late completion records the real outcome and duration while retaining the timeout timestamp. Incident payloads preserve the original run evidence observed when the incident was queued, even if the run completes before incident delivery.

Use Cronitor-style states

The same WarnFire ping URL supports either WarnFire path actions or Cronitor-style query states. The client uses path actions by default. To send the alternative state vocabulary, use:

WARNFIRE_HEARTBEAT_CONFIG=/etc/warnfire/heartbeat-url \
  warnfire run --protocol=cronitor -- /usr/local/bin/my-job

This sends state=run before the child starts, followed by state=complete or state=fail, with one series value and the child’s status_code. Existing instrumentation can send state=run|complete|fail|ok directly to the WarnFire URL. A series pairs start and terminal pings; without one, a terminal ping pairs only if there is exactly one running job. This is state compatibility on WarnFire’s endpoint; it does not send data to Cronitor’s service.

Verify it worked

Run the wrapped job once and verify its entry in Heartbeats → History. Check the raw exit status, recorded interpretation, last check-in, and next deadline. For a custom set, test an expected successful nonzero code, a mapped failure, and an unmapped code. Check the explanations, then edit the set and confirm that earlier runs retain their original interpretation. Test a failing job with the responders informed, then confirm the incident reaches the intended service and resolves after a successful run. A revoked monitor key stops accepting check-ins; create a replacement URL before retiring the old one.

Next steps

Use the configuration readiness check to test the service’s complete response path.

Generated ping URLs now include the workspace identity: /v1/ping/<workspace-id>~<key>/<slug>. The key must belong to that workspace; another workspace can use the same slug independently. Existing key-only URLs continue to work. Creating a monitor or rotating its key shows a separate one-time URL result above the monitor list; rotation does not reopen the monitor editor.

Read recovery evidence

A resolved incident shows its recorded resolution time, the duration from incident creation to resolution, and a View recovery event link when the monitoring source supplied a resolution event. Completed runs also show their successful check-in time. This describes the recorded incident; a later unrelated check-in does not replace its recovery evidence.

Inspect a recorded run

Open History, then select Details on a run. Run details open in a drawer on the right, keeping the history table in place. The drawer shows the run identity, start and finish times, outcome, duration, captured terminal output, and event messages. Close it with Close, Escape, or the shaded area outside the drawer. Keyboard focus returns to the Details button.

The Clients tab shows the independently pinned client version and full SHA-256 checksums directly on each platform card. Check the installer and binary checksums against your download. Containers show the manifest digest used by the pull command. macOS and Linux/Windows/container versions can differ; these are client versions, independent of the console’s server build.