Skip to main content

Trading Systems

Designing Alerts, Limits, and Recovery for Crypto Automation

Coordinate software limits, venue health, incident alerts, degraded states, and reconciliation before resuming a crypto automation workflow.

By Sunbot Labs

Updated

5 min read

Multi-exchange crypto automation using component-aware alerts, explicit limits, reconciliation, and controlled recovery.

How the workflow connects

  1. 1Detect a limit or health condition
  2. 2Prevent the affected new action
  3. 3Create one owned incident
  4. 4Query authoritative venue state
  5. 5Recover through guarded resume

Crypto venues operate continuously, but an automation system still needs bounded operating conditions. Rate limits, stale feeds, rejected orders, credential changes, maintenance windows, and stream gaps should move the system into explicit degraded or paused states with an accountable recovery path.

Separate configured limits from venue constraints

Client-defined software limits may restrict instruments, request size, order frequency, or aggregate exposure. Venue constraints include precision, minimums, permissions, and rate limits. Record which layer prevented an action and preserve the request context without retrying a policy rejection.

Model degraded operation by component

A public market stream can fail while private order updates remain connected. One exchange can be unavailable while another is healthy. Track component state and define which strategies or actions depend on each component rather than presenting one global green or red indicator.

  • Market-data freshness
  • Private account-stream state
  • REST request health and rate-limit budget
  • Clock synchronization
  • Credential and permission status

Turn alerts into incidents, not message storms

Group repeated detections by venue, connection, and condition. Create one incident with first and latest occurrence, affected workflows, current owner, acknowledgement, and escalation time. Update it as evidence changes rather than sending a separate notification for every reconnect attempt.

Require reconciliation before recovery

After an uncertain submission or connection gap, retrieve current orders, executions, balances, and positions supported by the venue. Compare them with local records, resolve discrepancies, confirm fresh data and healthy permissions, then request resume through the normal transition guard.

Separate a component failure from the system response

If one exchange market-data stream becomes stale, the affected strategy or instrument may need to stop creating new requests while account and order monitoring continue. A second healthy exchange does not make the first connection healthy, and a global shutdown may be unnecessary if the operating model supports isolated degradation.

Define the detection condition, affected scope, automatic protective action, alert owner, escalation time, and clearing test for each failure. Limits should identify whether they apply per request, strategy, account, instrument, venue, or time window. The dashboard should show the configured value and the observation that crossed it.

One degraded exchange connection isolated from healthy components with grouped alert evidence.

Visual guide

From detected condition to controlled recovery

An alert is useful when it identifies scope, protective action, owner, and the evidence required to resume.

  1. 1

    Detect

    Observe stale data, rejected requests, disconnect, or breached software limit.

  2. 2

    Contain

    Pause the affected scope and preserve permitted monitoring.

  3. 3

    Notify

    Create one owned incident with severity based on consequence.

  4. 4

    Investigate

    Review connection, events, open orders, and local state.

  5. 5

    Reconcile

    Compare venue facts before clearing uncertainty.

  6. 6

    Recover

    Resume through authorized guards and record the result.

Protective software limits support controlled operation; they do not remove market or financial risk.

Run recovery drills before they are needed

Test a dropped stream, expired credential, rate limit, partial venue outage, delayed order event, uncertain submission, and service restart with an open order. Confirm which workflows pause automatically, what remains visible, who receives the alert, and whether a person can determine the next safe action from the retained evidence.

A resume button should recheck the original clearing conditions. Closing an alert must not automatically enable submission. Review alert storms, repeated manual overrides, long reconciliation times, and incidents that reopen shortly after recovery. These patterns reveal weak thresholds, missing state, or an operating responsibility that has not been assigned.

  • Use sandbox, demo, or controlled test conditions where available
  • Record expected component and overall system states
  • Verify notifications without exposing secrets or account data
  • Confirm open-order and position reconciliation after restart
  • Keep recovery decisions separate from trading strategy decisions
A paused workflow moving through reconciliation and recovery checks before an authorized resume.

Practical questions

Questions that often come up

Should automation cancel orders when a data feed becomes stale?

That is a client-defined operational policy. The software should prevent unsafe new decisions, surface affected orders, and only perform cancellation when explicitly specified and authorized.

Can reconnect success automatically close an incident?

A connection can recover while missed order or account updates remain. Close only after the defined health and reconciliation checks pass.

Work with Sun Cluster

Planning a similar system for your organization?

Sun Cluster builds private crypto automation with exchange integrations, client-defined limits, operational alerts, and recovery workflows.