Skip to main content

Disconnect or end a connection epoch

This page covers the boundary at which one API connection epoch ends. It distinguishes an operator-requested disconnect, a Python or TWS/IB Gateway process restart, a broken local socket, an upstream nightly reset, and a socket-port reset. The next workflow step, reconnect and reconciliation, determines when a new epoch is safe to use.

Classify the boundary before acting​

BoundaryWhat the application observesWhat to do now
Deliberate IB.disconnect()The local API socket closes and disconnectedEvent is emitted.Persist an epoch handoff first, then disconnect. Do not treat disconnect as cancellation.
Python process exit or crashIn-memory owners, caches, and event handlers disappear.A graceful exit uses the same handoff; a crash must leave durable recovery work discoverable.
TWS/IB Gateway restart or exitThe local socket ends. Any untransmitted order is not a durable broker commitment.End readiness immediately and enter reconnect/reconciliation.
Local network or socket lossdisconnectedEvent follows the ready-client close path.Use the last durable checkpoint; wrapper caches have already been reset on this pinned path.
Upstream reset (1100)TWS/IB Gateway lost IBKR connectivity, but the local API socket may remain connected.Enter degraded mode. Await 1101 or 1102; do not restart the local process reflexively.
Socket-port reset (1300)TWS reports the new port and drops the API connection.Reconfigure the port, then reconnect as a new epoch.

Codes 1101 and 1102 distinguish restored upstream connectivity with data lost versus maintained. They do not prove application invariants by themselves. Code 1101 requires subscription replay; both paths still require critical order-state reconciliation before declaring readiness.

disconnectedEvent has two cache-orderings​

Pinned ib_async 2.1.0 deliberately exposes different ordering on its two disconnect paths:

IB.disconnect()
  -> client socket disconnect/reset
  -> disconnectedEvent
  -> wrapper.reset()

ready socket closes unexpectedly
  -> pending subscription events become done
  -> wrapper.connectionClosed()
  -> pending request futures fail and wrapper.reset()
  -> disconnectedEvent

Therefore, a disconnectedEvent handler must not read IB.trades(), IB.fills(), ticker maps, account caches, pending request maps, or subscription maps as its shutdown snapshot. Those values happen to remain visible during the deliberate event but are already gone during the socket-loss event. Persist ownership and correlation keys while the epoch is healthy; use the event only to clear readiness, mark the epoch ended, and enqueue recovery from the last durable checkpoint.

The public event is wired to the client's apiEnd event for a ready socket loss. IB.disconnect() also emits it explicitly. Calling IB.disconnect() after the client socket is already closed returns None and does not emit another event.

There is another important asymmetry. Unexpected ready-socket loss calls setEventsDone() and connectionClosed() before reset, so subscription events become done and pending request futures fail. Deliberate IB.disconnect() does neither: it resets the wrapper maps directly. An external task still awaiting a request future or subscription event can otherwise remain unresolved after the mapping that owned it disappears. Before deliberate disconnect, explicitly finish or cancel those awaiters, or durably hand their recovery to another owner and detach the local waiters.

What must cross the boundary​

Before an operator-requested disconnect or restart, durably record:

  • the connection-epoch ID, client ID, intended reason, and recovery owner;
  • each outstanding order's application intent ID, orderRef, account scope, latest status, and known permId/order ID/client ID values;
  • all observed execution IDs and whether their commission records are complete;
  • every subscription's logical key and replay specification, not its mutable Ticker or bar-list object;
  • unresolved request or write outcomes that make the epoch poisoned;
  • the last successful account-scope, contract, and readiness checkpoints needed to validate a new epoch.

IB.disconnect() sends no cancel-order request. A transmitted order may continue working after the API client disconnects, and a fill may occur while the application is offline. Explicitly cancel and confirm orders when the business action is cancellation; otherwise hand them to the next reconciliation owner. Never rely on an untransmitted order across a TWS/IB Gateway restart.

Mutable ib_async session objects do not cross the boundary. Deliberate disconnect resets account values, summaries, portfolios, positions, trades, fills, tickers, subscriptions, pending request futures/results, request-to-contract mappings, and timeout state. A new connection epoch must rebuild ownership from durable intent and fresh broker observations rather than reusing old objects.

Minimal deliberate shutdown​

This example is intentionally read-only and is safe only after the caller has proved there are no outstanding requests, subscription waiters, or orders. It persists the recovery marker before closing the API connection and calls disconnect() even if readiness was already cleared; the method safely returns None when its client socket is closed.

import json
from pathlib import Path
from typing import Any

from ib_async import IB


def persist_handoff(path: Path, record: dict[str, Any]) -> None:
    temporary = path.with_suffix(".tmp")
    temporary.write_text(json.dumps(record, sort_keys=True), encoding="utf-8")
    temporary.replace(path)


def stop_epoch(
    ib: IB,
    path: Path,
    epoch_id: str,
    client_id: int,
    reason: str,
) -> str | None:
    if not epoch_id or client_id < 0 or not reason.strip():
        raise ValueError("epoch_id, non-negative client_id, and reason are required")
    persist_handoff(
        path,
        {
            "kind": "disconnect-prepared",
            "connectionEpoch": epoch_id,
            "clientId": client_id,
            "reason": reason,
            "recoveryOwner": "reconnect-worker",
            "recoveryRequired": True,
            "outstandingOrders": [],
            "executionCommissions": [],
            "subscriptions": [],
            "unresolvedRequests": [],
            "unresolvedWrites": [],
            "readiness": {"state": "not-ready"},
        },
    )
    return ib.disconnect()

The atomic rename makes the small marker durable as one complete file, but production storage also needs an actual durability guarantee appropriate to its database or filesystem. An empty order list is safe only because this example never creates orders.

Production ownership pattern​

The coordinator below requires a complete durable ownership checkpoint and an awaited async-owner finalizer before disconnect. That finalizer must finish or cancel local awaiters, or durably hand them off and detach them; returning only a label is not a substitute for doing the work. The coordinator never snapshots wrapper caches in disconnectedEvent. A failure before the socket close propagates while the epoch remains connected; failure to persist an unexpected-loss marker poisons the owner so callers cannot mistake the loss for a clean shutdown.

from collections.abc import Awaitable, Callable
from copy import deepcopy
from typing import Any

from ib_async import IB


Record = dict[str, Any]


class EpochDisconnectOwner:
    def __init__(
        self,
        ib: IB,
        epoch_id: str,
        recovery_owner: str,
        persist: Callable[[Record], None],
    ) -> None:
        if not epoch_id or not recovery_owner:
            raise ValueError("epoch_id and recovery_owner are required")
        self.ib = ib
        self.epoch_id = epoch_id
        self.recovery_owner = recovery_owner
        self.persist = persist
        self.ready = True
        self.poisoned = False
        self.recovery_required = False
        self.clean_disconnect_observed = False
        self._checkpoint_id: str | None = None
        self._request_owner_keys: set[str] = set()
        self._subscription_owner_keys: set[str] = set()
        self._deliberate = False
        self._event_seen = False
        self._ended = False
        self.ib.disconnectedEvent += self._on_disconnected

    def checkpoint(
        self,
        checkpoint_id: str,
        client_id: int,
        outstanding_orders: list[Record],
        execution_commissions: list[Record],
        subscriptions: list[Record],
        unresolved_requests: list[Record],
        unresolved_writes: list[Record],
        readiness: Record,
    ) -> None:
        if self._ended or self.poisoned:
            raise RuntimeError("connection epoch cannot accept a checkpoint")
        if not checkpoint_id or client_id < 0:
            raise ValueError("checkpoint_id and non-negative client_id are required")
        request_owner_keys = self._owner_keys(unresolved_requests, "requests")
        subscription_owner_keys = self._owner_keys(subscriptions, "subscriptions")
        record: Record = {
            "kind": "epoch-ownership-checkpoint",
            "connectionEpoch": self.epoch_id,
            "checkpointId": checkpoint_id,
            "clientId": client_id,
            "recoveryOwner": self.recovery_owner,
            "outstandingOrders": deepcopy(outstanding_orders),
            "executionCommissions": deepcopy(execution_commissions),
            "subscriptions": deepcopy(subscriptions),
            "unresolvedRequests": deepcopy(unresolved_requests),
            "unresolvedWrites": deepcopy(unresolved_writes),
            "readiness": deepcopy(readiness),
        }
        self.persist(record)
        self._checkpoint_id = checkpoint_id
        self._request_owner_keys = request_owner_keys
        self._subscription_owner_keys = subscription_owner_keys

    async def disconnect(
        self,
        reason: str,
        finalize_async_owners: Callable[[], Awaitable[Record]],
    ) -> str | None:
        if self._ended:
            return None
        if not self._checkpoint_id:
            raise RuntimeError("persist an ownership checkpoint before disconnect")
        if not reason.strip():
            raise ValueError("disconnect reason is required")
        self.ready = False
        self.recovery_required = True
        try:
            finalization = await finalize_async_owners()
            self._validate_finalization(finalization)
            self.persist(
                {
                    "kind": "disconnect-prepared",
                    "connectionEpoch": self.epoch_id,
                    "checkpointId": self._checkpoint_id,
                    "recoveryOwner": self.recovery_owner,
                    "reason": reason,
                    "asyncOwnerFinalization": deepcopy(finalization),
                    "recoveryRequired": True,
                }
            )
        except BaseException:
            self.poisoned = True
            raise
        self._deliberate = True
        try:
            status = self.ib.disconnect()
        except BaseException:
            self.poisoned = True
            self.recovery_required = True
            raise
        finally:
            self._deliberate = False
        self._ended = True
        self.clean_disconnect_observed = status is not None and self._event_seen
        self.recovery_required = True
        return status

    @staticmethod
    def _owner_keys(records: list[Record], label: str) -> set[str]:
        keys = {str(record.get("ownerKey", "")).strip() for record in records}
        if "" in keys or len(keys) != len(records):
            raise ValueError(f"{label} require unique non-empty ownerKey values")
        return keys

    def _validate_finalization(self, finalization: Record) -> None:
        allowed_states = {"quiesced", "handed-off", "quiesced-and-handed-off"}
        allowed_outcomes = {"completed", "cancelled", "handed-off"}
        if finalization.get("state") not in allowed_states:
            raise RuntimeError("async owners are not quiesced or durably handed off")
        for field, expected in (
            ("requests", self._request_owner_keys),
            ("subscriptions", self._subscription_owner_keys),
        ):
            outcomes = finalization.get(field)
            if not isinstance(outcomes, list):
                raise RuntimeError(f"finalization must list {field}")
            actual = self._owner_keys(outcomes, field)
            if actual != expected or any(
                outcome.get("outcome") not in allowed_outcomes
                for outcome in outcomes
            ):
                raise RuntimeError(f"finalization does not cover checkpointed {field}")

    def _on_disconnected(self) -> None:
        self.ready = False
        self._event_seen = True
        if self._deliberate or self._ended:
            return
        self._ended = True
        self.recovery_required = True
        if self._checkpoint_id is None:
            self.poisoned = True
        try:
            self.persist(
                {
                    "kind": "unexpected-disconnect",
                    "connectionEpoch": self.epoch_id,
                    "checkpointId": self._checkpoint_id,
                    "recoveryOwner": self.recovery_owner,
                    "recoveryRequired": True,
                }
            )
        except BaseException:
            self.poisoned = True

    def close(self) -> None:
        self.ib.disconnectedEvent -= self._on_disconnected

The checkpoint write happens before the in-memory checkpoint pointer changes. Every unresolved request and subscription needs a unique ownerKey. disconnect() clears readiness, then awaits the application-specific finalizer and verifies that its requests and subscriptions outcomes cover exactly those keys with completed, cancelled, or handed-off. The callback must be idempotent because a subsequent durable write can fail. Any finalizer or pre-close write failure leaves the API socket open but the epoch degraded, poisoned, and recovery-required; it must not accept new work. The disconnect-prepared record deliberately remains recovery-required: the following reconciliation step, not a clean socket close, is what can prove outstanding state safe.

Nightly maintenance is not a local disconnect​

There is no universal fixed reset time. Regional IBKR maintenance windows and the configured local TWS/IB Gateway auto-restart or auto-logoff setting are separate schedules. Use the live IBKR system status, avoid hard-coded copied times, and plan for upstream degradation at least daily.

On 1100, stop new operations that depend on IBKR connectivity and mark readiness degraded, but do not assume the local socket has ended or working orders have been cancelled. On 1101, replay affected data subscriptions and reconcile. On 1102, avoid blind duplicate subscription replay, yet still reconcile critical order state. A farm-restored message is not proof that every application subscription is current.

ib_async.Watchdog restarts its controlled application on errors 100 and 1100 by design. That policy may be too aggressive for a system that prefers to keep the local process alive during known upstream maintenance. Separate process liveness, API-socket state, upstream connectivity, and application readiness before adopting it.

Source provenance​