Skip to main content

Temporal

Temporal owns the simulation lifecycle. The FastAPI app is a Temporal client; the actual orchestration runs in the temporal-worker service.

Why Temporal​

Before Temporal, the lifecycle had three independent actors (the main Celery task, a recovery beat task, and an orphan-cleanup beat task), each inferring the others' liveness from DB rows and pod HTTP responses. A deploy that killed the Celery task before the first heartbeat could leave a live GPU pod idle for hours. Temporal makes workflow liveness authoritative: the workflow state lives in Temporal's own store, not in heuristics.

Where it runs​

  • temporal — the server (temporalio/auto-setup) on 127.0.0.1:7233, namespace fishcloud, backed by the dedicated temporal-db Postgres container.
  • temporal-worker — runs python -m saas.workflows.worker. It hosts SimulationWorkflow (saas/workflows/sim_workflow.py) and its activities (saas/workflows/activities/). Configured via TEMPORAL_ADDRESS=temporal:7233 and TEMPORAL_NAMESPACE=fishcloud.

On job creation, POST /api/jobs starts the workflow with id=sim-{job_id} on the sim task queue.

The workflow​

SimulationWorkflow.run is pure orchestration over activities:

  1. Phase 1 (pre-GPU): optional enrich_seed (when enrich_web is set) and derive_markets. A failure here marks the job FAILED explicitly, since no pod exists yet.
  2. Phase 2 (GPU lifecycle): provision_pod → wait_for_worker_health → submit_and_poll → upload_and_finalize. The activities write user-visible state to Postgres synchronously.

The workflow has built-in resilience: a one-shot bad-host pod swap (vLLM never starts / silent host), pod-unreachable retries against the same pod (idempotent re-entry), and a one-shot pod swap when the LLM circuit breaker or slow-pod detector trips. Markers for these are defined in saas/constants/tiers.py.

Cancel, not terminate​

GPU teardown runs in the workflow's finally block. That finally only executes when the workflow is cancelled, not when it is terminated — terminate skips the cleanup path and would leave the pod billing. Always cancel a workflow:

temporal workflow cancel --workflow-id sim-<job_id>

Use terminate only as a last resort, knowing it will not run teardown and you must reap the pod manually.

Inspecting a workflow​

temporal workflow describe --workflow-id sim-<job_id>

The user-visible job status still lives in Postgres simulation_jobs (read by the API → frontend); Temporal owns workflow progress, retries, and heartbeats.