Recurring processor OOM and stream retry failure

1. 05:36:17

   Heavy order processing begins.

   Job context loads modified orders with 8–22 arrangement IDs. One task records reference-cache heap around 1,039 MiB.

2. 05:37:47

   First V8 fatal OOM.

   Mark-compact cannot recover enough old space near 6.8 GiB. Message finally blocks do not complete.

3. 05:43:51

   Memory warning alarm enters ALARM.

   Symptom detection trails the first process death by about six minutes.

4. 05:48:22

   Second processor OOM.

   The same org/job context is redelivered.

5. 05:50:27

   Critical memory alarm enters ALARM.

   The alarm name says “5min-critical,” but its configuration is three consecutive five-minute periods: a 15-minute condition.

6. 05:56–05:59

   A fresh task claims 20 and OOMs.

   Heap is only 214.5 MiB before reference loading and 215.5 MiB after it, then reaches the fatal limit in roughly three minutes. This is the strongest evidence against “slow leak alone.”

7. 06:06–06:08

   Fourth claim and OOM.

   The replay loop remains deterministic.

8. 06:16:21.568

   Processor consumer-alias claims 20 messages.

   The successful XCLAIM increments delivery count and resets ownership/idle age.

9. 06:16:21.893

   Another processor begins deleting the same 20.

   Logs show [1 / 1] for both XDEL and XACK, retry count 5, owner consumer-alias, and idle age about 324 ms.

10. 06:19:17

    The active processor OOMs after deletion.

    Its in-memory payload can still execute after the stream entry is gone, but it never reaches normal JobStatus completion.

Daily service evidence

Memory uses the AWS/ECS service-level Maximum statistic. OOM counts use fatal V8 signatures grouped by task stream. Delete counts use the retry-limit warning path; zero does not prove successful completion.

Fatal-event attribution ledger

Each fatal task stream was searched backward ten minutes for the last non-system logger context. The aliases cannot be reversed from this site.

Evidence sources and limits

1. Established the runtime topology. Described the cluster, services, task definitions, deployments, capacity providers, ASGs, desired/running counts, memory limits, health checks, and scaling state.
2. Built seven-day resource baselines. Queried AWS/ECS hourly Average and Maximum CPU/memory for processor, worker, and jobsQ.
3. Audited alarm behavior. Read thresholds, evaluation periods, dimensions, actions, and seven-day state transitions. This exposed the silent ProcessorCrash alarm and the misleading 5-minute name.
4. Located fatal signatures. Queried the ECS log group for V8 “JavaScript heap out of memory” and “ineffective mark-compacts” messages by processor log stream.
5. Attributed task streams. For each fatal timestamp, looked backward ten minutes in that exact log stream and parsed the last non-system (clientId)[jobId] context.
6. Correlated recovery events. Compared claim count, cache/heap checkpoints, max-retry deletion, owner, retry count, idle age, and subsequent fatal timestamps at millisecond precision.
7. Read the exact source paths. Verified processMessageBatch, pollForMessages, retry deletion, JobStatus.completeMessages, health monitor, runner signal handling, heap snapshot service, reference cache, and scheduled service refresh.
8. Tested competing explanations. The fresh-task baseline rejected “only a long-lived leak”; the low average CPU rejected sustained compute shortage; the repeated heavy job contexts supported workload amplification.

fields @timestamp, @logStream, @message
| filter @logStream like /prod-processor-b/
  and @message like /JavaScript heap out of memory/
| sort @timestamp asc

filter @logStream like /prod-processor-b/
  and @message like /hit the max retry count/
| stats count(*) as DeletedMessages by bin(1d) as day
| sort day asc

Continue with the memory and heap remediation playbook or the whole-system observability design.

Current processor budget

The 512 MiB margin is tight, but increasing it is not the first fix: the application demonstrated the ability to create more than 6 GiB of live heap from one batch.

1. Batch fan-out: processMessageBatch executes Promise.all for every returned stream item.
2. Per-message graph size: a modified SalesOrder can load many Revenue Arrangements. The incident contains repeated orders with 20–22 IDs, and each arrangement may contain elements, schedules, allocations, and expense data.
3. Per-order cloning: the worker intentionally deep clones organization reference data so mutations do not cross order boundaries. Correct isolation has a memory cost multiplied by concurrency.
4. Promise retention: the batch array, closures, AsyncContext state, raw message fields, intermediate results, and unresolved promise chains remain reachable until work settles.
5. Cache and entity reuse: task-local reference and entity caches may retain additional graphs after individual order work completes.
6. GC death spiral: as headroom shrinks, V8 spends seconds or minutes compacting. Application progress slows, event loop health degrades, but the active set remains live.
7. Replay: a crash skips ACK, cleanup, and completeMessages; reclaim reruns the same expensive allocation shape.

1. One shared bounded executor

• Use the same executor for normal polling, startup backlog, and auto-claimed messages.
• Start with concurrency 2; test 2, 4, and 5 against representative heavy and normal workloads.
• Separate scan/fetch count from active processing count. Scanning 100 IDs must not start 100 order graphs.
• Do not claim more work than the task can begin before the idle/ownership budget.
• Expose active count, queued-in-process count, and per-item wait time.

// Design sketch, not committed implementation.
async function processMessageBatch(connection, items) {
  await runWithConcurrency(items, maxInFlight, async (item) => {
    await waitForHeapHeadroom();
    await processOneMessage(connection, item);
  });
}

2. Add a memory admission watermark

• Warn when heap/limit remains above 70% after GC.
• Stop claiming new messages around 80% or when RSS leaves insufficient container margin.
• Finish in-flight work and advertise draining; restart only after ownership is safely released.
• Do not use a health-check 500 as the normal pressure control—abrupt replacement replays work.
• Combine heap, RSS, GC effectiveness, and event-loop delay; one gauge is insufficient.

3. Replace max-retry deletion with an explicit terminal path

• A delivery count is not proof that a message is idle; another consumer may have just claimed it.
• Use an idempotency key so competing reapers cannot create multiple terminal outcomes.
• Keep original message fields, raw payload reference, owner history, retry reasons, org/job/order IDs, and timestamps.
• Make JobStatus completion/error accounting exactly-once or safely idempotent.
• Provide a bounded, audited replay operation that restores ownership and counters coherently.

4. Reduce graph amplification after concurrency is safe

• Measure duplicate RevenueArrangement IDs across one batch before choosing a bulk/dedup strategy.
• Load only fields required by the current operation where the entity contract permits it.
• Avoid retaining full arrangement graphs on SalesOrder longer than the operation requires.
• Release references in per-item scope and verify AsyncContext does not retain completed chains.
• Bound cache by entries and estimated bytes; expose cache size and eviction.
• Do not blindly replace per-order isolation with one giant bulk load; that can move the peak rather than reduce it.

Continuous low-overhead series

Three-point heap comparison

1. A — clean baseline: after deploy/startup, idle, reference data initialized, and a full GC where available.
2. B — after representative normal work: complete enough ordinary jobs to warm caches, then idle and GC.
3. C — after one heavy job: process the sanitized high-arrangement fixture with safe concurrency, then idle for at least the ten-minute cache TTL and GC.

Compare retained size and dominator growth between A, B, and C. Focus on SalesOrder, RevenueArrangement, schedule arrays, Buffers/ArrayBuffers, Maps, cache entries, promises, AsyncContext/cls-hooked objects, Redis results, and logger payloads.

Canary rollout

1. Deploy metrics/event schema before or with the behavior change.
2. Canary one controlled task or profile with representative synthetic/sanitized work.
3. Compare heap/RSS/GC, throughput, queue age, retries, and job completion against baseline.
4. Increase task share in stages; stop on integrity delta, heap regression, or latency burn.
5. After full rollout, observe at least two known heavy schedule windows.

See the production evidence and impact and the whole-system observability design.