Production RCA

Recurring processor OOM and stream retry failure

The processors are not simply “using a lot of memory.” A repeatable workload amplification exhausts V8; the recovery path then has a demonstrated race that can delete active work.

Incident focus: 2026-09-09 05:36–06:19 UTC · Extended evidence: 2026-09-02 00:00–2026-09-09 17:38 UTC

Redacted engineering share

This public Pages build preserves evidence and counts while minimizing exposure. Customer, job, task, message, account, and raw-payload identifiers are excluded or replaced with stable aliases. Review the complete event ledger.

Verdict

Two coupled production defects

The OOM and the data-integrity risk are related but distinct. A memory fix alone does not make retry exhaustion safe.

Primary availability defect

Unbounded concurrent live-object amplification

processMessageBatch sends every returned message through Promise.all. Production normal polling was 20. Reclaim scans up to 100 and submits all successfully claimed messages to the same fan-out.

The observed heavy orders each loaded many full Revenue Arrangement graphs—commonly 20–22 IDs—plus derived schedules, cloned organization reference data, Redis/warehouse results, and processing intermediates. These remain live together until their promises settle.

Primary integrity defect

Retry-limit deletion races active ownership

When Redis reports delivery count at least five, the auto-claim scanner immediately runs XACK and XDEL. It neither claims ownership first nor requires the message to remain idle.

On Sep 9, one processor claimed 20 messages. A second processor deleted them 325 ms later after seeing delivery count five and idle age about 324 ms. The active processor continued from its returned payload and then OOMed before completion accounting.

What this is not

It is not established as an EC2 shortage, a single old task leaking slowly, or a problem that can be solved by adding RAM. Processor service CPU averaged 5.25% over seven days; a fresh task still consumed the heap in minutes; and the container already allows 7,384 MiB.

Latest incident

Five OOMs and one deletion race

All five OOM task streams were last active on organization Org A, job Job A2.

  1. Heavy order processing begins.

    Job context loads modified orders with 8–22 arrangement IDs. One task records reference-cache heap around 1,039 MiB.

  2. First V8 fatal OOM.

    Mark-compact cannot recover enough old space near 6.8 GiB. Message finally blocks do not complete.

  3. Memory warning alarm enters ALARM.

    Symptom detection trails the first process death by about six minutes.

  4. Second processor OOM.

    The same org/job context is redelivered.

  5. Critical memory alarm enters ALARM.

    The alarm name says “5min-critical,” but its configuration is three consecutive five-minute periods: a 15-minute condition.

  6. A fresh task claims 20 and OOMs.

    Heap is only 214.5 MiB before reference loading and 215.5 MiB after it, then reaches the fatal limit in roughly three minutes. This is the strongest evidence against “slow leak alone.”

  7. Fourth claim and OOM.

    The replay loop remains deterministic.

  8. Processor consumer-alias claims 20 messages.

    The successful XCLAIM increments delivery count and resets ownership/idle age.

  9. Another processor begins deleting the same 20.

    Logs show [1 / 1] for both XDEL and XACK, retry count 5, owner consumer-alias, and idle age about 324 ms.

  10. The active processor OOMs after deletion.

    Its in-memory payload can still execute after the stream entry is gone, but it never reaches normal JobStatus completion.

Root-cause chain

Why one batch can consume gigabytes

The fatal logs say “ineffective mark-compacts,” which means the collector ran but the object graph remained substantially live. That is consistent with concurrent batch retention.

Large sync arrives
Many modified orders and arrangements
Batch of 20
Normal poll or reclaimed pending work
Promise.all
All orders become live concurrently
Graph expansion
SalesOrder + 20–22 RAs + schedules
Old-space pressure
Objects survive GC while work is active
Fatal OOM
No ACK or completion counter update
Redelivery
Same allocation pattern repeats
Factor Evidence Confidence Role
Promise.all(msgItems.map(...)) Direct source inspection; every message in the returned batch starts concurrently. Proven Primary allocation multiplier
Arrangement-heavy modified orders Pre-OOM logs show many orders loading 20–22 IDs. Proven Per-message cost
Fresh-task exhaustion 215.5 MiB before reclaimed work; fatal heap within ~3 minutes. Proven Rules out slow-retention-only theory
Reference cache retention Ten-minute in-process cache; no size budget or proactive sweep in the dependency. Contributor Raises baseline and duplicates state per task
Only 512 MiB outside V8 old-space Container 7,384 MiB; MAX_OLD_SPACE=6872. Contributor Small margin for native buffers and runtime
True long-lived leak No paired post-full-GC baselines or heap-dominator diff captured during incident. Unproven Requires targeted profiling

Complete evidence ledger

All fatal events in the reviewed window

The ledger preserves every observed fatal processor event while replacing customer, job, task, and account identifiers with stable aliases.

27Processor fatal OOM events, Sep 2–9
23Fatal OOM events in rolling seven days
60Retry-limit stream entries removed
89%OOM events attributed to two org aliases

Daily service evidence

Date (2026 UTC)Processor memory peakFatal OOM eventsRetry-limit deletes
Sep 02Not retained in seven-day query4Not observed
Sep 0399.49%1140
Sep 0499.62%20
Sep 0599.19%30
Sep 0699.58%20
Sep 0799.47%00
Sep 0876.44%00
Sep 0999.00%520

Memory peak is the AWS/ECS service-level Maximum statistic. OOM count comes from fatal V8 signatures grouped by task log stream. Deletes come from the retry-limit warning path; zero means no matching warning, not a proof that every message completed.

Fatal-event attribution ledger

#Date (2026)Fatal timestampOrg aliasJob alias
01Sep 0208:56 UTCOrg AJob A3
02Sep 0209:06 UTCOrg AJob A3
03Sep 0209:17 UTCOrg AJob A3
04Sep 0209:27 UTCOrg CJob C1
05Sep 0305:27 UTCOrg BJob B1
06Sep 0305:27 UTCOrg BJob B1
07Sep 0305:37 UTCOrg BJob B1
08Sep 0308:53 UTCOrg AJob A1
09Sep 0308:53 UTCOrg AJob A1
10Sep 0309:03 UTCOrg DJob D1
11Sep 0309:04 UTCOrg AJob A1
12Sep 0309:13 UTCOrg AJob A1
13Sep 0309:13 UTCOrg AJob A1
14Sep 0309:23 UTCOrg AJob A1
15Sep 0309:24 UTCOrg CJob C2
16Sep 0408:57 UTCOrg AJob A4
17Sep 0409:07 UTCOrg AJob A4
18Sep 0505:32 UTCOrg BJob B2
19Sep 0505:32 UTCOrg BJob B2
20Sep 0505:32 UTCOrg BJob B2
21Sep 0605:27 UTCOrg BJob B3
22Sep 0605:30 UTCOrg BJob B3
23Sep 0905:37 UTCOrg AJob A2
24Sep 0905:48 UTCOrg AJob A2
25Sep 0905:59 UTCOrg AJob A2
26Sep 0906:08 UTCOrg AJob A2
27Sep 0906:19 UTCOrg AJob A2

Attribution method: for each fatal task stream, inspect the preceding ten minutes and select the last non-system logger context. Aliases are consistent within this site but cannot be reversed to customer identifiers.

Evidence inventory and provenance

Runtime

AWS observations

  • AWS/ECS CPU and memory Average/Maximum datapoints
  • ECS services, task definitions, deployments, health, and stop state
  • Application log streams and alarm configuration/history
  • Scalable targets, policies, ASGs, and capacity providers
Source

Code and configuration

  • Stream polling, claiming, concurrency, ACK, delete, and retry logic
  • Task memory and V8 old-space settings
  • Health checks, heap snapshots, metric publishing, and runner signals
  • Job schedules, logger context, and async context propagation
Limit

Unavailable during review

  • No paired heap snapshots or post-full-GC retained baseline
  • No authoritative Redis reconciliation of affected jobs
  • No warehouse credential for downstream row comparison
  • No customer-visible impact count or financial misstatement proof

Frequency

Why the spikes recur

The evidence shows scheduled workload concentration plus replay amplification, not random host behavior.

Recurring arrival windows

  • Sep 3, 5, 6, and 9 incidents cluster near 05:20–06:20 UTC.
  • Sep 2–4 also show a cluster around 08:50–09:25 UTC.
  • Two organizations account for 89% of extended-window OOMs.
  • Top job contexts repeat across several processor deaths.

Strong inference: large customer sync schedules repeatedly inject a similar shape of modified orders. Job metadata should be checked before asserting the exact upstream schedule.

Why each burst becomes a spike

  1. Shared Redis stream distributes heavy messages to processors.
  2. Each processor accepts a multi-message batch without a memory budget.
  3. All messages allocate concurrently.
  4. Long GC pauses raise CPU and make health checks time out.
  5. Unfinished messages become reclaim candidates.
  6. Reclaim reproduces the same allocation pattern.
How to smooth rather than merely move the spike

Bound per-process concurrency, estimate work weight from payload bytes and arrangement count, stop claiming near a heap watermark, apply tenant fairness, split oversized batches, and scale on backlog age/throughput. Staggering customer schedules helps, but it is a secondary control—not a substitute for bounded execution.

Impact and outcome

Availability loss is proven; financial impact is not yet quantified

The distinction matters. The code path creates a credible completeness risk, but repository and CloudWatch evidence alone cannot prove which downstream financial rows are missing.

Established

Operational impact

  • 23 processor deaths in the rolling seven-day window.
  • At least one separate worker OOM; no jobsQ OOM in that window.
  • Capacity consumed by multi-second to multi-minute mark-compact cycles.
  • Repeated processing delay and task replacement.
  • Memory alarms entered ALARM eight times.
Credible risk

Data and job impact

  • 60 stream entries permanently removed at retry limit.
  • Deleted messages have no durable DLQ payload.
  • Deletion path does not decrement JobStatus message count.
  • Deletion path does not explicitly fail or isolate the job.
  • Orders, recognition, warehouse, or posting may remain incomplete.
Not established

What cannot yet be claimed

  • No quantified number of customer-visible missing records.
  • No proven journal or revenue misstatement.
  • No authoritative final Redis JobStatus snapshot.
  • No downstream table reconciliation for the deleted IDs.
  • No proof that cache retention is the dominant heap owner.
Outcome of the Sep 9 loop

ECS returned the service to 30/30 healthy tasks and the repeated OOM burst ended. However, the system stopped retrying because 20 entries reached the deletion path. “No longer failing” is not the same outcome as “processed successfully.” Reconcile Redis JobStatus, raw-order data, message counters, and downstream outputs before replay or closure.

Tagging audit

Correlation exists, but it is fragmented

“Tagging” means three different things here: AWS resource tags, application correlation fields, and metric dimensions. They have different cost and cardinality rules.

Surface Available today Observed state Gap / recommendation
ECS service tags Yes Each investigated service has only Name. Add owner, component, environment, criticality, cost-center, data-classification, and runbook.
ECS managed task tags Yes aws:ecs:clusterName and aws:ecs:serviceName. Keep enabled; use enhanced Container Insights for task-level telemetry.
Service tag propagation Capability unused propagateTags=NONE. Set propagation deliberately for new tasks after defining a controlled tag schema.
Application logs Yes Logger prefix: (clientId)[jobId][level][userId][component][timestamp]. Emit JSON consistently and add traceId, taskId, messageId, stage, and outcome.
Redis stream fields Yes Message ID, salesOrderId, clientId, jobId, and type are available. Carry one correlation envelope through DLQ and all stage handoffs.
Custom metric context Partial MetricTracker can add TenantId, TenantName, and JobId based on sysconfig; current revlock metrics did not show those dimensions. Do not turn JobId into a broad metric dimension. Keep high-cardinality IDs in logs/spans.
Reporting metrics High cardinality present OrgId, UserId, and TemplateId produce 395 of 561 observed revlock metric identities. Review cost/value; move unbounded IDs to EMF properties or logs where possible.
Distributed tracing No No OpenTelemetry, X-Ray, traceparent, or correlation ID dependency was found. Add trace propagation across API, Camunda, jobs, streams, processors, and warehouse calls.
Cardinality boundary

Metric dimensions should be bounded: environment, profile, service, operation, stage, job type, outcome, driver, dependency, and message type. Put organization, job, order, message, task, user, and trace IDs in structured event properties. Every unique CloudWatch dimension combination creates another metric identity.

Investigation method

How the diagnosis was built

The method moved from infrastructure inventory to time correlation, then from time correlation to code-level causal checks.

  1. Established the runtime topology. Described the cluster, services, task definitions, deployments, capacity providers, ASGs, desired/running counts, memory limits, health checks, and scaling state.
  2. Built seven-day resource baselines. Queried AWS/ECS hourly Average and Maximum CPU/memory for processor, worker, and jobsQ.
  3. Audited alarm behavior. Read thresholds, evaluation periods, dimensions, actions, and seven-day state transitions. This exposed the silent ProcessorCrash alarm and the misleading 5-minute name.
  4. Located fatal signatures. Queried the ECS log group for V8 “JavaScript heap out of memory” and “ineffective mark-compacts” messages by processor log stream.
  5. Attributed task streams. For each fatal timestamp, looked backward ten minutes in that exact log stream and parsed the last non-system (clientId)[jobId] context.
  6. Correlated recovery events. Compared claim count, cache/heap checkpoints, max-retry deletion, owner, retry count, idle age, and subsequent fatal timestamps at millisecond precision.
  7. Read the exact source paths. Verified processMessageBatch, pollForMessages, retry deletion, JobStatus.completeMessages, health monitor, runner signal handling, heap snapshot service, reference cache, and scheduled service refresh.
  8. Tested competing explanations. The fresh-task baseline rejected “only a long-lived leak”; the low average CPU rejected sustained compute shortage; the repeated heavy job contexts supported workload amplification.
Representative read-only Logs Insights patterns 
fields @timestamp, @logStream, @message
| filter @logStream like /prod-processor-b/
  and @message like /JavaScript heap out of memory/
| sort @timestamp asc

filter @logStream like /prod-processor-b/
  and @message like /hit the max retry count/
| stats count(*) as DeletedMessages by bin(1d) as day
| sort day asc

Attribution then used filter-log-events against each exact task log stream with a bounded ten-minute lookback.

Known unknowns

What must be verified next

These are not reasons to delay the concurrency and retry-safety fixes. They determine customer remediation and later optimization.

Customer/job reconciliation

  • Authoritative Redis status and phase counters for the affected jobs.
  • Whether all 20 deleted raw-order payloads still exist.
  • Order and RevenueArrangement documents written before each crash.
  • Warehouse rows and staging status for the job.
  • Whether an idempotent replay can safely repair partial work.

Heap ownership

  • Retained-size dominators before and after a representative heavy batch.
  • Heap, RSS, external, and ArrayBuffer growth after forced/full GC.
  • Reference-cache entry count, estimated bytes, hit rate, and eviction behavior.
  • Duplicate arrangement IDs and graph cloning across concurrent orders.
  • Memory released after order completion and job finalization.

Continue with the memory and heap remediation playbook or the whole-system observability design.