Back to Signal Feed
CodeTracked since May 20, 2026

Cognee adds a worker-per-task pipeline executor with adaptive task concurrency

Replaces the old recursive `run_tasks_base` task execution path with a new worker-pipeline model that runs each stage through bounded queues, adds `Task(workers=...)` and `Task(timeout=...)` controls, and introduces fixed/adaptive worker strategies to tune throughput per task.

run_tasksworker_pipelineTask(workers, timeout)FixedWorkers

What Happened

  • Replaces the old recursive `run_tasks_base` task execution path with a new worker-pipeline model that runs each stage through bounded queues, adds `Task(workers=...)` and `Task(timeout=...)` controls, and introduces fixed/adaptive worker strategies to tune throughput per task.
  • Replaces the old recursive `run_tasks_base` task execution path with a new worker-pipeline model that runs each stage through bounded queues, adds `Task(workers=...)` and `Task(timeout=...)` controls, and introduces fixed/adaptive worker strategies to tune throughput per task.
  • 1 evidence item attached for review.

What is Different

Before

Scattered source updates, isolated context, and manual follow-up across multiple feeds.

Now

Introduces a configurable per-task concurrency engine: a bounded `asyncio.Queue` worker pipeline plus `FixedWorkers`/`AdaptiveWorkers` strategies, with a per-task timeout path, so Cognee can preserve ordered execution when needed and automatically adjust worker counts based on completion, throttle signals, and utilization.

Why Track This

Why It Matters

Pipeline operators and developers running large batch jobs like `cognify` should see faster, more controlled runs with fewer full-job stalls because the executor now scales work through per-stage worker pools instead of a recursive async chain, with reported warm-run timing dropping from 108.7s to about 94s. Technically, this is delivered by replacing stage-to-stage handoff logic in `run_tasks` with queue-based workers, adding adaptive hill-climb/shrink logic in `_AdaptivePool`, and reusing converged worker targets across runs; teams should watch whether adaptive scaling oscillates under bursty workloads, whether throttling detection misclassifies transient errors, and whether cross-run registry reuse causes stale targets after major workload shifts.

Impact

Pipeline operators and developers running large batch jobs like `cognify` should see faster, more controlled runs with fewer full-job stalls because the executor now scales work through per-stage worker pools instead of a recursive async chain, with reported warm-run timing dropping from 108.7s to about 94s. Technically, this is delivered by replacing stage-to-stage handoff logic in `run_tasks` with queue-based workers, adding adaptive hill-climb/shrink logic in `_AdaptivePool`, and reusing converged worker targets across runs; teams should watch whether adaptive scaling oscillates under bursty workloads, whether throttling detection misclassifies transient errors, and whether cross-run registry reuse causes stale targets after major workload shifts.

What To Watch Next

  • Watch whether run_tasks becomes a repeated pattern.
  • Track follow-up changes around AI Workflow Automation.
  • Compare future signals against this evidence trail.
  • Re-check risk flags: adaptive_worker_oscillation, throttling_detection_false_positive.
Open Topic TimelineOpen Technical EventOpen Original Sourceadaptive_worker_oscillation / throttling_detection_false_positive / cross_run_registry_staleness / timeout_suppresses_root_cause_visibility / order_regression_fixed_worker_mode

Supporting Evidence

GITHUB PULL REQUESTHigh Trust

topoteretes/cognee PR #2871: feat: worker-per-task pipeline executor with adaptive concurrency

The PR reports 200-file cognify runtime improving from 108.7s (baseline) to ~100s on first run and ~94s on warm reruns via the new worker pipeline and cross-run target reuse.