Back to Signal Feed
CodeTracked since May 20, 2026

cognee moves task execution to a worker-per-task pipeline with adaptive concurrency

The PR replaces the old recursive `run_tasks_base` executor with a queue-based worker pipeline (`run_worker_pipeline`) and adds `Task(workers=...)`/`Task(timeout=...)` control using `FixedWorkers` and `AdaptiveWorkers`, while adding a cross-run registry so adaptive tuning can resume from prior targets instead of re-discovering concurrency on every run.

run_tasksrun_worker_pipelineWorkerStrategyFixedWorkers

What Happened

  • The PR replaces the old recursive `run_tasks_base` executor with a queue-based worker pipeline (`run_worker_pipeline`) and adds `Task(workers=...)`/`Task(timeout=...)` control using `FixedWorkers` and `AdaptiveWorkers`, while adding a cross-run registry so adaptive tuning can resume from prior targets instead of re-discovering concurrency on every run.
  • The PR replaces the old recursive `run_tasks_base` executor with a queue-based worker pipeline (`run_worker_pipeline`) and adds `Task(workers=...)`/`Task(timeout=...)` control using `FixedWorkers` and `AdaptiveWorkers`, while adding a cross-run registry so adaptive tuning can resume from prior targets instead of re-discovering concurrency on every run.
  • 1 evidence item attached for review.

What is Different

Before

Scattered source updates, isolated context, and manual follow-up across multiple feeds.

Now

Introduces a configurable per-task concurrency engine in the public API and executor path, enabling fixed-order execution when required and adaptive worker scaling/rollback logic that can automatically increase, shrink, or reuse prior optimal concurrency for each task stage.

Why Track This

Why It Matters

Operators running large cognee `cognify` jobs should see faster and more predictable batch completion with clearer failure handling, because failed items can be reported individually instead of failing the whole run. The PR reports a runtime drop on a 200-file benchmark from 108.7s baseline to ~94s on warm, resumed runs, and adds per-task timeouts plus adaptive worker control to reduce stalled execution, but teams should watch for adaptive scale oscillation under changing throttling and for stale cross-run registry targets after process restarts or environment changes.

Impact

Operators running large cognee `cognify` jobs should see faster and more predictable batch completion with clearer failure handling, because failed items can be reported individually instead of failing the whole run. The PR reports a runtime drop on a 200-file benchmark from 108.7s baseline to ~94s on warm, resumed runs, and adds per-task timeouts plus adaptive worker control to reduce stalled execution, but teams should watch for adaptive scale oscillation under changing throttling and for stale cross-run registry targets after process restarts or environment changes.

What To Watch Next

  • Watch whether run_tasks becomes a repeated pattern.
  • Track follow-up changes around LLMOps.
  • Compare future signals against this evidence trail.
  • Re-check risk flags: adaptive_pool_oscillation_under_throttling, stale_convergence_target_after_restart.
Open Topic TimelineOpen Technical EventOpen Original Sourceadaptive_pool_oscillation_under_throttling / stale_convergence_target_after_restart / timeout_false_positive_on_slow_stages / order_regression_with_fixedworkers / cross_run_registry_cache_growth

Supporting Evidence

GITHUB PULL REQUESTHigh Trust

topoteretes/cognee PR #2871: feat: worker-per-task pipeline executor with adaptive concurrency

Rebuild the pipeline executor into bounded per-stage queues and task-specific worker pools, replacing the previous recursive async-for chain.