Why did my Celery task run twice with the Redis broker?

Celery Task Runs Twice with Redis: Causes and Fixes

Short answer

With the Redis broker, a message that is not acknowledged within the visibility timeout is delivered to another worker, and Celery's documentation gives the default as one hour. ETA, countdown and retry tasks that wait or run longer than that are redelivered, so the task executes again. Make the task idempotent first, then set acknowledgement mode, prefetch and time limits deliberately.

#Why it happens

Redis has no built-in acknowledgement protocol, so Celery emulates one with a visibility timeout: the time a worker has to acknowledge a message before it is redelivered to another worker. The Celery documentation gives the Redis default as 1 hour (3600 seconds). If a task is not acknowledged inside that window, it runs again. For ETA, countdown or retry tasks whose wait or run time exceeds the timeout, this can repeat in a loop.

The documentation suggests raising the timeout to cover the longest ETA, but says this is not recommended because it may hurt reliability. It also notes that the broker is not a database, so work scheduled far in the future belongs on periodic tasks, which the visibility timeout does not affect.

#The settings involved

Defaults from the Celery configuration reference, accessed 2026-10-28.
Setting	Default	What it changes
`task_acks_late`	Disabled	Default is to acknowledge just before the task runs. Enabled, the message is acknowledged after it finishes, so a worker crash causes redelivery
`worker_prefetch_multiplier`	4	Messages reserved per process ahead of time. A high value lets one worker hold messages while others idle, which hurts slow tasks
`task_soft_time_limit`	None	Raises `SoftTimeLimitExceeded` inside the task so it can clean up
`task_time_limit`	None	Hard limit. The worker process is killed and replaced
`task_reject_on_worker_lost`	Disabled	With late acks, re-queues a task when its process dies abruptly. The docs warn it can cause message loops
`worker_max_tasks_per_child`	No limit	Replaces a pool process after this many tasks

#Fixes, in order

Make the task idempotent. Running it twice must leave the same state as running it once. Key every write by a natural id and use upserts.
Check for existing output before paying for work. An LLM task should skip the model call when the result already exists.
Keep waits below the visibility timeout, or move far-future work to Celery beat periodic tasks.
Set a soft limit below the hard limit, so a hung provider call is recorded as a failure instead of holding a worker forever.
Choose late acknowledgement on purpose. It trades lost jobs for possible duplicates, so enable it only for idempotent tasks.
Set the prefetch multiplier to 1 for slow tasks so long jobs spread across workers.

Illustrative idempotent LLM task

@app.task(bind=True, acks_late=True)
def summarise_meeting(self, meeting_id):
    if summary_exists(meeting_id):     # redelivery: skip the paid model call
        return
    text = call_model(meeting_id)
    upsert_summary(meeting_id, text)   # keyed by meeting_id, safe to repeat

This is a sketch of the pattern, not code from a particular codebase. In Opaeron the narrative summary runs this way, on a worker, after the action items have already been returned to the user. See Celery and Redis for LLM Jobs and What is Opaeron?.

Sources

Celery documentation, Using RedisAccessed 2026-10-28. Visibility timeout default and redelivery of ETA and retry tasks.
Celery documentation, Configuration and defaultsAccessed 2026-10-28.

#Why it happens

#The settings involved

#Fixes, in order

Sources

Read the full write-up

More questions

What is Opaeron?

Should I use HNSW or IVFFlat in pgvector?