Celery v2.3 and chords..

Juust saw this on the Celery development branch for v2.3....

https://github.com/ask/celery/commit/332ac0fdf7627a42b50b76ca54722f8f7b0dee2b

http://ask.github.com/celery/userguide/tasksets.html#chords

This is how the taskset callback stuff is done:

Important Notes

By default the synchronization step is implemented by having a recurring task poll the completion of the taskset every second, applying the subtask when ready.

Example implementation:

def unlock_chord(taskset, callback, interval=1, max_retries=None):
    if taskset.ready():
        return subtask(callback).delay(taskset.join())
    unlock_chord.retry(countdown=interval, max_retries=max_retries)

This is used by all result backends except Redis, which increments a counter after each task in the header, then applying the callback when the counter exceeds the number of tasks in the set.

The Redis approach is a much better solution, but not easily implemented in other backends (suggestions welcome!)