Thursday, October 27, 2011

Celerybeat and celerybeat-schedule

In my effort to attempt to replace /etc/init.d/celerybeat with a version that worked more reliably with fabric, one of the discoveries is that celerybeat keeps firing off all tasks from the scheduled task list because Celerybeat stores the last_run time of a scheduled task usually in a celerybeat-schedule in the default dir)...if it hasn't been run in awhile (by virtue of using an older celerybeat-schedule) then you may see a lot of "Sending due task".

You can check the last_run_at apparently by using the shelve library from Python, which celerybeat uses to store all the scheduled tasks defined in your Celery configurations. Each time you restart Celerybeat, this celerybeat-schedule gets merged with your Celery scheduled tasks. Those that don't exist already are added.
sudo python
>>> import shelve
>>> a['entries']['my_task'].last_run_at
datetime.datetime(2011, 10, 28, 2, 1, 57, 717454)
The key is to specify explicitly define the celerybeat-schedule:

export CELERYBEAT_OPTS="--schedule=/var/run/celerybeat-schedule"

1 comment:

  1. Did you ever figure out a more reliable way of launching celerybeat via fabric? Doesn't work for me and I need sudo so can't do pty=False