celeryd-multi

The Celeryd documentation only includes examples of how celeryd-multi should be invoked, so here
they are:

1. You can use this init script as a template:
https://github.com/ask/celery/blob/master/contrib/generic-init.d/celeryd

Or you can invoke celeryd-multi the old-fashioned way:

a. You need a /var/run directory for celery. The pidfile depends on them.
b. You also need to create a /var/log/celeryd dir too.

2. If you type:

celeryd-multi show 2 -Q:1-1 default -Q:2-2 nightly --loglevel=INFO --pidfile=/var/run/celery/${USER}%n.pid --logfile=/var/log/celeryd.${USER}%n.log
> Starting nodes...
celeryd -Q celery,default --pidfile=/var/run/celery/me1.pid -n celery1.myhost-dev-0 --loglevel=INFO --logfile=/var/log/celeryd.me1.log
celeryd -Q nightly --pidfile=/var/run/celery/me2.pid -n celery2.myhost-dev-0 --loglevel=INFO --logfile=/var/log/celeryd.me2.log

...you can see how things will get invoked.

To start, you would do:

celeryd-multi start 2 -Q:1-1 celery,default -Q:2-2 nightly --loglevel=INFO --pidfile=/var/run/celery/${USER}%n.pid --logfile=/var/log/celeryd.${USER}%n.log

To stop:

This should start up two separate workers, one for celery, default and one for nightly. The celeryd aemon will fork up to # of CPU's, so if you have 4 CPU's per machine, you could have a total of 8 celery workers running.

...in both cases, CELERY_CONFIG_MODULE needs to be set.

Then you setup a queue in your Celery config similar to the following:

CELERY_QUEUES = {
    "default" : {
        "exchange" : "default",
        "binding_key" : "default" },
    "nightly" : {
        "exchange" : "nightly",
        "binding_key" : "nightly",
        "exchange_type" : "direct"
        }
    }

CELERY_DEFAULT_QUEUE = "default"
CELERY_DEFAULT_EXCHANGE = "default"
CELERY_DEFAULT_EXCHANGE_TYPE = "direct"
CELERY_DEFAULT_ROUTING_KEY = "default"