Friday, February 11, 2011

Installing dj-celery and modifying Django settings...

One of the recent changes with using django-celery (in v2.0.0+) is that you must now specify your Celery configuration with your Django settings file. The instructions are to import djcelery and then invoke djcelery.setup_loader(), which will inform Celery to use the djcelery.loaders.DjangoLoader class instead of the default Celery base loader.

Besides the fact that djcelery often grabs CELERY_RESULT_BACKEND variable from your Django settings file, the read_configuration() within this loader is what retrieves the rest of the config information back to Celery. Therefore, if you intend to use djcelery for Django management commands, you should know about how this works.

class DjangoLoader(BaseLoader):
    """The Django loader."""

    def read_configuration(self):
        """Load configuration from Django settings."""
        from django.conf import settings
        self.configured = True
        return settings

Normally Celery's default BaseLoader class invokes Celery as follows, which is to import the Celery configs from either the environment variable CELERY_CONFIG_MODULE or through its default celeryconfig.py file. The DjangoLoader overrides this part to retrieve the Celery configuration from the settings.py file.

def read_configuration(self):
        """Read configuration from ``celeryconfig.py`` and configure
        celery and Django so it can be used by regular Python."""
        configname = os.environ.get("CELERY_CONFIG_MODULE",
                                    DEFAULT_CONFIG_MODULE)
        try:
            celeryconfig = self.import_from_cwd(configname)
        except ImportError:
            warnings.warn("No celeryconfig.py module found! Please make "
                          "sure it exists and is available to Python.",

2 comments:

  1. Thanks! I just ran into this at work. We had a celery task that worked fine when invoked from a django-admin shell, or from within the web app, but would not work when invoked from a custom management command. Our solution was to 'import djcelery' at the top of our management command module.

    ReplyDelete
  2. Good to know! Yes, it appears that the djcelery.setup_loader() command seems to be extraneous if you import djcelery:

    https://github.com/ask/django-celery/blob/3cca8386c0a38e9b3e358ecee1a8bf36f3241fd2/djcelery/__init__.py

    These docs should be revised to only require "import djcelery"

    http://celeryproject.org/docs/django-celery/changelog.html#id8


    >>> import djcelery
    >>> import os
    >>> os.environ['CELERY_LOADER']
    'djcelery.loaders.DjangoLoader'

    ReplyDelete