Sunday, October 16, 2011

get_task_logger() in Celery...

If you looked at the Celery documentation, you'll notice that the get_task_logger() examples
constantly show up.

@task
def add(x, y):
    logger = add.get_logger()
    logger.info("Adding %s + %s" % (x, y))
    return x + y
What does this function do? Well, it turns out that it will create a separate logger instance specifically tied to the task name (submitted as a PR on https://github.com/ask/celery/issues/129). The propagate=False is always set, so that any messages passed to it will not move up the parent/ancestor chain.

Instead, a handler is always added to this task. If you wish to adjust the logger level,
you could do:

import logging
logging.getLogger('myproject.add').setLevel(logging.DEBUG)

If no loglevel is specified in get_logger(), then the default log level defined in CELERYD_LOG_LEVEL is used. Be careful though! The right way is to set the level number (not the level name) if you are modifying directly through Python:

from celery import current_app
from celery.utils import LOG_LEVELS
current_app.conf.CELERYD_LOG_LEVEL = LOG_LEVELS['DEBUG']  # pretty much the same as logging.DEBUG

What's the purpose of get_task_logger()? Well it appears the motivation is to allow logging by task names. If we were just to import the standard logging module, Celery will patch the logger module to add process-aware information (ensure_process_aware_logger()), and then add format/handlers to both the root logger and the logger defined by the multiprocessing module (the multiprocessing get_logger() does not use process shared-logs but it allows you to login things to the "multiprocessing" namespace, which adds SUBDEBUG/SUBWARNING debug levels).

def setup_logging_subsystem(self, loglevel=None, logfile=None, format=None, colorize=None, **kwargs):                                                             
        if Logging._setup:                   
            return                                                                                            
        loglevel = loglevel or self.loglevel      
        format = format or self.format                                                                         
        if colorize is None:                
            colorize = self.supports_color(logfile)                                                            
                                                                                                              
        if mputil and hasattr(mputil, "_logger"):                    
            mputil._logger = None                                                                              
        ensure_process_aware_logger()                               
        receivers = signals.setup_logging.send(sender=None,                    
                        loglevel=loglevel, logfile=logfile,                                 
                        format=format, colorize=colorize)                      
        if not receivers:                                           
            root = logging.getLogger()                              
                                                                                                              
            if self.app.conf.CELERYD_HIJACK_ROOT_LOGGER:                       
                root.handlers = []                                                                             
                                                                    
            mp = mputil.get_logger() if mputil else None                 
            for logger in filter(None, (root, mp)):                         
                self._setup_logger(logger, logfile, format, colorize, **kwargs)               
                logger.setLevel(loglevel)                        
                signals.after_setup_logger.send(sender=None, logger=logger,                                    
                                        loglevel=loglevel, logfile=logfile,                                    
                                        format=format, colorize=colorize)              

No comments:

Post a Comment