Django Logging Rotating files not working

timedrotatingfilehandler
django rotatingfilehandler
django json logging
django log sql queries
valueerror: unable to configure handler file
django error log file
python file rotation
django log to database

So i am having some issues with Django logging when it gets to the maxBytes size. Basically when that happens the file does not seem to rotate and create a new file.

Someone told me this could have something to do with the writing permissions of the server but i am not sure how to set that properly so that django is able to create a new log file when the old one is full.

my settings:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'filters': {
        'require_debug_false': {
            '()': 'django.utils.log.RequireDebugFalse'
        }
    },
    'formatters': {
        'verbose': {
            'format': "[%(asctime)s] %(levelname)s [%(name)s:%(lineno)s] %(message)s",
            'datefmt': "%d/%b/%Y %H:%M:%S"
        },
        'simple': {
            'format': '[%(levelname)-7s] %(asctime)s - %(message)s'
        },
    },
    'handlers': {
        'mail_admins': {
            'level': 'ERROR',
            'filters': ['require_debug_false'],
            'class': 'django.utils.log.AdminEmailHandler'
        },
        'console': {
            'level': 'INFO',
            'class': 'logging.StreamHandler'
        },
        'boom_file': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'maxBytes': 1024*1024*10,  # 10 MB
            'backupCount': 10,
            'filename': '/var/log/boom.log',
            'formatter': 'simple'
        },
        'binglaw_crawler_file': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'maxBytes': 1024*1024*10,  # 10 MB
            'backupCount': 10,
            'filename': '/var/log/boom-binglaw-crawler.log',
            'formatter': 'simple'
        },
    },
    'loggers': {
        'django.request': {
            'handlers': ['mail_admins'],
            'level': 'ERROR',
            'propagate': True,
        },
        'boom': {
            'handlers': ['console', 'boom_file'],
            'propagate': True,
            'level': 'DEBUG',
        },
        'boom.binglaw_crawler': {
            'handlers': ['binglaw_crawler_file', ],
            'propagate': False,
            'level': 'DEBUG',
        },
    }
}

i noticed my other log celeryd seems o be rotaing just fine.. isnt that strange?

-rw-r--rw- 1 root          root          10485721 Aug 18 12:12 boom-binglaw-crawler.log
-rw-r--r-- 1 root          root            403506 Nov  8 23:42 boom-celeryd.log
-rw-r--r-- 1 root          root             20201 Oct  2 12:47 boom-celeryd.log.1
-rw-r--rw- 1 root          root           1049478 Oct  1 18:49 boom-celeryd.log.2

UPDATE:

i am getting this error when i try to run the manage command that creates the log file

Traceback (most recent call last):
  File "/usr/lib/python2.7/logging/handlers.py", line 77, in emit
    self.doRollover()
  File "/usr/lib/python2.7/logging/handlers.py", line 142, in doRollover
    os.rename(self.baseFilename, dfn)
OSError: [Errno 13] Permission denied

When using the 'logging.handlers.RotatingFileHandler' in Django logging, I had this error:

Traceback (most recent call last):
  File "C:\Python33\lib\logging\handlers.py", line 73, in emit
    self.doRollover()
  File "C:\Python33\lib\logging\handlers.py", line 176, in doRollover
    self.rotate(self.baseFilename, dfn)
  File "C:\Python33\lib\logging\handlers.py", line 116, in rotate
    os.rename(source, dest)
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'log.txt.1'

This error was happening to me because when I start Django, it seems to start 2 different processes. Both processes setup logging, which means both get a handle to the same LOGGING files defined in 'settings.py' config file.

I added this line to my settings.py file, right before I set my LOGGING variable.

print("Initializing LOGGING in settings.py - if you see this more than once use 'runserver --noreload'")

If you start the app with 'manage.py runserver --noreload' parameter, it may cure your file contention issue.

python - Django Logging Rotating files not working, When using the 'logging.handlers.RotatingFileHandler' in Django logging, I had this error: Traceback (most recent call last): File  Turns out that error is because /var/log/myapp is owned by the gunicorn user (which serves the django app, and thus writes the logs, and thus owns the directory). The su directive lets you set the owner and group logrotate should run as to solve that problem.

One solution, which currently works for for me, was to use 'delay': True for the handler ids that write to files (the RotatingFileHandler in my case). This creates the files only when the first emit() is called from the loggers, and my understanding is that the thread contention between the main thread and the autoreload thread (which is the cause of the error) seems not to exist at that point.

Logging | Django documentation, It took me some time to figure out how to set up logging in Django. What I want to do is to log to a pair of rotating files each 1MB in size. Basically when that happens the file does not seem to rotate and create a new file. Someone told me this could have something to do with the writing permissions of the server but i am not sure how to set that properly so that django is able to create a new log file when the old one is full. my settings:

I had the same problem using a TimedRotatingFileHandler in combination with the local development server.

My solution which does not supply the --noreload flag is

if os.environ.get('RUN_MAIN') == "true":
    logging.config.dictConfig(LOGGING)
    logger = logging.getLogger(__name__)
else:
    logger = logging.getLogger('root')

It is not 100% clean as the second process gets a different logger but for me i wanted to have the auto reload feature enabled.

Logging to rotating files, Rotating logs with multiple workers in Django | The default Django The default Django logging settings make use of FileHandler which writes to a or even better TimedRotatingFileHandler to solve your problem, but alas  A common issue is when you first setup a daily logrotate.d entry, it will not rotate the first day. When you use a time based rotation (daily/weekly/monthly) logrotate scribbles a date stamp of the last date it saw the file in /var/lib/logrotate/status (or /var/lib/logrotate.status on RHEL systems).

Rotating logs with multiple workers in Django, The RotatingFileHandler class, located in the logging.handlers module, supports rotation of Problems. Logs missing when host in uwsgi with multiple process  Python’s logging library provides several techniques to configure logging, ranging from a programmatic interface to configuration files. By default, Django uses the dictConfig format . In order to configure logging, you use LOGGING to define a dictionary of logging settings.

RotatingFileHandler · Django Cook Book, For example, a logger named “scan” is the parent of loggers “scan.text”, “scan.​html” BaseRotatingHandler is the base class for handlers that rotate log files at a  Python’s logging module has lots of options. In this article, we will looks at the logging module’s ability to create Rotating Logs. Python supports two types of rotating logs: Rotate logs based on size (RotatingFileHandler) Rotate the logs based on some time interval (TimedRotatingFileHandler) Let’s spend some time learning how each of these two types … Continue reading Python: How to

15.6. logging — Logging facility for Python, How to log messages in Django, Python core logging concepts, LOGGING configuration For example, Python modules (i.e. .py files) that belong to a package named The log file is rotated every time a log file reaches 100 MB (i.e. maxBytes )  The WatchedFileHandler class, located in the logging.handlers module, is a FileHandler which watches the file it is logging to. If the file changes, it is closed and reopened using the file name. A file change can happen because of usage of programs such as newsyslog and logrotate which perform log file rotation.

Comments
  • have you tried disable_existing_loggers': True? And is your Django running under root or having write permission?
  • i am going to try it now @Anzel
  • yeah DJango is running under root @Anzel
  • it's the write permission of your log file, try this: chmod 777 boom-* and see if it fixes the problem. Then we will set a lower permission rights
  • @Anzel i already tried chmod 777 and nothing. i have updated my code above with the permission denied error i am getting when i try to run the command on production
  • Good catch. Do you think there's any other way to handle this situation without having to use --noreload?
  • Would be also intereset in a different way. Also the --noreload option still yields the error for me running on windows 10, django 2.1