Logging shell commands outputs with python

The analysis of sequencing data (at least in the first steps) is often a matter of starting multiple scripts one after an other.

I found out that to log the stdout and the stderr with python was not that much obivous for me. I struggled quite a bit with the logging module but finally I came with the following.


import os
import subprocess
import logging
import logging.handlers
import datetime as dt


class ShellLaunchAndLog:
    '''
    logging of shell commands output
    '''
    def __init__(self):
        self._logFolder = '/tmp'
    
    def PopenLog(self, commandL, logFile):
        '''
        Method to ensure shell scripts launch + logging
        '''
        p = subprocess.Popen(commandL, shell=False, 
            stdout=subprocess.PIPE, stderr=subprocess.PIPE)
        (stdout, stderr) = p.communicate()
        #create logger
        logger = logging.getLogger('sequence conversion')
        logger.setLevel(logging.DEBUG)
        # create file handler which logs even debug messages
        fh = logging.handlers.RotatingFileHandler(logFile, mode='a', 
            maxBytes=0, backupCount=0, encoding=None, delay=0)
        fh.setLevel(logging.DEBUG)
        # create formatter and add it to the handlers
        formatR = logging.Formatter('%(asctime)s-%(levelname)s : %(message)s')
        fh.setFormatter(formatR)
        logger.addHandler(fh)
        logger.info(stdout)
        logger.error(stderr)
        logger.removeHandler(fh)
    
    def FolderListing(self):
        '''
        lists the files contained in the current folder
        and logs the output into a file.
        '''
        print(dt.datetime.now().strftime("%Y-%m-%d %H:%M") + ' Script started !')
        self.PopenLog(['ls', '-al'], os.path.join(self._logFolder, 'ls.log'))
        print(dt.datetime.now().strftime("%Y-%m-%d %H:%M") + ' Script finished !')

Using this class in an interactive session of python allows you to log the results (and the errors) of the ‘ls -al’ in a file named ls.log in the /tmp folder.

This is of course just a starting point with a silly example of logging ‘ls -al’ shell command output.

If you really want to use this all you need to do is to replace ‘folderlisting’ method by a few one fitting to the scripts you would like to start and log the results.

It is fairly possible that the code presented is not the usual way to make it (see DISCLAIMER) but at least it has been working for me.

June 3, 2012 - 1 min read

Logging shell commands outputs with python