June 3, 2012
Logging shell commands outputs with python
The analysis of sequencing data (at least in the first steps) is often a matter of starting multiple scripts one after an other.
I found out that to log the stdout and the stderr with python was not that much obivous for me. I struggled quite a bit with the logging module but finally I came with the following.
import os import subprocess import logging import logging.handlers import datetime as dt class ShellLaunchAndLog: ''' logging of shell commands output ''' def __init__(self): self._logFolder = '/tmp' def PopenLog(self, commandL, logFile): ''' Method to ensure shell scripts launch + logging ''' p = subprocess.Popen(commandL, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE) (stdout, stderr) = p.communicate() #create logger logger = logging.getLogger('sequence conversion') logger.setLevel(logging.DEBUG) # create file handler which logs even debug messages fh = logging.handlers.RotatingFileHandler(logFile, mode='a', maxBytes=0, backupCount=0, encoding=None, delay=0) fh.setLevel(logging.DEBUG) # create formatter and add it to the handlers formatR = logging.Formatter('%(asctime)s-%(levelname)s : %(message)s') fh.setFormatter(formatR) logger.addHandler(fh) logger.info(stdout) logger.error(stderr) logger.removeHandler(fh) def FolderListing(self): ''' lists the files contained in the current folder and logs the output into a file. ''' print(dt.datetime.now().strftime("%Y-%m-%d %H:%M") + ' Script started !') self.PopenLog(['ls', '-al'], os.path.join(self._logFolder, 'ls.log')) print(dt.datetime.now().strftime("%Y-%m-%d %H:%M") + ' Script finished !') Using this class in an interactive session of python allows you to log the results (and the errors) of the ‘ls -al’ in a file named ls.log in the /tmp folder.
Read more