The analysis of sequencing data (at least in the first steps) is often a matter of starting multiple scripts one after an other.
I found out that to log the stdout and the stderr with python was not that much obivous for me. I struggled quite a bit with the logging module but finally I came with the following.
import os import subprocess import logging import logging.handlers import datetime as dt class ShellLaunchAndLog: ''' logging of shell commands output ''' def __init__(self): self._logFolder = '/tmp' def PopenLog(self, commandL, logFile): ''' Method to ensure shell scripts launch + logging ''' p = subprocess.Popen(commandL, shell=False, stdout=subprocess.PIPE, stderr=subprocess.PIPE) (stdout, stderr) = p.communicate() #create logger logger = logging.getLogger('sequence conversion') logger.setLevel(logging.DEBUG) # create file handler which logs even debug messages fh = logging.handlers.RotatingFileHandler(logFile, mode='a', maxBytes=0, backupCount=0, encoding=None, delay=0) fh.setLevel(logging.DEBUG) # create formatter and add it to the handlers formatR = logging.Formatter('%(asctime)s-%(levelname)s : %(message)s') fh.setFormatter(formatR) logger.addHandler(fh) logger.info(stdout) logger.error(stderr) logger.removeHandler(fh) def FolderListing(self): ''' lists the files contained in the current folder and logs the output into a file. ''' print(dt.datetime.now().strftime("%Y-%m-%d %H:%M") + ' Script started !') self.PopenLog(['ls', '-al'], os.path.join(self._logFolder, 'ls.log')) print(dt.datetime.now().strftime("%Y-%m-%d %H:%M") + ' Script finished !')
Using this class in an interactive session of python allows you to log the results (and the errors) of the ‘ls -al’ in a file named ls.log in the /tmp folder.
This is of course just a starting point with a silly example of logging ‘ls -al’ shell command output.
If you really want to use this all you need to do is to replace ‘folderlisting’ method by a few one fitting to the scripts you would like to start and log the results.
It is fairly possible that the code presented is not the usual way to make it (see DISCLAIMER) but at least it has been working for me.