tmhmm_and_signalp: tools/protein_analysis/seq_analysis

annotate tools/protein_analysis/seq_analysis_utils.py @ 31:3c9bbbc6cbbe draft

v0.2.11 take two; removed tabs in README file

author	peterjc
date	Thu, 21 Sep 2017 11:23:01 -0400
parents	6d9d7cdf00fc
children	20da7f48b56f

rev	line source
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	1 """A few useful functions for working with FASTA files and running jobs.
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	2
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	3 This module was originally written to hold common code used in both the TMHMM
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	4 and SignalP wrappers in Galaxy.
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	5
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	6 Given Galaxy currently supports Python 2.4+ this cannot use the Python module
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	7 multiprocessing so the function run_jobs instead is a simple pool approach
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	8 using just the subprocess library.
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	9 """
30 6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	10
6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	11 from __future__ import print_function
6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	12
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	13 import os
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	14 import subprocess
30 6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	15 import sys
6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	16
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	17 from time import sleep
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	18
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	19 __version__ = "0.0.2"
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	20
7 5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	21 try:
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	22 from multiprocessing import cpu_count
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	23 except ImportError:
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	24 # Must be under Python 2.5, this is copied from multiprocessing:
7 5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	25 def cpu_count():
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	26 """Returns the number of CPUs in the system."""
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	27 if sys.platform == 'win32':
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	28 try:
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	29 num = int(os.environ['NUMBER_OF_PROCESSORS'])
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	30 except (ValueError, KeyError):
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	31 num = 0
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	32 elif 'bsd' in sys.platform or sys.platform == 'darwin':
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	33 comm = '/sbin/sysctl -n hw.ncpu'
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	34 if sys.platform == 'darwin':
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	35 comm = '/usr' + comm
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	36 try:
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	37 with os.popen(comm) as p:
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	38 num = int(p.read())
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	39 except ValueError:
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	40 num = 0
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	41 else:
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	42 try:
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	43 num = os.sysconf('SC_NPROCESSORS_ONLN')
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	44 except (ValueError, OSError, AttributeError):
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	45 num = 0
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	46
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	47 if num >= 1:
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	48 return num
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	49 else:
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	50 raise NotImplementedError('cannot determine number of cpus')
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	51
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	52
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	53 def thread_count(command_line_arg, default=1):
30 6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	54 """Determine number of threads to use from the command line args."""
7 5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	55 try:
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	56 num = int(command_line_arg)
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	57 except ValueError:
7 5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	58 num = default
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	59 if num < 1:
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	60 sys.exit("Threads argument %r is not a positive integer" % command_line_arg)
3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	61 # Cap this with the pysical limit of the machine,
7 5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	62 try:
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	63 num = min(num, cpu_count())
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	64 except NotImplementedError:
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	65 pass
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	66 # For debugging,
3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	67 # hostname = os.environ.get("HOSTNAME", "this machine")
3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	68 # sys.stderr.write("Using %i cores on %s\n" % (num, hostname))
7 5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	69 return num
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	70
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	71
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	72 def fasta_iterator(filename, max_len=None, truncate=None):
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	73 """Simple FASTA parser yielding tuples of (title, sequence) strings."""
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	74 handle = open(filename)
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	75 title, seq = "", ""
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	76 for line in handle:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	77 if line.startswith(">"):
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	78 if title:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	79 if truncate:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	80 seq = seq[:truncate]
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	81 if max_len and len(seq) > max_len:
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	82 raise ValueError("Sequence %s is length %i, max length %i"
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	83 % (title.split()[0], len(seq), max_len))
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	84 yield title, seq
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	85 title = line[1:].rstrip()
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	86 seq = ""
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	87 elif title:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	88 seq += line.strip()
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	89 elif not line.strip() or line.startswith("#"):
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	90 # Ignore blank lines, and any comment lines
3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	91 # between records (starting with hash).
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	92 pass
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	93 else:
25 41a42022f815 Uploaded v0.2.6, embedded citations peterjc parents: 7 diff changeset	94 handle.close()
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	95 raise ValueError("Bad FASTA line %r" % line)
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	96 handle.close()
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	97 if title:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	98 if truncate:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	99 seq = seq[:truncate]
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	100 if max_len and len(seq) > max_len:
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	101 raise ValueError("Sequence %s is length %i, max length %i"
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	102 % (title.split()[0], len(seq), max_len))
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	103 yield title, seq
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	104 raise StopIteration
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	105
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	106
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	107 def split_fasta(input_filename, output_filename_base, n=500, truncate=None, keep_descr=False, max_len=None):
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	108 """Split FASTA file into sub-files each of at most n sequences.
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	109
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	110 Returns a list of the filenames used (based on the input filename).
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	111 Each sequence can also be truncated (since we only need the start for
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	112 SignalP), and have its description discarded (since we don't usually
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	113 care about it and some tools don't like very long title lines).
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	114
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	115 If a max_len is given and any sequence exceeds it no temp files are
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	116 created and an exception is raised.
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	117 """
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	118 iterator = fasta_iterator(input_filename, max_len, truncate)
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	119 files = []
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	120 try:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	121 while True:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	122 records = []
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	123 for i in range(n):
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	124 try:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	125 records.append(iterator.next())
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	126 except StopIteration:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	127 break
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	128 if not records:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	129 break
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	130 new_filename = "%s.%i.tmp" % (output_filename_base, len(files))
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	131 handle = open(new_filename, "w")
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	132 if keep_descr:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	133 for title, seq in records:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	134 handle.write(">%s\n" % title)
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	135 for i in range(0, len(seq), 60):
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	136 handle.write(seq[i:i + 60] + "\n")
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	137 else:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	138 for title, seq in records:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	139 handle.write(">%s\n" % title.split()[0])
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	140 for i in range(0, len(seq), 60):
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	141 handle.write(seq[i:i + 60] + "\n")
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	142 handle.close()
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	143 files.append(new_filename)
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	144 # print "%i records in %s" % (len(records), new_filename)
30 6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	145 except ValueError as err:
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	146 # Max length failure from parser - clean up
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	147 try:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	148 handle.close()
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	149 except Exception:
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	150 pass
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	151 for f in files:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	152 if os.path.isfile(f):
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	153 os.remove(f)
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	154 raise err
7 5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	155 for f in files:
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	156 assert os.path.isfile(f), "Missing split file %r (!??)" % f
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	157 return files
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	158
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	159
30 6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	160 def run_jobs(jobs, threads, pause=10, verbose=False, fast_fail=True):
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	161 """Takes list of cmd strings, returns dict with error levels."""
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	162 pending = jobs[:]
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	163 running = []
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	164 results = {}
30 6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	165 skipped = []
7 5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	166 if threads == 1:
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	167 # Special case this for speed, don't need the waits
7 5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	168 for cmd in jobs:
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	169 results[cmd] = subprocess.call(cmd, shell=True)
5e62aefb2918 Uploaded v0.1.2 to Test Tool Shed peterjc parents: 6 diff changeset	170 return results
30 6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	171 failed = False
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	172 while pending or running:
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	173 # See if any have finished
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	174 for (cmd, process) in running:
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	175 return_code = process.poll() # non-blocking
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	176 if return_code is not None:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	177 results[cmd] = return_code
30 6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	178 if return_code:
6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	179 failed = True
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	180 running = [(cmd, process) for (cmd, process) in running
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	181 if cmd not in results]
3 fe10f448d641 Migrated tool version 0.0.6 from old tool shed archive to new tool shed repository peterjc parents: 0 diff changeset	182 if verbose:
30 6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	183 print("%i jobs pending, %i running, %i completed" %
6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	184 (len(pending), len(running), len(results)))
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	185 # See if we can start any new threads
30 6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	186 if pending and failed and fast_fail:
6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	187 # Don't start any more jobs
6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	188 if verbose:
6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	189 print("Failed, will not start remaining %i jobs" % len(pending))
6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	190 skipped = pending
6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	191 pending = []
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	192 while pending and len(running) < threads:
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	193 cmd = pending.pop(0)
3 fe10f448d641 Migrated tool version 0.0.6 from old tool shed archive to new tool shed repository peterjc parents: 0 diff changeset	194 if verbose:
30 6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	195 print(cmd)
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	196 process = subprocess.Popen(cmd, shell=True)
a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	197 running.append((cmd, process))
29 3cb02adf4326 v0.2.9 Python style improvements peterjc parents: 26 diff changeset	198 # Loop...
6 39a6e46cdda3 Migrated tool version 0.0.9 from old tool shed archive to new tool shed repository peterjc parents: 3 diff changeset	199 sleep(pause)
3 fe10f448d641 Migrated tool version 0.0.6 from old tool shed archive to new tool shed repository peterjc parents: 0 diff changeset	200 if verbose:
30 6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	201 print("%i jobs completed" % len(results))
6d9d7cdf00fc v0.2.11 Job splitting fast-fail; RXLR tools supports HMMER2 from BioConda; Capture more version information; misc internal changes peterjc parents: 29 diff changeset	202 assert set(jobs) == set(results).union(skipped)
0 a2eeeaa6f75e Migrated tool version 0.0.1 from old tool shed archive to new tool shed repository peterjc parents: diff changeset	203 return results

Mercurial > repos > peterjc > tmhmm_and_signalp

annotate tools/protein_analysis/seq_analysis_utils.py @ 31:3c9bbbc6cbbe draft