Mercurial > repos > drosofff > fetch_fasta_from_ncbi

annotate retrieve_fasta_from_NCBI.py @ 14:54941746784b draft default tip

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
author drosofff
date Fri, 16 Jun 2017 05:28:45 -0400
parents 639daa4c3c1a
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
1 #!/usr/bin/env python
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
2 # -*- coding: utf-8 -*-
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
3 """
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
4 From a taxonomy ID retrieves all the nucleotide sequences
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
5 It returns a multiFASTA nuc/prot file
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
6
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
7 Entrez Database UID common name E-utility Database Name
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
8 Nucleotide GI number nuccore
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
9 Protein GI number protein
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
10
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
11 Retrieve strategy:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
12
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
13 esearch to get total number of UIDs (count)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
14 esearch to get UIDs in batches
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
15 loop untile end of UIDs list:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
16 epost to put a batch of UIDs in the history server
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
17 efetch to retrieve info from previous post
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
18
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
19 retmax of efetch is 1/10 of declared value from NCBI
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
20
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
21 queries are 1 sec delayed, to satisfy NCBI guidelines (more than what they request)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
22
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
23
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
24 """
2
e9df554f7725 Uploaded
drosofff
parents: 1
diff changeset
25 import sys
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
26 import logging
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
27 import optparse
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
28 import time
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
29 import urllib
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
30 import urllib2
7
cd7de2d6c716 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 5
diff changeset
31 import httplib
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
32 import re
12
0bec3cba5c56 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit b6de14061c479f0418cd89e26d6f5ac26e565a07
drosofff
parents: 10
diff changeset
33
0bec3cba5c56 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit b6de14061c479f0418cd89e26d6f5ac26e565a07
drosofff
parents: 10
diff changeset
34
14
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
35 class QueryException(Exception):
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
36 pass
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
37
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
38
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
39 class Eutils:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
40
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
41 def __init__(self, options, logger):
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
42 """
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
43 Initialize retrieval parameters
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
44 """
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
45 self.logger = logger
12
0bec3cba5c56 planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit b6de14061c479f0418cd89e26d6f5ac26e565a07
drosofff
parents: 10
diff changeset
46 self.base = "https://eutils.ncbi.nlm.nih.gov/entrez/eutils/"
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
47 self.query_string = options.query_string
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
48 self.dbname = options.dbname
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
49 if options.outname:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
50 self.outname = options.outname
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
51 else:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
52 self.outname = 'NCBI_download' + '.' + self.dbname + '.fasta'
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
53 self.ids = []
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
54 self.retmax_esearch = 100000
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
55 self.retmax_efetch = 500
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
56 self.count = 0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
57 self.webenv = ""
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
58 self.query_key = ""
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
59
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
60 def retrieve(self):
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
61 """
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
62 Retrieve the fasta sequences corresponding to the query
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
63 """
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
64 self.get_count_value()
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
65
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
66 # If no UIDs are found exit script
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
67 if self.count > 0:
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
68 self.get_uids_list()
14
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
69 try:
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
70 self.get_sequences()
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
71 except QueryException as e:
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
72 self.logger.error("Exiting script.")
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
73 raise e
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
74 else:
14
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
75 self.logger.error("No UIDs were found. Exiting script.")
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
76 raise Exception("")
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
77
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
78 def get_count_value(self):
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
79 """
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
80 just to retrieve Count (number of UIDs)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
81 Total number of UIDs from the retrieved set to be shown in the XML
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
82 output (default=20). By default, ESearch only includes the first 20
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
83 UIDs retrieved in the XML output. If usehistory is set to 'y',
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
84 the remainder of the retrieved set will be stored on the History server;
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
85
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
86 http://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.EFetch
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
87 """
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
88 self.logger.info("retrieving data from %s" % self.base)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
89 self.logger.info("for Query: %s and database: %s" %
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
90 (self.query_string, self.dbname))
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
91 querylog = self.esearch(self.dbname, self.query_string, '', '', "count")
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
92 self.logger.debug("Query response:")
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
93 for line in querylog:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
94 self.logger.debug(line.rstrip())
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
95 if '</Count>' in line:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
96 self.count = int(line[line.find('<Count>')+len('<Count>') : line.find('</Count>')])
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
97 self.logger.info("Found %d UIDs" % self.count)
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
98
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
99 def get_uids_list(self):
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
100 """
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
101 Increasing retmax allows more of the retrieved UIDs to be included in the XML output,
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
102 up to a maximum of 100,000 records.
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
103 from http://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.ESearch
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
104 """
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
105 retmax = self.retmax_esearch
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
106 if (self.count > retmax):
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
107 num_batches = (self.count / retmax) + 1
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
108 else:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
109 num_batches = 1
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
110 self.logger.info("Batch size for esearch action: %d UIDs" % retmax)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
111 self.logger.info("Number of batches for esearch action: %d " % num_batches)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
112 for n in range(num_batches):
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
113 querylog = self.esearch(self.dbname, self.query_string, n*retmax, retmax, '')
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
114 for line in querylog:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
115 if '<Id>' in line and '</Id>' in line:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
116 uid = (line[line.find('<Id>')+len('<Id>') : line.find('</Id>')])
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
117 self.ids.append(uid)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
118 self.logger.info("Retrieved %d UIDs" % len(self.ids))
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
119
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
120 def esearch(self, db, term, retstart, retmax, rettype):
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
121 url = self.base + "esearch.fcgi"
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
122 self.logger.debug("url: %s" % url)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
123 values = {'db': db,
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
124 'term': term,
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
125 'rettype': rettype,
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
126 'retstart': retstart,
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
127 'retmax': retmax}
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
128 data = urllib.urlencode(values)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
129 self.logger.debug("data: %s" % str(data))
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
130 req = urllib2.Request(url, data)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
131 response = urllib2.urlopen(req)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
132 querylog = response.readlines()
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
133 response.close()
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
134 time.sleep(1)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
135 return querylog
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
136
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
137 def epost(self, db, ids):
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
138 url = self.base + "epost.fcgi"
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
139 self.logger.debug("url_epost: %s" % url)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
140 values = {'db': db,
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
141 'id': ids}
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
142 data = urllib.urlencode(values)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
143 req = urllib2.Request(url, data)
2
e9df554f7725 Uploaded
drosofff
parents: 1
diff changeset
144 serverResponse = False
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
145 nb_trials = 0
2
e9df554f7725 Uploaded
drosofff
parents: 1
diff changeset
146 while not serverResponse:
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
147 nb_trials += 1
2
e9df554f7725 Uploaded
drosofff
parents: 1
diff changeset
148 try:
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
149 self.logger.debug("Try number %s for opening and readin URL %s" % ( nb_trials, url+data ))
2
e9df554f7725 Uploaded
drosofff
parents: 1
diff changeset
150 response = urllib2.urlopen(req)
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
151 querylog = response.readlines()
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
152 response.close()
2
e9df554f7725 Uploaded
drosofff
parents: 1
diff changeset
153 serverResponse = True
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
154 except urllib2.HTTPError as e:
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
155 self.logger.info("urlopen error:%s, %s" % (e.code, e.read() ) )
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
156 self.logger.info("Retrying in 1 sec")
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
157 serverResponse = False
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
158 time.sleep(1)
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
159 except urllib2.URLError as e:
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
160 self.logger.info("urlopen error: Failed to reach a server")
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
161 self.logger.info("Reason :%s" % ( e.reason ) )
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
162 self.logger.info("Retrying in 1 sec")
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
163 serverResponse = False
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
164 time.sleep(1)
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
165 except httplib.IncompleteRead as e:
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
166 self.logger.info("IncompleteRead error: %s" % ( e.partial ) )
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
167 self.logger.info("Retrying in 1 sec")
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
168 serverResponse = False
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
169 time.sleep(1)
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
170 self.logger.debug("query response:")
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
171 for line in querylog:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
172 self.logger.debug(line.rstrip())
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
173 if '</QueryKey>' in line:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
174 self.query_key = str(line[line.find('<QueryKey>')+len('<QueryKey>'):line.find('</QueryKey>')])
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
175 if '</WebEnv>' in line:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
176 self.webenv = str(line[line.find('<WebEnv>')+len('<WebEnv>'):line.find('</WebEnv>')])
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
177 self.logger.debug("*** epost action ***")
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
178 self.logger.debug("query_key: %s" % self.query_key)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
179 self.logger.debug("webenv: %s" % self.webenv)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
180 time.sleep(1)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
181
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
182 def efetch(self, db, query_key, webenv):
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
183 url = self.base + "efetch.fcgi"
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
184 self.logger.debug("url_efetch: %s" % url)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
185 values = {'db': db,
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
186 'query_key': query_key,
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
187 'webenv': webenv,
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
188 'rettype': "fasta",
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
189 'retmode': "text"}
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
190 data = urllib.urlencode(values)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
191 req = urllib2.Request(url, data)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
192 self.logger.debug("data: %s" % str(data))
8
cc43a7a11324 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 7
diff changeset
193 serverTransaction = False
cc43a7a11324 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 7
diff changeset
194 counter = 0
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
195 response_code = 0
8
cc43a7a11324 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 7
diff changeset
196 while not serverTransaction:
cc43a7a11324 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 7
diff changeset
197 counter += 1
cc43a7a11324 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 7
diff changeset
198 self.logger.info("Server Transaction Trial: %s" % ( counter ) )
2
e9df554f7725 Uploaded
drosofff
parents: 1
diff changeset
199 try:
e9df554f7725 Uploaded
drosofff
parents: 1
diff changeset
200 response = urllib2.urlopen(req)
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
201 response_code = response.getcode()
8
cc43a7a11324 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 7
diff changeset
202 fasta = response.read()
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
203 response.close()
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
204 if ( (response_code != 200) or ("Resource temporarily unavailable" in fasta)
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
205 or ("Error" in fasta) or (not fasta.startswith(">") ) ):
8
cc43a7a11324 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 7
diff changeset
206 serverTransaction = False
14
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
207 if ( response_code != 200 ):
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
208 self.logger.info("urlopen error: Response code is not 200")
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
209 elif ( "Resource temporarily unavailable" in fasta ):
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
210 self.logger.info("Ressource temporarily unavailable")
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
211 elif ( "Error" in fasta ):
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
212 self.logger.info("Error in fasta")
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
213 else:
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
214 self.logger.info("Fasta doesn't start with '>'")
8
cc43a7a11324 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 7
diff changeset
215 else:
cc43a7a11324 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 7
diff changeset
216 serverTransaction = True
2
e9df554f7725 Uploaded
drosofff
parents: 1
diff changeset
217 except urllib2.HTTPError as e:
8
cc43a7a11324 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 7
diff changeset
218 serverTransaction = False
2
e9df554f7725 Uploaded
drosofff
parents: 1
diff changeset
219 self.logger.info("urlopen error:%s, %s" % (e.code, e.read() ) )
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
220 except urllib2.URLError as e:
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
221 serverTransaction = False
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
222 self.logger.info("urlopen error: Failed to reach a server")
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
223 self.logger.info("Reason :%s" % ( e.reason ) )
8
cc43a7a11324 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 7
diff changeset
224 except httplib.IncompleteRead as e:
cc43a7a11324 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 7
diff changeset
225 serverTransaction = False
cc43a7a11324 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 7
diff changeset
226 self.logger.info("IncompleteRead error: %s" % ( e.partial ) )
14
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
227 if (counter > 500):
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
228 serverTransaction = True
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
229 if (counter > 500):
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
230 raise QueryException({"message":"500 Server Transaction Trials attempted for this batch. Aborting."})
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
231 fasta = self.sanitiser(self.dbname, fasta)
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
232 time.sleep(0.1)
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
233 return fasta
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
234
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
235 def sanitiser(self, db, fastaseq):
5
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
236 if db not in "nuccore protein" : return fastaseq
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
237 regex = re.compile(r"[ACDEFGHIKLMNPQRSTVWYBZ]{49,}")
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
238 sane_seqlist = []
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
239 seqlist = fastaseq.split("\n\n")
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
240 for seq in seqlist[:-1]:
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
241 fastalines = seq.split("\n")
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
242 if len(fastalines) < 2:
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
243 self.logger.info("Empty sequence for %s" % ("|".join(fastalines[0].split("|")[:4]) ) )
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
244 self.logger.info("%s download is skipped" % ("|".join(fastalines[0].split("|")[:4]) ) )
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
245 continue
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
246 if db == "nuccore":
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
247 badnuc = 0
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
248 for nucleotide in fastalines[1]:
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
249 if nucleotide not in "ATGC":
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
250 badnuc += 1
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
251 if float(badnuc)/len(fastalines[1]) > 0.4:
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
252 self.logger.info("%s ambiguous nucleotides in %s or download interrupted at this offset | %s" % ( float(badnuc)/len(fastalines[1]), "|".join(fastalines[0].split("|")[:4]), fastalines[1]) )
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
253 self.logger.info("%s download is skipped" % (fastalines[0].split("|")[:4]) )
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
254 continue
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
255 fastalines[0] = fastalines[0].replace(" ","_")[:100] # remove spaces and trim the header to 100 chars
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
256 cleanseq = "\n".join(fastalines)
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
257 sane_seqlist.append(cleanseq)
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
258 elif db == "protein":
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
259 fastalines[0] = fastalines[0][0:100]
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
260 fastalines[0] = fastalines[0].replace(" ", "_")
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
261 fastalines[0] = fastalines[0].replace("[", "_")
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
262 fastalines[0] = fastalines[0].replace("]", "_")
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
263 fastalines[0] = fastalines[0].replace("=", "_")
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
264 fastalines[0] = fastalines[0].rstrip("_") # because blast makedb doesn't like it
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
265 fastalines[0] = re.sub(regex, "_", fastalines[0])
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
266 cleanseq = "\n".join(fastalines)
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
267 sane_seqlist.append(cleanseq)
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
268 self.logger.info("clean sequences appended: %d" % (len(sane_seqlist) ) )
4ff395248db4 planemo upload for repository https://bitbucket.org/drosofff/gedtools/
drosofff
parents: 2
diff changeset
269 return "\n".join(sane_seqlist)
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
270
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
271 def get_sequences(self):
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
272 """
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
273 Total number of records from the input set to be retrieved, up to a maximum
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
274 of 10,000. Optionally, for a large set the value of retstart can be iterated
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
275 while holding retmax constant, thereby downloading the entire set in batches
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
276 of size retmax.
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
277
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
278 http://www.ncbi.nlm.nih.gov/books/NBK25499/#chapter4.EFetch
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
279
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
280 """
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
281 batch_size = self.retmax_efetch
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
282 count = self.count
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
283 uids_list = self.ids
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
284 self.logger.info("Batch size for efetch action: %d" % batch_size)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
285 self.logger.info("Number of batches for efetch action: %d" % ((count / batch_size) + 1))
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
286 with open(self.outname, 'w') as out:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
287 for start in range(0, count, batch_size):
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
288 end = min(count, start+batch_size)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
289 batch = uids_list[start:end]
13
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
290 if self.epost(self.dbname, ",".join(batch)) != -1:
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
291 mfasta = ''
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
292 while not mfasta:
639daa4c3c1a planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit ca132a4b5d5d7175e6e8bd62cc518397d14dad17
drosofff
parents: 12
diff changeset
293 self.logger.info("retrieving batch %d" % ((start / batch_size) + 1))
14
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
294 try:
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
295 mfasta = self.efetch(self.dbname, self.query_key, self.webenv)
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
296 out.write(mfasta + '\n')
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
297 except QueryException as e:
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
298 self.logger.error("%s" % e.message)
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
299 raise e
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
300
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
301
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
302 LOG_FORMAT = '%(asctime)s|%(levelname)-8s|%(message)s'
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
303 LOG_DATEFMT = '%Y-%m-%d %H:%M:%S'
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
304 LOG_LEVELS = ['DEBUG', 'INFO', 'WARNING', 'ERROR', 'CRITICAL']
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
305
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
306
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
307 def __main__():
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
308 """ main function """
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
309 parser = optparse.OptionParser(description='Retrieve data from NCBI')
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
310 parser.add_option('-i', dest='query_string', help='NCBI Query String')
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
311 parser.add_option('-o', dest='outname', help='output file name')
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
312 parser.add_option('-l', '--logfile', help='log file (default=stderr)')
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
313 parser.add_option('--loglevel', choices=LOG_LEVELS, default='INFO', help='logging level (default: INFO)')
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
314 parser.add_option('-d', dest='dbname', help='database type')
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
315 (options, args) = parser.parse_args()
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
316 if len(args) > 0:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
317 parser.error('Wrong number of arguments')
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
318
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
319 log_level = getattr(logging, options.loglevel)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
320 kwargs = {'format': LOG_FORMAT,
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
321 'datefmt': LOG_DATEFMT,
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
322 'level': log_level}
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
323 if options.logfile:
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
324 kwargs['filename'] = options.logfile
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
325 logging.basicConfig(**kwargs)
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
326 logger = logging.getLogger('data_from_NCBI')
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
327
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
328 E = Eutils(options, logger)
14
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
329 try:
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
330 E.retrieve()
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
331 except Exception as e:
54941746784b planemo upload for repository https://github.com/ARTbio/tools-artbio/tree/master/tools/fetch_fasta_from_ncbi commit 11ca680184135ef39a6c552d9f3cc427a8ed6c4c
drosofff
parents: 13
diff changeset
332 sys.exit(1)
0
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
333
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
334
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
335 if __name__ == "__main__":
4b34f2b5c14e Uploaded
drosofff
parents:
diff changeset
336 __main__()