Mercurial > repos > devteam > data_manager_rsync_g2

annotate data_manager/data_manager_rsync.py @ 0:70afa70bba41 draft

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
author devteam
date Fri, 06 Nov 2015 14:18:13 -0500
parents
children 861c071a6cd5
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
1 #!/usr/bin/env python
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
2 #Dan Blankenberg
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
3
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
4 import sys
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
5 import os
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
6 import tempfile
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
7 import shutil
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
8 import optparse
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
9 import urllib2
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
10 import subprocess
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
11 import datetime
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
12 from os.path import basename
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
13 from json import loads, dumps
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
14 from xml.etree.ElementTree import tostring
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
15
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
16 import logging
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
17 _log_name = __name__
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
18 if _log_name == '__builtin__':
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
19 _log_name = 'toolshed.installed.g2.rsync.data.manager'
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
20 log = logging.getLogger( _log_name )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
21
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
22 # Get the Data from the Galaxy Project rsync server
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
23 RSYNC_CMD = 'rsync'
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
24 RSYNC_SERVER = "rsync://datacache.g2.bx.psu.edu/"
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
25 LOCATION_DIR = "location"
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
26 INDEX_DIR = "indexes"
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
27
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
28 # Pull the Tool Data Table files from github
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
29 # FIXME: These files should be accessible from the rsync server directly.
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
30 TOOL_DATA_TABLE_CONF_XML_URLS = { 'main':"https://raw.githubusercontent.com/galaxyproject/usegalaxy-playbook/master/files/galaxy/usegalaxy.org/config/tool_data_table_conf.xml",
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
31 'test':"https://raw.githubusercontent.com/galaxyproject/usegalaxy-playbook/master/files/galaxy/test.galaxyproject.org/config/tool_data_table_conf.xml" }
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
32
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
33 # Replace data table source entries with local temporary location
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
34 GALAXY_DATA_CANONICAL_PATH = "/galaxy/data/"
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
35 TOOL_DATA_TABLE_CONF_XML_REPLACE_SOURCE = '<file path="%slocation/' % ( GALAXY_DATA_CANONICAL_PATH )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
36 TOOL_DATA_TABLE_CONF_XML_REPLACE_TARGET = '<file path="%s/'
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
37
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
38 # Some basic Caching, so we don't have to reload and download everything every time
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
39 CACHE_TIME = datetime.timedelta( minutes=10 )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
40 TOOL_DATA_TABLES_LOADED_BY_URL = {}
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
41
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
42 # Entries will not be selected by default
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
43 DEFAULT_SELECTED = False
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
44
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
45 # Exclude data managers without 'path' column or that are in the manual exclude list
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
46 PATH_COLUMN_NAMES = ['path']
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
47 EXCLUDE_DATA_TABLES = []
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
48 # TODO: Make additional handler actions available for tables that can't fit into the the basic
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
49 # "take the value of path" as a dir and copy contents.
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
50 # e.g. mafs. Although this maf table is goofy and doesn't have path defined in <table> def,
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
51 # but it does exit in the .loc.
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
52
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
53 # --- These methods are called by/within the Galaxy Application
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
54
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
55 def exec_before_job( app, inp_data, out_data, param_dict, tool=None, **kwd ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
56 # Look for any data tables that haven't been defined for this data manager before and dynamically add them to Galaxy
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
57 param_dict = dict( **param_dict )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
58 param_dict['data_table_entries'] = param_dict.get( 'data_table_entries', [] )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
59 if not isinstance( param_dict['data_table_entries'], list ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
60 param_dict['data_table_entries'] = [param_dict['data_table_entries']]
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
61 param_dict['data_table_entries'] = ",".join( param_dict['data_table_entries'] )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
62 if tool:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
63 tool_shed_repository = tool.tool_shed_repository
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
64 else:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
65 tool_shed_repository = None
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
66 tdtm = None
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
67 data_manager = app.data_managers.get_manager( tool.data_manager_id, None )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
68 data_table_entries = get_data_table_entries( param_dict )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
69 data_tables = load_data_tables_from_url( data_table_class=app.tool_data_tables.__class__ ).get( 'data_tables' )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
70 for data_table_name, entries in data_table_entries.iteritems():
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
71 #get data table managed by this data Manager
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
72 has_data_table = app.tool_data_tables.get_tables().get( data_table_name )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
73 if has_data_table:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
74 has_data_table = bool( has_data_table.get_filename_for_source( data_manager, None ) )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
75 if not has_data_table:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
76 if tdtm is None:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
77 from tool_shed.tools import data_table_manager
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
78 tdtm = data_table_manager.ToolDataTableManager( app )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
79 target_dir, tool_path, relative_target_dir = tdtm.get_target_install_dir( tool_shed_repository )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
80 #Dynamically add this data table
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
81 log.debug( "Attempting to dynamically create a missing Tool Data Table named %s." % data_table_name )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
82 data_table = data_tables[data_table_name]
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
83 repo_info = tdtm.generate_repository_info_elem_from_repository( tool_shed_repository, parent_elem=None )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
84 if repo_info is not None:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
85 repo_info = tostring( repo_info )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
86 tmp_file = tempfile.NamedTemporaryFile()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
87 tmp_file.write( get_new_xml_definition( app, data_table, data_manager, repo_info, target_dir ) )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
88 tmp_file.flush()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
89 app.tool_data_tables.add_new_entries_from_config_file( tmp_file.name, None, app.config.shed_tool_data_table_config, persist=True )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
90 tmp_file.close()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
91
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
92 def galaxy_code_get_available_data_tables( trans ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
93 #list of data tables
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
94 found_tables = get_available_tables( trans )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
95 rval = map( lambda x: ( ( x, x, DEFAULT_SELECTED ) ), found_tables )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
96 return rval
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
97
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
98 def galaxy_code_get_available_data_tables_entries( trans, dbkey, data_table_names ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
99 #available entries, optionally filtered by dbkey and table names
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
100 if dbkey in [ None, '', '?' ]:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
101 dbkey = None
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
102 if data_table_names in [ None, '', '?' ]:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
103 data_table_names = None
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
104 found_tables = get_available_tables_for_dbkey( trans, dbkey, data_table_names )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
105 dbkey_text = '(%s) ' % ( dbkey ) if dbkey else ''
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
106 rval = map( lambda x: ( "%s%s" % ( dbkey_text, x[0] ), dumps( dict( name=x[0].split( ': ' )[0], entry=x[1] ) ).encode( 'base64' ).rstrip(), DEFAULT_SELECTED ), found_tables.items() )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
107 return rval
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
108
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
109 # --- End Galaxy called Methods ---
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
110
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
111
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
112 def rsync_urljoin( base, url ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
113 # urlparse.urljoin doesn't work correctly for our use-case
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
114 # probably because it doesn't recognize the rsync scheme
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
115 base = base.rstrip( '/' )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
116 url = url.lstrip( '/' )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
117 return "%s/%s" % ( base, url )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
118
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
119 def rsync_list_dir( server, dir=None, skip_names=[] ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
120 #drwxr-xr-x 50 2014/05/16 20:58:11 .
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
121 if dir:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
122 dir = rsync_urljoin( server, dir )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
123 else:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
124 dir = server
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
125 rsync_response = tempfile.NamedTemporaryFile()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
126 rsync_stderr = tempfile.NamedTemporaryFile()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
127 rsync_cmd = [ RSYNC_CMD, '--list-only', dir ]
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
128 return_code = subprocess.call( rsync_cmd, stdout=rsync_response, stderr=rsync_stderr )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
129 rsync_response.flush()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
130 rsync_response.seek(0)
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
131 rsync_stderr.flush()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
132 rsync_stderr.seek(0)
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
133 if return_code:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
134 msg = "stdout:\n%s\nstderr:\n%s" % ( rsync_response.read(), rsync_stderr.read() )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
135 rsync_response.close()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
136 rsync_stderr.close()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
137 raise Exception( 'Failed to execute rsync command (%s), returncode=%s. Rsync_output:\n%s' % ( rsync_cmd, return_code, msg ) )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
138 rsync_stderr.close()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
139 rval = {}
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
140 for line in rsync_response:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
141 perms, line = line.split( None, 1 )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
142 line = line.strip()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
143 size, line = line.split( None, 1 )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
144 line = line.strip()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
145 date, line = line.split( None, 1 )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
146 line = line.strip()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
147 time, line = line.split( None, 1 )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
148 name = line.strip()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
149 if name in skip_names:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
150 continue
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
151 size = line.strip()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
152 rval[ name ] = dict( name=name, permissions=perms, bytes=size, date=date, time=time )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
153 rsync_response.close()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
154 return rval
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
155
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
156 def rsync_sync_to_dir( source, target ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
157 rsync_response = tempfile.NamedTemporaryFile()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
158 rsync_stderr = tempfile.NamedTemporaryFile()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
159 rsync_cmd = [ RSYNC_CMD, '-avzP', source, target ]
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
160 return_code = subprocess.call( rsync_cmd, stdout=rsync_response, stderr=rsync_stderr )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
161 rsync_response.flush()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
162 rsync_response.seek(0)
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
163 rsync_stderr.flush()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
164 rsync_stderr.seek(0)
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
165 if return_code:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
166 msg = "stdout:\n%s\nstderr:\n%s" % ( rsync_response.read(), rsync_stderr.read() )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
167 rsync_response.close()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
168 rsync_stderr.close()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
169 raise Exception( 'Failed to execute rsync command (%s), returncode=%s. Rsync_output:\n%s' % ( rsync_cmd, return_code, msg ) )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
170 rsync_response.close()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
171 rsync_stderr.close()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
172 return return_code
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
173
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
174
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
175 def data_table_needs_refresh( cached_data_table, url ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
176 if cached_data_table is None:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
177 return True, {}
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
178 if datetime.datetime.now() - cached_data_table.get( 'time_loaded' ) > CACHE_TIME:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
179 data_table_text = urllib2.urlopen( url ).read()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
180 if cached_data_table.get( 'data_table_text', None ) != data_table_text:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
181 return True, {'data_table_text':data_table_text}
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
182 loc_file_attrs = rsync_list_dir( RSYNC_SERVER, LOCATION_DIR )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
183 if cached_data_table.get( 'loc_file_attrs', None ) != loc_file_attrs:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
184 return True, {'loc_file_attrs':loc_file_attrs}
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
185 return False, {}
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
186
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
187 def load_data_tables_from_url( url=None, site='main', data_table_class=None ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
188 if not url:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
189 url = TOOL_DATA_TABLE_CONF_XML_URLS.get( site, None )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
190 assert url, ValueError( 'You must provide either a URL or a site=name.' )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
191
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
192 cached_data_table = TOOL_DATA_TABLES_LOADED_BY_URL.get( url, None )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
193 refresh, attribs = data_table_needs_refresh( cached_data_table, url )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
194 if refresh:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
195 data_table_text = attribs.get( 'data_table_text' )or urllib2.urlopen( url ).read()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
196 loc_file_attrs = attribs.get( 'loc_file_attrs' ) or rsync_list_dir( RSYNC_SERVER, LOCATION_DIR )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
197
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
198 tmp_dir = tempfile.mkdtemp( prefix='rsync_g2_' )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
199 tmp_loc_dir = os.path.join( tmp_dir, 'location' )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
200 os.mkdir( tmp_loc_dir )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
201 rsync_sync_to_dir( rsync_urljoin( RSYNC_SERVER, LOCATION_DIR ), os.path.abspath( tmp_loc_dir ) )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
202
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
203
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
204 new_data_table_text = data_table_text.replace( TOOL_DATA_TABLE_CONF_XML_REPLACE_SOURCE, TOOL_DATA_TABLE_CONF_XML_REPLACE_TARGET % ( tmp_loc_dir ) )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
205 data_table_fh = tempfile.NamedTemporaryFile( dir=tmp_dir, prefix='rysnc_data_manager_data_table_conf_' )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
206 data_table_fh.write( new_data_table_text )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
207 data_table_fh.flush()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
208 tmp_data_dir = os.path.join( tmp_dir, 'tool-data' )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
209 os.mkdir( tmp_data_dir )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
210 data_tables = data_table_class( tmp_data_dir, config_filename=data_table_fh.name )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
211 for name, data_table in data_tables.data_tables.items():
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
212 if name in EXCLUDE_DATA_TABLES or not data_table_has_path_column( data_table ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
213 log.debug( 'Removing data table "%s" because it is excluded by name or does not have a defined "path" column.', name )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
214 del data_tables.data_tables[name]
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
215 cached_data_table = { 'data_tables': data_tables, 'tmp_dir': tmp_dir, 'data_table_text': data_table_text, 'tmp_loc_dir': tmp_loc_dir, 'loc_file_attrs': loc_file_attrs, 'time_loaded': datetime.datetime.now() }
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
216 TOOL_DATA_TABLES_LOADED_BY_URL[ url ] = cached_data_table
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
217 #delete the files
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
218 data_table_fh.close()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
219 cleanup_before_exit( tmp_dir )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
220 return cached_data_table
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
221
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
222 def data_table_has_path_column( data_table ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
223 col_names = data_table.get_column_name_list()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
224 for name in PATH_COLUMN_NAMES:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
225 if name in col_names:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
226 return True
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
227 return False
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
228
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
229 def get_available_tables( trans ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
230 #list of data tables
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
231 data_tables = load_data_tables_from_url( data_table_class=trans.app.tool_data_tables.__class__ )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
232 return data_tables.get( 'data_tables' ).get_tables().keys()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
233
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
234 def get_new_xml_definition( app, data_table, data_manager, repo_info=None, location_file_dir=None ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
235 sub_dict = { 'table_name': data_table.name, 'comment_char': '', 'columns': '', 'file_path': '' }
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
236 sub_dict.update( data_manager.get_tool_shed_repository_info_dict() )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
237 if data_table.comment_char:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
238 sub_dict['comment_char'] = 'comment_char="%s"' % ( data_table.comment_char )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
239 for i, name in enumerate( data_table.get_column_name_list() ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
240 if name is not None:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
241 sub_dict['columns'] = "%s\n%s" % ( sub_dict['columns'], '<column name="%s" index="%s" />' % ( name, i ) )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
242 location_file_dir = location_file_dir or app.config.galaxy_data_manager_data_path
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
243 for filename in data_table.filenames.keys():
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
244 sub_dict['file_path'] = basename( filename )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
245 sub_dict['file_path'] = os.path.join( location_file_dir, sub_dict['file_path'] ) #os.path.abspath?
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
246 if not os.path.exists( sub_dict['file_path'] ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
247 # Create empty file
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
248 open( sub_dict['file_path'], 'wb+' ).close()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
249 break
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
250 sub_dict[ 'repo_info' ] = repo_info or ''
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
251 return """
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
252 <tables><table name="%(table_name)s" %(comment_char)s>
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
253 %(columns)s
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
254 <file path="%(file_path)s" />
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
255 %(repo_info)s
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
256 </table></tables>
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
257 """ % sub_dict
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
258
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
259 def get_available_tables_for_dbkey( trans, dbkey, data_table_names ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
260 my_data_tables = trans.app.tool_data_tables.get_tables()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
261 data_tables = load_data_tables_from_url( data_table_class=trans.app.tool_data_tables.__class__ )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
262 rval = {}
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
263 for name, data_table in data_tables.get( 'data_tables' ).get_tables().iteritems():
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
264 if ( not data_table_names or name in data_table_names ): #name in my_data_tables.keys() and
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
265 #TODO: check that columns are similiar
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
266 if not dbkey:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
267 entry_getter = data_table.get_named_fields_list()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
268 else:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
269 entry_getter = data_table.get_entries( 'dbkey', dbkey, None, default=[] )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
270 for entry in entry_getter:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
271 name = "%s: %s" % ( data_table.name, dumps( entry ) )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
272 rval[name] = entry
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
273 return rval
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
274
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
275 def split_path_all( path ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
276 rval = []
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
277 path = path.rstrip( '/' )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
278 while True:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
279 head, tail = os.path.split( path )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
280 if tail:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
281 rval.append( tail )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
282 path = head
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
283 elif head:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
284 rval.append( head )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
285 break
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
286 else:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
287 break
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
288 rval.reverse()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
289 return rval
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
290
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
291 def get_data_for_path( path, data_root_dir ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
292 # We list dir with a /, but copy data without
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
293 # listing with / gives a . entry when its a dir
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
294 # cloning without the / will copy that whole directory into the target,
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
295 # instead of just that target's contents
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
296 if path.startswith( GALAXY_DATA_CANONICAL_PATH ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
297 path = path[ len( GALAXY_DATA_CANONICAL_PATH ):]
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
298 make_path = path
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
299 rsync_source = rsync_urljoin( rsync_urljoin( RSYNC_SERVER, INDEX_DIR ), path )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
300 if rsync_source.endswith( '/' ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
301 rsync_source = rsync_source[:-1]
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
302 try:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
303 dir_list = rsync_list_dir( rsync_source + "/" )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
304 except Exception, e:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
305 dir_list = None
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
306 while not dir_list or '.' not in dir_list:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
307 head, tail = os.path.split( make_path )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
308 if not head:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
309 head = tail
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
310 make_path = head
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
311 rsync_source = rsync_urljoin( rsync_urljoin( RSYNC_SERVER, INDEX_DIR ), head ) #if we error here, likely due to a connection issue
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
312 if rsync_source.endswith( '/' ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
313 rsync_source = rsync_source[:-1]
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
314 dir_list = rsync_list_dir( rsync_source + "/" )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
315 split_path = split_path_all( make_path )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
316 target_path = data_root_dir
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
317 for p in split_path[:-1]:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
318 target_path = os.path.join( target_path, p )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
319 if not os.path.exists( target_path ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
320 os.mkdir( target_path )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
321 rsync_sync_to_dir( rsync_source, target_path )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
322 return path
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
323
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
324 def get_data_and_munge_path( data_table_name, data_table_entry, data_root_dir ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
325 path_cols = []
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
326 for key, value in data_table_entry.iteritems():
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
327 if key in PATH_COLUMN_NAMES:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
328 path_cols.append( ( key, value ) )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
329 found_data = False
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
330 if path_cols:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
331 for col_name, value in path_cols:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
332 #GALAXY_DATA_CANONICAL_PATH
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
333 if value.startswith( GALAXY_DATA_CANONICAL_PATH ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
334 data_table_entry[col_name] = get_data_for_path( value, data_root_dir )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
335 found_data = True
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
336 else:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
337 print 'unable to determine location of rsync data for', data_table_name, data_table_entry
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
338 return data_table_entry
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
339
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
340 def fulfill_data_table_entries( data_table_entries, data_manager_dict, data_root_dir ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
341 for data_table_name, entries in data_table_entries.iteritems():
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
342 for entry in entries:
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
343 entry = get_data_and_munge_path( data_table_name, entry, data_root_dir )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
344 _add_data_table_entry( data_manager_dict, data_table_name, entry )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
345 return data_manager_dict
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
346
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
347 def _add_data_table_entry( data_manager_dict, data_table_name, data_table_entry ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
348 data_manager_dict['data_tables'] = data_manager_dict.get( 'data_tables', {} )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
349 data_manager_dict['data_tables'][data_table_name] = data_manager_dict['data_tables'].get( data_table_name, [] )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
350 data_manager_dict['data_tables'][data_table_name].append( data_table_entry )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
351 return data_manager_dict
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
352
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
353 def cleanup_before_exit( tmp_dir ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
354 if tmp_dir and os.path.exists( tmp_dir ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
355 shutil.rmtree( tmp_dir )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
356
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
357 def get_data_table_entries( params ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
358 rval = {}
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
359 data_table_entries = params.get( 'data_table_entries', None )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
360 if data_table_entries :
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
361 for entry_text in data_table_entries.split( ',' ):
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
362 entry_text = entry_text.strip().decode( 'base64' )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
363 entry_dict = loads( entry_text )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
364 data_table_name = entry_dict['name']
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
365 data_table_entry = entry_dict['entry']
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
366 rval[ data_table_name ] = rval.get( data_table_name, [] )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
367 rval[ data_table_name ].append( data_table_entry )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
368 return rval
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
369
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
370 def main():
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
371 #Parse Command Line
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
372 parser = optparse.OptionParser()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
373 (options, args) = parser.parse_args()
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
374
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
375 filename = args[0]
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
376
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
377 params = loads( open( filename ).read() )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
378 target_directory = params[ 'output_data' ][0]['extra_files_path']
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
379 os.mkdir( target_directory )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
380 data_manager_dict = {}
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
381
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
382 data_table_entries = get_data_table_entries( params['param_dict'] )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
383
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
384 # Populate the data Tables
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
385 data_manager_dict = fulfill_data_table_entries( data_table_entries, data_manager_dict, target_directory )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
386
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
387 #save info to json file
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
388 open( filename, 'wb' ).write( dumps( data_manager_dict ) )
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
389
70afa70bba41 planemo upload for repository https://github.com/galaxyproject/tools-devteam/tree/master/data_managers/data_manager_rsync_g2 commit 86cf90107482cab1cb47fc0d42d6705f8077daa7
devteam
parents:
diff changeset
390 if __name__ == "__main__": main()