Mercurial > repos > kls286 > chap_test_20230328

annotate MLaaS/README.md @ 0:cbbe42422d56 draft

Find changesets by keywords (author, files, the commit message), revision number or hash, or revset expression.
planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
author kls286
date Tue, 28 Mar 2023 15:07:30 +0000
parents
children
Ignore whitespace changes - Everywhere: Within whitespace: At end of lines:
rev   line source
0
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
1 ## MLaaS end-to-end example using MNIST dataset
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
2 MLaaS stands for Machine Learning as a Service, and here we'll provide
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
3 end-to-end example based on MNIST dataset using
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
4 Python [Keras](https://keras.io/) ML framework for training
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
5 part, and [TFaas](https://github.com/vkuznet/TFaaS) ML framework
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
6 for inference part.
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
7
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
8 ### Requirements (environment)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
9 To proceed with ML trainig we need to acquire MNIST dataset.
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
10 We will assume that you have a box where recent version of python is installed,
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
11 please note that instructions were tested with `Python 3.10.10`
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
12
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
13 ```
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
14 # create mnist_env, here python refers to python 3.10.10
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
15 python -m venv mnist_env
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
16
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
17 # download mnist dataset for training purposes in numpy gziped arrays
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
18 curl -ksLO https://storage.googleapis.com/tensorflow/tf-keras-datasets/mnist.npz
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
19
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
20 # download MNIST dataset for training purposes in pkl.gz data-format
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
21 curl -ksLO https://s3.amazonaws.com/img-datasets/mnist.pkl.gz
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
22
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
23 # download MNIST images
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
24 # download MNIST actual images which we will use within inference
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
25 curl -O http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
26 ```
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
27
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
28 ### Train ML model
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
29 Below you can see fully tested Keras mased ML codebase to train
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
30 simple convolutional neural network over MNIST dataset (save
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
31 this code as `ktrain.py`):
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
32 ```
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
33 #!/usr/bin/env python
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
34 #-*- coding: utf-8 -*-
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
35 #pylint: disable=
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
36 """
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
37 File : ktrain.py
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
38 Author : Valentin Kuznetsov <vkuznet AT gmail dot com>
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
39 Description: Keras based ML network to train over MNIST dataset
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
40 """
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
41
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
42 # system modules
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
43 import os
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
44 import sys
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
45 import json
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
46 import gzip
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
47 import pickle
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
48 import argparse
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
49
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
50 # third-party modules
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
51 import numpy as np
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
52 import tensorflow as tf
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
53 from tensorflow import keras
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
54 from tensorflow.keras import layers
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
55 from tensorflow.keras import backend as K
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
56 from tensorflow.python.tools import saved_model_utils
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
57
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
58
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
59 def modelGraph(model_dir):
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
60 """
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
61 Provide input/output names used by TF Graph along with graph itself
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
62 The code is based on TF saved_model_cli.py script.
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
63 """
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
64 input_names = []
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
65 output_names = []
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
66 tag_sets = saved_model_utils.get_saved_model_tag_sets(model_dir)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
67 for tag_set in sorted(tag_sets):
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
68 print('%r' % ', '.join(sorted(tag_set)))
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
69 meta_graph_def = saved_model_utils.get_meta_graph_def(model_dir, tag_set[0])
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
70 for key in meta_graph_def.signature_def.keys():
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
71 meta = meta_graph_def.signature_def[key]
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
72 if hasattr(meta, 'inputs') and hasattr(meta, 'outputs'):
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
73 inputs = meta.inputs
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
74 outputs = meta.outputs
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
75 input_signatures = list(meta.inputs.values())
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
76 input_names = [signature.name for signature in input_signatures]
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
77 if len(input_names) > 0:
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
78 output_signatures = list(meta.outputs.values())
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
79 output_names = [signature.name for signature in output_signatures]
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
80 return input_names, output_names, meta_graph_def
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
81
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
82 def readData(fin, num_classes):
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
83 """
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
84 Helper function to read MNIST data and provide it to
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
85 upstream code, e.g. to the training layer
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
86 """
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
87 # Load the data and split it between train and test sets
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
88 # (x_train, y_train), (x_test, y_test) = keras.datasets.mnist.load_data()
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
89 f = gzip.open(fin, 'rb')
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
90 if sys.version_info < (3,):
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
91 mnist_data = pickle.load(f)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
92 else:
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
93 mnist_data = pickle.load(f, encoding='bytes')
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
94 f.close()
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
95 (x_train, y_train), (x_test, y_test) = mnist_data
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
96
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
97 # Scale images to the [0, 1] range
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
98 x_train = x_train.astype("float32") / 255
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
99 x_test = x_test.astype("float32") / 255
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
100 # Make sure images have shape (28, 28, 1)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
101 x_train = np.expand_dims(x_train, -1)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
102 x_test = np.expand_dims(x_test, -1)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
103 print("x_train shape:", x_train.shape)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
104 print(x_train.shape[0], "train samples")
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
105 print(x_test.shape[0], "test samples")
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
106
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
107
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
108 # convert class vectors to binary class matrices
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
109 y_train = keras.utils.to_categorical(y_train, num_classes)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
110 y_test = keras.utils.to_categorical(y_test, num_classes)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
111 return x_train, y_train, x_test, y_test
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
112
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
113
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
114 def train(fin, fout=None, model_name=None, epochs=1, batch_size=128, h5=False):
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
115 """
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
116 train function for MNIST
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
117 """
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
118 # Model / data parameters
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
119 num_classes = 10
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
120 input_shape = (28, 28, 1)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
121
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
122 # create ML model
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
123 model = keras.Sequential(
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
124 [
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
125 keras.Input(shape=input_shape),
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
126 layers.Conv2D(32, kernel_size=(3, 3), activation="relu"),
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
127 layers.MaxPooling2D(pool_size=(2, 2)),
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
128 layers.Conv2D(64, kernel_size=(3, 3), activation="relu"),
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
129 layers.MaxPooling2D(pool_size=(2, 2)),
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
130 layers.Flatten(),
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
131 layers.Dropout(0.5),
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
132 layers.Dense(num_classes, activation="softmax"),
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
133 ]
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
134 )
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
135
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
136 model.summary()
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
137 print("model input", model.input, type(model.input), model.input.__dict__)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
138 print("model output", model.output, type(model.output), model.output.__dict__)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
139 model.compile(loss="categorical_crossentropy", optimizer="adam", metrics=["accuracy"])
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
140
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
141 # train model
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
142 x_train, y_train, x_test, y_test = readData(fin, num_classes)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
143 model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, validation_split=0.1)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
144
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
145 # evaluate trained model
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
146 score = model.evaluate(x_test, y_test, verbose=0)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
147 print("Test loss:", score[0])
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
148 print("Test accuracy:", score[1])
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
149 print("save model to", fout)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
150 writer(fout, model_name, model, input_shape, h5)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
151
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
152 def writer(fout, model_name, model, input_shape, h5=False):
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
153 """
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
154 Writer provide write function for given model
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
155 """
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
156 if not fout:
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
157 return
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
158 model.save(fout)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
159 if h5:
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
160 model.save('{}/{}'.format(fout, h5), save_format='h5')
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
161 pbModel = '{}/saved_model.pb'.format(fout)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
162 pbtxtModel = '{}/saved_model.pbtxt'.format(fout)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
163 convert(pbModel, pbtxtModel)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
164
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
165 # get meta-data information about our ML model
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
166 input_names, output_names, model_graph = modelGraph(model_name)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
167 print("### input", input_names)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
168 print("### output", output_names)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
169 # ML uses (28,28,1) shape, i.e. 28x28 black-white images
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
170 # if we'll use color images we'll use shape (28, 28, 3)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
171 img_channels = input_shape[2] # last item represent number of colors
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
172 meta = {'name': model_name,
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
173 'model': 'saved_model.pb',
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
174 'labels': 'labels.txt',
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
175 'img_channels': img_channels,
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
176 'input_name': input_names[0].split(':')[0],
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
177 'output_name': output_names[0].split(':')[0],
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
178 'input_node': model.input.name,
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
179 'output_node': model.output.name
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
180 }
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
181 with open(fout+'/params.json', 'w') as ostream:
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
182 ostream.write(json.dumps(meta))
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
183 with open(fout+'/labels.txt', 'w') as ostream:
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
184 for i in range(0, 10):
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
185 ostream.write(str(i)+'\n')
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
186 with open(fout + '/model.graph', 'wb') as ostream:
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
187 ostream.write(model_graph.SerializeToString())
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
188
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
189 def convert(fin, fout):
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
190 """
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
191 convert input model.pb into output model.pbtxt
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
192 Based on internet search:
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
193 - https://www.tensorflow.org/guide/saved_model
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
194 - https://www.programcreek.com/python/example/123317/tensorflow.core.protobuf.saved_model_pb2.SavedModel
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
195 """
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
196 import google.protobuf
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
197 from tensorflow.core.protobuf import saved_model_pb2
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
198 import tensorflow as tf
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
199
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
200 saved_model = saved_model_pb2.SavedModel()
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
201
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
202 with open(fin, 'rb') as f:
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
203 saved_model.ParseFromString(f.read())
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
204
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
205 with open(fout, 'w') as f:
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
206 f.write(google.protobuf.text_format.MessageToString(saved_model))
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
207
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
208
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
209 class OptionParser():
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
210 def __init__(self):
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
211 "User based option parser"
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
212 self.parser = argparse.ArgumentParser(prog='PROG')
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
213 self.parser.add_argument("--fin", action="store",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
214 dest="fin", default="", help="Input MNIST file")
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
215 self.parser.add_argument("--fout", action="store",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
216 dest="fout", default="", help="Output models area")
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
217 self.parser.add_argument("--model", action="store",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
218 dest="model", default="mnist", help="model name")
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
219 self.parser.add_argument("--epochs", action="store",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
220 dest="epochs", default=1, help="number of epochs to use in ML training")
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
221 self.parser.add_argument("--batch_size", action="store",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
222 dest="batch_size", default=128, help="batch size to use in training")
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
223 self.parser.add_argument("--h5", action="store",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
224 dest="h5", default="mnist", help="h5 model file name")
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
225
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
226 def main():
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
227 "Main function"
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
228 optmgr = OptionParser()
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
229 opts = optmgr.parser.parse_args()
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
230 train(opts.fin, opts.fout,
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
231 model_name=opts.model,
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
232 epochs=opts.epochs,
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
233 batch_size=opts.batch_size,
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
234 h5=opts.h5)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
235
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
236 if __name__ == '__main__':
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
237 main()
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
238 ```
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
239
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
240 ### Training process
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
241 We will train our model using the following command (for simplicity we skip
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
242 warning messages from TF and irrelevant printouts):
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
243 ```
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
244 # here fout=mnist represents mnist directory where we'll stored our trained model
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
245 # and model=mnist is the name of the model we'll use later in inference
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
246 ./ktrain.py --fin=./mnist.pkl.gz --fout=mnist --model=mnist
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
247 ...
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
248 x_train shape: (60000, 28, 28, 1)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
249 60000 train samples
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
250 10000 test samples
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
251 Model: "sequential"
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
252 _________________________________________________________________
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
253 Layer (type) Output Shape Param #
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
254 =================================================================
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
255 conv2d (Conv2D) (None, 26, 26, 32) 320
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
256
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
257 max_pooling2d (MaxPooling2D (None, 13, 13, 32) 0
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
258 )
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
259
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
260 conv2d_1 (Conv2D) (None, 11, 11, 64) 18496
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
261
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
262 max_pooling2d_1 (MaxPooling (None, 5, 5, 64) 0
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
263 2D)
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
264
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
265 flatten (Flatten) (None, 1600) 0
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
266
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
267 dropout (Dropout) (None, 1600) 0
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
268
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
269 dense (Dense) (None, 10) 16010
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
270
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
271 =================================================================
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
272 Total params: 34,826
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
273 Trainable params: 34,826
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
274 Non-trainable params: 0
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
275 _________________________________________________________________
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
276
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
277 422/422 [==============================] - 37s 84ms/step - loss: 0.3645 - accuracy: 0.8898 - val_loss: 0.0825 - val_accuracy: 0.9772
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
278 Test loss: 0.09409885853528976
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
279 Test accuracy: 0.9703999757766724
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
280 save model to mnist
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
281
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
282 ### input ['serving_default_input_1:0']
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
283 ### output ['StatefulPartitionedCall:0']
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
284 ```
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
285 When this process is over you'll find `mnist` directory with the following
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
286 content:
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
287 ```
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
288 shell# ls mnist
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
289
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
290 assets keras_metadata.pb model.graph saved_model.pb variables
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
291 fingerprint.pb labels.txt params.json saved_model.pbtxt
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
292 ```
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
293 - `saved_model.pb` represents trained ML model in protobuffer data-format
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
294 - `saved_model.pbtxt` represents trained ML model in text protobuffer representation
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
295 - `labels.txt` contains our image labels
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
296 - `params.json` contains meta-data used by TFaaS and it has the following content:
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
297 ```
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
298 cat mnist/params.json | jq
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
299 {
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
300 "name": "mnist",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
301 "model": "saved_model.pb",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
302 "labels": "labels.txt",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
303 "img_channels": 1,
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
304 "input_name": "serving_default_input_1",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
305 "output_name": "StatefulPartitionedCall",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
306 "input_node": "input_1",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
307 "output_node": "dense/Softmax:0"
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
308 }
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
309 ```
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
310 Here you see, that our ML model is called `mnist`, the model is stored in
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
311 `saved_model.pb` file, and more importantly this file contains the input and
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
312 output tensor names and nodes which we need to provide for TFaaS to server
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
313 our predictions.
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
314
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
315 ### Inference server
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
316 Now, it is time to start our inference server. You can find its code in `src/go` area.
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
317 To build the code you need
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
318 ```
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
319 # download TF library and includes for your OS, e.g. macOS build
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
320 curl -ksLO https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-darwin-x86_64-2.11.0.tar.gz
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
321 # or linux build
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
322 curl -ksLO https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-cpu-linux-x86_64-2.11.0.tar.gz
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
323 # or linux GPU build
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
324 curl -ksLO https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-gpu-linux-x86_64-2.11.0.tar.gz
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
325
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
326 # provide TF include area location to go build command
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
327 # the /opt/tensorflow/include is are where TF includes are
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
328 export CGO_CPPFLAGS="-I/opt/tensorflow/include"
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
329
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
330 # compile the code
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
331 make
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
332
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
333 # it will produce tfaas executable
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
334
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
335 # to run the code we need to setup `DYLD_LIBRARY_PATH`
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
336 export DYLD_LIBRARY_PATH=/opt/tensorflow/lib
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
337 ./tfaas -config config.json
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
338 ```
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
339 where `config.json` has the following form (please refer for more details):
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
340 ```
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
341 {
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
342 "port": 8083,
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
343 "modelDir": "models",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
344 "staticDir": "static",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
345 "configProto": "",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
346 "base": "",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
347 "serverKey": "",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
348 "serverCrt": "",
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
349 "verbose": 1
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
350 }
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
351 ```
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
352
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
353 ### Serving predictions with TFaaS inference server
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
354 Finally, we are ready for the inference part.
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
355 - upload your ML model to TFaaS server
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
356 ```
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
357 # create tarball of your mnist ML trained model
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
358 tar cfz mnist.tar.gz mnist
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
359
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
360 # upload tarball to TFaaS server
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
361 curl -v -X POST -H "Content-Encoding: gzip" \
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
362 -H "Content-Type: application/octet-stream" \
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
363 --data-binary @./mnist.tar.gz \
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
364 http://localhost:8083/upload
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
365
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
366 # check your model presence
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
367 curl http://localhost:8083/models
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
368
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
369 # generate image from MNIST dataset you want to use for prediction
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
370 # img1.png will contain number 1, img4.png will contain number 4
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
371 ./mnist_img.py --fout img1.png --imgid=3
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
372 ./mnist_img.py --fout img4.png --imgid=2
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
373
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
374 # ask for prediction of your image
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
375 curl http://localhost:8083/predict/image -F 'image=@./img1.png' -F 'model=mnist'
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
376 [0,1,0,0,0,0,0,0,0,0]
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
377
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
378 curl http://localhost:8083/predict/image -F 'image=@./img4.png' -F 'model=mnist'
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
379 [0,0,0,0,1,0,0,0,0,0]
cbbe42422d56 planemo upload for repository https://github.com/CHESSComputing/ChessAnalysisPipeline/tree/galaxy commit 1401a7e1ae007a6bda260d147f9b879e789b73e0-dirty
kls286
parents:
diff changeset
380 ```