Downloading the Model to my drive FAILED

[pumpkinband]

`!pip install -U -q PyDrive
from pydrive.auth import GoogleAuth
from pydrive.drive import GoogleDrive
from google.colab import auth
from oauth2client.client import GoogleCredentials

auth.authenticate_user()
gauth = GoogleAuth()
gauth.credentials = GoogleCredentials.get_application_default()
drive = GoogleDrive(gauth)

model.save('model.h5')
model_file = drive.CreateFile({'title' : 'model.h5'})
model_file.SetContentFile('model.h5')
model_file.Upload()

drive.CreateFile({'id': model_file.get('id')})`

Used this code to Download the model but Failed

ERROR :-

NameError: name 'model' is not defined

[Santosh-Gupta]

It looks like you used the train your own models notebook, and now you're trying to save the model you trained? Or is it something else?

[pumpkinband]

yes sir i want to download model.h5 , can you help me please.
@Santosh-Gupta

[JayYip]

What are you trying to download? The model weights trained by yourself or the weights that we provide?

[pumpkinband]

Model weight i trained i guess. (i meant running the CELLS by my self) please it would be a great help @JayYip

[JayYip]

May I know which notebook are you trying to run?

[pumpkinband]

https://colab.research.google.com/drive/1FSti89LtRIzuQapRpwv2Lih4Dm3dENHb#scrollTo=qZ110ZqNfBac @JayYip

[JayYip]

Seems I cannot open the link you provide. That might be a copy of the original colab notebook and I cannot access it. Which notebook in this folder?

[pumpkinband]

DocProductPresentationV6-0.2.0.ipynb

[JayYip]

Sorry, but I can't find the corresponding code in the notebook.

BTW, can you download the model files from OneDrive? Seems it's broken again.

[pumpkinband]

Sure sir but how can i download model.h5 of your model from OneDrive, can you please direct me.

[JayYip]

I'm not sure what is model.h5. As for the model weights that we provide, you can download here.

[pumpkinband]

ok but how to download the model weight can you please help me step by step

[JayYip]

The zip file that I linked above contains the model weights. That is the same file as DataAndCheckpoint.zip in DocProductPresentationV6-0.2.0.ipynb notebook.

First, you download the zip file. Then you need to upload the file you downloaded to colab and rename it to DataAndCheckpoint.zip and execute the notebook.

I am still confused though, what is the model in your original post? Where do you get that chunk of code?

[harshit-saraswat]

Hello sir, actually I had an issue with the model. I've downloaded the weights and embedding file from your drive and when i tried to run it on my machine(windows os) i got an error in predictor.py file as it imports faiss which isn't available on windows yet. So, I was thinking is there a way to run the model on windows? Also I'm trying to deploy the model on an android based application and the video Siraj posted on his channel doesn't actually covers it neither his repository. Mind shedding in some light?

[pumpkinband]

same issue as @harshit-saraswat

[JayYip]

So, I was thinking is there a way to run the model on windows?

Please take a look at this issue. Seems they're working on a Windows version.

Also I'm trying to deploy the model on an android based application and the video Siraj posted on his channel doesn't actually covers it neither his repository

I don't have many experience of developing apps but as far as I can tell, retrieval based model is not very suitable to put the whole model to mobile device because the embedding vector size is huge. I would suggest you to serve the model on a server.

BTW, if you guys have experience developing apps, maybe you can work with us! If you're interested please contact me via email: junpang.yip@gmail.com and let us get to know you and your skillset.

[harshit-saraswat]

@JayYip there's the problem right, for the model to be used as a server I do need it to be run on my machine as for as far as I know, colab doesn't allow webhooks or post requests from other consoles/webservices such as dialogflow. TBH integrating it with Dialogflow as a chatbot would really be a great deal. I'm really trying on howto send my inputted query string to your model online as an server and get the ouput as result. Any ideas about how can I use this functionality? It needn't be necessary to be used as a dialogflow integration/ backend. I just need some guidance about using the model as a server and sending request and retrieving the output back, since as of now colab doesn't serve that type of services/purposes. Thanks!

[pumpkinband]

on that topic i want to convert it into .tflite can you help me with that @harshit-saraswat

[harshit-saraswat]

@pumpkinband Follow the following steps:

1. Convert the model checkpoints which you've downloaded from the one drive link given by the creators here, to a h5/pb file first. These links will help- https://www.tensorflow.org/tutorials/keras/save_and_restore_models,https://stackoverflow.com/questions/44466066/how-can-i-convert-a-trained-tensorflow-model-to-keras, https://blog.metaflow.fr/tensorflow-how-to-freeze-a-model-and-serve-it-with-a-python-api-d4f3596b3adc.
2. Now use this link to convert the saved graph instances of the model to a tflite file. https://www.tensorflow.org/lite/convert/python_api

Hope it helped.

[pumpkinband]

thanks alot @harshit-saraswat and sir @JayYip thank you so much

[Santosh-Gupta]

@jay 'model.h5' is the name of the file, what that person is doing trying to use the keras function model.save('model.h5') to save the weights into Keras's h.5 format.

So the issue is that either Keras is not saving the model correctly, or that Pydrive is not uploading it correctly.

@pumpkinband did you get your issue fixed?

[harshit-saraswat]

Anyone able to run the model?or deploy it on android yet?

[JayYip]

what that person is doing trying to use the keras function model.save('model.h5') to save the weights into Keras's h.5 format

Yeah. I know. I'm just confused what's the value of that file since I cannot find that chunk of code in the notebook. And he did not provide any info about model object.

Anyone able to run the model?

You should be able to run the model like we did in the notebooks once you downloaded the zip file and get the environment right.

[pumpkinband]

so can anyone tell me the name of model it is asking, i just want to download the Model in keras (h.5) ??

[harshit-saraswat]

@pumpkinband can you send a link to your code?

[pumpkinband]

@harshit-saraswat Thanks a Lot https://colab.research.google.com/drive/19_ZL7Wr-R0ZvO-ZJkkMeg9WSGsS5rSrl#scrollTo=rVBFeL7Abhiq

[pumpkinband]

can you tell me where did i go wrong @harshit-saraswat

[harshit-saraswat]

@pumpkinband allow access for the link please. maybe change the folders sharing to allow everyone to view.

[pumpkinband]

https://colab.research.google.com/drive/19_ZL7Wr-R0ZvO-ZJkkMeg9WSGsS5rSrl yup did it @harshit-saraswat

[JayYip]

@pumpkinband Your code doesn't make much sense to me. What do you want to download to your drive exactly? The model object just came out of nowhere and thus it yields NameError. If you want the model weight, it's in the path newFolder/models/bertffn_crossentropy/bertffn.

[harshit-saraswat]

@pumpkinband you don't need to convert the model to a json or h5 format because the model weight and graph are stored in the BioBert folder and also the new folder as .ckpt/.chkp and .meta file along with some .json file. Now use those files and convert them into a .pb and .pbtxt format and then convert it into a .tflite format.
This link clearly explains how you can convert the files into a .pb and .pbtxt file. To convert those into .tflite you can use

`import tensorflow as tf

converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
tflite_model = converter.convert()
open("converted_model.tflite", "wb").write(tflite_model)`

[pumpkinband]

I am trying to convert it into .pb but I just need one thing, what is output_node_names of this project ?? @harshit-saraswat

[Santosh-Gupta]

I need to brush up machine learning model storage types. What is the difference between how it's currently stored and .pb? Also, what are 'output_node_names '. I'm not familiar with these, but if you can give more details I may be able to figure out what parts of DocProduct these belong in.

[pumpkinband]

3 different ways to find output_node_names, but I can’t do any of this :

1, If you still have your python file describing the model, you can always add an my_var = tf.indentity(my_var, name="my_new_name") and then load your previous model and save it again with this new added node

2, Also note that the graph_def is holding nodes in order (per connected graph), so output nodes should be close to the end of the array.

3, Finally, you can always use tensorboard to load an existing graph and just look at the name of your “output” nodes.
@Santosh-Gupta

[pumpkinband]

can anyone please upload graph.pbtxt to tensorboard and tell me the "output_node_names" ??

[Santosh-Gupta]

What are 'output_node_names '? I'm not sure what this is. Is it the name of the output variables in the Tensorflow graph?

"3 different ways to find output_node_names, but I can’t do any of this :"

What issues have you run into doing these?

[JayYip]

Please take a look at this issue. #6

[Chrome3912]

try the api: https://pocket-derm.firebaseapp.com/