Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

suggestions and questions on training the data #6

Open
shwangdev opened this issue Apr 9, 2019 · 2 comments
Open

suggestions and questions on training the data #6

shwangdev opened this issue Apr 9, 2019 · 2 comments

Comments

@shwangdev
Copy link

  1. The default batch size is 16 which is the root cause of taking too much time to train the data.
    when I changed the batch size as 512, the convergence is much quicker.
  2. I have adjusted your code to train data with keras by showing up the loss value and acc value of each iteration. But when loss value is less than 0.04 and acc value is hight than 0.99. I want to generate the weight file. How do I to control when to generate that weight file?
@shwangdev shwangdev changed the title suggestions on training the data suggestions and questions on training the data Apr 9, 2019
@kwonmha
Copy link
Owner

kwonmha commented Apr 9, 2019

  1. Thanks for informing. I guess I have tried different batch sizes but I'm not sure how large I have tried. Batch size depends on the memory size of GPU. Nevertheless, It's worth knowing.

  2. I think you can get loss value which is written as cost in my original code.

cost = self.model.train_on_batch(batch[0], batch[1])

cost = self.model.train_on_batch(batch[0], batch[1])

And I can't imagine how you can get accuracy but if you can have loss and accuracy after performing each iteration, it seems easy to save weight file by adding simple condition.

if acc > 0.99 or loss < 0.04: 
    SAVE_MODEL

Related code is here :

Maybe modifying codes in that block would work.

@shwangdev
Copy link
Author

  1. Yep, it depends on GPU performance.
  2. After digging into the source code, I found there was 2 parameter from the command line to force save a weight file in iterations. --progress and --min-iter. I have set the two parameters as 500. the weight file can be generated out quickly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants