Trained model accuracy!

[mmazloom]

Hello every body,

I hav trained a model using the SimplePlantLeafDiseaseLoading sample in Lazarus. It shows that the trained model has abou 98% accuracy in the train phase.
But when I use the compute method and pass an image to model, almost all predictions are wrong!
What is the problem I have?
Regrads

[joaopauloschuler]

@mmazloom,
Thank you for the bug report.

I'll work on it (reproduce and fix it if required) along the next week.

[joaopauloschuler]

Is your source code available for me to look at?

[joaopauloschuler]

@mmazloom ?

[mmazloom]

Sorry for the delay.
As I said, I used the original example in Lazarus to train the model. Then I loaded the model in delphi and use the following procedure to test:

procedure TForm1.Button1Click(Sender: TObject);
var
NN: TNNet;
I: integer;
ImgVolumes: TNNetVolumeList;
Volume, AuxVolume: TNNetVolume;
pOutput, vOutput, vDisplay: TNNetVolume;
hit, miss: integer;
NumClasses: integer;

FNewSizeX, FNewSizeY: integer;
Image: TJpegImage;
bmp: TBitmap;

begin

ImgVolumes := TNNetVolumeList.Create();

NumClasses := 39;

NN := TNNet.Create();

NN.LoadFromFile('d:\SimplePlantLeafDisease.nn');

NN.EnableDropouts(false);

pOutput := TNNetVolume.Create(NumClasses, 1, 1);
vOutput := TNNetVolume.Create(NumClasses, 1, 1);

Volume := TNNetVolume.Create();
AuxVolume := TNNetVolume.Create();

OpenDialog1.Execute();

Image := TJpegImage.Create;
Image.LoadFromFile(OpenDialog1.FileName);

bmp := TBitmap.Create;
bmp.Assign(Image);

LoadBitmapIntoVolume(bmp, Volume);

FNewSizeX := 128;
FNewSizeY := 128;
if (FNewSizeX > 0) and (FNewSizeY > 0) then
begin
if (Volume.SizeX <> FNewSizeX) or (Volume.SizeY <> FNewSizeY) then
begin
AuxVolume.Copy(Volume);
Volume.CopyResizing(AuxVolume, FNewSizeX, FNewSizeY);
end;
end;

NN.Compute(Volume);
NN.GetOutput(pOutput);

Memo1.Lines.Add(pOutput.GetClass.ToString());
for i := 0 to NumClasses - 1 do
Memo1.Lines.Add(i.ToString() + ': ' + FloatToStr(pOutput.Raw[i]));

end;

---

Extra info:
I used the SimplePlantLeafDiseaseLoading sample to train the model and used the Plant_with_augmentation images as train samples.

[joaopauloschuler]

@mmazloom , thank you for sharing the code!

I miss something like this before calling Compute:
Volume.RgbImgToNeuronalInput(FColorEncoding);

If you are using RGB images, FColorEncoding will be 0 (or csEncodeRGB).

What is the result from: Volume.PrintDebug() ?

[mmazloom]

I used the Image (1).jpg from folder Plant_leave_diseases_dataset_with_augmentation\Apple___Black_rot as input image, this the volume debug output:
'(128,128,3) - Min: -2 Min Pos:563 Max:1.984375 Max Pos:462 Avg:-0.258774429559708 Non Zero:48879 Size:49152'

and this the RAW array output (class 0 is winner instead of class 1):
0: 0.579165101051331
1: 0.208285823464394
2: 0.0502395406365395
3: 0.162309527397156
4: 0
5: 3.29359397392871E-37
6: 1.85352550473172E-40
7: 0
8: 5.60519385729927E-45
9: 5.32653584857903E-39
10: 4.40814865713444E-39
11: 0
12: 0
13: 3.29359397392871E-37
14: 1.85352550473172E-40
15: 0
16: 7.00649232162409E-45
17: 4.86734995999829E-39
18: 2.93881715236665E-39
19: 6.72623262875912E-44
20: 1.40129846432482E-44
21: 0
22: 0
23: 0
24: 0
25: 0
26: 0
27: 0
28: 0
29: 0
30: 0
31: 0
32: 0
33: 0
34: 0
35: 0
36: 0
37: 0
38: 0

[joaopauloschuler]

@mmazloom , thank you sharing.

I'll only be able to look in more details along the next week. Anyway, I'm wondering if is there any problem with the labeling... (such as, is class 1 really the class 1)? Also, are all images giving the same output?

Anyway, thank you for the feedback. I'll look at it.

[mmazloom]

No, the output for different images is different. But I expect to return the 0 for the first folder, 1 for the scond folde and ...
because the training phase shows a good performance (above 98%). Also the images i have used to test, belongs to the train set and must get high score! right?
Thank you again for the greate library and waiting for the result.

[joaopauloschuler]

Quick update: I'm about to start working on this bug report.

[joaopauloschuler]

@mmazloom,
It seems to be working for me.

You can uncomment the fitting method if you need to train again.

Give it a go at your end to this code please.

///This file has an implementation to classify
//plant leaf diseases. You can get the dataset at
//https://data.mendeley.com/datasets/tywbtsjrjv/1/files/d5652a28-c1d8-4b76-97f3-72fb80f94efc/Plant_leaf_diseases_dataset_without_augmentation.zip?dl=1 .
//Folders with plant diseases will need to be stored inside of a folder named "plant".
program SimplePlantLeafDiseaseLoadingAPI;
(*
 Coded by Joao Paulo Schwarz Schuler.
 https://github.com/joaopauloschuler/neural-api
*)
{$mode objfpc}{$H+}

uses {$IFDEF UNIX} {$IFDEF UseCThreads}
  cthreads, {$ENDIF} {$ENDIF}
  Classes, SysUtils, CustApp, neuralnetwork, neuralvolume, Math, neuraldatasets,
  neuralfit;

type
  TTestCNNAlgo = class(TCustomApplication)
  protected
    FSizeX, FSizeY: integer;
    FTrainingFileNames, FValidationFileNames, FTestFileNames: TFileNameList;
    procedure DoRun; override;
    procedure Test(ImageFileName:string; classIdx: integer);
  end;

  procedure TTestCNNAlgo.DoRun;
  var
    NN: TNNet;
    NeuralFit: TNeuralImageLoadingFit;
    ProportionToLoad: Single;
    I: integer;
  begin
    FSizeX := 64;
    FSizeY := 64;

    ProportionToLoad := 0.1;
    WriteLn('Loading ', Round(ProportionToLoad*100), '% of the Plant leave disease file names into memory.');

    // for testing with Tiny Imagenet 200, do this (http://cs231n.stanford.edu/tiny-imagenet-200.zip):
    // {FolderName=}'tiny-imagenet-200/train', {pImageSubFolder=}'images',

    // for testing with places standard small images
    // http://places2.csail.mit.edu/download.html
    // Download small images (256 * 256) with easy directory structure)
    // http://data.csail.mit.edu/places/places365/places365standard_easyformat.tar
    // {FolderName=}'places_folder/train', {pImageSubFolder=}'',

    CreateFileNameListsFromImagesFromFolder(
      FTrainingFileNames, FValidationFileNames, FTestFileNames,
      {FolderName=}'plant', {pImageSubFolder=}'',
      {TrainingProp=}0.9*ProportionToLoad,
      {ValidationProp=}0.05*ProportionToLoad,
      {TestProp=}0.05*ProportionToLoad
    );

    WriteLn('Creating Neural Network...');
    NN := TNNet.Create();
    NN.AddLayer([
      TNNetInput.Create(FSizeX, FSizeY, 3),
      TNNetConvolutionLinear.Create({Features=}64, {FeatureSize=}5, {Padding=}4, {Stride=}2),
      TNNetMaxPool.Create(2),
      TNNetMovingStdNormalization.Create(),
      TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1),
      TNNetConvolutionLinear.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1),
      TNNetMaxPool.Create(2),
      TNNetConvolutionReLU.Create({Features=}64, {FeatureSize=}3, {Padding=}1, {Stride=}1),
      TNNetConvolutionReLU.Create({Features=}256, {FeatureSize=}3, {Padding=}1, {Stride=}2),
      TNNetConvolutionLinear.Create({Features=}512, {FeatureSize=}3, {Padding=}1, {Stride=}2),
      TNNetMaxPool.Create(2),
      TNNetFullConnectLinear.Create(FTrainingFileNames.ClassCount),
      TNNetSoftMax.Create()
    ]);
    NN.DebugStructure();

    WriteLn
    (
      'Training Images:', FTrainingFileNames.Count,
      ' Validation Images:', FValidationFileNames.Count,
      ' Test Images:', FTestFileNames.Count
    );

    NeuralFit := TNeuralImageLoadingFit.Create;
    NeuralFit.TrainingVolumeCacheEnabled := false;
    NeuralFit.FileNameBase := 'SimplePlantLeafDiseaseAPI';
    NeuralFit.InitialLearningRate := 0.001;
    NeuralFit.LearningRateDecay := 0.01;
    NeuralFit.StaircaseEpochs := 10;
    NeuralFit.Inertia := 0.9;
    NeuralFit.L2Decay := 0.0000001;
    NeuralFit.AvgWeightEpochCount := 0;

    NeuralFit.MaxThreadNum := 1;
    //NeuralFit.FitLoading(NN, FSizeX, FSizeY, FTrainingFileNames, FValidationFileNames, FTestFileNames, {BatchSize}256, {Epochs}10);
    NeuralFit.Free;

    for I := 0 to 10 do
    begin
      Test(FTrainingFileNames[I], FTrainingFileNames.Integers[I]);
      ReadLn();
    end;
    FTrainingFileNames.Free;
    FValidationFileNames.Free;
    FTestFileNames.Free;
    NN.Free;

    ReadLn;

    Terminate;
  end;

  procedure TTestCNNAlgo.Test(ImageFileName:string; classIdx: integer);
  var
    NN: TNNet;
    V, AuxVolume, VOutput: TNNetVolume;
  begin
    WriteLn('Loading Saved Neural Network...');
    NN := TNNet.Create();
    V := TNNetVolume.Create();
    VOutput := TNNetVolume.Create();
    AuxVolume := TNNetVolume.Create();
    NN.LoadFromFile('SimplePlantLeafDiseaseAPI.nn');
    WriteLn;WriteLn('Loading Image ',ImageFileName,' of class ',classIdx);
    LoadImageFromFileIntoVolume(ImageFileName, V);
    AuxVolume.CopyResizing(V, FSizeX, FSizeY);
    AuxVolume.RgbImgToNeuronalInput(csEncodeRGB);
    WriteLn;WriteLn('Input is:');
    AuxVolume.PrintDebug();
    NN.Compute(AuxVolume, VOutput);
    WriteLn;WriteLn('Result is: ', VOutput.GetClass());
    VOutput.PrintWithIndex();
    VOutput.Free;
    AuxVolume.Free;
    V.Free;
    NN.Free;
  end;

var
  Application: TTestCNNAlgo;
begin
  Application := TTestCNNAlgo.Create(nil);
  Application.Title:='Plant Leaf Disease Classification';
  Application.Run;
  Application.Free;
end.

[joaopauloschuler]

@mmazloom ,
You might be interest in this example coded by @DrDub
https://github.com/joaopauloschuler/neural-api/tree/master/examples/ResNet/server