Skip to content

cypherics/IVU

Repository files navigation

IVU

Requirements

Check the requirement.txt for project specific requirements. To install project specific requirements run the following command, It is highly recommended to create a seperate python enviorment before running the command.

pip install -r requirements.txt

DATASET

The original data set can be found here original dataset. The complete processed dataset can be downloaded from complete dataset. Dataset which is split into train and test can be downloaded from split data for training and testing.

Description of the data

Value Description
key_points 17 body keypoints
normalized_key_points key points are normalized with respect to body position
distance_matrix pair wise distance between key points
normalized_distance_matrix pair wise distance between key points is normalized with respect to body position
class_label name of the class
class_label_idx integer equivalent of class_label
file name of the file
frame_number respective frame of the file
frame_details details with respect to over all files

The size of features that are used for training for key_points & normalized_key_points is 51 , and for distance_matrix & normalized_distance_matrix is 136.

Training

  • Config SetUp

    In order to start the training, all the required training parameters, must be specified in a config file.

    data:
      validation_split: 0.2
      
      # Number of frames that are to considered as a single sequence during training 
      stride: 32
    
    optimizer:
      name: Adam
      parameters:
        lr: 0.001
    
    callbacks:
      ReduceLROnPlateau:
        parameters:
          patience: 3
          verbose: 1
          factor: 0.2
          min_lr: 0.000001
    
      EarlyStopping:
        parameters:
          min_delta: 0.001
          patience: 8
          verbose: 1
    
      ModelCheckpoint:
        parameters:
          verbose: 1
          save_best_only: True
          save_weights_only: False
    loss:
      name: CategoricalCrossentropy
      parameters:
        from_logits: True
    
    batch_size: 32
    epochs: 300
    shuffle: True
    
    model :
      # name of the model to use for training
      name : 
      # model input parameters
      parameters:
        # size of input features
        input_features: 136 
        # total number of classes
        n_classes: 7
    
    log_dir : logs/
    • LSTM model configuration

    model :
      name : lstm_kar_model
      parameters:
        hidden_units: 64
        input_features: # value based on type of data
        n_classes: 7
        penalty: 0.0001
    • Convolution model configuration

    model:
      name: temporal_model
      parameters:
        input_features: # value based on type of data
        n_classes: 7

    Based on the type of data used for training, the size of input_features will vary. If data used for training is either key_points or normalized_key_points set input_features: 51 , else if data used is distance_matrix or normalized_distance_matrix set input_features: 136.

  • Start Training

    Once the config is set, then based on the choice of the type of the data that is being used one can perform the training.

    Train when data is Normalized Distance Matrix

    from ivu.trainer import Trainer
    
    trainer = Trainer.train_with_normalized_distance_matrix(train_pth=r"path_to_train_pickle",
                                                            test_pth=r"path_to_test_pickle",
                                                            conf_pth=r"path_to_config")
    trainer.start_training()

    Train when data is Distance Matrix

    from ivu.trainer import Trainer
    
    trainer = Trainer.train_with_distance_matrix(train_pth=r"path_to_train_pickle",
                                                            test_pth=r"path_to_test_pickle",
                                                            conf_pth=r"path_to_config")
    trainer.start_training()

    Train when data is Normalized KeyPoints

    from ivu.trainer import Trainer
    
    trainer = Trainer.train_with_normalized_key_points(train_pth=r"path_to_train_pickle",
                                                            test_pth=r"path_to_test_pickle",
                                                            conf_pth=r"path_to_config")
    trainer.start_training()

Run Experiments

Predefined config files can be found in config/, which can be used to perform experiments. Just run the train_experiments.py, with paths to the processed training and test set data, which will start performing experiments on those configurations present in the folder.

If both train_key_points.pickle and test_key_points.pickle are available then run the following command to perform experiments using those files

python run_train_and_testing.py path/to/train_key_points.pickle path/to/test_key_points.pickle

If just train_key_points.pickle is available then run the following command to perform experiments without test set.

python run_train_and_testing.py path/to/train_key_points.pickle None

The results of the experiment can be viewed in logs/ folder, which is generated in the present working directly. The folder contains saved model in the folder named chk, training graphs saved in the folder name graphs, a pdf containing the confusion matrix and its test metric.

To visualize the training graph run the following command in the terminal

tensorboard --logdir path/till/graph

Inference

INFER_FOR = ["normalized_distance_matrix", "distance_matrix", "normalized_key_points"]

Based on what type of data the model was trained on choose the appropriate mode, below command is when mode is normalized_distance_matrix and stride of 32

python run_example.py normalized_distance_matrix path/till/the/folder/chk path/where/my/testing/videos/are path/where/to/save/results 32

Run Demo Using pretrained weights

  • Download pretrained weights from here
  • Download the videos for demo here
  • Extract the downloaded model.zip, the extracted folder will contain a chk folder, the path till the chk folder is the model path
  • Extract the downloaded demo_videos.zip
  • Run the following command to run inference on the downloaded model.
python run_example.py normalized_key_points path/till/the/extracted/model/chk path/till/the/extracted/demo_videos path/where/to/save/results 64