Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong Logic for HopSize? #1099

Open
fyuvb opened this issue Dec 6, 2021 · 2 comments
Open

Wrong Logic for HopSize? #1099

fyuvb opened this issue Dec 6, 2021 · 2 comments
Labels

Comments

@fyuvb
Copy link

fyuvb commented Dec 6, 2021

Thank you for the great work in this project.
I was trying to use Meyda for MFCC feature collection. However, I went through the code here in scriptprocessor onaudioprocess callback:
https://github.com/meyda/meyda/blob/main/src/meyda-wa.ts#L163-L184

   this._m.spn.onaudioprocess = (e) => {
      var buffer;
      if (this._m.inputData !== null) {
        this._m.previousInputData = this._m.inputData;
      }
      this._m.inputData = e.inputBuffer.getChannelData(this._m.channel);
      if (!this._m.previousInputData) {
        buffer = this._m.inputData;
      } else {
        buffer = new Float32Array(
          this._m.previousInputData.length +
            this._m.inputData.length -
            this._m.hopSize
        );
        buffer.set(this._m.previousInputData.slice(this._m.hopSize));
        buffer.set(
          this._m.inputData,
          this._m.previousInputData.length - this._m.hopSize
        );
      }
      var frames = utilities.frame(buffer, this._m.bufferSize, this._m.hopSize);
      frames.forEach((f) => {
        this._m.frame = f;
        var features = this._m.extract(
          this._m._featuresToExtract,
          this._m.frame,
          this._m.previousFrame
        );
        // call callback if applicable
        if (
          typeof this._m.callback === "function" &&
          this._m.EXTRACTION_STARTED
        ) {
          this._m.callback(features);
        }
        this._m.previousFrame = this._m.frame;
      });
    };

I made a drawing to illustrate:
Screen Shot 2021-12-06 at 8 26 31 PM

My question is,

  1. To compute accuracy features, we might not want to pad zeros. We would want to queue enough samples until we could start again.
  2. we should not take the newly input buffer as the previous input buffer. Instead, we should start at where Frame5 started. Otherwise the feature extraction window will mess up.

I understand that the current implementation could give similar result as my proposal, but I am not quite sure how big the difference is. I was trying to use Meyda as my feature extraction preprocessing step for my Tensorflow MFCC. The extracted features does not match and my model gave me bad results. Please advice if I am making a mistake here. Many thanks!

@hughrawlinson
Copy link
Member

Hey @fyuvb - Happy to help!

Good catch! Thanks for catching the bug. Definitely something we should fix. If you'd like to give it a go, I'd be happy to review a PR. Otherwise, I'll try and get to it.

@fyuvb
Copy link
Author

fyuvb commented Dec 7, 2021

Hi @hughrawlinson. Thank you for confirming this problem. I am not very good at javascript so thank you so much for fixing this problem. Much appreciated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Development

No branches or pull requests

2 participants