Simple audio mixer implementation

[leo150]

Hey. First of all, thanks a lot for adding all the audio enhancements in the past several months. They help a lot with the audio experience, especially handling various sample rates.

Regarding further audio features, would it be possible to add a simple audio mixer at some point? This would greatly help with improving screen sharing functionality. This way, users can stream their mic and ReplayKit audio at the same time. I'm currently using a mixer based on the VideoCast implementation. There are also a couple of open-source implementations, such as PLAudioMixer, VideoCore, SwiftVideo.

I can create a PR with one of the implementations, but there are architectural decisions I'm not sure about in the context of HaishinKit. First, audio formats need to be converted to match each other before mixing. HK already does this, so the mixer needs to be placed in a way that avoids doing the conversion twice. Second, HK is using queues; I'm not sure if some of them can be reused as well. Any advice will be appreciated. Thanks.

[shogo4405]

You're welcome. Regarding audio mixing, I'm considering implementing it since macOS provides multiple resources. However, I'd like to first add a lightweight mixing feature for videos. Therefore, I anticipate that the implementation might take some time.

When utilizing other open-source libraries, it's advisable to maintain loose coupling by inputting the CMSampleBuffer or AVAudioBuffer produced by the library into HaishinKit.

CMSampleBuffer (ReplayKit audio). ──┐
	              Mixing Audio Library ─> CMSampleBuffer or AVAudioBuffer ─> HaishinKit(append(_ sampleBuffer: CMSampleBuffer) or append(audioBuffer: AVAudioBuffer))
CMSampleBuffer (ReplayKit mic) ───┛

Continuing processing on the HaishinKit side involves adjusting the sample count and preserving silent parts. I consider this a necessary process and a cost that should be paid.

[dumojo]

I can create a PR with one of the implementations, but there are architectural decisions I'm not sure about in the context of HaishinKit. First, audio formats need to be converted to match each other before mixing. HK already does this, so the mixer needs to be placed in a way that avoids doing the conversion twice. Second, HK is using queues; I'm not sure if some of them can be reused as well. Any advice will be appreciated. Thanks.

Hi if you have a properly working audio mixer implementation for CMSampleBuffer/AVAudioPCMBuffer it would be great if you could share it with me.

[leo150]

#1386

[shogo4405]

Considering ReplayKit, I feel there is a volume difference between appAudio and micAudio. Specifically, appAudio seems too loud. Is there an easy way to control the volume? #1386

[leo150]

Yes here's a stub method here for the future implementation. I don't recall testing it, but it should work right away.

[shogo4405]

I was able to confirm that the volume can be changed using the method you provided. Thank you.
https://github.com/shogo4405/HaishinKit.swift/blob/main/Sources/IO/IOAudioMixerByMultiTrack.swift#L13-L15