Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add JVM Support #103

Open
wants to merge 21 commits into
base: main
Choose a base branch
from
Open

Add JVM Support #103

wants to merge 21 commits into from

Conversation

aschulz90
Copy link

I only tested this on a Windows PC connecting with an Android device. Video/Screensharing and Audio works with the sample apps. DataChannel was only tested via Junit.
Since this relies on webrtc-java for native interop, it should theoretically work on any platform it supports as well.

@shepeliev
Copy link
Owner

@aschulz90 great job. Let me take some time to review.

@shepeliev shepeliev self-requested a review February 11, 2024 16:37
Copy link
Owner

@shepeliev shepeliev left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. I left a couple of comments.

name = deps.webrtc.java.get().name,
version = deps.webrtc.java.get().version,
classifier = "macos-x86_64"
)
Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've got on my Mac M2

java.lang.UnsatisfiedLinkError: /private/var/folders/sp/8h47vr6s72v4hngsrxrfwpzm0000gn/T/libwebrtc-java1779612920915696582.dylib: dlopen(/private/var/folders/sp/8h47vr6s72v4hngsrxrfwpzm0000gn/T/libwebrtc-java1779612920915696582.dylib, 0x0001): tried: '/private/var/folders/sp/8h47vr6s72v4hngsrxrfwpzm0000gn/T/libwebrtc-java1779612920915696582.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64')), '/System/Volumes/Preboot/Cryptexes/OS/private/var/folders/sp/8h47vr6s72v4hngsrxrfwpzm0000gn/T/libwebrtc-java1779612920915696582.dylib' (no such file), '/private/var/folders/sp/8h47vr6s72v4hngsrxrfwpzm0000gn/T/libwebrtc-java1779612920915696582.dylib' (mach-o file, but is an incompatible architecture (have 'x86_64', need 'arm64'))

But it's okay if I comment this declaration. I'll try to figure out how to solve this if you don't have M1/M2 at the moment.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I only have a M1 Mac Mini, so no camera or microphone to test. But also got it at least running, by, like you mentioned, removing the x86 dependency.

I think adding this native dependencies would need to be done by the implementing application. It still worked after I removed the import from the library build.gradle and added it to the sample app build.gradle.

I will remove the import from the library and update the README to reflect that it is neccessary for implementing applications to add the right dependency on their own (like you need to add the WebRTC pod to iOS apps right now).

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we have to leave import with classifier in order to run JVM tests in CI. I guess we could solve it as following:

    val osName = System.getProperty("os.name").lowercase()
    val hostOS = if (osName.contains("mac")) {
        "macos"
    } else if (osName.contains("linux")) {
        "linux"
    } else if (osName.contains("windows")) {
        "windows"
    } else {
        throw IllegalStateException("Unsupported OS: $osName")
    }
    val hostArch = System.getProperty("os.arch").lowercase()
    jvmMainApi(deps.webrtc.java)
    jvmMainImplementation(
        group = deps.webrtc.java.get().group!!,
        name = deps.webrtc.java.get().name,
        version = deps.webrtc.java.get().version,
        classifier = "$hostOS-$hostArch"
    )

It works well on my Mac. However, I'd like to test it on Windows/Linux too. What do you think?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the CI tests in this repo? Then as jvmTestImplementation? I am not that familiar with publishing artifacts and am not sure what the published artifact with one of them defined as a dependency would look like.
But maybe the webrtc-kmp jvm artifact could also be published with every classifier and dependency separately instead?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had to add a switch for the architecture, because "os.arch" returned "amd64" on my windows device.

val hostArch = when(val arch = System.getProperty("os.arch").lowercase()) {
      "amd64" -> "x86_64"
      else -> arch
  }

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's move tests to the commonTest source set so that they run on all platforms in CI.

However, I think we'll have problem with MediaDevicesTests. It will fail on different platforms as Android, iOS and Mac require permissions to access microphone or camera. Perhaps, we could leave it but add @Ignore annotation for now.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds good. The MediaDevicesTests where just fo testing the implementation.

@shepeliev
Copy link
Owner

shepeliev commented Mar 4, 2024

I've tried to run the JVM sample app on Mac. Unfortunately, there is exception getting user media:

java.lang.Error: Unhandled Exception
        at dev.onvoid.webrtc.media.video.VideoDeviceSource.start(Native Method)
        at com.shepeliev.webrtckmp.LocalVideoStreamTrack.<init>(LocalVideoStreamTrack.kt:13)
        at com.shepeliev.webrtckmp.MediaDevicesImpl.getUserMedia(MediaDevices.kt:89)
        at com.shepeliev.webrtckmp.MediaDevices$Companion.getUserMedia(MediaDevices.kt)
        at com.shepeliev.webrtckmp.sample.shared.RoomComponent$ViewModel$openUserMedia$2.invokeSuspend(RoomComponent.kt:86)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

Also, JVM tests fail on Mac with java.lang.UnsatisfiedLinkError yet. I'll try to figure out what is the reason as soon as be able.

It's really great to add JVM platform, however I think we should make it work properly on any OS (windows, macos, linux).

@tamimattafi
Copy link

@shepeliev @aschulz90 Any updates on this?

@aschulz90
Copy link
Author

@tamimattafi
I don't have any Mac (neither Intel nor M1) or Linux device with a webcam and microphone to test on. Maybe you could help by checking the tests and demo on them, if you have any of those device types.

@tamimattafi
Copy link

@aschulz90 Hello! I have both Mac on Intel and M1, I can help on Saturday if it's suitable for you

@tamimattafi
Copy link

tamimattafi commented Apr 15, 2024

@aschulz90 We tried on both M1 and Intel macs, here's a report:

1. Sample run from main():

  1. Crash when the PC on Intel chip is missing a camera or microphone:
java.util.NoSuchElementException: List is empty.
        at kotlin.collections.CollectionsKt___CollectionsKt.first(_Collections.kt:214)
        at com.shepeliev.webrtckmp.sample.shared.RoomComponent$ViewModel.createPeerConnection(RoomComponent.kt:180)
        at com.shepeliev.webrtckmp.sample.shared.RoomComponent$ViewModel.createRoom(RoomComponent.kt:122)
  1. Selecting Speakers on M1 MacBook throws an exception:
java.lang.Error: Set playout device failed
	at dev.onvoid.webrtc.media.audio.AudioDeviceModule.setPlayoutDevice(Native Method)
	at com.shepeliev.webrtckmp.WebRtc.setAudioOutputDevice(WebRtc.kt:78)
	at com.shepeliev.webrtckmp.SelectMicrophoneCameraScreenKt$SelectMicrophoneCameraScreen$4$1$3$1$1.invoke(SelectMicrophoneCameraScreen.kt:166)
	at com.shepeliev.webrtckmp.SelectMicrophoneCameraScreenKt$SelectMicrophoneCameraScreen$4$1$3$1$1.invoke(SelectMicrophoneCameraScreen.kt:163)
	at com.shepeliev.webrtckmp.SelectMicrophoneCameraScreenKt$DeviceSelector$1$4$2$1.invoke(SelectMicrophoneCameraScreen.kt:235)
	at com.shepeliev.webrtckmp.SelectMicrophoneCameraScreenKt$DeviceSelector$1$4$2$1.invoke(SelectMicrophoneCameraScreen.kt:234)
  1. The confirm button on M1 MacBook doesn't react to clicks, even tho all devices were selected, seems like it's missing a permission
  2. Share desktop asks for permission, denying the permission, keeps showing the screen sharing, and by clicking on create (room) it throws this exception:
java.util.NoSuchElementException: List is empty.
	at kotlin.collections.CollectionsKt___CollectionsKt.first(_Collections.kt:214)
	at com.shepeliev.webrtckmp.sample.shared.RoomComponent$ViewModel.createPeerConnection(RoomComponent.kt:179)
	at com.shepeliev.webrtckmp.sample.shared.RoomComponent$ViewModel.createRoom(RoomComponent.kt:122)
	at com.shepeliev.webrtckmp.sample.shared.RoomComponent.createRoom(RoomComponent.kt)
	at com.shepeliev.webrtckmp.VideoScreenKt$VideoScreen$2$1$2$2$1$1.invoke(VideoScreen.kt:122)
	at com.shepeliev.webrtckmp.VideoScreenKt$VideoScreen$2$1$2$2$1$1.invoke(VideoScreen.kt:122)
	at androidx.compose.foundation.ClickablePointerInputNode$pointerInput$3.invoke-k-4lQ0M(Clickable.kt:898)

Because of these issues, I couldn't really test the actual connection. Not sure how to give this permission manually since I don't see the app there, probably I need to build a .dmg and install the app, or something is missing in the manifest / plist for macOS.

2. Tests in commonTest run on jvm:

  1. IceServerTest fails both on M1 and Intel:
java.lang.UnsatisfiedLinkError: 'void dev.onvoid.webrtc.media.audio.AudioDeviceModule.initialize(dev.onvoid.webrtc.media.audio.AudioLayer)'
	at dev.onvoid.webrtc.media.audio.AudioDeviceModule.initialize(Native Method)
	at dev.onvoid.webrtc.media.audio.AudioDeviceModule.<init>(AudioDeviceModule.java:36)
	at com.shepeliev.webrtckmp.WebRtc.initializePeerConnectionFactory(WebRtc.kt:42)
	at com.shepeliev.webrtckmp.WebRtc.initialize(WebRtc.kt:37)
	at com.shepeliev.webrtckmp.WebRtc.getPeerConnectionFactory$webrtc_kmp(WebRtc.kt:16)
	at com.shepeliev.webrtckmp.PeerConnection$native$2.invoke(PeerConnection.kt:40)
	at com.shepeliev.webrtckmp.PeerConnection$native$2.invoke(PeerConnection.kt:39)
	....
  1. PeerConnectionTest fails both on M1 and Intel:
java.lang.UnsatisfiedLinkError: 'void dev.onvoid.webrtc.media.audio.AudioDeviceModule.initialize(dev.onvoid.webrtc.media.audio.AudioLayer)'
	at dev.onvoid.webrtc.media.audio.AudioDeviceModule.initialize(Native Method)
	at dev.onvoid.webrtc.media.audio.AudioDeviceModule.<init>(AudioDeviceModule.java:36)
	at com.shepeliev.webrtckmp.WebRtc.initializePeerConnectionFactory(WebRtc.kt:42)
	at com.shepeliev.webrtckmp.WebRtc.initialize(WebRtc.kt:37)
	at com.shepeliev.webrtckmp.WebRtc.getPeerConnectionFactory$webrtc_kmp(WebRtc.kt:16)
	at com.shepeliev.webrtckmp.PeerConnection$native$2.invoke(PeerConnection.kt:40)
	at com.shepeliev.webrtckmp.PeerConnection$native$2.invoke(PeerConnection.kt:39)
	...

@shepeliev @aschulz90 I would love to help with any further steps to accelerate the integration on JVM

@shepeliev
Copy link
Owner

@tamimattafi thanks for the tests. Unfortunately, I don't have any quick answer for these issues on mac. It definitely requires more researching. I'll back to this as soon as be able.

@aschulz90
Copy link
Author

Yeah, I haven't accounted for permissions on desktop. A quick search wasn't that conclusive on how to request them for java applications on macOS. Unfortunately I don't think I can be of much help in that area.

@tricknology
Copy link

tricknology commented Apr 16, 2024

@tamimattafi - Are you using OpenJDK by any chance? And perhaps running this from the IDE and not a built package?

Check this: https://bugs.openjdk.org/browse/JDK-8272639

They've used the --resource-dir option when building the package and pointed it to a custom info.plist containing the proper key/val

<key>NSMicrophoneUsageDescription</key>
<string>The application is requesting access to the microphone.</string> 

@tamimattafi
Copy link

@tricknology Yes indeed, I'm using OpenJDK 20 on my machine, and JetBrains Runtime version 17 on the IDE. And indeed I run the sample from main() method from the IDE as stated in the test report.

My guess is, as you stated that it is better to build a package and install it properly for proper testing with permissions.

@its-Chiedu
Copy link

@tamimattafi I feel like there should be a way to pass options to the compiler in run config.

You might also specify a makefile to do the same.. unfortunately I'm not exactly sure on the step-by-step.

@aschulz90
Copy link
Author

Selecting Speakers on M1 MacBook throws an exception:

This should be fixed now.

@tamimattafi
Copy link

@its-Chiedu There is a way to pass every detail including info.plist
https://github.com/JetBrains/compose-multiplatform/blob/master/tutorials/Native_distributions_and_local_execution/README.md#customizing-infoplist-on-macos

However, I'm having some issues building a .dmg using java 17 and java 20, I will look through the issue when I find some free time.

@aschulz90
Copy link
Author

aschulz90 commented Apr 24, 2024

@shepeliev I have merged the latest changes from main and updated the sample app for jvm support. Can you please review that part again?

@shepeliev
Copy link
Owner

Unfortunately, still have an exception on both of Intel and M2 macs:

java.lang.Error: Unhandled Exception
        at dev.onvoid.webrtc.media.video.VideoDeviceSource.start(Native Method)
        at com.shepeliev.webrtckmp.LocalVideoStreamTrack.<init>(LocalVideoStreamTrack.kt:13)
        at com.shepeliev.webrtckmp.MediaDevicesImpl.getUserMedia(MediaDevices.kt:89)
        at com.shepeliev.webrtckmp.MediaDevices$Companion.getUserMedia(MediaDevices.kt)
        at com.shepeliev.webrtckmp.sample.shared.RoomComponent$ViewModel$openUserMedia$2.invokeSuspend(RoomComponent.kt:86)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:108)
        at kotlinx.coroutines.scheduling.CoroutineScheduler.runSafely(CoroutineScheduler.kt:584)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.executeTask(CoroutineScheduler.kt:793)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.runWorker(CoroutineScheduler.kt:697)
        at kotlinx.coroutines.scheduling.CoroutineScheduler$Worker.run(CoroutineScheduler.kt:684)

I guess some problem is in native WebRTC SDK that embeded with WebRTC java lib.
There are two options for now:

  • make basic sample Java app which reproduces the problem and raise an issue at https://github.com/devopvoid/webrtc-java
  • build webrtc-java from the sources adding some logs into low-level C code and try to figure out by myself

I think we could start with the first option for now. Java background devs help is appreciated :)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants