Seeing the preview while streaming

[maxgrrs]

Hey Pedro,
first - thank you for the effort you put in this solution and providing it to the open source world!
second - I have a question regarding an application I develop using Root Encoder:

The main screen of the application should be divided in two halves.
One half contains the buttons like start/stop streaming and switch camera.
The other half contains the preview of the camera like it will be streamed.

I do this using a CamaraPreview Composable:

@Composable
fun CameraPreview(cameraProviderFuture: ListenableFuture<ProcessCameraProvider>) {
    val lifecycleOwner = LocalLifecycleOwner.current
    AndroidView(
        factory = { context ->
            PreviewView(context).apply {
                setBackgroundColor(Color.GREEN)
                layoutParams = LinearLayout.LayoutParams(MATCH_PARENT, MATCH_PARENT)
                scaleType = PreviewView.ScaleType.FILL_START
                implementationMode = PreviewView.ImplementationMode.COMPATIBLE
                post {
                    cameraProviderFuture.addListener(Runnable {
                        val cameraProvider = cameraProviderFuture.get()
                        bindPreview(
                            cameraProvider,
                            lifecycleOwner,
                            this,
                        )
                    }, ContextCompat.getMainExecutor(context))
                }
            }
        }
    )
}

fun bindPreview(
    cameraProvider: ProcessCameraProvider,
    lifecycleOwner: LifecycleOwner,
    previewView: PreviewView,
) {
    val preview: Preview = Preview.Builder()
        .build()

    val cameraSelector: CameraSelector = CameraSelector.Builder()
        .requireLensFacing(CameraSelector.LENS_FACING_BACK)
        .build()

    preview.setSurfaceProvider(previewView.surfaceProvider)

    var camera = cameraProvider.bindToLifecycle(lifecycleOwner, cameraSelector, preview)
}

This CameraPreview Composable is integrated into a HomeScreen Composable:

@OptIn(ExperimentalPermissionsApi::class)
@Composable
fun HomeScreen(
    modifier: Modifier = Modifier
){
    val currentContext = LocalContext.current
    val connectChecker = AppConnectChecker(currentContext)

    val multiplePermissionsState = rememberMultiplePermissionsState(
        listOf(
            android.Manifest.permission.RECORD_AUDIO,
            android.Manifest.permission.CAMERA,
        )
    )

    val rtmpCamera2 = RtmpCamera2(currentContext, false, connectChecker)
    rtmpCamera2.streamClient.setReTries(10)

    var streamButtonLabel by remember { mutableIntStateOf(R.string.start_button) }
    var streamingTarget by remember { mutableStateOf("rtmp://somelink") }

    val cameraProviderFuture : ListenableFuture<ProcessCameraProvider> = ProcessCameraProvider.getInstance(currentContext)

    Column(
        modifier = Modifier.fillMaxWidth(),
        horizontalAlignment = Alignment.CenterHorizontally,
        verticalArrangement = Arrangement.spacedBy(
            dimensionResource(id = R.dimen.padding_medium)
        )
    ) {
        DefaultButton(
            labelResourceId = streamButtonLabel,
            onClick = { onStreamClicked(currentContext, rtmpCamera2, multiplePermissionsState, streamingTarget, changeButtonLabel = { streamButtonLabel = it }) }
        )
        DefaultButton(
            labelResourceId = R.string.switch_camera_button,
            onClick = { onSwitchCameraClicked(currentContext, rtmpCamera2) }
        )
        EditStreamingTargetField(streamingTarget, onTargetChange = { streamingTarget = it })

        CameraPreview(cameraProviderFuture = cameraProviderFuture)
    }
}

Im still in the phase of trying around so many things are not really best practise.
Before I integrated the CamaraPreview and the related cameraProviderFuture the application was able to start a stream on twitch with audio and video working.
When I first integrated the preview, it did not work because the Camera was blocked.
It is only working when the RtmpCamera2 instance is not instanciated (outcommented).

I somehow tried to find out how to tell the Preview to use the already instantiated stuff from Camera2, but I did not get behind it.
I also tried to find out how to use your startPrievew() function in a way that suit the requirments but I also couldnt figure it out.

Do you know a way on how to reach the desired state?

If you miss some more background information I will provide further files and so on of course...
Thanks, Max

[pedroSG94]

Hello,

When I first integrated the preview, it did not work because the Camera was blocked.
It is only working when the RtmpCamera2 instance is not instanciated (outcommented).

This is because RtmpCamera2 already contain a camera instance and you can't use 2 cameras instances at the same time.
You need to use camera from RtmpCamera2 instead of CameraPreview with ProcessCameraProvider.
For example:

class MainActivity : ComponentActivity(), ConnectChecker {

    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)
        setContent {
            val rtmpCamera2 = RtmpCamera2(LocalContext.current, true, this)
            MyApplicationTheme {
                // A surface container using the 'background' color from the theme
                Surface(
                    modifier = Modifier.fillMaxSize(),
                    color = MaterialTheme.colorScheme.background
                ) {
                    AndroidView(factory = {
                        OpenGlView(it).apply {
                            val view = this
                            view.setAspectRatioMode(AspectRatioMode.Adjust)
                            view.holder.addCallback(object : SurfaceHolder.Callback {
                                override fun surfaceCreated(p0: SurfaceHolder) {

                                }

                                override fun surfaceChanged(p0: SurfaceHolder, p1: Int, p2: Int, p3: Int) {
                                    rtmpCamera2.replaceView(view) //attach preview to camera
                                    rtmpCamera2.startPreview() //open camera
                                }

                                override fun surfaceDestroyed(p0: SurfaceHolder) {
                                    rtmpCamera2.stopPreview() //close camera
                                }
                            })
                        }
                    }, modifier = Modifier.fillMaxSize())
                }
            }
        }
    }

    .... //ConnectChecker implementation
}

[maxgrrs]

Hey Pedro, thank you for your fast reply 🙏🏼
I copied youre surface code over to the CameraPreview composable and handed in the camera instance as parameter:

@Composable
fun CameraPreview(cameraInstance: RtmpCamera2) {
    Surface(
        modifier = Modifier.fillMaxSize(),
        color = MaterialTheme.colorScheme.background
    ) {
        AndroidView(factory = {
            OpenGlView(it).apply {
                val view = this
                view.setAspectRatioMode(AspectRatioMode.Adjust)
                view.holder.addCallback(object : SurfaceHolder.Callback {
                    override fun surfaceCreated(p0: SurfaceHolder) {

                    }

                    override fun surfaceChanged(p0: SurfaceHolder, p1: Int, p2: Int, p3: Int) {
                        cameraInstance.replaceView(view) //attach preview to camera
                        cameraInstance.startPreview() //open camera
                    }

                    override fun surfaceDestroyed(p0: SurfaceHolder) {
                        cameraInstance.stopPreview() //close camera
                    }
                })
            }
        }, modifier = Modifier.fillMaxSize())
    }
}

When I now open the app it shows me the preview in the right way and when I press start streaming everything works fine as well.
Thank you for that at first. I just have two small issues when I want to stop the stream.

One thing is that the rtmpSender stops sending video packages as wanted, but continues to write audio packages (which was so before implementing the preview stuff and is not related to that question so I will check it again and maybe open another request).

The other related thing is that the preview is freezing. I guess this is because the stopStream() function from youre Camera2Base.java file is closing the camera:

public void stopStream() {
    if (streaming) {
      streaming = false;
      stopStreamRtp();
    }
    if (!recordController.isRecording()) {
      onPreview = !isBackground;
      if (audioInitialized) microphoneManager.stop();
      if (glInterface != null) {
        glInterface.removeMediaCodecSurface();
        if (glInterface instanceof OffScreenGlThread) {
          glInterface.stop();
          cameraManager.closeCamera();
        }
      } else {
        if (isBackground) {
          cameraManager.closeCamera();
          onPreview = false;
        } else {
          cameraManager.stopRepeatingEncoder();
        }
      }
      videoEncoder.stop();
      if (audioInitialized) audioEncoder.stop();
      recordController.resetFormats();
    }
  }

It would be nicer if the preview stays alive.
Do you recommend to just use startPreview() again when the stream is stopped or is a better approach existing?
I couldnt find any parameter at stopStream() to keep the camera alive for preview or so.
Thanks, Max

[pedroSG94]

Hello,

First a tip. Surface of the component can be removed if you want. I used it because it is wrote when you create a new jetpack compose project.

About the packets send. Check that you correctly stop the stream. You have an example code here:
https://github.com/pedroSG94/RootEncoder/blob/master/app/src/main/java/com/pedro/streamer/openglexample/OpenGlRtmpActivity.java#L472
So you can use something like this:

if (rtmpCamera1.isRecording()) {
      rtmpCamera1.stopRecord();
}
if (rtmpCamera1.isStreaming()) {
      rtmpCamera1.stopStream();
}
rtmpCamera1.stopPreview(); //remove this line if you don't want stop preview

The preview should be alive if you call only stopStream or stopRecord. The preview stop when you call stopPreview or your preview view (OpenGlView) is destroyed (surfaceDestroyed callback is called). You can check it in my app examples.

I recommend you use breakpoints or logs to make sure that your preview is not destroyed or stopPreview is not called.

[maxgrrs]

Hey Pedro,

when I remove the Surface the upper half with the buttons becomes somehow overwritten and completly black.
I guess some sort of container is needed, but I dont know.

Regarding the rest of the explanation I had some problems figuring out the reasons for the problem.
The thing was, that when I pressed "stop stream", the stopStream() function was not triggered.
It was just triggered when I pressed it the second time, which was the moment when the image freezed and the (audio) stream continued nevertheless.

I tried out several things until I realized that maybe a new instance of the rtmpCamera2 gets created on each recomposition due to the weird implementation I did in the HomeScreen composable:

@OptIn(ExperimentalPermissionsApi::class)
@Composable
fun HomeScreen(
    modifier: Modifier = Modifier
){
    val currentContext = LocalContext.current
    val connectChecker = AppConnectChecker(currentContext)

    val multiplePermissionsState = rememberMultiplePermissionsState(
        listOf(
            android.Manifest.permission.RECORD_AUDIO,
            android.Manifest.permission.CAMERA,
        )
    )

    val rtmpCamera2 = RtmpCamera2(currentContext, false, connectChecker)
    rtmpCamera2.streamClient.setReTries(10)

    var streamButtonLabel by remember { mutableIntStateOf(R.string.start_button) }
    var streamingTarget by remember { mutableStateOf("rtmp://somelink") }

    val cameraProviderFuture : ListenableFuture<ProcessCameraProvider> = ProcessCameraProvider.getInstance(currentContext)

    Column(
        modifier = Modifier.fillMaxWidth(),
        horizontalAlignment = Alignment.CenterHorizontally,
        verticalArrangement = Arrangement.spacedBy(
            dimensionResource(id = R.dimen.padding_medium)
        )
    ) { ...

Im still not always 100% aware of how to handle the state in the new compose way. An easy but also not best practice approach I did was hoisting the camera instance to the MainActivity as the caller instance:

class MainActivity : ComponentActivity() {
    override fun onCreate(savedInstanceState: Bundle?) {
        super.onCreate(savedInstanceState)

        val connectChecker = AppConnectChecker(applicationContext)
        val rtmpCamera2 = RtmpCamera2(applicationContext, true, connectChecker)
        rtmpCamera2.streamClient.setReTries(10)

        setContent {
            Broadcast4allTheme {
                // A surface container using the 'background' color from the theme
                Surface(
                    modifier = Modifier.fillMaxSize(),
                    color = MaterialTheme.colorScheme.background
                ) {
                    HomeScreen(rtmpCamera2)
                }
            }
        }
    }
}

This solved both problems for me so everything is fine now.
Thank you again, Max

[pedroSG94]

Closing then.
You can open other if needed