Camera recording choppy, not smooth. Min/max frame rate difference high

[kenle]

Describe the bug
Hi @pedroSG94,

My video recordings are looking not smooth, when panning you can see the frames freeze and jump sometimes. I compared this to the native camera recording with the exact same settings, and the native camera recording is much smoother.

When looking at the media information of the RtmpCamera2 recording, it seems strange that the min frame rate is 15.139 FPS, while the max frame rate is 72.139 FPS, with an average of 29.834 FPS. The native camera recording has min frame rate of 30.000 FPS and max frame rate of 30.010 FPS with average of 30.000 FPS.

I also tried for RtmpCamera1 and it appears the same, with frames that freeze and jump.

I provided detailed information below regarding this case. Hope you can help me figure this out, I tried many different settings without any success.

Thank you very much,
-Ken

To Reproduce

Code Summary

rtmpCamera2.setVideoCodec(VideoCodec.H265);

if(rtmpCamera2.prepareAudio() && rtmpCamera2.prepareVideo(
                                    1280,
                                     720,
                                    30,
                                    6997000, // copied bitrates native phone was using
                                    2,
                                    CameraHelper.getCameraOrientation(context),
                                    CodecProfileLevel.HEVCProfileMain,
                                    CodecProfileLevel.HEVCMainTierLevel31
                                ) {
    rtmpCamera2.startRecord(captureFilePath);
}

Since programmatically I only have access to a SurfaceTexture and not a SurfaceView, I modified Camera2Base to have a constructor that took in a Surface object, and I used a modified version of OffScreenGlThread as the glInterface. The run function was modified to closely resemble OpenGlView's run function, while using my surface.

 public Camera2Base(Surface surface, Context context) {
    this.context = context;
    glInterface = new MyOffScreenGlThread(surface, context);
    isBackground = false;
    init(context);
  }

  public MyOffScreenGlThread(Surface surface, Context context) {
    this.context = context;
    this.surface = surface;
  }

  @Override
  public void run() {
    surfaceManager.release();
    //surfaceManager.eglSetup();
    surfaceManager.eglSetup(this.surface);
    surfaceManager.makeCurrent();
    managerRender.initGl(context, encoderWidth, encoderHeight, encoderWidth, encoderHeight);
    managerRender.getSurfaceTexture().setOnFrameAvailableListener(this);
    surfaceManagerPhoto.release();
    surfaceManagerPhoto.eglSetup(encoderWidth, encoderHeight, surfaceManager);
    semaphore.release();
    try {
      while (running) {
        fpsLimiter.setFrameStartTs();
        if (frameAvailable || forceRender) {
          frameAvailable = false;
          surfaceManager.makeCurrent();
          managerRender.updateFrame();
          managerRender.drawOffScreen();
          managerRender.drawScreen(encoderWidth, encoderHeight, AspectRatioMode.NONE, 0, isPreviewVerticalFlip, isPreviewHorizontalFlip);
          surfaceManager.swapBuffer();

          if (!filterQueue.isEmpty()) {
            Filter filter = filterQueue.take();
            managerRender.setFilterAction(filter.getFilterAction(), filter.getPosition(), filter.getBaseFilterRender());
          }

          synchronized (sync) {
            if (surfaceManagerEncoder.isReady() && !fpsLimiter.limitFPS()) {
              int w = muteVideo ? 0 : encoderWidth;
              int h = muteVideo ? 0 : encoderHeight;
              surfaceManagerEncoder.makeCurrent();
              managerRender.drawScreen(w, h, AspectRatioMode.NONE,
                  streamRotation, isStreamVerticalFlip, isStreamHorizontalFlip);
              surfaceManagerEncoder.swapBuffer();
            }
            if (takePhotoCallback != null && surfaceManagerPhoto.isReady()) {
              surfaceManagerPhoto.makeCurrent();
              managerRender.drawScreen(encoderWidth, encoderHeight, AspectRatioMode.NONE,
                  streamRotation, isStreamVerticalFlip, isStreamHorizontalFlip);
              takePhotoCallback.onTakePhoto(GlUtil.getBitmap(encoderWidth, encoderHeight));
              takePhotoCallback = null;
              surfaceManagerPhoto.swapBuffer();
            }
          }
        }
        synchronized (sync) {
          long sleep = fpsLimiter.getSleepTime();
          if (sleep > 0) sync.wait(sleep);
        }
      }
    } catch (InterruptedException ignore) {
      Thread.currentThread().interrupt();
    } finally {
      managerRender.release();
      surfaceManagerPhoto.release();
      surfaceManagerEncoder.release();
      surfaceManager.release();
    }
  }

And I use it as such

rtmpCamera2 = new RtmpCamera2(new Surface(mySurfaceTexture), RtmpConnectionHandler.instance);

Resulting Galaxy A15 RtmpCamera2 File Recording:

https://www.youtube.com/watch?v=2t28JEvseqQ

Media Info for RtmpCamera2 File Recording:

Compared to Galaxy A15 Native Camera Recording:

https://www.youtube.com/watch?v=7idj-g_QfbE

Media Info for Native Camera Recording:

Logs related to Video Encoder Setup

I/VideoEncoder(16616): Prepare video info: SURFACE, 1280x720
I/VideoEncoder(16616): set bitrate mode CBR
I/MediaCodec(16616): MediaCodec will operate in async mode
D/CodecSeeding(16616): Seed: codec c2.mtk.hevc.encoder, mediatype video/hevc, overrideable 1
D/CodecProperties(16616): setTuningValue(vq-target-bpp,0)
D/CodecProperties(16616): setTuningValue(vq-target-bpp-1080p,1.50)
D/CodecProperties(16616): setTuningValue(vq-target-bpp-720p,1.80)
D/CodecProperties(16616): setTuningValue(vq-target-bpp-540p,2.10)
D/CodecProperties(16616): setTuningValue(vq-target-bpp-480p,2.30)
D/CodecProperties(16616): setTuningValue(vq-target-bpp-320x240,0)
D/CodecProperties(16616): setTuningValue(vq-target-qpmax,-1)
D/CodecProperties(16616): setTuningValue(vq-target-qpmax-1080p,45)
D/CodecProperties(16616): setTuningValue(vq-target-qpmax-720p,44)
D/CodecProperties(16616): setTuningValue(vq-target-qpmax-540p,43)
D/CodecProperties(16616): setTuningValue(vq-target-qpmax-480p,42)
D/CodecProperties(16616): setTuningValue(vq-bitrate-phaseout,1.75)
D/CodecProperties(16616): setTuningValue(vq-boost-missing-qp,0.20)
D/CodecProperties(16616): setFeatureValue(_vq_eligible.device,1)
D/CodecProperties(16616): setFeatureValue(_quality.target,1)
D/CodecSeeding(16616): Seed: codec c2.mtk.hevc.encoder, mediatype video/hevc, overrideable 0
D/VQApply (16616): minquality: applies only to VBR encoding
D/MediaCodec(16616): shapeMediaFormat: deltas(0): AMessage(what = 0x00000000) = {
D/MediaCodec(16616):   }
D/MediaCodec(16616): flushMediametrics
D/CCodec  (16616): [c2.mtk.hevc.encoder] buffers are bound to CCodec for this session
I/CCodec  (16616): appPid(16616) width(1280) height(720)
W/CCodec  (16616): can't get ro.hardware.chipname
W/CCodec  (16616): can't setup for [mt6835]
W/CCodec  (16616): can't get ro.hardware.chipname
W/CCodec  (16616): can't setup for [mt6835]
W/CCodec  (16616): can't get ro.hardware.chipname
W/CCodec  (16616): can't setup for [mt6835]
D/CCodecConfig(16616): no c2 equivalents for color-format
D/CCodecConfig(16616): no c2 equivalents for flags
D/CCodecConfig(16616): no c2 equivalents for encoder
D/CCodecConfig(16616): c2 config diff is   c2::u32 algo.bitrate-mode.value = 1
D/CCodecConfig(16616):   c2::u32 coded.bitrate.value = 6997000
D/CCodecConfig(16616):   c2::u32 coded.pl.level = 24580
D/CCodecConfig(16616):   c2::i64 coding.sync-frame-interval.value = 2000000
D/CCodecConfig(16616):   c2::u32 raw.pixel-format.value = 34
D/CCodecConfig(16616):   c2::u32 raw.size.height = 720
D/CCodecConfig(16616):   c2::u32 raw.size.width = 1280
W/ColorUtils(16616): expected specified color aspects (0:0:0:0)
W/Codec2Client(16616): query -- param skipped: index = 3254781982.
D/CCodec  (16616): Setting encoder options for disabling wrapping
D/CCodec  (16616): encoding statistics level = 0
D/CCodec  (16616): setup formats input: AMessage(what = 0x00000000) = {
D/CCodec  (16616):   int32_t android._color-format = 2130708361
D/CCodec  (16616):   Rect crop(0, 0, 1279, 719)
D/CCodec  (16616):   int32_t color-standard = 0
D/CCodec  (16616):   int32_t color-range = 0
D/CCodec  (16616):   int32_t color-transfer = 0
D/CCodec  (16616):   int32_t frame-rate = 30
D/CCodec  (16616):   int32_t height = 720
D/CCodec  (16616):   float input.time-stretch = 1.000000
D/CCodec  (16616):   int32_t intra-refresh-period = 0
D/CCodec  (16616):   int32_t latency = 8
D/CCodec  (16616):   string mime = "video/raw"
D/CCodec  (16616):   int32_t prepend-sps-pps-to-idr-frames = 0
D/CCodec  (16616):   int32_t priority = 0
D/CCodec  (16616):   int32_t video-qp-average = 0
D/CCodec  (16616):   int32_t width = 1280
D/CCodec  (16616):   int32_t android._dataspace = 0
D/CCodec  (16616):   int64_t android._C2MemoryUsage = 0
D/CCodec  (16616):   int32_t color-format = 2130708361
D/CCodec  (16616):   int32_t bytebuffer-wrapping-disable = 1
D/CCodec  (16616): }
D/CCodec  (16616): setup formats output: AMessage(what = 0x00000000) = {
D/CCodec  (16616):   int32_t bitrate = 6997000
D/CCodec  (16616):   int32_t bitrate-mode = 2
D/CCodec  (16616):   Rect crop(0, 0, 1279, 719)
D/CCodec  (16616):   int32_t width = 1280
D/CCodec  (16616):   int32_t color-standard = 0
D/CCodec  (16616):   int32_t color-range = 0
D/CCodec  (16616):   int32_t color-transfer = 0
D/CCodec  (16616):   int32_t frame-rate = 30
D/CCodec  (16616):   int32_t height = 720
D/CCodec  (16616):   float input.time-stretch = 1.000000
D/CCodec  (16616):   int32_t intra-refresh-period = 0
D/CCodec  (16616):   int32_t latency = 8
D/CCodec  (16616):   int32_t max-bitrate = 6997000
D/CCodec  (16616):   string mime = "video/hevc"
D/CCodec  (16616):   int32_t prepend-sps-pps-to-idr-frames = 0
D/CCodec  (16616):   int32_t priority = 0
D/CCodec  (16616):   int32_t profile = 1
D/CCodec  (16616):   int32_t video-qp-average = 0
D/CCodec  (16616):   int32_t level = 256
D/CCodec  (16616): }
I/CCodecConfig(16616): query failed after returning 23 values (BAD_INDEX)
D/CCodecConfig(16616): c2 config diff is   c2::u32 coded.tile-layout.columns = 5
D/CCodecConfig(16616):   c2::u32 coded.tile-layout.rows = 3
W/ColorUtils(16616): expected specified color aspects (0:0:0:0)
W/CCodec  (16616): can't get ro.hardware.chipname
W/CCodec  (16616): can't setup for [mt6835]
W/CCodec  (16616): can't get ro.hardware.chipname
W/CCodec  (16616): can't setup for [mt6835]
D/CCodec  (16616): [CreateInputSurface] surface in app process
D/BufferQueueConsumer(16616): [GraphicBufferSource](id:40e800000004,api:0,p:-1,c:16616) connect: controlledByApp=false
D/CCodecConfig(16616): set default color aspects for surface (R:2(Limited), P:1(BT709_5), M:1(BT709_5), T:3(SMPTE170M))
D/CCodec  (16616): input format changed to AMessage(what = 0x00000000) = {
D/CCodec  (16616):   int32_t android._color-format = 2130708361
D/CCodec  (16616):   Rect crop(0, 0, 1279, 719)
D/CCodec  (16616):   int32_t color-standard = 1
D/CCodec  (16616):   int32_t color-range = 2
D/CCodec  (16616):   int32_t color-transfer = 3
D/CCodec  (16616):   int32_t frame-rate = 30
D/CCodec  (16616):   int32_t height = 720
D/CCodec  (16616):   float input.time-stretch = 1.000000
D/CCodec  (16616):   int32_t intra-refresh-period = 0
D/CCodec  (16616):   int32_t latency = 8
D/CCodec  (16616):   string mime = "video/raw"
D/CCodec  (16616):   int32_t prepend-sps-pps-to-idr-frames = 0
D/CCodec  (16616):   int32_t priority = 0
D/CCodec  (16616):   int32_t video-qp-average = 0
D/CCodec  (16616):   int32_t width = 1280
D/CCodec  (16616):   int32_t android._dataspace = 260
D/CCodec  (16616):   int64_t android._C2MemoryUsage = 0
D/CCodec  (16616):   int32_t color-format = 2130708361
D/CCodec  (16616):   int32_t bytebuffer-wrapping-disable = 1
D/CCodec  (16616): }
D/C2OMXNode(16616): actual buffer count(16), input delay(0), pipeline delay(8)
D/GraphicBufferSource(16616): setting dataspace: 0x104, acquired=0
D/CCodec  (16616): ISConfig not changed
I/VideoEncoder(16616): prepared
I/System.out(16616): getMaxRecommendedRecordBitRateFor(SD:30) = 6997000
D/MPEG4Writer(16616): PreAllocation enabled
I/VideoEncoder(16616): started

Logs related to FPS

/BufferPoolAccessor2.0(16616): bufferpool2 0xb400007627261c28 : 5(35280 size) total buffers - 4(28224 size) used buffers - 121/126 (recycle/alloc) - 5/243 (fetch/transfer)
I/System.out(16616): FPS: 30
I/System.out(16616): FPS: 30
I/System.out(16616): FPS: 30
I/System.out(16616): FPS: 30
I/System.out(16616): FPS: 30
I/System.out(16616): FPS: 30
D/BufferPoolAccessor2.0(16616): bufferpool2 0xb400007627261c28 : 5(35280 size) total buffers - 4(28224 size) used buffers - 251/256 (recycle/alloc) - 5/503 (fetch/transfer)
I/System.out(16616): FPS: 31
I/System.out(16616): FPS: 30
I/System.out(16616): FPS: 30
I/System.out(16616): FPS: 31
I/System.out(16616): FPS: 30
D/BufferPoolAccessor2.0(16616): bufferpool2 0xb400007627261c28 : 5(35280 size) total buffers - 4(28224 size) used buffers - 380/385 (recycle/alloc) - 5/761 (fetch/transfer)
I/System.out(16616): FPS: 30
I/System.out(16616): FPS: 31
I/System.out(16616): FPS: 31
I/System.out(16616): FPS: 30
I/System.out(16616): FPS: 30
D/BufferPoolAccessor2.0(16616): bufferpool2 0xb400007627261c28 : 5(35280 size) total buffers - 4(28224 size) used buffers - 507/512 (recycle/alloc) - 5/1016 (fetch/transfer)
I/System.out(16616): FPS: 30
I/System.out(16616): FPS: 30
I/System.out(16616): FPS: 31
I/System.out(16616): FPS: 31
I/System.out(16616): FPS: 30
D/BufferPoolAccessor2.0(16616): bufferpool2 0xb400007627261c28 : 5(35280 size) total buffers - 4(28224 size) used buffers - 635/640 (recycle/alloc) - 5/1271 (fetch/transfer)
I/System.out(16616): FPS: 30
I/System.out(16616): FPS: 30
I/System.out(16616): FPS: 30
I/System.out(16616): FPS: 30

Also tried RtmpCamera1

https://www.youtube.com/watch?v=a9zm5NNZX_4

Media Info for RtmpCamera1

Smartphone (please complete the following information):

• Library version: 2.3.5
• Device: Galaxy A15
• OS: Android Version 14, One UI 6.0
• Class used: RtmpCamera2

Additional context
Add any other context about the problem here.

[pedroSG94]

Hello,

Do you have the same problem using RtmpCamera2 in OpenGl example without modifications?
If yes, did you tried using rotate example, change video source to CameraX and then record?

[kenle]

Hi @pedroSG94 ,

I recorded with the OpenGl example (editing the resolution and Codec profiles since the default is 640x480). It seems smoother although I still notice some hickups (for example if you focus on the middle vertical red shelf bar around time 0:23 to 0:27 you can see some glitches). I'm not sure if it's just me or if I'm being too critical.

https://www.youtube.com/watch?v=l97P8HyFzHg

The Rotation CameraX example seems to have some choppiness as well (for example around 0:16 to 0:20)
https://www.youtube.com/watch?v=HaqU_GvHHEs

Compare that to Native again:
https://www.youtube.com/watch?v=7idj-g_QfbE

Is there a reason why the min and max frame rates are different? In the native camera min and max stay at 30fps.

Thank you very much for your time,
-Ken

OpenGL Example

Rotation Camera X Example

[pedroSG94]

Hello,

I think that the fps of the metadata is not related with the problem. I can't reproduce your case but I have a similar metadata min and max fps.

According with internet. That min and max fps are calculated depend of the timestamp in the frames but the audio and video have a good sync so I think I don't need to worry about it (fix it could be interested but it is not a priority since the video is not affected).

About your last video examples. I can see the same problem in native camera video but with less frequent for example in second 2-4 you can see a fps jump.

Can you see the same problem (fps) in the activity preview or the problem is only in the video recorded?
If you try with other device, can you reproduce it?

[pedroSG94]

I detected a problem related that should be fixed here:
23aacf9
Get last master branch commit and let me know if the result is better. I will create and give you a gradle compile while I upload the next release if all is working fine

[kenle]

I detected a problem related that should be fixed here: 23aacf9 Get last master branch commit and let me know if the result is better. I will create and give you a gradle compile while I upload the next release if all is working fine

Thank you so much @pedroSG94 ! I will try it and get back to you.

[kenle]

Hi @pedroSG94,

I did some re-tests a few times because at first it seemed ok, but then this seems to happen very randomly sometimes. Overall I'm not sure if it's any better because I went back to the old version to test again and it seems ok sometimes, and sometimes not.

To be extra sure, I tested the native camera again a few times, and I believe it seems overall smooth.

Again, at this point I'm not sure if I'm being too critical. Although only a few glitches, I believe the OpenGlView is much better than my modified OffScreenGlThread that I showed you at first, so if you're busy with other tasks perhaps I can continue with OpenGlView since I found a way to integrate it into my app.

Thank you very much for your time. - Ken

OpenGL Example Re-test 1 AFTER availableFrames fix (seems smooth overall)

https://www.youtube.com/watch?v=aoEn89OPD6U

OpenGL Example Re-test 2 AFTER availableFrames fix (glitches 1:58 - 2:07)

https://www.youtube.com/watch?v=gRrdAFPoL1E

Opengl Example Re-test BEFORE availableFrame fix (glitches 0:50 - 0:57)

https://www.youtube.com/watch?v=RVJOGCxqcy4

Native Camera Re-test (seems smooth overall)

https://www.youtube.com/watch?v=z3m3RXvrhVg

Native Camera Re-test 2 (seems smooth overall)

https://www.youtube.com/watch?v=DUNvWbVWAug

[pedroSG94]

Hello,

Lets try this last idea. Try to compile using this branch:
#1414

Let me know if all is working fine. This remove thread sleep so the performance should be better.
Test with rotation example, please

[kenle]

Hi @pedroSG94,

Looks like you just merged it? I'll try the master branch.

Thank you,
-Ken

[kenle]

Hi @pedroSG94 ,

Here's result of the latest master branch:
https://www.youtube.com/watch?v=x46sJdpSYCc

In my opinion I still see the glitches (for example at 1:16 - 1:20) . Which file did you remove the thread sleep from?

Again, thank you for all your effort.

-Ken

[pedroSG94]

Which file did you remove the thread sleep from?

I removed it from GlStreamInterface
I replaced the thread with a loop that sleep when no frameavailable to a ExecutorService. Old file version:
https://github.com/pedroSG94/RootEncoder/blob/2.3.6/library/src/main/java/com/pedro/library/view/GlStreamInterface.kt

[kenle]

Here's another video: https://www.youtube.com/watch?v=NrWCSDqMoiU

In my opinion it doesn't happen often enough anymore where it would bother the user or be too noticeable.

I noticed the glitches happen usually first few seconds of the film, then it smooths out, and it might happens very small now once every minute or two.

-Ken

[pedroSG94]

Well, at least it is better than on the start.
I will keep an eye in the way to improve it but for now I can't detect more code problems.

In my devices I can't reproduce this glitches anymore but I think my devices are betters, maybe it is only noticeable on low spec devices.

[kenle]

Yes, thank you @pedroSG94 .

I really appreciate your library, it is great.

• Ken