Port ZXing Android to Camera2

[Denrage]

During investigation of adding new features to this library we noticed that it's still using the obsolet Android.Hardware.Camera library. To have it future proof i refactored the code of the CameraController.android to use the Camera2 API now. This is just a port and it's not removing functionality or adding any.
The only hiccup i had was the FastJavaByteArray. Because the ImageReader is not giving out a pointer on receiving an image, the FastJavaByteArray can't be used anymore (or it would slow it down)

[Redth]

This is great! thanks for the work put into this! I'm going to get an update out first that fixes some issues and still allows for < android api 21 support and then will merge this after.

The only issue I'm seeing so far on this PR is that the orientation seems off 90 degrees... The sample which doesn't auto rotate doesn't find 1D barcodes unless i rotate the device, so perhaps the buffer needs a quick rotate to be in the expected orientation...

[jameswestgate]

I just want to say this is an awesome pull request but it seemed to break Autofocus for me and I couldn’t get qr code scanning to work at all on Pixel 5 running Android Q. Other scenarios worked well though.

[Denrage]

I just want to say this is an awesome pull request but it seemed to break Autofocus for me and I couldn’t get qr code scanning to work at all on Pixel 5 running Android Q. Other scenarios worked well though.

At least in the sample autofocus is working perfectly on my Huawei Mate 10 Pro. The Touch event to force a focus is also working. One caveat: the touch event can only be handled if there is no overlay bc currently the overlay is not propagating the touch event down to the surface view.

Need to test it in simulator for Pixel 5, rlly have no idea why it wouldn't work on that

[Denrage]

This is great! thanks for the work put into this! I'm going to get an update out first that fixes some issues and still allows for < android api 21 support and then will merge this after.

The only issue I'm seeing so far on this PR is that the orientation seems off 90 degrees... The sample which doesn't auto rotate doesn't find 1D barcodes unless i rotate the device, so perhaps the buffer needs a quick rotate to be in the expected orientation...

Do you have an example barcode which is not working? Bc for me it's working however the bar code is orientated or the phone is orientated. I chose a random barcode from the internet for that test
https://cdn.discordapp.com/attachments/396196956830957571/801015315843776522/SVID_20210119_100809_1.mp4

[jameswestgate]

Need to test it in simulator for Pixel 5, rlly have no idea why it wouldn't work on that

Thank you for responding! I will put together a minimum example for you in the next few days, to validate that it's not something else.

[jameswestgate]

I ran the camera2 branch unmodified on my physical Pixel5 and it did not scan a QR code. I then got my Galaxy s9 and it did scan. The s9 is running Android 9. However, the latest master from @Redth works fine on Pixel 5.

I will debug and see if I can find out what's happening.

[Denrage]

I will debug and see if I can find out what's happening.
That would be great!

The only thing i can imagine is that the pixel has an android which is too old for camera2. Please tell me any finding you will have there

[jameswestgate]

This is the latest Pixel5 running Android 11.
I have looked into this further, specifically CameraAnalyser.DecodeFrame

On @Redth 's master branch (has some updated commits), the frame is being decoded as follows:

Decode Time: 80.6448 ms (width: 800, height: 600, degrees: 90, rotate: True)

On your branch

Decode Time: 92.926 ms (width: 352, height: 288)

There is a difference in how the width and height for decoding are calculated in these two codebases, and I think this is where the problem lies. Can I suggest maybe you open Issues on your branch of the code and we can collaborate further there and resolve the issue outside of this pull request?

It could also be helpful to merge the latest commits from @Redth 's master into this branch?

All the best, James

[Redth]

I wonder if there's some dpi calculation missing?

[jameswestgate]

I'll look into how the two repos work these values out and see if that has something to do with it

[jameswestgate]

The reason for the width and height is because this is calculated to be the best aspect ratio although they are quite small values. I am tryng to see how this relates to the data provided to the barcodeReader as well as PlanarYUVLuminanceSource which seems to require the width and height of the data.

[jameswestgate]

Well, this was quite a rabbit hole to go down, but I have this working on my Pixel 5.

I adjusted the algorithm to return the same size as in the master branch (by excluding smaller sizes), I looked at the ImageReader because no matter what I tried, the decoder did not want to find a QR code.

When outputting the yuv bytes to disk as a jpeg, it become apparent there was a yuv conversion issue on the Pixel5.
It turned out the code in CameraEventsListener.ImageToByteArray may not always work correctly, and I replaced it with code converted from this answer:

https://stackoverflow.com/questions/52726002/camera2-captured-picture-conversion-from-yuv-420-888-to-nv21.

Now I know what the issue is, I will see if there is a better way of getting a LuminanceSource from the preview image, if not I will optimise and provide the code as a pull request to the camera2 branch.

[jameswestgate]

At risk of pull request inception, see Denrage#1 for changes

[Denrage]

So the main problem here was not the difference in size but the conversion of the YUV to Nv21 on the Pixel5? I still need to implement the forwarding of a desired size in regard of the options used (as you mentioned in an earlier comment). I want to merge master into the branch anyway in the end but will do that ofc too

[jameswestgate]

I actually left the size in the end, because it was working fine and the performance was really good as a result. I think in the master branch when selecting the most appropriate size for an aspect ratio, it skips very small sizes because it is for the preview.

I want to merge master into the branch anyway in the end but will do that ofc too

The nuget package dependancy on FastAndroidCamera as well as FastJavaArrayEx.android.cs and FastJavaByteArrayYUVLuminanceSource.android.cs can also be removed

[Denrage]

• Merged YUV changes of jameswestgate
• rotated YUV output (saw it was 90° off in the jpeg debug)
• merged master and added the changes from master in the CameraAnalyzer
• Removed dependencies to the FastAndroidCamera

[DarkIrata]

Long story short, a customer found some problems scanning bar codes. I did improve it a little bit and changed a few more lines to make it a little bit faster (also for qr codes) but there is still some improvements to make which i will try to implement. Whil i don't have much hope it will be merged anytime soon.. at least there is a reason now to wait.

Edit: Tests properly if you want to use it productive. The base functionallity worked fine for me in my tests, still..
Edit 2: I will be on vacation and after that will try to fix and improve it further. Mainy having problems with "code 39" Barcodes.

[DarkIrata]

From this point i can't do much more. I need a few people to test on different devices. Especially tablets. Current state is:

• Reduced Resolutions from decode frames (Better results for barcodes, faster decode)
• Changed LuminanceSource to a Custom NV21 LuinanceSource for decoder with a proper rotation function
• Frames now get (if AutoRotate is false) automaticlly rotated based on Orientation and Sensor Rotation on initial as close as possible to what you see
• Added a Debugging Helper so you can kind of see what the decoder see
• Focus is not based on continous video initially

Thing which could be improved are:

• (Theory) 2 ImageReader. One for the low resolution and one for a higher resolution. I don't know if it would be possible by OS restrictions, but in theory this would allow lower end devices get a proper quality picture. Since i don't have many devices to test, i don't know what the quality right now is
• Cropping, this would skyrock the decode time further.

Tested with:

• OnePlus 7 Pro (~30-150ms Decode time)
• Huawei M20 (~10-130ms Decode time)