Import Camera config error: content too large

[clrozeboom]

Describe the bug
We calibrated one of our cameras using 100+ images and ended up with a 63MB file when we exported the settings. We'd like to be able to restore this calibration if needed, but we get an error saying the Content is too large.

Wondering if there is a way to calibrate this larger, by hand, or if it can be updated in a hotfix?
Are there alternative ways to upload the data that we could employ?
Thanks!

To Reproduce
You can see the file we're importing and the log here: https://github.com/FRC5010/FRCLibrary/tree/Crescendo2024/PhotonVision
This is a pretty standard Arducam Global Shutter camera.

Platform:

• Beelink N5095
• Arducam global shutter camera
• PhotonVision Version: v2024.2.9
• Browser: Edge
• Camera(s) Used: Arducam 120fps Mono Global Shutter USB Camera, 720P OV9281
• https://a.co/d/hMY4ipE

Additional context

Mar 11 21:55:43 photonvision java[714]: [JettyServerThreadPool-41] WARN io.javalin.Javalin - Body greater than max size (50000000 bytes)
Mar 11 21:55:43 photonvision java[714]: [2024-03-11 21:55:43] [WebServer - RequestHandler] [ERROR] An error occurred while uploading calibration data: Content Too Large
Mar 11 21:55:43 photonvision java[714]: [2024-03-11 21:55:43] [WebServer - RequestHandler] [ERROR] io.javalin.http.HttpResponseException: Content Too Large
Mar 11 21:55:43 photonvision java[714]:         at io.javalin.http.servlet.JavalinServletContextKt.throwContentTooLargeIfContentTooLarge(JavalinServletContext.kt:178)
Mar 11 21:55:43 photonvision java[714]:         at io.javalin.http.Context.bodyAsBytes(Context.kt:126)
Mar 11 21:55:43 photonvision java[714]:         at io.javalin.http.servlet.JavalinServletContext.access$bodyAsBytes$s-1678783089(JavalinServletContext.kt:34)
Mar 11 21:55:43 photonvision java[714]:         at io.javalin.http.servlet.JavalinServletContext$body$2.invoke(JavalinServletContext.kt:97)
Mar 11 21:55:43 photonvision java[714]:         at io.javalin.http.servlet.JavalinServletContext$body$2.invoke(JavalinServletContext.kt:97)
Mar 11 21:55:43 photonvision java[714]:         at kotlin.SynchronizedLazyImpl.getValue(LazyJVM.kt:74)
Mar 11 21:55:43 photonvision java[714]:         at io.javalin.http.servlet.JavalinServletContext.getBody(JavalinServletContext.kt:97)
Mar 11 21:55:43 photonvision java[714]:         at io.javalin.http.servlet.JavalinServletContext.bodyAsBytes(JavalinServletContext.kt:98)
Mar 11 21:55:43 photonvision java[714]:         at io.javalin.http.Context.body(Context.kt:117)
Mar 11 21:55:43 photonvision java[714]:         at org.photonvision.server.RequestHandler.onDataCalibrationImportRequest(RequestHandler.java:518)
Mar 11 21:55:43 photonvision java[714]:         at io.javalin.routing.HandlerEntry.handle(HandlerEntry.kt:19)
Mar 11 21:55:43 photonvision java[714]:         at io.javalin.http.servlet.DefaultTasks.HTTP$lambda$8$lambda$6$lambda$5(DefaultTasks.kt:39)
Mar 11 21:55:43 photonvision java[714]:         at io.javalin.http.servlet.JavalinServlet.handleTask(JavalinServlet.kt:88)
Mar 11 21:55:43 photonvision java[714]:         at io.javalin.http.servlet.JavalinServlet.handleSync(JavalinServlet.kt:53)
Mar 11 21:55:43 photonvision java[714]:         at io.javalin.http.servlet.JavalinServlet.service(JavalinServlet.kt:41)
Mar 11 21:55:43 photonvision java[714]:         at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587)
Mar 11 21:55:43 photonvision java[714]:         at io.javalin.jetty.JavalinJettyServlet.service(JavalinJettyServlet.kt:58)
Mar 11 21:55:43 photonvision java[714]:         at jakarta.servlet.http.HttpServlet.service(HttpServlet.java:587)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:529)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
Mar 11 21:55:43 photonvision java[714]:         at io.javalin.jetty.JettyServer$start$wsAndHttpHandler$1.doHandle(JettyServer.kt:61)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1303)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:173)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.server.Server.handle(Server.java:563)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.server.HttpChannel.lambda$handle$0(HttpChannel.java:505)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:762)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:497)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:282)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:416)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:385)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:272)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.lambda$new$0(AdaptiveExecutionStrategy.java:140)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
Mar 11 21:55:43 photonvision java[714]:         at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
Mar 11 21:55:43 photonvision java[714]:         at java.base/java.lang.Thread.run(Thread.java:829)
Mar 11 21:55:43 photonvision java[714]: 
Mar 11 21:55:43 photonvision java[714]: [2024-03-11 21:55:43] [WebServer - Server] [DEBUG] Handled HTTP request of type POST from endpoint /api/calibration/importFromData for host 10.50.10.194 in 16.426361 ms

[mcm001]

Yeah, there's a 50MB upload limit set in Server.java. You could increase this limit by changing the code, or you can strip images from the calibration JSON using #1244

[clrozeboom]

That looks promising, is that implemented? If so, could you point to documentation or explain how to use it? Thanks!
Edit - is it the python extraction process talked about here? https://docs.photonvision.org/en/latest/docs/calibration/calibration.html#accessing-calibration-images

[5010TigerDynasty]

Hey, when trying to use the calibrationUtils.py, it requires a package for mrcal. How can that be installed?

[clrozeboom]

This is as far as I was able to get with the calibrationUtils.py utility after installing it on my photonvision server.

python3 calibrationUtils.py photon_calibration_Arducam_OV9281_USB_Camera_1280x720.json camera_cal/
Traceback (most recent call last):
File "/home/pv/calibrationUtils.py", line 255, in
main()
File "/home/pv/calibrationUtils.py", line 251, in main
convert_photon_to_mrcal(args.input, args.output_folder)
File "/home/pv/calibrationUtils.py", line 228, in convert_photon_to_mrcal
mrcal_model = __convert_cal_to_mrcal_cameramodel(camera_cal_data)
File "/home/pv/calibrationUtils.py", line 135, in __convert_cal_to_mrcal_cameramodel
[
File "/home/pv/calibrationUtils.py", line 139, in
).reshape((cal.calobjectSize.width, cal.calobjectSize.height, 3))
TypeError: 'float' object cannot be interpreted as an integer

[clrozeboom]

I think we are going to redo our calibration with only 60+ images to cut down the size of the export. Being able to have a restorable calibration for this level of effort in calibrating is crucial.

[mcm001]

You can also go in and replace the observations array in the json with an empty list. I would accept a PR increasing the max upload size as well.

And yeah looks like a bug in the script in the PR you posted above. Should be super easy to just cast to int?

[mcm001]

You could also patch the frontend to clear observation snapshot data if the file size is above 50Mb, that's super easy too