deckGL crash when loading data

[nomans1]

Hello,

Recently been having issues with deckGL which I think is related to performance. I have a huge data set that I stream using an express server and deckGL consumes. My problem is, the JSON file is 2GB (2.2m records) and Chrome tab just crashes with Error 5.

Although when I try to stream a 500mb JSON file (1.1m records) it performs fine. Can you let me know how I can display the larger JSON file without it crashing? I've tried async iterables and currently using fetchFile from loaders.gl to get the data.

Any help related to improving my code both streaming and deckGL is welcome

loading data in deckGL: https://imgur.com/78TrIET
Scatterplot layer: https://imgur.com/HaRsW0v
Express JSON stream: https://imgur.com/kxSIGFF

Thank you

[Pessimistress]

Chrome by default limits the memory used per tab to 1-2GB, depending on your system. There are workarounds (e.g. using a binary representation of your data) to save memory, but ultimately, you can't load and render an arbitrarily big GeoJSON all at once.

Since you have a backend you should consider splitting your data into tiles and use the TileLayer or MVTLayer which would only fetch the data that is visible in the current viewport.

[nomans1]

Thanks for your reply, correct me if I'm wrong but TileLayer can only bed used with GeoJson layer right? My use case is for heatmap and scatterplot layers. What would you recommend there? Is the best way to do it is to use binary representation of my data or is there anything else?

Question 2:

Another way of dealing with it is to send a subset of my data. For instance filtering by area. So I will only send data for a particular area using a REST API and query strings. How would I update the data object then? Because as I look at it, updateTriggers doesn't work for data object.

[Pessimistress]

TileLayer can only be used with GeoJsonLayer

No. See the code sample on
https://deck.gl/docs/api-reference/geo-layers/tile-layer

HeatmapLayer doesn't work across tile boundaries, but you can use the onViewportLoad callback of TileLayer to get all the points loaded.

updateTriggers doesn't work for data object

updateTriggers is for accessors. When you change the data prop all attributes are recalculated.

[Pessimistress]

If all you need is to show this 2GB JSON file then you can certainly use binary to reduce the memory footprint. See https://deck.gl/docs/developer-guide/performance#supply-attributes-directly