Generating mvt for big geojson files

[tdyku]

Reading big files in node

I would like to generate mvt pyramids with your package for really big geojson file(2GB). It is not possible with fs.readFileSync, because of file size. How can I deal with this problem?
Is it possible to generate piramides for many geojson files? Or maybe I should use streams?

Is it use case for multiple layers? Should I cut geojson on many smaller?

[hampelm]

Hey Tom! Late reply, but for files that large I'd recommend using Tippecanoe to build vector tiles: https://github.com/mapbox/tippecanoe. It's got a lot of setting you can tweak to work with large datasets. You can then use https://github.com/mapbox/mbutil to get the tiles out as files.

If that doesn't work for your usecase, I'd suggest chunking your geojson into smaller collections

[tdyku]

Hi hampelm!
Thank you for response! I already figured it out, here's solution for people with same problem:

1. Install tippencanoe (https://github.com/mapbox/tippecanoe) and generate mbtiles:
   tippecanoe -o mbt_file.mbtiles -z18 input_file.geojson -pC

2. Install mb-util(https://github.com/mapbox/mbutil) and convert mbtiles to mvt tiles:
   ./mbutil/mb-util mbt_file.mbtiles mvt_dir --image_format=pbf

3. All files are now in proper format, but you have to change their extension:
   find mvt_dir -iname "*.pbf" -exec bash -c 'mv "$0" "${0%\.pbf}.mvt"' {} \;

Just be careful with -pC option, because not always you want to compress tiles.
Regards,
Tomasz

Issue can be closed

[rahafo]

hi @Tomkee,
I follow your steps and I successfully convert my .Mbtiles file to .mvt, thank you!
However, when I add it to my map it doesn't appear, do you have any idea why is that happen?
I didn't get the compressing tiles that you mentioned, so I think the problem is about it.
If you kindly explain it to me,
Thank you!

[tdyku]

Could you show browser log (F12 in chrome)? Try generating tiles both with -pC and without. Also maybe there's no point in renaming files to .mvt(omit step 3.)

[rahafo]

Thanks @Tomkee for your response.

Here is the log:

This log appear when I didn't use -pC, but when I use it, nothing appears in the map and the console doesn't have any errors!

[tdyku]

Is it possible to reproduce it somehow?

[rahafo]

I didn't get what do you mean by reproducing it, reproducing what?

[tdyku]

Can I find your tiles somewhere?

[rahafo]

the geojson looks like something like this:

{"type": "FeatureCollection", "features": [{"type": "Feature",
"properties": {p1": 20180401, "p2": 942, "p3": 966, "p4": 1467, "p5": 995, "p6": 2, "p7": 96, "p8": 5},
"geometry": {
"type": "LineString",
"coordinates": [
[-122.48369693756104, 37.83381888486939],
[-122.48348236083984, 37.83317489144141]] }
}]}

but with more than 2 million feature.