Welcome to Synth Discussions!

[christos-h]

👋 Welcome!

We’re using Discussions as a place to connect with other members of our community. We hope that you:

• Ask questions you’re wondering about 🤔
• Share ideas. 💡
• Talk about larger projects, roadmap items and RFCs. 👍

To get started, comment below with an introduction of yourself and tell us about what you do with this community.

[sharisiri]

Hi!

First, hats off to a great project. Really impressed with the amount of data one can generate in under 30 seconds.

I'm new to Rust and would love to use this repo as a starting point. Is there a way to bypass the RAM constraints when generating very large CSV datasets? Ideally I would like to generate around 2b rows and then save the output to a folder of partitioned CSVs.

Thanks in advance and have a great week.

[iamwacko]

Hi!

First, hats off to a great project. Really impressed with the amount of data one can generate in under 30 seconds.

I'm new to Rust and would love to use this repo as a starting point. Is there a way to bypass the RAM constraints when generating very large CSV datasets? Ideally I would like to generate around 2b rows and then save the output to a folder of partitioned CSVs.

Thanks in advance and have a great week.

Make a large swap file. Or is there some other kind of limit you are talking about?

[sharisiri]

@iamwacko Thanks for the quick response. Reading up on swap files and will give it a shot. I am running Synth in a dev container locally with ~8gb of ram. The program crashes when I generate more data than can be fit into available RAM.