Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Provide Stream output for transformers #1176

Open
snuyanzin opened this issue Apr 26, 2024 · 0 comments
Open

Provide Stream output for transformers #1176

snuyanzin opened this issue Apr 26, 2024 · 0 comments

Comments

@snuyanzin
Copy link
Collaborator

The problem with current method like net.datafaker.transformations.JsonTransformer#generate(net.datafaker.transformations.Schema<IN,?>, int)
it generates the whole String and then returns it. As a result for bigger numbers it consumes larger amount of memory and e.g.
such test fails with OutOfMemory

@Test
    void test2() {
        BaseFaker faker = new BaseFaker(new Random(10L));
        Schema<Object, ?> schema = Schema.of(
            field("Text", () -> faker.name().firstName()),
            field("Bool", () -> faker.name().lastName())
        );

        JsonTransformer<Object> transformer = JsonTransformer.builder().build();
        String json = transformer.generate(schema, 50_000_000);
        System.out.println(json);
    }

There is not so much we can do about this method since anyway with such approach we need somehow to store that giant string value.

Another approach is instead of generation the final string value we could generate a stream of values and return it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant