Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] Format concats lines when reading from multiple streams to one file #934

Open
1 task done
krystian-wolarek opened this issue Apr 5, 2024 · 0 comments
Open
1 task done
Assignees
Labels

Comments

@krystian-wolarek
Copy link

krystian-wolarek commented Apr 5, 2024

Describe the bug
The issue occurs when parsing from multiple streams and writing to a single stream.
The first line from the second stream is appended to some line from the first stream but should be put in a new line.
Sometimes it's appended in a new line but mostly not.
When I use csv-stringify to format stream then it works well, everything is in a new line
I read streams simultaneously then problem occurs, when I do it one by one then it works

Parsing or Formatting?

  • Formatting

To Reproduce
Full code in the sandbox. Code is in src/my-processor.js
https://codesandbox.io/p/sandbox/streams-merge-ldw56f

To demonstrate the issue I've created in /temp-files two csv-s 'file1.csv' and file2.csv.
Formatting is done in two methods transformStreamFastCSV- problematic and transformStreamAndFormatStringify - working.
Results are merged into two files temp-files/merge-fast-csv.csv and temp-files/merge-stringify.csv.

Short version (more like pseudocode):

const filesToDownload = streams.length;
const writer = fs.createWriteStream('...');

formatToCSVStream() {
    const options = isFirstResponse
        ? { headers: true, includeEndRowDelimiter: true }
        : { headers: true, writeHeaders: false, includeEndRowDelimiter: true };
    return csv.format(options);
};

fastCSVParse(index) {
    return csv
        .parse({ headers: true })
        .transform((data) => {...data, file: 'file-'+index})
        .on("error", (error) => {
            console.error(error);
        });
}
let streamsEndCounter = 0;
streams.forEach((stream, index) => {
    stream
        .pipe(fastCSVParse(index))
        .pipe(formatToCSVStream())
        .on("end", () => {
            streamsEndCounter++;

            if (streamsEndCounter === filesToDownload) {
              writer.end();
            }
        })
        .pipe(writer, { end: false });

    isFirstResponse = false;
});

Expected behavior
Fast csv creates 1 row that has no newline and record 100,Riannon... is mostly appended to some line:
199,Britte,Stacy,Britte.Stacy@yopmail.com,dummyBritte.Stacy@gmail.com,worker,file-0100,Riannon,Gino,Riannon.Gino@yopmail.com,dummyRiannon.Gino@gmail.com,doctor,file-1
but should be placed in new line like result from csv-stringify
199,Britte,Stacy,Britte.Stacy@yopmail.com,dummyBritte.Stacy@gmail.com,worker,file-0 100,Riannon,Gino,Riannon.Gino@yopmail.com,dummyRiannon.Gino@gmail.com,doctor,file-1

Screenshots
Zrzut ekranu 2024-04-5 o 12 55 15

Desktop (please complete the following information):

  • Node Version 18.19.0 and in codesandbox 20.something
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

3 participants