Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Retweeted text is split over several lines in CSV #2

Open
nicsuzor opened this issue Mar 16, 2023 · 2 comments
Open

Retweeted text is split over several lines in CSV #2

nicsuzor opened this issue Mar 16, 2023 · 2 comments
Assignees
@nicsuzor
Labels
bug Something isn't working

Comments

@nicsuzor
Copy link
Contributor

Very infrequently, a tweet will be split over several lines in the CSV prior to uploaded in GBQ. See, for example, tweet id: 1517074359571075072
@nicsuzor nicsuzor added the bug Something isn't working label Mar 16, 2023
@nicsuzor nicsuzor self-assigned this Mar 16, 2023
@nicsuzor
Copy link
Contributor Author

Here is a reproducible failing test: f8f8a5b

@nicsuzor
Copy link
Contributor Author 

self.obj = DataFrame(loads(json, precise_float=self.precise_float), dtype=None
E               UnicodeEncodeError: 'utf-8' codec can't encode characters in position 3579-3582: surrogates not allowed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@nicsuzor nicsuzor

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@nicsuzor