Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix: use unicode error symbol � instead of failing to import data #2296

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

spodlesny
Copy link

Encoding detection is not a silver bullet, and some files can be broken inside, which ends up failing to import the whole batch of files (due to one broken file). For some tasks (sentiment detection), missing character is worth tradeoff. For other use-cases, at least the issue with encoding is presented to user in standardized way and on front-end - without looking into code, like I had to.

Replacement of null character (\x00) for � is required to successfully save data into PostgreSQL.

@sweep-ai
Copy link

sweep-ai bot commented Oct 26, 2023

Apply Sweep Rules to your PR?

  • Apply: Leftover TODOs in the code should be handled.
  • Apply: All new business logic should have corresponding unit tests in the tests/ directory.
  • Apply: Any clearly inefficient or repeated code should be optimized or refactored.

@spodlesny
Copy link
Author

Let me know if I should open and new Issue in bug tracker first. I had to fix it for my use-case, so I wanted to share my fix with community in case other have the same issue.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant