-
Notifications
You must be signed in to change notification settings - Fork 663
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix bulk2 upload and download utf-8 encoding errors. #688
Merged
jon-wobken
merged 1 commit into
simple-salesforce:release/v1.12.6
from
calbanese11:fix-bulk2-utf8-encoding
Oct 24, 2023
Merged
fix bulk2 upload and download utf-8 encoding errors. #688
jon-wobken
merged 1 commit into
simple-salesforce:release/v1.12.6
from
calbanese11:fix-bulk2-utf8-encoding
Oct 24, 2023
+4
−4
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
calbanese11
changed the title
fix bulk2 upload and download utf-8 encoding errors.
fix bulk2 upload and download utf-8 encoding errors. Fixes #687
Oct 9, 2023
calbanese11
changed the title
fix bulk2 upload and download utf-8 encoding errors. Fixes #687
fix bulk2 upload and download utf-8 encoding errors.
Oct 10, 2023
jon-wobken
added a commit
that referenced
this pull request
Feb 26, 2024
* fix bulk2 upload and download utf-8 encoding errors. (#688) Co-authored-by: codyalbanese <calbanese@appannie.com> * Pass the column datatype along with current value to pd.Series (#692) Prevents pd.Series from issuing a FutureWarning when value is empty and it is unable to infer a data type. gh-691 * added max retries to recursive refresh token functionality (#685) * remove pendulum depeendency and use native datetime * adding query_all function to fetch soft deleted records in sf (#697) Co-authored-by: Prajwal Pokharel <prajwalpokharel@ip-192-168-4-22.ec2.internal> * Handle non-XML responses during error conditions (#695) ``` File "/home/web/project/env/lib/python3.10/site-packages/simple_salesforce/api.py", line 146, in __init__ self._refresh_session() File "/home/web/project/env/lib/python3.10/site-packages/simple_salesforce/api.py", line 283, in _refresh_session self.session_id, self.sf_instance = self._salesforce_login_partial() File "/home/web/project/env/lib/python3.10/site-packages/simple_salesforce/login.py", line 221, in SalesforceLogin return soap_login(soap_url, login_soap_request_body, File "/home/web/project/env/lib/python3.10/site-packages/simple_salesforce/login.py", line 231, in soap_login except_code = getUniqueElementValueFromXmlString( File "/home/web/project/env/lib/python3.10/site-packages/simple_salesforce/util.py", line 21, in getUniqueElementValueFromXmlString xmlStringAsDom = xml.dom.minidom.parseString(xmlString) File "/usr/lib/python3.10/xml/dom/minidom.py", line 2000, in parseString return expatbuilder.parseString(string) File "/usr/lib/python3.10/xml/dom/expatbuilder.py", line 925, in parseString return builder.parseString(string) File "/usr/lib/python3.10/xml/dom/expatbuilder.py", line 223, in parseString parser.Parse(string, True) xml.parsers.expat.ExpatError: syntax error: line 1, column 0 ``` Where `string` == "upstream connect error or disconnect/reset before headers. reset reason: connection termination" * Adding utf-8 encoding when performing bulk2 data uploads. (#683) * add functionality to return detailed results from batch request/result * simplify worker variable from batch_results to result * Update bulk2.py * pass instance_url to login (#711) * Require cryptography as optional dependency of pyjwt (#680) * Update .travis.yml - remove support for 3.6 and 3.7 Remove support for python 3.6 and 3.7 * Update README.rst Remove python 3.6 and 3.7 Add python 3.12 * Add typing annotations (#660) * Add type annotations for util.py * Add type annotations for metadata.py * Fix up typing issues in metadata.py * Add type annotations for login.py * Fix up typing issues in login.py and add errorhandling * Add type annotations for format.py * Add type annotations for exceptions.py * Fix up typing issues in login.py * Add type annotations for bulk.py * Fix up typing issues in bulk.py * Add type annotations for api.py * Fix up typing issues in api.py * Fix up unittests * Use MutableMapping instead of dict * Fix formatting (long lines) * Fix up wrong typing * Fix up typing issues * Fixup long lines * Fixup typing issues in login.py * Add mypy type checking to tox testenv:static script * Use old type annotation syntax since python 3.6 doesn't support __future__ annotations * Use old type annotation syntax * Add py.typed to create a PEP 561 compatible package * Add type alias for headers * Fix unused import * Add type alias for proxies * Add TypeVar T for generic functions * Add types for BulkDataAny and BulkDataStr * Use standard HTTPMethod StrEnum * Use standard HTTPMethod StrEnum * Revert "Use standard HTTPMethod StrEnum" This reverts commit b99d34a. * Revert "Use standard HTTPMethod StrEnum" This reverts commit d0d1d6e. * Make Usage and PerAppUsage more type safe * Fix Missing class docstring * Add typing to bulk2 api * Finish typing of bulk2 * Fixup * Add future import and fixup DateTime annotation * Maybe fix imports * Move TypedDict import to typing_extensions * Add dependency so also python 3.6 works * Remove from future annotations * Use older syntax * Adjust pipeline to use py.test instead of setup.py test (since it is deprecated) * Fixup test dependencies and use pytest instead of setup.py test * Fixup python 3.6 error * Fixup python 3.6 error * Fix formatting and add pip cache * Fix typing issues --------- Co-authored-by: JonWobken <49450080+jon-wobken@users.noreply.github.com> * Update api.py Fix formatting Update default api version * Update bulk.py Fix code format Remove unused "operation" in _get_batch_request_with_batch_results * Update bulk2.py fix code formatting * Update login.py line to long fix * Update api.py * Fix CI for release/v1.12.6 (#713) * Fix pipeline * Fix pipeline * Include "fields" as a query param in GET requests (#705) * Include fields as a query param in get requests * fix line length from CI build * Refactor to pass **kwargs Simplifies use to **kwargs so fields can be passed as "params='fields=field1,field2,field3'" * Update api.py * Update api.py --------- Co-authored-by: JonWobken <49450080+jon-wobken@users.noreply.github.com> * Update bulk.py added detail result argument * Update setup.py * Update bulk.py * Update test_requirements.txt * Update test_requirements.txt * Update .travis.yml * Update .travis.yml * Update .travis.yml * Update CHANGES --------- Co-authored-by: Cody Albanese <43307196+calbanese11@users.noreply.github.com> Co-authored-by: codyalbanese <calbanese@appannie.com> Co-authored-by: Chris Caldwell <chrisolof@gmail.com> Co-authored-by: Rob O <140531024+rob-avestruz-kp@users.noreply.github.com> Co-authored-by: prazwalp007 <prazwalp007@gmail.com> Co-authored-by: Prajwal Pokharel <prajwalpokharel@ip-192-168-4-22.ec2.internal> Co-authored-by: John-Scott Atlakson <24574+jsma@users.noreply.github.com> Co-authored-by: Saqib Rokadia <saqib@saqibr.com> Co-authored-by: jshadle-vectr <132278887+jshadle-vectr@users.noreply.github.com> Co-authored-by: Giorgos Drosos <56369797+gdrosos@users.noreply.github.com> Co-authored-by: Felix Schnabel <fschnabel007@gmail.com> Co-authored-by: framarzar <86966421+framarzar@users.noreply.github.com>
jon-wobken
added a commit
that referenced
this pull request
Mar 26, 2024
* fix bulk2 upload and download utf-8 encoding errors. (#688) Co-authored-by: codyalbanese <calbanese@appannie.com> * Pass the column datatype along with current value to pd.Series (#692) Prevents pd.Series from issuing a FutureWarning when value is empty and it is unable to infer a data type. gh-691 * added max retries to recursive refresh token functionality (#685) * remove pendulum depeendency and use native datetime * adding query_all function to fetch soft deleted records in sf (#697) Co-authored-by: Prajwal Pokharel <prajwalpokharel@ip-192-168-4-22.ec2.internal> * Handle non-XML responses during error conditions (#695) ``` File "/home/web/project/env/lib/python3.10/site-packages/simple_salesforce/api.py", line 146, in __init__ self._refresh_session() File "/home/web/project/env/lib/python3.10/site-packages/simple_salesforce/api.py", line 283, in _refresh_session self.session_id, self.sf_instance = self._salesforce_login_partial() File "/home/web/project/env/lib/python3.10/site-packages/simple_salesforce/login.py", line 221, in SalesforceLogin return soap_login(soap_url, login_soap_request_body, File "/home/web/project/env/lib/python3.10/site-packages/simple_salesforce/login.py", line 231, in soap_login except_code = getUniqueElementValueFromXmlString( File "/home/web/project/env/lib/python3.10/site-packages/simple_salesforce/util.py", line 21, in getUniqueElementValueFromXmlString xmlStringAsDom = xml.dom.minidom.parseString(xmlString) File "/usr/lib/python3.10/xml/dom/minidom.py", line 2000, in parseString return expatbuilder.parseString(string) File "/usr/lib/python3.10/xml/dom/expatbuilder.py", line 925, in parseString return builder.parseString(string) File "/usr/lib/python3.10/xml/dom/expatbuilder.py", line 223, in parseString parser.Parse(string, True) xml.parsers.expat.ExpatError: syntax error: line 1, column 0 ``` Where `string` == "upstream connect error or disconnect/reset before headers. reset reason: connection termination" * Adding utf-8 encoding when performing bulk2 data uploads. (#683) * add functionality to return detailed results from batch request/result * simplify worker variable from batch_results to result * Update bulk2.py * pass instance_url to login (#711) * Require cryptography as optional dependency of pyjwt (#680) * Update .travis.yml - remove support for 3.6 and 3.7 Remove support for python 3.6 and 3.7 * Update README.rst Remove python 3.6 and 3.7 Add python 3.12 * Add typing annotations (#660) * Add type annotations for util.py * Add type annotations for metadata.py * Fix up typing issues in metadata.py * Add type annotations for login.py * Fix up typing issues in login.py and add errorhandling * Add type annotations for format.py * Add type annotations for exceptions.py * Fix up typing issues in login.py * Add type annotations for bulk.py * Fix up typing issues in bulk.py * Add type annotations for api.py * Fix up typing issues in api.py * Fix up unittests * Use MutableMapping instead of dict * Fix formatting (long lines) * Fix up wrong typing * Fix up typing issues * Fixup long lines * Fixup typing issues in login.py * Add mypy type checking to tox testenv:static script * Use old type annotation syntax since python 3.6 doesn't support __future__ annotations * Use old type annotation syntax * Add py.typed to create a PEP 561 compatible package * Add type alias for headers * Fix unused import * Add type alias for proxies * Add TypeVar T for generic functions * Add types for BulkDataAny and BulkDataStr * Use standard HTTPMethod StrEnum * Use standard HTTPMethod StrEnum * Revert "Use standard HTTPMethod StrEnum" This reverts commit b99d34a. * Revert "Use standard HTTPMethod StrEnum" This reverts commit d0d1d6e. * Make Usage and PerAppUsage more type safe * Fix Missing class docstring * Add typing to bulk2 api * Finish typing of bulk2 * Fixup * Add future import and fixup DateTime annotation * Maybe fix imports * Move TypedDict import to typing_extensions * Add dependency so also python 3.6 works * Remove from future annotations * Use older syntax * Adjust pipeline to use py.test instead of setup.py test (since it is deprecated) * Fixup test dependencies and use pytest instead of setup.py test * Fixup python 3.6 error * Fixup python 3.6 error * Fix formatting and add pip cache * Fix typing issues --------- Co-authored-by: JonWobken <49450080+jon-wobken@users.noreply.github.com> * Update api.py Fix formatting Update default api version * Update bulk.py Fix code format Remove unused "operation" in _get_batch_request_with_batch_results * Update bulk2.py fix code formatting * Update login.py line to long fix * Update api.py * Fix CI for release/v1.12.6 (#713) * Fix pipeline * Fix pipeline * Include "fields" as a query param in GET requests (#705) * Include fields as a query param in get requests * fix line length from CI build * Refactor to pass **kwargs Simplifies use to **kwargs so fields can be passed as "params='fields=field1,field2,field3'" * Update api.py * Update api.py --------- Co-authored-by: JonWobken <49450080+jon-wobken@users.noreply.github.com> * Update bulk.py added detail result argument * Update setup.py * Update bulk.py * Update test_requirements.txt * Update test_requirements.txt * Update .travis.yml * Update .travis.yml * Update .travis.yml * Update CHANGES * Update __version__.py --------- Co-authored-by: Cody Albanese <43307196+calbanese11@users.noreply.github.com> Co-authored-by: codyalbanese <calbanese@appannie.com> Co-authored-by: Chris Caldwell <chrisolof@gmail.com> Co-authored-by: Rob O <140531024+rob-avestruz-kp@users.noreply.github.com> Co-authored-by: prazwalp007 <prazwalp007@gmail.com> Co-authored-by: Prajwal Pokharel <prajwalpokharel@ip-192-168-4-22.ec2.internal> Co-authored-by: John-Scott Atlakson <24574+jsma@users.noreply.github.com> Co-authored-by: Saqib Rokadia <saqib@saqibr.com> Co-authored-by: jshadle-vectr <132278887+jshadle-vectr@users.noreply.github.com> Co-authored-by: Giorgos Drosos <56369797+gdrosos@users.noreply.github.com> Co-authored-by: Felix Schnabel <fschnabel007@gmail.com> Co-authored-by: framarzar <86966421+framarzar@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Changes to correct the utf-8 encoding issues with the new Bulk2 implementation. Merging into Master since a version 1.12.6 branch has not yet been created.
Add charset=UTF-8 specification to the CSV_CONTENT_TYPE header to ensure proper request encoding/decoding across all Bulk2 Operations
When downloading results... Access results.content instead of results.text. Results.content byte stream is encoded as utf-8 and can then be properly decoded and returned as a string.
Fixes #687