[BUG] SalesForce integration problem with datatypes

[diatco]

Mage version

v0.9.68

Describe the bug

1. When creating a integration pipeline between S3 and SalesForce the upload to SalesForce fails due to datatypes of certain columns being different in Mage and in SalesForce.
   More precisely BillingAddress has a string/object dtype in Mage but Address dtype in SF and pipeline fails if data contains this column. Also YearStarted has string/object dtype in Mage, but Text(4) in SF and pipeline also fails if data contains this column.
   Another SF dtype that is incompatible is Currency(18, 0).

raise Exception(message)

Exception: 2024-05-07T07:57:04 Salesforce process failed with error The incomming schema is incompatable with your Account object. {"error": "The incomming schema is incompatable with your Account object", "errors": ["  File \"/usr/local/lib/python3.10/site-packages/mage_integrations/destinations/salesforce/__init__.py\", line 37, in <module>

    destination.process(sys.stdin.buffer)

", "  File \"/usr/local/lib/python3.10/site-packages/mage_integrations/destinations/base.py\", line 364, in process

    errors=traceback.format_stack(),

"], "message": "Traceback (most recent call last):

  File \"/usr/local/lib/python3.10/site-packages/mage_integrations/destinations/salesforce/target_salesforce/sinks.py\", line 74, in _validate_schema_against_object

    validate_schema_field(

  File \"/usr/local/lib/python3.10/site-packages/mage_integrations/destinations/salesforce/target_salesforce/utils/validation.py\", line 41, in validate_schema_field

    raise InvalidStreamSchema(

mage_integrations.destinations.salesforce.target_salesforce.utils.exceptions.InvalidStreamSchema: BillingAddress is not updatable for your Account Object, invalid for upsert action

2. Disable column type check doesn't work. I have toggled it on and I still get the error.

3. I have 2 columns in the dataset (country and city). These columns are missing in the Account object in SF. I turned on Automatically add new fields option but I receive an error when running the pipeline

raise Exception(message)

Exception: 2024-05-07T08:57:45 Salesforce process failed with error The incomming schema is incompatable with your Account object. {"error": "The incomming schema is incompatable with your Account object", "errors": ["  File \"/usr/local/lib/python3.10/site-packages/mage_integrations/destinations/salesforce/__init__.py\", line 37, in <module>

    destination.process(sys.stdin.buffer)

", "  File \"/usr/local/lib/python3.10/site-packages/mage_integrations/destinations/base.py\", line 364, in process

    errors=traceback.format_stack(),

"], "message": "Traceback (most recent call last):

  File \"/usr/local/lib/python3.10/site-packages/mage_integrations/destinations/salesforce/target_salesforce/sinks.py\", line 74, in _validate_schema_against_object

    validate_schema_field(

  File \"/usr/local/lib/python3.10/site-packages/mage_integrations/destinations/salesforce/target_salesforce/utils/validation.py\", line 37, in validate_schema_field

    raise InvalidStreamSchema(f\"{field_name} does not exist in your {stream_name} Object\")

mage_integrations.destinations.salesforce.target_salesforce.utils.exceptions.InvalidStreamSchema: Country does not exist in your Account Object

These are the fields in my Account Object

To reproduce

This is the dataset to reproduce the problemAccount_Lead.csv
I import it from S3 but I guess it doesn't matter what is the source.

these are the Select Stream options:

Select all columns except _s3_last_modified

For destination select SalesForce and activate Automatically add new fields and Disable column type check

Expected behavior

Expected to upload the data, and add the columns that are missing.

Screenshots

No response

Operating system

-OS: Windows 11
-Browser: Brave Version 1.65.114 Chromium: 124.0.6367.60

Additional context

No response

[diatco]

Upsert option in the yaml config also doesn't work, It duplicates my records.