fix: use pandas function to check for NaN

[LinuxChristian]

Starting with pandas 1.0, an experimental pandas.NA value (singleton) is available to represent scalar missing values as
opposed to numpy.nan. Comparing the variable with itself results in a pandas.NA value that doesn't support type-casting
to boolean. Using the build-in pandas.isna function handles all pandas supported NaN values (None, pd.NA, np.nan, NaT).

Thank you for opening a Pull Request! Before submitting your PR, there are a few things you can do to make sure it goes smoothly:

• Make sure to open an issue as a bug/issue before writing your code! That way we can discuss the change, evaluate designs, and agree on the general idea
• Ensure the tests and linter pass
• Code coverage does not decrease (if any source code was changed)
• Appropriate docs were updated (if necessary)

Fixes #729

[plamut]

Thanks for the fix and the test, it generally looks good! Just made two improvement suggestions.

[plamut]

FWIW, the prerelease-deps failure is flakiness, while Python 3.6 unit tests failure is legitimate - we use the oldest supported depenency versions there, and the new test failed with an attribute error:

 AttributeError: module 'pandas' has no attribute 'NA'

[LinuxChristian]

I added one commit per comment so it's easier to review. If you would like me to squash a final version let me know.

[plamut]

Neat to also include the reason why a minimum version is needed. 👍

[plamut]

LGTM, thanks for the quick update!

Nothing to worry about squashing the commits, we do that when the PR is merged.

[plamut]

It appears that the test was not skipped?

    def test_dataframe_to_json_generator(module_under_test):
        utcnow = datetime.datetime.utcnow()
        df_data = collections.OrderedDict(
            [
>               ("a_series", [pandas.NA, 2, 3, 4]),
                ("b_series", [0.1, float("NaN"), 0.3, 0.4]),
                ("c_series", ["a", "b", pandas.NA, "d"]),
                ("d_series", [utcnow, utcnow, utcnow, pandas.NaT]),
                ("e_series", [True, False, True, None]),
            ]
        )
E       AttributeError: 'NoneType' object has no attribute 'NA'

Also reproducible locally.

Edit: Ah, typo in the decorator name, I actually remember seeing that before. Fixed it myself, should be fine now.