Reconcile Historical IncluCivics (Employee Demographic) Data

[combinatorist]

Impetus

@kazshak created a way to pull data for inclucivics from here in our own data portal (instead of directly from the city's portal). 🎉

#Advantages
The advantage is that we have more control of how we collect the data. We don't have to reload data in sequence, which also means we can make changes or fill in holes after the fact. For example, we were able to make a public records request to back fill a bunch data we missed. 🥇

Problem

However, this also meant that we discovered some discrepancies between what we downloaded at the time and what the city populated in the public records request later. Most are formatting differences, but some are content changes.
This is the commit where he pulled from the open data portal: code-for-nashville/inclucivics@f35c6d4.

Todo

We need to reconcile these differences. Of course, we can't know what's correct in a real content change (like someone's salary), but we can make a thoughtful guess about which is correct. We can also add missing data (e.g. counties isn't always there).

[combinatorist]

I described some specific differences in the PR linked above, but here's a list to just one file to give an idea of what we're talking about:

• input/20150419.csv renamed to input/20150401.csv
• column names changed slightly
• last column (county) dropped
• row order changed
• some salaries, titles, etc changed
• two rows removed

Meanwhile, in input/20151001.csv which wasn't renamed there were the following formatting changes:

• salary: was a quoted string with 1000s place comma and became a number (no trailing decimal 0s)
• date started: was M/D/YY (e.g. 1/1/13) and became MM/DD/YYYY (e.g. 01/01/2013)

There may be other differences in these and other files. We need to determine for each difference whether it is good or bad.