Refactor cleancalaccessrawfile to use lower-level command-line tools or other methods that are faster than the current slow Python loop through each file

[palewire]

No description provided.

[martinburch]

Could you point me to the slow loop that's involved here? Is it clean() being invoked at https://github.com/california-civic-data-coalition/django-calaccess-raw-data/blob/master/calaccess_raw/management/commands/cleancalaccessrawfile.py#L75 ?

[gordonje]

Line 126 in cleancalaccessrawfile.py is where we start looping over each line in a given .tsv file.

Line 285 in updatecalaccessrawdata.py is where we loop over each .tsv file, calling the above management command.

Either of these what you are looking for @martinburch?

[martinburch]

Yes, I think the line iteration in cleancalaccessrawfile.py is where we can try applying the multiprocessing imap as discussed in slack.