feat: refactor ingest

[rbiseck3]

Description

This refactors the current ingest CLI process to support better granularity in how the steps are ran

• Both multiprocessing and async now supported. Given that a lot of the steps are IO-bound, such as downloading and uploading content, we can achieve better parallelization by using async here
• Destination step broken up into a stager step and an upload step. This will allow for steps that require manipulation of the data between formats, such as converting the elements json into a csv format to upload for tabular destinations, to be pulled out of the step that does the actual upload.
• The process of writing the content to a local destination was now pulled out as it's own dedicated destination connector, meaning you no longer need to persist the content locally once the process is done if the content was uploaded elsewhere.
• Quick update to the chunker/partition step to use the python client.
• Move the uncompress suppport as a pipeline step since this can arbitrarily apply to any concrete files that have been downloaded, regardless of where they came from.
• Leverage last modified date to mark files to be reprocessed, even if the file already exists locally.

Callouts

Retry configs haven't been moved over yet. This is an open question because the intent was for it to wrap potential connection errors but now any of the other steps that leverage an API might run into network connection issues. Should those be isolated in each of the steps and wrapped with the same retry configs? Or do we need to expose a unique retry config for each step? This would bloat the input params even more.

Testing

• If you want to run the new code as an SDK, there's an example file that was added to highlight how to do that: example.py
• If you want to run the new code as an isolated CLI:

PYTHONPATH=. python unstructured/ingest/v2/main.py --help

• If you want to see which commands have been migrated to the new version, there's now a v2 short help text next to those commands when running the current cli:

PYTHONPATH=. python unstructured/ingest/main.py --help
Usage: main.py [OPTIONS] COMMAND [ARGS]...main.py --help   

Options:
  --help  Show this message and exit.

Commands:
  airtable
  azure
  biomed
  box
  confluence
  delta-table
  discord
  dropbox
  elasticsearch
  fsspec
  gcs
  github
  gitlab
  google-drive
  hubspot
  jira
  local          v2
  mongodb
  notion
  onedrive
  opensearch
  outlook
  reddit
  s3             v2
  salesforce
  sftp
  sharepoint
  slack
  wikipedia

You can run any of the local or s3 specific ingest tests and these should now work.

[potter-potter]

This is really cool! And its also a lot to understand. Could you add notes on how to test it out in the description?

[ryannikolaidis]

are there doc changes that need to happen for users of the connectors via Python? guessing nothing breaks yet for them since this is only additive, but we'll just need to once we switch over

[ryannikolaidis]

got it, so we just defer to v2 for cli if there exists a v2 command

[rbiseck3]

Yep, at least right now as soon as the new connector exists, it replaces the old one.

[ryannikolaidis]

Yea, this is actually pretty nice because it means that nothing breaks as far as users know. Also means we can add deprecations warnings for the Python users for whom it will break once we switch over.

[ryannikolaidis]

Maybe worth doing that for the ones that you're bumping here? And probably bump the docs for the respective connectors?

[rbiseck3]

Where do we manage the docs for connectors now? I thought that changed recently and not sure if that's still being managed by this repo.

[ryannikolaidis]

good question! so, we now host the docs themselves in this repo: https://github.com/Unstructured-IO/docs

and it pulls source code snippets from here: https://github.com/Unstructured-IO/unstructured/tree/main/docs/source/ingest/source_connectors/code
and
https://github.com/Unstructured-IO/unstructured/tree/main/docs/source/ingest/destination_connectors/code

[ryannikolaidis]

Overall I think there are a lot of good changes in here! This does come at the cost of quite a bit of complexity (though to be fair some of this is coming back in a bit cold). My biggest concern is that it's going to be tough for an average person to walk in and debug or be effective here, and currently you have to do a lot of tracing to grok what's going on. My main ask: can we equivalently (in size and scope) push the documentation? Including a lot more comments and docstrings as well as a comprehensive breakdown of everything happening with maybe some diagrams as well?

[ryannikolaidis]

Probably should mention in the PR description the bug fix (which is actually what caused the PR's surface area to bloat a bunch)

Thanks for the additions so far. Just noting here from conversation, more extensive documentation (mentioned in comments) will be added before we switch over from original code and current flow. In the meantime this should be non-breaking for users.