Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: Add clarifai as ingest destination connector #2569

Closed
wants to merge 9 commits into from
Closed

feat: Add clarifai as ingest destination connector #2569

wants to merge 9 commits into from

Conversation

mogith-pn
Copy link

This PR intends to add Clarifai as a ingest destination connector.

  • Access via CLI and programmatic.
  • Documentation and Examples.
  • Integration test script.

@mogith-pn
Clarifai : Added clarifai as destination connector (#2)
e58ff48 
* added initial intg codes

* Added test cases

* Delete oryx-build-commands.txt

* Delete testnotebook.ipynb

* Clarifai- Added examples in docs

* Clarifai-Modified element_dict parsing

* Clarifai: Modified clarifai.py

* Clarifai-modified test
@mogith-pn
Clarifai-Modified makefile
e805ed1
@mogith-pn
Merge branch 'main' into main
dc22aea
@mogith-pn
Copy link
Author

@potter-potter
Could you please take a look at this PR ?
TIA !

@potter-potter
Copy link
Contributor

@mogith-pn This looks very nice! I will definitely get to reviewing this but may take a bit. (we are heads-down on some other work.) Out of curiosity what is your relation to Clarifai?

@mogith-pn
Copy link
Author

@potter-potter ,
Thanks for your response.
I'm working as ML Ops engineer at Clarifai, actively driving integrations with external platforms. Feel free to reach out to me for any information.
Happy to contribute to Unstructured !

@potter-potter
Copy link
Contributor

@mogith-pn I'm reviewing the pr this week.

@potter-potter
Copy link
Contributor

potter-potter commented Mar 3, 2024 •
edited

I'm getting this when I run thetest_unstructured_ingest/dest/clarifai.sh
Can you give me some insight on why?
WARNING clarifai.client.input: code: INPUT_UNSUPPORTED_FORMAT description: "Input format unsupported" details: "Input type text is not supported by the workflow"

@mogith-pn
Copy link
Author

mogith-pn commented Mar 4, 2024 •
edited

I'm getting this when I run thetest_unstructured_ingest/dest/clarifai.sh Can you give me some insight on why? WARNING clarifai.client.input: code: INPUT_UNSUPPORTED_FORMAT description: "Input format unsupported" details: "Input type text is not supported by the workflow"

Hi @potter-potter,
Thanks for pointing this error, this is an expected behaviour using API endpoint. since our clarifai app by default uses an workflow which is used for image type inputs, so we need to explicitly define the type of default_workflow as "universal" so that it could accepts any type of inputs. I have modified the cmd block to accommodate this change.

potter-potter
potter-potter reviewed Mar 6, 2024
View reviewed changes
test_unstructured_ingest/dest/clarifai.sh Outdated

done

EXPECTED=8729
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's switch this to 8 and do a smaller document. (see above)

potter-potter
potter-potter reviewed Mar 6, 2024
View reviewed changes
test_unstructured_ingest/dest/clarifai.sh Outdated

PYTHONPATH=. ./unstructured/ingest/main.py \
local \
--input-path example-docs/book-war-and-peace-1225p.txt \
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Lets do example-docs/book-war-and-peace-1.txt

Also, can you make this file executable chmod 755...

@potter-potter
Copy link
Contributor

@mogith-pn Almost there. This is great! Super fun destination connector.

Once you make the changes above, I'll take over and get this merged into the repo. To do that I will essentially copy your files and make a new branch. Its the easiest way we have found to deal with CI/CD and tests, etc...

@mogith-pn
Copy link
Author

@potter-potter ,
Thanks for your review. I hope this is fine. please review it and let me know.
TIA :)

@potter-potter
Copy link
Contributor

@mogith-pn Looks great! I'll work on getting this merged. Nice work.

@potter-potter
Copy link
Contributor

potter-potter commented Mar 21, 2024 •
edited

@mogith-pn The clarifai pr has been merged. Can you close yours? Nice work!

#2633

@mogith-pn
Copy link
Author

@mogith-pn The clarifai pr has been merged. Can you close yours? Nice work!

#2633

@potter-potter , Thanks for your continuous support throughout the PR.

@mogith-pn mogith-pn closed this Mar 22, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@potter-potter potter-potter Awaiting requested review from potter-potter

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
2 participants
@mogith-pn @potter-potter