(Constructs): Web RAG - Web Crawler, Chatting with Web Pages and Search

[spugachev]

Describe the feature

Many RAG experiences are built around websites. Users want to crawl one or more websites, retrieve content from pages, schedule periodic updates, and inject results into OpenSearch to enable RAG requests based on website data.

To support this scenario, a WebCrawler construct can be created. It should be capable of creating new OpenSearch indexes or using existing ones.

This construct can also be used to obtain data from websites in real-time. For example, a user could ask a chatbot to summarize a specific webpage. In this case, the web crawler should extract data from the webpage and provide it to the chatbot.

We should also consider web search scenarios, where users want to use a search engine to obtain results. The results found by the search engine should be parsed and returned to the chatbot.

Use Case

RAG over websites

Proposed Solution

No response

Other Information

No response

Acknowledgements

• I may be able to implement this feature request
• This feature might incur a breaking change

[krokoko]

As discussed, assigning it temporarily to you @spugachev , thanks ! :)

[github-actions]

This issue is now marked as stale because it hasn't seen activity for a while. Add a comment or it will be closed soon. If you wish to exclude this issue from being marked as stale, add the "backlog" label.

[github-actions]

Closing this issue as it hasn't seen activity for a while. Please add a comment @mentioning a maintainer to reopen. If you wish to exclude this issue from being marked as stale, add the "backlog" label.