This is the Repository containing the material required for the 2nd assignment of Lizmotors mobility for the role of AI/ML Engineering Intern.
Building a Basic RAG (Retrieval-Augmented-Generation) or Vector Search System for an EV based Company called Canoo. We can Define RAG in simple terms: When connecting a Language Model (LLM) to a datastore, we augment it by adding extra data to a vector database (DB). The prompt is meticulously crafted to enable the LLM to not only consider the original input but also consult the vector DB for the most relevant response.
Data Extraction -> Chunks -> Vector Embeddings -> Vector Database
Retrieval: User Query -> Vector Embeddings -> Search (Vector similarity search) from Vector Database -> Result (summarized, keywords, etc..)
Result from Retrieval stage is then synthesized with the LLM
1) Based on 4 queries, find relevant web links of each query using Internet search APIs
2) Scrap relevant data from those web links and store as CSV files
Based on similarity or disimilarity between the data of all 4 queries the decision to make single or 4 CSV will be decided.
-
For queries within the list, extract 10 web links each using duckduckgo API and store them in a text file consisting of links and their respective query
-
Read the Text file and extract the non_link and link part in sepeate lists. Topic = non_link part or the Queries
-
Scrape each link using the combination of Selenium and BeautifulSoup. I am extracting their p and span tag.
-
Then I am using GEMINI API to extract relevant information on the basis of respective Topic from scraped text in clean and clear format. Helps in reducing the task of data cleaning.
-
Storing Data in a CSV with following Structure. Information Column Data is in Json Format
Query / Topic url Information
Note: csv files contains extracted Information based on respective Query on each and every respevtive url so some NaN results are expected
- duckduckgo API
- csv
- google_api_core.exceptions
- google.generativeai
- genai
- BeautifulSouo
- Selenium