How to capture missing values in a category when appending two categories' data from the wanted list

[bakaito]

I'm trying to build a dataframe from two different categories of a website's data using autoscraper library.

import pandas as pd
from autoscraper import AutoScraper
df=pd.DataFrame()

#scrape data
url = "https://www.grabon.in/makemytrip-coupons/"
wanted= ["Flight Tickets - Get Up To Rs 1250 OFF", "Rs.3000 OFF"]
scraper = AutoScraper()
df["description"] = scraper.build(url, wanted[0])
df["amount saved"] = scraper.build(url, wanted[1])

One of the categories, "amount saved" seems to have missing values corresponding to "description" category. Hence the following error:

Traceback (most recent call last):
  File "C:/Users/Samhitha/Desktop/Sam-GrabOn/readibility/TEXTHERO_py.py", line 9, in <module>
    df["amount saved"] = scraper.build(url, list_1[1])
  File "C:\Users\Samhitha\Desktop\Sam-GrabOn\PycharmProjects\kw_rank_GWT_Emailers\venv\lib\site-packages\pandas\core\frame.py", line 3163, in __setitem__
    self._set_item(key, value)
  File "C:\Users\Samhitha\Desktop\Sam-GrabOn\PycharmProjects\kw_rank_GWT_Emailers\venv\lib\site-packages\pandas\core\frame.py", line 3239, in _set_item
    value = self._sanitize_column(key, value)
  File "C:\Users\Samhitha\Desktop\Sam-GrabOn\PycharmProjects\kw_rank_GWT_Emailers\venv\lib\site-packages\pandas\core\frame.py", line 3896, in _sanitize_column
    value = sanitize_index(value, self.index)
  File "C:\Users\Samhitha\Desktop\Sam-GrabOn\PycharmProjects\kw_rank_GWT_Emailers\venv\lib\site-packages\pandas\core\internals\construction.py", line 752, in sanitize_index
    "Length of values "
ValueError: Length of values (171) does not match length of index (173)

Is there a way to get autoscraper capture null values for "amount saved" appropriately corresponding to its "description" so that I can append both of them into a single dataframe?

[alirezamika]

You should pass a list to the build method. like scraper.build(url, [wanted[0]])
Also you may want to use get_result_similar method with unique=False argument.

[bakaito]

Thanks for a prompt response. I tried with building a list, but there's still a mismatch in the length. when manually checked, few of the values from wanted[1] corresponding to wanted[0] are being omitted (both wanted[0] and wanted[1] are being scraped from the same url). Here's my code.

from autoscraper import AutoScraper
df=pd.DataFrame()
af=pd.DataFrame()

#scrape data
url = "https://www.grabon.in/makemytrip-coupons/"
wanted = ["Flight Tickets - Get Up To Rs 1250 OFF", "Rs.3000 OFF"]
scraper = AutoScraper()
scraper.get_result_similar(url, unique=False)
df["description"] = scraper.build(url, [wanted[0]])
af["amount saved"] = scraper.build(url, [wanted[1]])
print(len(df), len(af))

output:

C:\Users\Samhitha\Desktop\Sam-GrabOn\PycharmProjects\kw_rank_GWT_Emailers\venv\Scripts\python.exe C:/Users/Samhitha/Desktop/Sam-GrabOn/readibility/TEXTHERO_py.py
39 22

Process finished with exit code 0

[alirezamika]

from autoscraper import AutoScraper
df=pd.DataFrame()
af=pd.DataFrame()

#scrape data
url = "https://www.grabon.in/makemytrip-coupons/"
wanted = ["Flight Tickets - Get Up To Rs 1250 OFF", "Rs.3000 OFF"]
scraper = AutoScraper()
scraper.build(url, [wanted[0]])
df["description"] = scraper.get_result_similar(url, unique=False)
scraper.build(url, [wanted[1]])
af["amount saved"] = scraper.get_result_similar(url, unique=False)
print(len(df), len(af))

[bakaito]

Many thanks. My bad, I didn't have a clear idea of what the build was actually doing. Now I do.
May I also know that if I'm building on an example element (wanted[1]) that is repeated multiple times on the webpage, the entire series (af["amount saved"]) that I get from get_result_similar is also repeated the same number of times (which creates a mismatch in the lengths again). Is there a way autoscraper build handles this automatically without a manual check and passing a different example element that only appears once?

[alirezamika]

You can pass grouped=True when calling get_result_similar method to see the results grouped by rules and use the ones which you need only. Then you can use keep_rules or remove_rules method to finetune the rules.

[Priyasganesan]

I am too facing similar kind of problem. I need to combine the wanted list in question and answer way. Some of the contents in the answer dataframe is missing i.e there is a length mismatch and also because of that the question and the answer are mapped in a wrong way. Is there anything that can be done to map the two items in the wanted list correctly?