SEO Crawl Analysis Template : question

[httpwebmedia]

Hello Elias, I wished I could contact you through mail but for some reasons I can't DM people on twitter.

So here I am and ask you a little help on your great tool and more specifically on https://www.kaggle.com/code/eliasdabbas/seo-crawl-analysis-template

I've been watching your video https://youtu.be/IZUBvDfDbqo

and I would like to make the code works with the following interactive tool :

In Jupyter notebook, when you execute the code :
def count_column_values(col_name, show_top=10, sort_others=False):
series = crawl_df[col_name].str.split('@@').explode()
print(f'Number of {series.name}s: {series.count():,}')
print(f'Number of pages: {len(crawl_df):,}')
print(f'{series.name}s per page: {series.count()/len(crawl_df):.2f}')
display(value_counts_plus(series, sort_others=sort_others, dropna=False, show_top=show_top))

widgets.interact(count_column_values,
col_name=crawl_df.select_dtypes('object').columns,
show_top=widgets.IntSlider(min=1, max=50, value=10));

it returns a line partially hidden like this :

So my question is, how do I get the interactive code like in your video please?

Thanks a lot for your great work!!

[httpwebmedia]

I forgot to mention, if I run the widget code in python interpreter, it returns this line :
interactive(children=(Dropdown(description='col_name', options=('url', 'title', 'meta_desc', 'viewport', 'charset', 'h1', 'h2', 'h3', 'canonical', 'alt_href', 'og:site_name', 'og:type', 'og:description', 'og:title', 'og:url', 'og:image', 'twitter:site', 'twitter:image', 'twitter:card', 'jsonld_@context', 'jsonld_@graph', 'body_text', 'download_slot', 'links_url', 'links_text', 'links_nofollow', 'nav_links_url', 'nav_links_text', 'nav_links_nofollow', 'header_links_url', 'header_links_text', 'header_links_nofollow', 'footer_links_url', 'footer_links_text', 'footer_links_nofollow', 'img_loading', 'img_src', 'img_srcset', 'img_alt', 'img_width', 'img_sizes', 'img_height', 'ip_address', 'resp_headers_date', 'resp_headers_content-type', 'resp_headers_link', 'resp_headers_x-cache', 'resp_headers_x-server', 'resp_headers_cache-control', 'resp_headers_cf-cache-status', 'resp_headers_last-modified', 'resp_headers_report-to', 'resp_headers_nel', 'resp_headers_vary', 'resp_headers_server', 'resp_headers_cf-ray', 'request_headers_accept', 'request_headers_accept-language', 'request_headers_user-agent', 'request_headers_accept-encoding', 'redirect_urls', 'h4', 'h5'), value='url'), IntSlider(value=10, description='show_top', max=50, min=1), Checkbox(value=False, description='sort_others'), Output()), _dom_classes=('widget-interact',))

but I can't run it in Jupyter...

[eliasdabbas]

1. Great to know it worked! Enjoy the interactivity.
2. I think the crawler is behaving correctly. Otherwise you wouldn't have known about this issue. You can use custom extraction, XPath or CSS selectors, if you want only title tags within the head. Does that make sense?

[httpwebmedia]

yes, it makes sense :) thanks for all!

One last, will you make the chat gpt entity extractor code available?

[eliasdabbas]

Great! Happy it worked, and I'm still not sure why on earth the Economist has so many titles on their pages.

Yes, I've written an article about how to train ChatGPT for the entity extraction app and it will be published hopefully soon.

[httpwebmedia]

Hello Elias, I still have great fun using your great tool.

I was wondering how I could possibly use https proxies with robotstxt_to_df?
I'm blocked by the site if I use it with that method...

Thanks :)

[eliasdabbas]

Great to know you're having fun with it!
At the moment there is no way to use proxies, but you can try the following:

1. Change request headers:

import advertools as adv
adv.robotstxt.headers  # <-- change this dictionary as needed

2. Manually download the robots file (right click, save), and then run robotstxt_to_df on the absolute file path on your local computer
3. Try running from a different location using Google Colab, Kaggle, or any other online notebook service.
   Hope this helps!