How to pull all URLs modified today?

[gk2go]

today= constant.get_now()
sitemap_df = adv.sitemap_to_df(sitemap_url)

This gives me a dataframe with all the urls in the sitemap. How to filter this down to only get the entries where the lastmod is today?

[eliasdabbas]

You can create a variable with the desired date, for example today, and then filter the sitemap df where the extracted date from lastmod is equal to today.

For example:

import datetime

import advertools as adv
import pandas as pd
sitemap = adv.sitemap_to_df('https://www.nytimes.com/sitemaps/new/news.xml.gz')

today = datetime.datetime(2021, 1, 19)

sitemap[pd.to_datetime(sitemap['lastmod'].dt.date).eq(today)]

	loc	lastmod	publication_name	news_publication_date
0	https://www.nytimes.com/live/2021/01/19/us/inauguration-day-biden	2021-01-19 22:08:17+00:00	The New York Times	2021-01-19T13:30:02Z
1	https://www.nytimes.com/2021/01/11/us/ca-covid-surge.html	2021-01-19 22:07:33+00:00	The New York Times	2021-01-11T13:48:56Z
2	https://www.nytimes.com/2021/01/19/opinion/politics/new-york-city-homicide.html	2021-01-19 22:04:19+00:00	The New York Times	2021-01-19T10:00:07Z
3	https://www.nytimes.com/2021/01/19/world/europe/italy-government-collapse-renzi-conte.html	2021-01-19 22:03:02+00:00	The New York Times	2021-01-19T21:37:46Z
4	https://www.nytimes.com/interactive/2020/obituaries/people-died-coronavirus-obituaries.html	2021-01-19 22:02:51+00:00	The New York Times	2020-04-16T22:28:14Z

[gk2go]

Has anything changed in how this works?
Try it on https://www.aljazeera.net/sitemap.xml?yyyy=2022&mm=01&dd=11 and it should return this url for example, but it doesn't:

<loc>https://www.aljazeera.net/turath/longform/2022/1/11/%d8%b9%d9%84%d9%85%d8%a7%d8%a1-%d9%85%d9%88%d8%b3%d9%8a%d9%82%d9%8a%d9%88%d9%86</loc>
<lastmod>2023-05-22T00:09:29+00:00</lastmod>

Or if you try it on https://www.aljazeera.net/sitemap.xml?yyyy=2021&mm=08&dd=31 it should return this url for example:

<loc>https://www.aljazeera.net/turath/2021/8/31/%d9%85%d8%aa%d8%b5%d9%88%d9%81%d8%a9-%d8%aa%d9%8a%d9%85%d9%8a%d9%91%d9%88%d9%86-%d9%88%d8%a7%d9%84%d8%b0%d9%87%d8%a8%d9%8a-%d9%8a%d9%84%d8%a8%d8%b3-%d8%a7%d9%84%d8%ae%d8%b1%d9%82%d8%a9</loc>
<lastmod>2023-05-22T00:07:14+00:00</lastmod>

[eliasdabbas]

Nothing changed.

I just tried with the two sitemaps you shared, and it seems they actually contain the links you mentioned:

What's the output you are getting? Can you please share the code?

[gk2go]

sitemap_url = 'https://www.aljazeera.com/sitemap.xml'
date = constant.get_now()
date = datetime.datetime(date.year, date.month, date.day)
sitemaps_df = []
xml_sitemaps = adv.sitemap_to_df(sitemap_url, recursive=False)
xml_sitemaps = xml_sitemaps[pandas.to_datetime(xml_sitemaps['lastmod'].dt.date).eq(date)]
for sitemap in xml_sitemaps['loc']:
     temp = adv.sitemap_to_df(sitemap)
     sitemaps_df.append(temp[pandas.to_datetime(temp['lastmod'].dt.date).eq(date)])
     time.sleep(10)

The resulting sitemaps_df will only have the urls from https://www.aljazeera.net/sitemap.xml?yyyy=2023&mm=05&dd=23 eventhough it goes through the other sitemaps.

[eliasdabbas]

That seems to be the correct behavior.

You have define date:

date = datetime.datetime(date.year, date.month, date.day)

Then filtered for only URLs where lastmod is equal to date:

xml_sitemaps = xml_sitemaps[pandas.to_datetime(xml_sitemaps['lastmod'].dt.date).eq(date)]

So you are specifying that you only want to get such sitemaps.

You also append a subset that only contains URLs where lastmod is equal to date:

sitemaps_df.append(temp[pandas.to_datetime(temp['lastmod'].dt.date).eq(date)])

Basically, you have ensure that you don't get, and then, that you don't append any other sitemaps.

What is your intention? If you want all the sitemaps, then just run

xml_sitemaps = adv.sitemap_to_df(sitemap_url)

[gk2go]

Hi Elias, Is there a wait to add a one second pause/sleep between each sitemap url retrieval? Let me know Thanks

[eliasdabbas]

You can write a loop and introduce an interval with time.sleep

import time

xml_sitemaps = [
  "https://example.com/sitemap_1.xml",
  "https://example.com/sitemap_2.xml",
  "https://example.com/sitemap_3.xml",


sitemaps_df = []

for sitemap in xml_sitemaps:
    temp = adv.sitemap_to_df(sitemap)
    sitemaps_df.append(temp)
    time.sleep(1)

final_df = pd.concat(sitemaps_df, ignore_index=True)

In order to get a list of sitemaps from a sitemap index file, you can use the recursive=False option

xml_sitemaps = adv.sitemap_to_df('https://example.com/sitemap_index.xml', recursive=False)