A lot of subreddit feeds not working in v21.2.0

[markochk]

IMPORTANT

Read and tick the following checkbox after you have created the issue or place an x inside the brackets ;)

• I have read the CONTRIBUTING.md and followed the provided tips
• I accept that the issue will be closed without comment if I do not check here
• I accept that the issue will be closed without comment if I do not fill out all items in the issue template.

Explain the Problem

I upgraded Nextcloud News app to v21.2.0 and since then only a few subreddit RSS feeds were fetching articles, while the rest weren't. I use about 15 subreddit RSS feeds. Non-reddit related RSS feeds are working properly.

I don't think it's relevant, but some subreddit feeds that are working:

• https://www.reddit.com/r/programming/.rss?limit=10
• https://www.reddit.com/r/linux/top/.rss?limit=10
• https://www.reddit.com/r/linux_gaming/top/.rss?limit=10

and some that aren't:

• https://www.reddit.com/r/technology/top/.rss?limit=10
• https://www.reddit.com/r/hardware/top/.rss?sort=top&t=week&limit=50
• https://www.reddit.com/r/nba/top/.rss?limit=10
• https://www.reddit.com/r/formula1/top/.rss?limit=10

I've checked that reserved_at field in oc_jobs table is 0. I've restarted apache, php-fpm, server itself and nothing worked. Cron is working fine. I didn't find any errors in logs, but it's possible I've looked at the wrong ones. Only after downgrading to v21.1.0 were all RSS feeds working properly again.

Steps to Reproduce

1. Install News app v21.2.0
2. Add about 10 subreddit RSS feeds by adding .rss at the end of subreddit url
3. Check that some subreddit feeds aren't fetching articles

System Information

• News app version: 21.2.0
• Nextcloud version: 26.0.1
• Cron type: system cron
• PHP version: php8.1-fpm
• Database: 10.6.12-MariaDB
• Browser and version: Firefox 112.0.2
• Server OS and version: Ubuntu 22.04.2 LTS kernel v5.15.0

Contents of nextcloud/data/nextcloud.log

Only data I have in it is 2 years old, so it's not meaningful for this issue.

[SMillerDev]

https://validator.w3.org/feed/check.cgi?url=https://www.reddit.com/r/programming/.rss?limit%3D10 says it's invalid on the first one already. Seeing how Reddit wants to charge for their API access I'm expecting this to just break over time.

Additionally, without logs of any failures it's near impossible to reproduce.

[Grotax]

The main difference between the versions that might affect this is that we now send the last modified date as http header.

If you don't see any errors in the logs then this can have two reasons, their feed is broken or they just don't want to give any data or both.

I don't think there is much we can do about that. We don't have the capacity to have special handling for websites that are not compatible with web and RSS/atom standards.

[quentinus95]

Hello, I'm facing the same issue on my side, for some other feeds. For instance, this feed: https://hnrss.org/newest?points=200 has been broken by the update. I would say that 30-50% of my feeds are currently out of order.

I have been able to unlock feeds content by editing the table like this:

update oc_news_feeds set http_last_modified = "2023-05-01" where id = <id>;

I then got all the articles that were published recently.

I'm wondering if there is not a bug where articles that have a publication date before last modified but that are added to the feed on a later date are ignored.

It can happen for Reddit or HackerNews as article may be added to the feed only if they match a specific condition. Would it be possible to roll back this change while a proper fix is designed?

[quentinus95]

Do you want me to create an issue to track this need in the meantime?
Also, do you think an option to disable this behavior or a rollback could be possible to mitigate the issue?

[Grotax]

I don't think a rollback is the way to go, we need someone to create a PR to improve the situation or to change feed-io. If we rollback the code we just have the old bug again.

You can open an issue but that won't change much at least for me. If you do make sure to document the important details and what the background is.

[quentinus95]

I can understand, from a project point of view, that a rollback is not acceptable, however do you know if a user can roll back the package version? I tried deleting the app files and restoring an older version, but a restart of Nextcloud forced the update to the latest. Do you know if there is a change required in db to achieve it?

[Grotax]

Not sure maybe you need to delete the entries in oc_appconfig and oc_migrations see the docs.

But I'm not sure maybe check the forums on how to downgrade an app. Nextcloud normally doesn't support this.

[quentinus95]

Bug report created: #2236

[Grotax]

Should be solved with latest version 23.0.0