Wrong display in wallabag (spiegel.de) - Android cannot fetch paywall content, Desktop is working

[PackElend]

We have a lot of requests about fetching config issue. It'll help us A LOT if you give a try to fix it on your own following the doc.
If you failed to fix it yourself, tick the following boxes:

• I've tried myself without success
• I've replaced HOST in the issue title with the host of the URL that can't be fetched (ie: nytimes.com, 20minutes.fr, bbc.com, etc.)

Content related:

• URL: https://www.spiegel.de/wirtschaft/soziales/javier-milei-und-sein-radikalprogramm-fuer-argentinien-wie-eine-wurzelbehandlung-ohne-narkose-a-c342ea68-d1f6-4470-8d2a-42ff68a5094e
• wallabag version: 2.6.6 // https://app.wallabag.it/

Describe what's wrong:
On desktop, using the browser extension, I can easily fetch content from spiegel.de's paid content but it fails on the Android app. tyring to re-fetch content fails either.

At least you can add spiegel.de with restrictions to #3439

By the way, how to report working paywalls?

[HolgerAusB]

The standard way is sending just the article URL to wallabag, while wallabag server then tries to fetch the content. This is not working for paywalls or for sites that uses JavaScript to build the html on client side.

Wallabagger is an extension for desktop browsers with the option to grab the already rendered html article and send the whole content to wallabag instead of just the URL. This works great for most paywall sites and JavaScript sites. But there are no extensions/plugins at all for Android or iOS browsers. So that method is not possible for mobile browsers yet.

The third option is to store your credentials of a subscribed paywall site within wallabag's UI. Unfortunately this needs a site-dependend config, which needs a definition how to log in. We don't have this for spiegel.de at the moment. But these parameters are not easy to find out. Somebody needs the skills to do so AND a valid subscription. I don't even have the skill to do. I tried several times on different sites and just got one success. And I also don't have a Spiegel subscription.

But I agree, that it would be worthy to have option 3 for spiegel.de as this is one of the largest weekly political print papers in Germany.

Re-fetching an article automatically uses method 1 (or 3 if available).

By the way, how to report working paywalls?

I think you should leave a comment in the appropriate issue. But it seems, that 3439 is not an official list.

[PackElend]

The third option is to store your credentials of a subscribed paywall site within wallabag's UI

I did that, forgot to mention that. I thought it worked on a desktop due to this.

By the way, how to report working paywalls?

I think you should leave a comment in the appropriate issue. But it seems, that 3439 is not an official list.

--> This conversation has been locked and limited to collaborators.

Unfortunately this needs a site-dependend config, which needs a definition how to log in.

as I use the hosted app, wallabag maintainers need to kick in here as I cannot do https://doc.wallabag.org/en/developer/paywall

[HolgerAusB]

wallabag maintainers are focused on other things at the moment. And as i wrote, login settings aren't that easy as it might look.

Spiegel uses a two-step-login. The first login form only has a email-field and (beside others) a hidden field with a generated secret one-time-token. After sending your mail address and that token to the server, you get a second form for the password and a hidden field with the email you transferred in the previous step and a NEW hidden secret TOKEN, which you need to send again to the server.

AFAIK, wallabag is not able to do such a two-step-verification at the moment, sorry. Spiegel does not want bots or crawlers grabbing their content.

BTW, wallabag re-uses this repo for its site depended things

[PackElend]

BTW, wallabag re-uses this repo for its site depended things

at least spiegel.de is listed: https://github.com/fivefilters/ftr-site-config/blob/master/spiegel.de.txt

[HolgerAusB]

Yes but only to select relevant content, but not for defining credentials ☹️

[PackElend]

can we raise an issue over there ?

[HolgerAusB]

No! FiveFilters don't use credentials at all. If there are configs with login parameters these are for wallabag. So wallabag is the correct place for your issue. But I don't think, that we could help here.