[deviantart] API responded with 429 Too Many Requests constantly, --sleep-request 5.0-10.10 and even 30.0 doesn't help

[unmnae]

About day or two ago i had been downloading galleries from deviantart. Usually it goes for a few pictures, then starts to increase delays up to 30 seconds, then starts to download one picture per ~ 3 to 7 429 errors, which were slow, but fine enough for me. This way it had worked for a year at least.
At some moment i had noticed, that 429 error messages had started to go without messages of a picture being downloaded, right during download. Since that time it now doesn't download anything but keeps returning 429 errors.
I had tried to go back to 12.6.5 of gallerydl, but it didn't help. Site itself opens normally, pictures are downloaded normally through browser. Could it be that deviantart had started to filter out requests from gallerydl specifically?

[Hrxn]

Are you using your own OAuth credentials?
I can't tell, you did not provide a verbose log.

[GoAwayNow]

Can confirm, two days ago Deviantart started responding with 429 to any request I made. This was tested with credentials that came from my already-logged-in instance, as well as with credentials spawned by private browsing instances and entirely new browsers I don't use.

I'm not using my own OAuth application.

I'm running through Hydownloader, so I don't know how to get a verbose log, but this is a (mildly redacted) log produced by it.

single-urls-6685-gallery-dl-latest - Copy.txt

EDIT: I've registered my own application and used it instead, which worked. I think this implies that the issue is exactly what it says, the default OAuth application is over its limits.

Follow the instructions in the configuration docs to set up your own application. BTW, this link is to the raw file because Github fails to properly render large files. Search for "extractor.deviantart.client-id" to find what you're looking for.

[unmnae]

Follow the instructions in the configuration docs to set up your own application. BTW, this link is to the raw file because Github fails to properly render large files. Search for "extractor.deviantart.client-id" to find what you're looking for.

Yes, this seems to fix my problem. But it is still looks like a strange change of deviantart's side.

UPD: now it seem to download *.jpg files instead of full-sized *.png in some cases, even when there is "download" button, that manually allows to download original png through browser.

UPD2: had seen several config options like "quality": png, but then it starts to fail to download pictures without "download" button on their page, that is available only in jpg. And links, that download jpg instead of png when "download" button is present on page and leads to png, still download jpg. As if he ignores download button on some of them.

Here is my config

{
"extractor":
{
"deviantart":
{
"#": "download 'gallery' and 'scraps' images for user profile URLs",
"include": "gallery,scraps",

    "mature": true,
    "original": true,
        "quality": "png",
    "public": false,
		
        "#": "put description texts into a separate directory",
        "metadata": true,
        "postprocessors": [
            {
                "name": "metadata",
                "mode": "custom",
                "directory"       : "Descriptions",
                "content-format"  : "{description}\n",
                "extension-format": "descr.txt"
            }
          ],
		"cookies": "path_to_cookie_file/cookies-deviantart-com.txt",
		"cookies-update": true,
		"client-id": MY_CORRENT_CLIEND_ID,
		"client-secret": "MY_CLIENT_SECRET"
        }
}

}

Here are commands i use:
gallery-dl.exe -i my_path_to_my_own_deviantart_links_input_file\input.txt --config my_path_to_config\gallerydl\config.json -o public=0 --sleep-request 5.0-10.0 --dest path_to_destination --download-archive my_path_to_archive\archive-DA.sqlite3

gallery-dl.exe --config my_path_to_config\config.json -o public=0 --dest path_to_destination link_to_page_with_download_button_and_png_picture

[Hrxn]

EDIT: I've registered my own application and used it instead, which worked. I think this implies that the issue is exactly what it says, the default OAuth application is over its limits.

Yes, that's exactly the case.

Follow the instructions in the configuration docs to set up your own application. BTW, this link is to the raw file because Github fails to properly render large files. Search for "extractor.deviantart.client-id" to find what you're looking for.

Rendering of rst seems still somehwat broken on GitHub.

Here's the same content with proper formatting etc:
https://gdl-org.github.io/docs/configuration.html#api-tokens-ids

[GoAwayNow]

Hey, turns out, this isn't just an "Oopsy, default app got used too much!"

Deviantart seems to have SIGNIFICANTLY lowered their API limits. My new app, made two days ago, is already returning 429.

EDIT: And based on their supposed "adaptive rate limiting" and the fact that the default app has consistently returned 429 for days, it does seem plausible that Deviantart is attempting to block Gallery-DL use.

[mikf]

UPD: now it seem to download *.jpg files instead of full-sized *.png in some cases, even when there is "download" button, that manually allows to download original png through browser.

You probably need a refresh-token. (gallery-dl oauth:deviantart)

[GoAwayNow]

You probably need a refresh-token. (gallery-dl oauth:deviantart)

I thought refresh tokens were stored in cache.sqlite3. Do they need to be copied into the config file as well?

[unmnae]

gallery-dl oauth:deviantart

That i do before each time i start new download, just in case. It is there is some pictures, whose previews can be expanded (when viewing through browser). Some of this expandable pictures can be expanded once (and disable RMB context menu), some can be expanded twice - not disabling this menu. And for this pictures, that can be expanded twice AND which are jpg in preview, but have download button, which will downloads png, gallerydl NOW had started to download jpg expanded previews. Problem is that it doesn't work like that for all this types of pictures, but for some of them. As soon as i will find an sfw example, i will add it to this post.
It wasn't an issue before i had started to receieve 429 constantly and were forced to use this

Follow the instructions in the configuration docs to set up your own application.

solution to download anything at all. Before that login/password in config file and oauth command had worked nicely for me.

UPD1: Btw, could it be connected to sfw/nsfw setting? Because it seem like in gallery with some images marked by nswf (when logged out) specifically this images get their jpeg preview being downloaded instead of png original despite availability of a download button when logged in and png behing this download button despite "mature" being enabled both in account on site and have "mature": true in my config.