Add option to reconstruct XML from the pages API #429
+255
−16
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add support to backup pages using API:Query instead of Special:Export WikiTeam#280 into recent version of dumpgenerator.py Adds additional parameter: --apiexport which uses a query request instead of submit on api.php which works without Special:Export which is disabled on some sites.
-renamed --apiexport to '--apiquery' and gave it a more useful help text -renamed function 'getXMLPage_' to 'selectXMLQuerMode' to make it distinct enough from 'getXMLPage' -added in checks so that '--apiquery' can only be used together with '--curonly' -dialed back some timeouts from 120 to 60 which is still more than the previous 30 Improvements that could still be implemented: -Checking the availability of Special:Export before trying to download the list of titles and images wasting resources
'--images' and '--xmlrevisions' too, accidentally disabled that in previous commit
|
Il 12/04/22 01:54, Gernot Zacharias ha scritto:
I stumbled into the problem when trying to downloadhttps://pokewiki.de where Special:Export wasn't available and i tried the old version from 2016 with the pull request #280 which seems to work so far.
Thank you but I don't understand, have you tried --xmlrevisions which
uses the API and should work just the same? Did it fail and how?
|
nemobis
changed the title
|
I tried --xmlrevisions which always failed and resuming basically made it write the same pages over and over into the output file |
|
Il 12/04/22 11:02, Gernot Zacharias ha scritto:
I tried --xmlrevisions which always failed
Thanks. Please open an issue with the command you used and ideally a
tail of the XML file after the failure, if you still have it.
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merged: Add support to backup pages using API:Query instead of Special:Export #280 into recent version of dumpgenerator.py
(so, most credit should go to NyaMisty who tried to merge this back in 2016 and closed the pull request back in 2018)
Adds additional parameter: --apiquery
which uses a query request instead of submit request on api.php which works without Special:Export which is disabled on some sites.
Also adds: --ignore-api-check
which i just copied from the earlier mentioned pull request and it doesn't do much so it could be removed as well
I stumbled into the problem when trying to download https://pokewiki.de where Special:Export wasn't available and i tried the old version from 2016 with the pull request #280 which seems to work so far. I am still doing some testing as I am mostly just trying to archive game wikis and have not much coding experience. I didn't wanted all revisions and only the current once which didn't work because of the missing 'Special:Export' page. Also '--xmlrevisions' wasn't working either as it failed after some time and resuming the progress only lead to it downloading the same pages over again creating many duplicates inside the output file and not going further.
so some improvements I haven't done would include:
Thank you all for the great work :)
I am just trying to give something back to the community even if it is just recycling old pull requests that got closed.