Make Wikidata primary source for Pakistan

[lucychambers]

Limit initially to:

• Current term (14th Assembly)
• Lower House

This will involve creating multiple prompts to make sure that Wikidata is sufficient:

• Wikidata ⇆ Wikipedia
• Wikidata ⇆ Official site
• (Maybe, whilst in progress) Wikidata ⇆ EP

Acceptance Criteria

• All membership sources (morph/official.csv and archive/official-term-14.csv) are replaced by information from Wikidata.
• No information is lost (unless it was incorrect) — anything not in Wikidata, and not suitable for adding there can be added to a new person source.

[tmtmtmtm]

For now it's OK to make a morph scraper to get the membership/P39 information out of Wikidata — we won't know the correct abstractions for doing something more general until we've done quite a few more of these.

[chrismytton]

I've made an initial prompt which compares what's in the Wikipedia scraper with Wikidata items that have a current (no end date) Member of the 14th National Assembly of Pakistan (Q33512801) P39 entry.

https://www.wikidata.org/wiki/User:Chris_Mytton/sandbox/prompts/Pakistan_National_Assembly

[lucychambers]

Outstanding:

• Scraper for historical members (within current term).
  These are the people in the "Membership changes" section of the List of members of the 14th National Assembly of Pakistan page on Wikipedia.
• Prompt for the scraped historical members.
• Generate Quickstatements TSV to add missing memberships for historical members
• Write a membership scraper to get term 14 data from Wikidata
• EP comparison prompt. EP master branch <-> Wikidata.
• Switch over EveryPolitician to treat Wikidata as the main source, but don't merge yet
• Once any problems have been fixed, merge the EP branch to complete the switch

[chrismytton]

I've also created a prompt for the official site.

That prompt uses a manually generated CSV that takes the output from the scraper and combines it with EveryPolitician reconciliation information using something similar to the following command, run from data/Pakistan/Assembly in ep-data. ~/Downloads/pakistan-national-assembly.csv is the scraper output.

q -H -d, -O 'select w.id as wikidata, name
 from ~/Downloads/pakistan-national-assembly.csv o
 left join sources/idmap/official.csv oid on oid.id = o.id
 left join sources/reconciliation/wikidata.csv w on w.uuid = oid.uuid'

[chrismytton]

Once #53037 has been merged that should fix a couple of issues with the official site prompt.

I've also been seeing a strange error with the official site prompt, sometimes the SPARQL will return 330 results, but then clicking through an running the SPARQL manually returns 339 results, as expected. I'm not sure if there's anything that can be done or if it's just transient, but worth watching out for.

[chrismytton]

I've updated the Wikipedia scraper to pick up historic members from the "Membership changes" table in everypolitician-scrapers/pakistan-national-assembly-wikipedia@62d4ac5.

[chrismytton]

Prompt for the historic members of the 14th term is here:

https://www.wikidata.org/wiki/User:Chris_Mytton/sandbox/prompts/Pakistan_National_Assembly_historic

[chrismytton]

I've generated Quickstatements (docs) for the missing historic term 14 members here:

https://gist.github.com/chrismytton/aa224963a46b92dc273569af7355a512

[chrismytton]

I've now run the that batch of Quickstatements, so the people on the historic term 14 prompt should now all have a "Member of the 14th National Assembly of Pakistan" P39 statement.

[chrismytton]

The members that I've just added a term 14 P39 for are missing start and end dates, because they weren't simple to scrape from the Wikipedia page. @lucychambers has kindly volunteered to go through and manually add them for the 22 members on the prompt, thanks Lucy!

[chrismytton]

Before we can switch EveryPolitician over to using Wikidata as the primary membership source we need to create a scraper. The tonga-assembly-wikidata scraper is probably the best example we have to work from, that scraper was created on a previous attempt to switch a country to using Wikidata, so should in theory have all the fields we need.

[chrismytton]

I've created the scraper for getting membership information from Wikidata:

• everypolitician-scrapers/pakistan-national-assembly-sparql
• morph.io scraper

[chrismytton]

Prompt created at User:Chris_Mytton/sandbox/prompts/Pakistan_National_Assembly_EveryPolitician which compares the term-14.csv file for Pakistan on EveryPolitician's master branch with what's currently in Wikidata.