Utilities for processing Wikipedia and Wikidata dumps in Go

A Go package providing utilities for processing Wikipedia and Wikidata dumps.

Features:

Supports Wikidata entities JSON dumps.
Supports Wikimedia Enterprise HTML dumps.
Supports Wikimedia Commons entities dumps.
Supports SQL dumps (database layout).
Decompression and JSON decoding is parallelized for maximum throughput on a single machine.
Parses into idiomatic Go structs, with no loss of information.
Can download and process a dump at the same time.
Can cache downloaded files locally.
Supports GZIP and BZIP2.
Supports data in JSON arrays, NDJSON, and SQL.

Installation

This is a Go package. You can add it to your project using go get:

go get gitlab.com/tozd/go/mediawiki

It requires Go 1.21 or newer.

See full package documentation on pkg.go.dev.

There is also a read-only GitHub mirror available, if you need to fork the project there.

Name		Name	Last commit message	Last commit date
Latest commit History 145 Commits
.github		.github
testdata		testdata
.gitattributes		.gitattributes
.gitignore		.gitignore
.gitlab-ci.yml		.gitlab-ci.yml
.gitlab-conf.yml		.gitlab-conf.yml
.golangci.yml		.golangci.yml
.markdownlint.yml		.markdownlint.yml
.nancy-ignore		.nancy-ignore
.release-it.yml		.release-it.yml
.sops.yaml		.sops.yaml
CHANGELOG.md		CHANGELOG.md
LICENSE		LICENSE
Makefile		Makefile
README.md		README.md
article.go		article.go
commons.go		commons.go
commons_test.go		commons_test.go
config.go		config.go
docs.go		docs.go
entity.go		entity.go
entity_test.go		entity_test.go
errors.go		errors.go
go.mod		go.mod
go.sum		go.sum
html.go		html.go
process.go		process.go
process_test.go		process_test.go
wikidata.go		wikidata.go
wikidata_test.go		wikidata_test.go
wikipedia.go		wikipedia.go
wikipedia_test.go		wikipedia_test.go