PDF version of the Guidelines

[kepper]

Dear all,

I'm working on bringing back a PDF version of the Guidelines. Much of that has been prepared in our transition back to ODD already, and I'm quite close to making a PR for this. Just for giggles: Right now, we're looking at ~5700 pages, but that can easily go up or down by a couple hundred pages. A more relevant number is probably the size of the Guideline chapters (as opposed to the specs), which is still some 350+ pages. I'm deeply impressed by what the community as a whole has achieved with this!

There are some open questions about this, though, mostly related to the workflow with this.

I would like to automatically publish each PDF, even for the development versions. With Zenodo, it seems possible to get a reserved DOI through their API. That DOI could be embedded in the PDF before upload. I already have access to all contributors on GitHub, and we could probably add them automatically as contributors to the publication on Zenodo. If we'd have Orcid IDs or GNDs, we could even identify them properly. It also seems possible to use the obsoletes property in their API to link all dev versions.

I know that this is will create a huge number of MEI publications. However, if we manage to keep them structures (using the obsoletes property or some such), I think this is worthwhile, because it only reflects the community's activity. The huge benefit is that we can give public recognition to the work that community members contribute to MEI.

While I will continue to work on the shape of the PDF, I would like to get some feedback on this. I must say I'm not married to Zenodo, but haven't found a similar API functionality for Humanities Commons. In any case, I would require assistance to set up the corresponding workflow for this. So, what do people think? Just mentioning @bwbohl, @musicEnfanthen, and @ahankinson here because I would approach them first for assistance, but everyone else's feedback and help is equally appreciated :-)

[ahankinson]

I'm not sure about generating it for the development versions. We recently went through an exercise to try and get a handle on the Verovio repo because we built a new version on each commit, and it ended up being gigabytes in size. That was a lot of space, and it was 99% the same from one revision to the next.

It may also mean that a wayward search engine finds and promotes a development version of the docs over the 'stable' version, for reasons known only to itself -- maybe someone stumbled across a version, linked to it in a tweet, which then got shared, and then Google thought "hey! this is important!". This isn't very user-friendly for someone Googling it, but it happens quite frequently for all sorts of software, and it's a really hard feedback loop to break once it does happen (dev version gets cited, google finds it, people find it with google and cite it more, google makes it even more important, etc. etc.).

An alternate proposal might be to add a [build-docs] flag or something like that to a commit message, which would explicitly enable documentation building for a given commit. (You can see how this works in the Verovio actions for [skip-ci]: https://github.com/rism-digital/verovio/blob/develop/.github/workflows/ci_build.yml#L62-L64).

Ultimately, though, I think I would be in favour of only building a version of the PDF for merges made to the stable branch, if we're assigning a DOI. A DOI is designed primarily to make things citeable, and I think we would only want things that have been reviewed for accuracy and completeness to be cited. If we mint a DOI, we are also obliged to keep the thing that the DOI references indefinitely, and I don't think we necessarily need to keep a PDF that was built from this commit in perpetuity, to pick one of many examples. (Also, isn't the MEI Board supposed to approve publications? If we're assigning a DOI to every commit, does this count as a publication?)

Can I tag @annakijas1 to see what she thinks, and also so she sees the question about Humanities Commons?

[ahankinson]

PS: Can I convert this to a discussion, since there's no actual change proposed or bug reported?

[kepper]

I was hoping that the obsoletes flag would help to make crawlers behave nicely (there is also an reversed flag that we could attach to older commits). I agree however that this is something to consider. We've been thinking to build documentation and schemata for branches as well (so that it get's easier to prepare things for a PR), but we surely don't want to publish every commit there with a DOI, so we need to find the right balance anyway.
Regardign keeping old versions: Zenodo would take care of the archiving. On our own webspace / repo, I would only keep the latest version. File size is ~50mb right now, but I still need to check whether all images are included properly or not, so that may change.

[musicEnfanthen]

What about having the complete repo as versioned snapshots on Zenodo? I did this for our Webern repo (see https://zenodo.org/record/5035258) and it is pretty straightforward and automated from Github when aligned with Github Releases. Additionally, it is really nice that you get a separate DOI for every release, but also an "overarching" DOI that always points to the latest release.
Not sure if it would make sense to have a PDF with DOI for every commit to develop, though. So maybe generate a PDF that is served from and hosted on our website on every commit, but then only "archive" the latest state of it with Zenodo when a release is made?

[bwbohl]

First of all: many thanks for your efforts @kepper !!

I think @ahankinson pointed out a very dangerous pitfall considering the dev-versions and pushed importance through search engines AND commit quality. As long as we're not strictly regulating the latter, a "proper" publication of dev-status should be out of sight I guess. Publishing the commits to stable i.e. releases through e.g. Zenodo would be great, though!

One thing that bothers me a bit since our transition to our continuous integration workflow (and does so again in this context) is the fact, that the source and the product do not live in the same repo… but never mind!

The pre-dominant questions concerning the integration of your source2PDF stuff are:

• what technology does your workflow run on?
• what are the build requirements?
• what are the inputs you need from GitHub (e.g. contributors)?
  ** follow up on contributors: are those contributors to the whole repo, to specific parts of the repo, how os their contribution evaluate etc.?

And of course, I'd be happy to assist in making this happen!
Which brings me to the second thing that bothers me since CI-ifying the music-encoding repo:
Should we switch to a docker-based build, i.e. create a docker image that holds everything we need for all our building and testing stuff? This could easily be run within GitHub Actions and probably easier (at least for some) on local machines? (But this might be worth a second discussion)

[kepper]

Hi @bwbohl, thanks for your input. I'm just responding to the direct questions for now.

• what technology does your workflow run on?

It's a combination of the existing XSLT (which already prepares a single file HTML version of the Guidelines), and the PrinceXML library, which is based on CSS3 and allows quite professional layouts, and is free for non-commercial purposes.

• what are the build requirements?

We need to run PrinceXML on the commandline. Should be equally possible as GitHub Action or in a Docker image.

• what are the inputs you need from GitHub (e.g. contributors)?

I'm already getting me the stuff I need from within the XSLT. If we plan to include DOIs at least for some versions, I would need that to be feeded into the loop at some point.

• • follow up on contributors: are those contributors to the whole repo, to specific parts of the repo, how os their contribution evaluate etc.?

I would recommend to make no distinction between the levels of contribution. Everyone who ever contributed in some sense will be listed once and forever. We do have a verbal summary over at 1.1.2 Acknowledgments, which will not change automatically. Right now, I'm thinking about an additional index page at the end of the PDF that will have links to the GitHub accounts. Maybe we want to clean that list and provide additional info in a separate file. We could probably drop the DDMAL-LabManager (assuming that's a bot…), and may want to give full names for some of those mentioned in the sources etc.

[bwbohl]

please see #959 for current status

[musicEnfanthen]

PDF is now available. Thanks @kepper for the amazing work!

Further improvements/workflows will be addressed in #1162.

Closing this discussion.