Support non-github organizations and repositories

[mirodietiker]

Github might be the most dominant player, but it is not representing the whole Open universe.
This causes bias.

For instance drupal.org maintains its own repository source, meanwhile managed through a gitlab.
We currently can't add them.
Also drupal.org does not provide a concept like an organization.

The idea to define Organizations and allow them to self-manage their ecosystem makes sense.
Thus we only have one pull request here to add the org and we can then lookup all repositories in the orgs self-owned channel.

Currently this is only done by supporting githug org lookups.

The proposal to improve this is this:
Orgs are a generic lookup resource. It can be a github org or an external URL with an org manifest.
The org manifest would be a YML based lookup that lists all their repositories (with optional metadata)

As a reference i like to look at composer.json. This is a definition of a repo dependency. That's a similar list.
We should choose a format / structure that allows us to extend metadata easily, such as adding tags / categories and other information to a repository.

A manifest can be published under any own URL such as example.com/open/repositories.yml

• We can still encourage orgs to publish such a manifest file as a github repository / file and we can fetch that through the Github raw feature.
• The org owner will create his own manifest, publish it and register it at the benchmark.
• If the org owner has a new repo, he will extend his manifest.
• The github record is then like a special org type with auto-discovery.
• (If needed, we will be able to support other platforms similarly as well, such as gitlab.)
• (Similarly, we could also expand and collect open datasets)

Todo:

• Define an example manifest file with a repo collection
• Add support for the manifest in benchmark code
• Promote the availability
• Add non-github orgs that publish their own manifest
• Enjoy broader stats :-)

[mathildedd]

Proposal for a minimal yml file for an organization outside of github: (for now, we thought about gitlab or raw git)

organization_name: "MD Systems"
description: "We love open source"
location: "Zurich"
email: info@md-systems.ch
url: https://www.drupal.org/md-systems
date_creation: 2008
logo: https://www.md-systems.ch/themes/mds_theme/logo.svg
members:
  miro_dietiker:
    email: email_address@md-systems.ch
  mathilde_dumond:
    email: email_address@md-systems.ch
repositories:
  paragraphs:
    url: https://git.drupalcode.org/project/paragraphs
  entity_reference_revision:
    url: https://git.drupalcode.org/project/entity_reference_revisions

We get this way some of the metadata that is by default available on github:

• organization name
• description
• location
• email
• url to the organization where the repositories can be consulted
• the date of creation of the organization
• a logo
• a list of the members. For this we think that a list should be available in the yml document. Gitlab also has a members list per repository (not per organization). It can still be the fallback if no member is given in the yml. Git can provide the list of all committers with git shortlog --summary --numbered --email but this is probably too broad for our purpose.
• a list of the repositories, with an address to either the gitlab repo or the git repo.

[mathildedd]

the reason to add the email address is to be able to fetch the commits done by someone in the org. optionally, this can also be done by feeding the usernames used in the specific repositories