Using crates.io Categories to automatically populate crates

[elpiel]

I've been thinking about this for a while and I think it would be very nice to have an automated way of populating the crates, instead of manually adding each of them.

The AeroRust Website

We can integrate www.areweinspaceyet.org as part of the AeroRust website (e.g. like a menu item) and populate the crates using the crates.io Categories - categories.toml using the same design.

Rust re-write

Instead of the Ruby app we can rewrite the server using Rust.
It will be easier to collect, parse and serve the website and easier to maintain as no Ruby knowledge will be required.

Tasks:

• We need the same information that the Ruby script currently generates
• Build the crates list using a list of categories to be included
• Whitelist & Blacklist for crates that are missing or wrongly categorized
• Decide if we should use a database dump (with https://crates.io/crates/db-dump) or crates.io. API (https://crates.io/crates/crates_io_api)

You can see an example binary for collecting the crates' information by categories from crates.io here

✔️ More categories on crates.io

We can ask in the crates.io repository how we can extend the categories that can be used for crates.
This will allow us the collect these crates from the registry and show them on www.areweinspaceyet.org

Currently, these categories exist, which are very insufficient for our needs:

• [science]
• [science.categories.robotics]
• [simulation]
• [network-programming]

There should be categories related to Aerospace and simulations related to aerospace, so it might be possible to create categories like:

• [aerospace]
• [aerospace.categories.space-protocols]
• [aerospace.categories.drones]
• [aerospace.categories.unmanned-aerial-vehicles]
• [aerospace.categories.simulation]
• [aerospace.categories.protocols]
• etc.

[elpiel]

A few students worked on this idea to learn Rust for a university task.
The code can be found here: https://github.com/elpiel/crates-categories

We should take the code that retrieves all crates from a given category and integrate it into the current setup of this repository.
Ideally, we would use Rust for the whole generation process and have the option to exclude or include specific crate names in a given category.

[MoralCode]

has anyone written a jekyll plugin in rust before?

Also: would it be preferred to separate the autogenerated crates from the manually curated ones? like by listing them below the others?

Also it seems like this is a pretty simple query to https://crates.io/api/v1/crates?category=aerospaceto fetch crates. thsis may need some deduping with current crates in the curated list though

[elpiel]

We're moving this issue to the aerorust official website.
We'll be making the catalogue of crates inside the website alongside some information about the community.

[anowell]

Instead of the Ruby app we can rewrite the server using Rust.

I was just reading this and thought I'd point out that Are We Learning Yet fully removed Ruby in anowell/are-we-learning-yet#96. Notably, we switched from Jekyll to cobalt.rs and we implemented a standalone rust-based scraper to replace the plugin. See also the AWLY README to see the simpler, Ruby-free, local setup. I'd be open to explore moving the scraper out into it's own project and making it generic, configurable, or composable enough to accommodate AeroRust if we can settle on a design that would serve both projects. Or you're welcome to just copy it and modify to your liking.

[MoralCode]

@anowell wow this is interesting on a number of levels. Is there a blog post about your process of switching from ruby/jekyll to cobalt.rs? i might do the same with https://kubos-preservation-group.github.io/ if it wont change the website UI too much.

Ive also been thinking of forking AWISY/AWLY to make a community registry of all the various kubos projects and forks that people are making. So id be interested in a generic tool that can help create sites like that with enough flexibility that it could even be used by pretty much any community, regardless of if they use crates/io or not.

Sounds like yall are doing some super cool work.

[elpiel]

@anowell thanks for the info. I did follow your progress of switching to cobolt with rust scrapper.

Static site generator & Design

I did start on a slightly different path through - new design. This is why we've picked up zola instead of cobolt, plus some time ago I tried using cobolt and there were some missing pieces there. Zola on the other hand has excellent and feature-rich capabilities on top of Tera too. Bootstrap (as it's probably the easiest path) and some js packages were also necessary for the SASS and its features.

Scraper

As for the scraper - it's an excellent idea. I was thinking of using crates.io API or ... a dump (https://github.com/dtolnay/db-dump) of the DB.
I did leave the scraper to a few people that want to learn Rust, but I will also take a look at the one you have.

Let's see how these things progress and if not much happens with the scraper we can try to reuse the one in AWLY.

Categories

I think this is actually the most important part. I guess the path forward on this topic would be (as I mentioned) to have a list of Whitelist/Blacklist crates per category. The categories on crates.io are not full at all and it can take some time to figure them out and extend the list upstream and surface the crates for people to find over aligning on crates.io (something you mentioned anowell/are-we-learning-yet#120 (comment))

It might be a good opportunity to also have that in the design - our custom categories to have a special label and the crates.io ones to not have (or have a different one) but have a link to crates.io.

So far, I thought about listing each crate in its respective category (which might be tedious now that I think of it) so a better approach would be to have a list of crates listing their respective categories (mimicking the way you set them in Cargo.toml, i.e. per crate)

[elpiel]

Other categories that can be included:

• Mission planning (primarily space related) - thanks @simbleau for suggesting this category
• Motion planning (related to autonomy, also known as path planning) - see wikipedia https://en.wikipedia.org/wiki/Motion_planning