Assemble an IPLD Working Group

[RangerMauve]

• Identify folks from teams in the PL Network that might want to attend
• Initial survey of IPLD needs from people that expressed interest
• Put together GH org or repo for tracking meetings
• Sketch up how meetings could be run based on what other groups do
• Schedule initial meeting (scoping the WG)
• Document WG goals and processes.

Over the course of LabWeek 2022, we had some useful conversations on the state of IPLD and some potential next steps.

One thing I noticed is that IPLD was historically an R&D project and over the past year or so hasn't been getting much attention as focus shifted at PL on other things.

I think now would be a good time to move IPLD from being an isolated R&D project which some others might use, and turn it into a collaborative project that aims to provide building blocks for specific teams trying to build on top of content addressed linked data in the PL Network.

To that end, I'd like to start by getting some stakeholders from various groups in PL to come together and talk about needs that we can address collectively.

One thing I think we can focus on early on is the state of IPLD in Rust which has been neglected due to nobody having time to bring it further even though there might be some useful functions that the ecosystem could use. This could be a good way to scope out what needs to be done, and to hire capacity within the IPLD team to build up APIs for other teams to make use of.

But, all that said, this is something we could discuss within the first meeting to start scoping out what the actual needs of the community are.

First step is to find who's interested in participating, and to schedule the first meeting. Ideally by the end of the month.

[b5]

I'm down to rep number zero / iroh in in this WG!

cc @2color, who has been tracking working groups, and I think would benefit from a report-out from these meetings, should they happen.

[willscott]

this is relevant to my interests. I'm happy to attend.

[10d9e]

[mikevoronov]

this is super relevant to Fluence Labs needs, would love to attend

[Zorlin]

I, too, choose you.

(I'm interested)

[alvin-reyes]

This is definitely something I want to get involve in and contribute so I'm in.

[Gozala]

I'd be down to represent dag.house in this WG!

I also think that representation from Fission would be of great benefit, especially since they are rewriting pretty much everything in Rust! @expede is this something you or someone else on the team would be interested in ?

[Gozala]

I would also mention @vmx who started attempt to reconcile CIDv2 / Fat Pointers effort

[en0ma]

I am so much interested in this

[expede]

@Gozala oh yeah, I'm sure we'll have interested folks at Fission 💯

[vmx]

I'm surely also interested :)

[RangerMauve]

So, just some more context from the IPLD team side of things.

Right now we don't have any staff other than Rod (and I guess myself) and even though we'd like to build up more capacity, we need to first figure out what those folks would actually be doing.

With that in mind, I'd like our first meeting or two to discuss some of the needs we have in the ecosystem, in particular stuff that existing teams don't have time to refine or do themselves. From there it'd be nice to scope out some things that the IPLD team could work on which would enable others in the ecosystem to build on linked data.

Alternately we could look at how existing teams could commit developer time to working on "common good" things and skip hiring up devs for the IPLD team.

It might be good to do a combination of brainstorming some more urgent things like accessible docs and Rust implementations, as well as "nice to haves" like better high level tooling that would make some work easier but hasn't been a priority for anyone else to implement.

It might also be good to see if there's far reaching changes to IPLD that is needed by a bunch of teams like rethinking the feature set of IPLD Schemas to better serve different groups, or seeing if some of the Datalog stuff that Fission is working on could be standardized across the ecosystem.

That said, would folks have time for a call on Wednesday November 23rd at 18:00 UTC? I'm thinking for the first one it'd be good to get an hour and a half or even two hours so we can take our time getting on the same page on our contexts before talking about needs.

Gimmi a 👍 if it works or a 👎 if it doesn't.

[willscott]

I believe that the move the bytes data transfer working group is scheduled for 18:30 UTC that day: https://lu.ma/8kk9i628

[RangerMauve]

Ty @willscott How about same time Thursday the 23rd?

[b5]

With that in mind, I'd like our first meeting or two to discuss some of the needs we have in the ecosystem, in particular stuff that existing teams don't have time to refine or do themselves.

Any chance we can gather these needs async? I think this meeting will be more successful if we solicited IPLD needs beforehand & then reviewed as a group

[willscott]

Ty @willscott How about same time Thursday the 23rd?

That's thanksgiving in the US

[RangerMauve]

Ty @willscott How about same time Thursday the 23rd?

That's thanksgiving in the US

Yikes. 😅 Any suggestions for times in November? Ideally after 15:00 UTC and before 04:00 UTC (which is my availability).

With that in mind, I'd like our first meeting or two to discuss some of the needs we have in the ecosystem, in particular stuff that existing teams don't have time to refine or do themselves.

Any chance we can gather these needs async? I think this meeting will be more successful if we solicited IPLD needs beforehand & then reviewed as a group

Great idea, here's a Hackmd with a bit of a template folks can follow to add themselves and their needs.

https://hackmd.io/ugjp9NdTSoy91hcd5rsPxg

[willscott]

• moving up half an hour from your initial proposal for the hour before the move the bytes WG could work in general (17:30 - 18:30 UTC)
• alternatively starting after move the bytes (19:30 UTC - 20:30 UTC) could work though gets late for europeans

[rvagg]

I'm probably out for a live sync given timezones, but I also suspect it might be difficult to regularly get folks together for sync chats? Happy to look at notes and participate async; that's my preferred mode anyway tbh.

[RangerMauve]

Oo, that's a good idea. Maybe we can commit to doing async chats on a given day just to have a deadline for folks to add comments ahead of time?

[RangerMauve]

Would it make sense to meet the week after thanksgiving in the US? Maybe some time on the 29th like 16:00 UTC?

Also, if folks could post their IPLD needs and thoughts sooner than later that would be very much appreciated.

Just copy-paste the template in this hackmd and add your name, affiliation, and list of needs/thoughts. https://hackmd.io/ugjp9NdTSoy91hcd5rsPxg

[aschmahmann]

I'm interested as well, although also won't be able to participate synchronously for a while.

[dmitrizagidulin]

I'd like to paticipate! :)

[RangerMauve]

Doesn't look like anybody has added needs to the hackmd and it doesn't seem like folks are super eager for finding a time to meet. Should we maybe postone this for post-holidays around Mid-January?

In the meantime it'd be very helpful if folks could post IPLD neads in here. If we don't get more posts, I'll try to schedule some one on ones and collect stuff through that. 😁

[AlexxNica]

Awesome initiative, @RangerMauve! 👏

I'm up for a meeting even if it's just us @RangerMauve. We could start planning the logistics, documentation, etc. What do you think? 😁

[RangerMauve]

@AlexxNica Ty for the offer! I was thinking of doing some one on ones to start and writing the notes down. I'll be reaching out to folks that expressed interest in this thread asynchornously. 😁

[cdata]

Hi all, @vmx showed up in our Discord and shared this thread with me. We're using libipld extensively in our project https://github.com/subconsciousnetwork/noosphere.

I would be happy to represent the perspective of Subconscious Inc.

[RangerMauve]

Hey folks, here's a big post summarizing the interviews I made and a proposal for what we should do next.

---

IPLD WG Proposal

TL;DR

People want to use IPLD and there are needs within projects that can be addressed by it. Addressing the needs will act as a force multiplier for speeding up dev time in the ecosystem and saving $ on dev time. We need people dedicated to tooling and docs to meet those needs. The WG will exist to collect those needs and help fulfill them.

Summary of interviews

The overall feeling is that people are excited to use IPLD and see it as an important building block for applications in the PL Network. However, for those that are unconvinced, it seems to be a matter of the "Why" of IPLD not being explicit in the documentation / framing.

On top of that, there is consensus on IPLD documentation lacking clear onboarding for people new to IPLD, and it being hard to search through. Specifically onboarding for devs that have some idea of IPFS, and docs for devs that already know about web2 development wanting to understand web3/IPLD.

Tooling for Rust is on a lot of people's mind and while there's some out there it leaves much to be desired. IPLD in different languages should integrate with how those languages work rather than sticking to golang paradigms accross the board. Rust is also important for the WebAssembly story of integrating IPLD in that Rust can be compiled to WASM and exposed to different "host" langages more easily than other languages like JS or Golang.

The core of IPLD seems to be pretty stable and not many folks want to change it. This involves the data model and the popular codecs being used like dag-cbor. There is however room for improvment around pathing and schema generics. Higher level developer tools like visualizers for IPLD data would make it easier for folks to work with the data as well. There exist some tools already, but most of them are woefully out of date and need an overhaul or to be replaced entirely. More standardization on ADLs in languages and use out-of-the box is important. In particular HAMTs, Prolly Trees, and encrypted DAGs.

People generally have time for once a month meetings, at most every other week. This is with the caveat that they would prefer to attend meetings which are talking about things that are specifically relevant to what they're working on. However even if they don't attend folks are interested in recordings and transcripts being made of the calls so that they can be reviewed async.

IPLD Vision

IPLD should make it easier for applications to author and exchange structed, linked, content-addressed data. In particular it should be easy to load data from IPLD (via schemas etc) in different programming languages, and we should have answers for when an application needs to deal with efficiently authoring and loading a large dataset.

In order to enable this, the IPLD Team will exist as a "service shop" for enabling teams to build on top of IPLD in order to save everyone on time to getting their applications into users hands and to make it easier to onboard even more users of IPLD and FIL. As the FIL ecosystem grows to make ingestion and egreess of data smoother, having an answer to "How do I get my data onto FIL?" becomes more important.

Without IPLD, applications using structured data in decentralized systems would need to reinvent something like IPLD if they want to make their data interoperable and portable. It's not just IPLD at the codec layer, but it being used with replicating data, visualizing it, and building communties that build on top of each other. Having higher level APIs with ADLs for large datasets means that applications can plug them into their stack and save on time and money while delivering their vision to users.

Purpose

The working group's focus is to find needs accross groups building on top of linked data in the PL Network, and to find ways of fulfilling those needs either by members contributing their time or by hiring capacity into the IPLD team.

The working group exists to enable IPLD development which supports cross-cutting needs accross groups building on top of it in the PL Network.

E.g. Work on tooling, documentation, building relations between groups.

Format

• Monthly or twice-a-montly calls
• Scope topics ahead of time via GH discussions
• Topics should be actionable (e.g. "Decide on who will work on this new schema feature")
• Video recordings of calls
• Stream to Youtube
• Automatic transcriptions (or a person after the fact)
• Notes recorded in Github discussions

Template

• Pick a facilitator for the meeting at the end of the last meeting
• The facilitator should start asking for topics to discuss a week or two ahead of the meeting
• Instead of passively waiting for comments, they should actively reach out to people based on past meetings or general needs
• Ensure topics have at least two people interested in discussing them and that it wouldn't be easier to discuss async
• Propose agenda ahead of the meeting along with a template for notes
• Ask for general checkins to be written into the notes ahead of the meeting to save on time
• Timebox agenda items ahead of time and have further discussion moved async or to have smaller calls
• Review agenda and checkins at start of call
• Review results of action items from last call
• Proceed through the agenda
• After getting through the agenda
  • collect action items / outcomes
  • collect potential topics for next time
  • figure out the next facilitator
• After the call, the facilitator should paste the notes on Github and get a review before merging it in

Initial Topics

• IPLD Team Hiring
• Docs needs
  • Improve searchability
  • Onboarding for beginners
  • More concrete examples (with APIs?)
  • Clear link to "official" libraries in different languages
• Rust needs
  • Support for Schemas in encoding/decoding
  • Straightforward path to WASM
  • Generate FFI bindings
• High level API that exposes all functionality
• Schema Generics
  • Tracking issue: Support for generics in schema #248
  • Adding syntax to the DMT and DSM
  • Who can work on this?
• Review state of selectors and whether they're meeting our needs
  • Talk about alternatives like Datalog
• IPLD URLs in more places
• IPLD Database needs (prolly trees, etc)
• DAG encryption between projects (reuse what WNFS is doing?)

IPLD Team Rampup

Why hire a team?

IPLD is a core part of all the technology in the PL Ecosystem and most projects are using it one way or another.

Currently, even though people want to use it, they end up wasting a lot of time trying to get it to do what they need. In particular, bindings between data structures in the host programming language and IPLD can be found lacking in anything outside of Go-IPLD-Prime (and even that might need some more attention).

All that said, people are excited to use IPLD and can see it being useful if there were improvements being made to the DX and docs.

However, even though most groups see value in improving IPLD, they don't generally have time to contribute to this development on their own time due to budget and timeline constraints.

This is an opportunity for PL to step in and act as a force multiplier in saving people on hours of developer time accross the board.

Roles

Specifically, there is a need for the following roles to be filled in as support for the ecosystem.

Firstly, we would need a dedicated TMP to wrangle all of the people needed to work on IPLD. They don't necessarily need to have deep knowledge of IPLD or a specific vision ahead of time. This will be necessary to keep the development going smoothly and to ensure things brought up at the WG will be followed up on.

Next, we would need a technical writer + docs person. A big part of why people are spending more time on trying to figure out IPLD than is necessary is that our docs aren't currently geared towards newcomers and are more focused on abstract documentation of all the features of IPLD. This person's first job would be to fill in some of the TODOs and make the site more searchable. There has already been work started on the IPLD website refactoring to doks PR, however we need also need to write new sections and better intros for beginners to IPLD.

We will also need some developers to work on improving tooling. The most pressing needs are in the Rust IPLD library which is starting to become more important due to all the Rust work being done at Fission, Outercore, FVM, et al. Although there is some rust tooling in place, it's severely lagging behind in important features like IPLD Schemas and ADLs. This is where we'd need one or two Rust engineers to start working on bringing the tooling on par (or better!) with what we have in Go and JavaScript.

Speaking of JavaScript, DAGHaus (and others) are actively working on web-facing tooling and are building on top of things like IPLD Schemas and finding needs for functionality. It would be useful to get another engineer working purely on the JavaScript side of IPLD tooling and making sure to keep up with the needs of the ecosystem and to quickly unblock problems that folks are facing. This would also involve finishing up migrating our codebases to ESM-only and helping with building higher level APIs for working with IPLD such as js-ipld-url-resolve.

Finally, it would be useful to have a dedicated Golang developer to keep up with needs in the Go implementations of IPLD and to work on improving existing tooling.

Outside of development, it would be useful to have a community-focused person who will be there to help coordinate meetings and engage with people on social media and in the various PL chats. This person may also be important for reaching out to other Open Source communities such as Microsoft and the Linux foundation to build up relationships and get more groups involved in building on top of IPLD and its specifications.

Conclusion / Next Steps

Hopefully this will give folks a sense of the needs the ecosystem has for IPLD and some ideas on how to address them.

Our next step is to start scoping out the initial working group call and it's topic and starting to fulfill some of these needs so we can advance the entire ecosystem together.

[RangerMauve]

Hey folks,

My contract with PL for work on IPLD is expiring so I won't have time to continue my work on pushing this forward.

I think that the post I made above still makes sense, and it might be good for some of the folks that expressed interest to take the wheel and organize the initial call.

I also still believe strongly in the initial suggestions of coordinating work on docs and Rust as something the WG should focus on to start.

I'm not going to be leaving the ecosystem or anything like that so I might be able to attend and contribute an hour here or there of my spare time on this, but I won't be able to dedicate as much time since I'll need to fill it with other contracts.

If anyone has questions feel free to ask them here or hop into my DMs/email me. :)

[sstelfox]

If this gets revived I would like to be involved. Was there anyone active inside of PL still pushing this?

[vmx]

Currently no one is pushing for it. Though there's at least an IPLD community call every 4 weeks, where you can meet other IPLD/Multiformats interested folks.