feat: build schema, inject into Wasm

[chadoh]

Wouldn't it be great to always have schemas available right on-chain?

Then tooling can plug right in. Tools like ts-codegen can build TS wrappers for on-chain contracts. Frontends like RAEN Admin can generate interactive docs. You can build interactive CLIs for specific deployed contracts. Imagine!

Perhaps this hasn't been done out of fear of bloating the size of deployed Wasm files. But if we compress the JSON with brotli, it ends up adding just a tiny number of kB for most contracts. The largest compressed schemas in the cw-plus repo weigh in at 6kB, for the multisig contracts. That's 6kB on (uncompressed) contracts weighing in at 343kB (fixed multisig) and 438kB (flex multisig). Not bad!

Since the point here is to only keep around schema'd Wasms, I just decided to update in-place. We could keep the old un-schema'd files around, though, if y'all prefer.

To do

• Single contracts, not just workspaces (honestly not sure why there are two scripts in the first place; we can do it all with Rust, get rid of the bash scripts, and make it check if root Cargo.toml defines members or not).
• Possibly make file-lookup logic more robust (currently hard-coding paths to built binaries, rather than using a library to do it, and have only tested on macOS)
• Demonstrate hooking into this with an adapted version of RAEN Admin — this already grabs an on-chain schema out of a Wasm custom section and decompresses it with brotli, but for NEAR, and the CosmWasm schema is a different shape than what the react json-schema form component consumes, so some translation is necessary.
• Add tests
• Don't panic/abort if project doesn't have a schema binary / alias. In these cases, just don't add the schema.
• Version the schema. That is, embed version info into the schema or into the Wasm custom section name, so consumers can decode it correctly even as the schema format continues to evolve.

[webmaster128]

Hey @chadoh. thank you for your proposal. This was discussed multiple times before and we don't want to embed the schema into the Wasm for various reasons:

• Larger Wasm files increase compile cost and the module loaded into memory when instantiated
• It's hard to find this data
• Not all contracts use the cargo schema alias
• The schema format is not necessarily stable

It would be much better to either let wasmd store the schemas or have them off-chain.

makes the generated Wasm files smaller, rather than larger

It would be interesting to understand why this is the case. If you tested cw-plus then probably because you compared cosmwasm/workspace-optimizer:0.12.8 (Rust 1.63) with cosmwasm/workspace-optimizer:0.12.11 (Rust 1.66).

honestly not sure why there are two scripts in the first place; we can do it all with Rust, get rid of the bash scripts, and make it check if root Cargo.toml defines members or not

Historical reasons. I'd be very happy to explore if this can be consolidated to one builder image.

[chadoh]

Thanks for the quick response, @webmaster128! Even on a draft.

And sorry to re-open a discussion that has happened before. I talked to a few people in the Cosmos ecosystem and none of them were aware of previous discussions about this, and they all liked the idea. Is there somewhere I can read the previous discussions?

I actually realized some errors in my original post and have updated it. Re: "makes the generated Wasm files smaller, rather than larger" – that's only somewhat true. While testing locally, I was only running the build_workspace script, and forgot that it does not run the wasm-opt step. After running wasm-opt, I expect that the schema'd Wasm files will be somewhat larger. I can get you good numbers on that tomorrow.

Larger Wasm files increase compile cost and the module loaded into memory when instantiated

Yes, a little. Is it worth it, though?

I'll get you numbers on how much space this adds. With optimized contract sizes on the order of 100-200kB, adding <10kB to a custom section containing the schema doesn't seem bad to me.

And the benefits are huge! Building in the CosmWasm ecosystem will become 10x easier.

5-10% "bloat" for 10x dev experience seems worth it, imo.

I can see the argument for storing them off-chain. How responsive is IPFS these days? That could be a solution. But it will slow down the optimize process, as well as the fetching-of-schemas. Given the small size of the embedded schema, I'm not sure it's worth it.

It's hard to find this data

Not sure what you mean by this.

Not all contracts use the cargo schema alias

Yes, and I actually didn't use it here, either. I used cargo run --bin schema 😛

The tongue-out face because I assume not every project implements a schema binary, either! And yeah, thanks for the reminder, I added a TODO item for this. It definitely needs to deal gracefully with such projects, and just not worry about building or injecting the schema.

The schema format is not necessarily stable

Thanks again for the reminder! I added another TODO item for this. The embedded schema absolutely needs to be versioned, so that the schema format can continue to evolve, independently of the tooling that consumes it.

[chadoh]

Here's the current output when building the cw-plus repo:

Found workspace member entries: ["packages/*", "contracts/*"]
Package directories: ["contracts/cw1-subkeys", "contracts/cw1-whitelist", "contracts/cw20-base", "contracts/cw20-ics20", "contracts/cw3-fixed-multisig", "contracts/cw3-flex-multisig", "contracts/cw4-group", "contracts/cw4-stake", "packages/controllers", "packages/cw1", "packages/cw2", "packages/cw20", "packages/cw3", "packages/cw4"]
Contracts to be built: ["contracts/cw1-subkeys", "contracts/cw1-whitelist", "contracts/cw20-base", "contracts/cw20-ics20", "contracts/cw3-fixed-multisig", "contracts/cw3-flex-multisig", "contracts/cw4-group", "contracts/cw4-stake"]

Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-subkeys"...
  1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
   Compiling cw1-whitelist v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist)
   Compiling cw1-subkeys v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-subkeys)
    Finished release [optimized] target(s) in 9.44s
  2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
   Compiling cw1-whitelist v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist)
   Compiling cw1-subkeys v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-subkeys)
    Finished dev [unoptimized + debuginfo] target(s) in 1.25s
     Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-subkeys/schema/cw1-subkeys.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw1-subkeys/schema/cw1-subkeys.json
  3. inject compressed JSON into Wasm
     updating in place: target/wasm32-unknown-unknown/release/cw1_subkeys.wasm
     compressed schema size:       3kB
     original Wasm size:         331kB
     Wasm with injected schema:  314kB

Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist"...
  1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
   Compiling cw1-whitelist v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist)
    Finished release [optimized] target(s) in 5.67s
  2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
   Compiling cw1-whitelist v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist)
    Finished dev [unoptimized + debuginfo] target(s) in 0.66s
     Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist/schema/cw1-whitelist.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw1-whitelist/schema/cw1-whitelist.json
  3. inject compressed JSON into Wasm
     updating in place: target/wasm32-unknown-unknown/release/cw1_whitelist.wasm
     compressed schema size:       2kB
     original Wasm size:         216kB
     Wasm with injected schema:  204kB

Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw20-base"...
  1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
    Finished release [optimized] target(s) in 0.05s
  2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
     Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw20-base/schema/cw20-base.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw20-base/schema/cw20-base.json
  3. inject compressed JSON into Wasm
     updating in place: target/wasm32-unknown-unknown/release/cw20_base.wasm
     compressed schema size:       3kB
     original Wasm size:         367kB
     Wasm with injected schema:  349kB

Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw20-ics20"...
  1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
    Finished release [optimized] target(s) in 0.05s
  2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
     Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw20-ics20/schema/cw20-ics20.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw20-ics20/schema/cw20-ics20.json
  3. inject compressed JSON into Wasm
     updating in place: target/wasm32-unknown-unknown/release/cw20_ics20.wasm
     compressed schema size:       2kB
     original Wasm size:         390kB
     Wasm with injected schema:  367kB

Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig"...
  1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
    Finished release [optimized] target(s) in 0.05s
  2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
   Compiling cw3-fixed-multisig v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig)
    Finished dev [unoptimized + debuginfo] target(s) in 0.83s
     Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig/schema/cw3-fixed-multisig.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig/schema/cw3-fixed-multisig.json
  3. inject compressed JSON into Wasm
     updating in place: target/wasm32-unknown-unknown/release/cw3_fixed_multisig.wasm
     compressed schema size:       6kB
     original Wasm size:         343kB
     Wasm with injected schema:  327kB

Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-flex-multisig"...
  1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
   Compiling cw3-fixed-multisig v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig)
   Compiling cw3-flex-multisig v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-flex-multisig)
    Finished release [optimized] target(s) in 16.43s
  2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
   Compiling cw3-fixed-multisig v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-fixed-multisig)
   Compiling cw3-flex-multisig v1.0.0 (/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-flex-multisig)
    Finished dev [unoptimized + debuginfo] target(s) in 1.36s
     Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw3-flex-multisig/schema/cw3-flex-multisig.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw3-flex-multisig/schema/cw3-flex-multisig.json
  3. inject compressed JSON into Wasm
     updating in place: target/wasm32-unknown-unknown/release/cw3_flex_multisig.wasm
     compressed schema size:       6kB
     original Wasm size:         438kB
     Wasm with injected schema:  417kB

Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw4-group"...
  1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
    Finished release [optimized] target(s) in 0.06s
  2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
     Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw4-group/schema/cw4-group.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw4-group/schema/cw4-group.json
  3. inject compressed JSON into Wasm
     updating in place: target/wasm32-unknown-unknown/release/cw4_group.wasm
     compressed schema size:       0kB
     original Wasm size:         242kB
     Wasm with injected schema:  226kB

Building "/Users/chadoh/code/w3ba/cw-plus/contracts/cw4-stake"...
  1. compile Wasm
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
    Finished release [optimized] target(s) in 0.05s
  2. build schema JSON
warning: profile package spec `cw1155-base` in profile `release` did not match any packages
    Finished dev [unoptimized + debuginfo] target(s) in 0.05s
     Running `/Users/chadoh/code/w3ba/cw-plus/target/debug/schema`
Removing "/Users/chadoh/code/w3ba/cw-plus/contracts/cw4-stake/schema/cw4-stake.json" …
Exported the full API as /Users/chadoh/code/w3ba/cw-plus/contracts/cw4-stake/schema/cw4-stake.json
  3. inject compressed JSON into Wasm
     updating in place: target/wasm32-unknown-unknown/release/cw4_stake.wasm
     compressed schema size:       2kB
     original Wasm size:         295kB
     Wasm with injected schema:  278kB

[chadoh]

Another complication with using something like IPFS or other off-chain storage is that uploading contracts requires "logging in" / adding keys to two different systems: the target blockchain and the schema-storage location.

[pyramation]

Hey @chadoh. thank you for your proposal. This was discussed multiple times before and we don't want to embed the schema into the Wasm for various reasons:

• Larger Wasm files increase compile cost and the module loaded into memory when instantiated

is this already compressed with the best tool for compression? We were wondering if we can use gzip or something to reduce it further?

     updating in place: target/wasm32-unknown-unknown/release/cw1_subkeys.wasm
     compressed schema size:       3kB
     original Wasm size:         331kB
     Wasm with injected schema:  314kB

• It's hard to find this data

would there be an way to access this and/or make an API endpoint to get the schema so it's not hard to find?

• Not all contracts use the cargo schema alias

maybe we can make it a part of the build so it's always run and developers don't have to manually create schemas. Seems that abstracting this step from rust devs could be a good thing.

• The schema format is not necessarily stable

Since we now have IDL versions, we can always detect the schema version. If it changes, the tooling can know how to parse the schema based on the IDL which is in the schemas now.

cc @ethanfrey this is the PR I was mentioning

[webmaster128]

honestly not sure why there are two scripts in the first place; we can do it all with Rust, get rid of the bash scripts, and make it check if root Cargo.toml defines members or not

This was now changed and will be shipped with the next release 0.15.0. Then we only have one image for x86_64 and one for ARM64.