[wip] feat: Halo2 compatible vanilla Hashers #1560

DrPeterVanNostrand · 2022-02-03T17:39:16Z

Pasta compatible vanilla hashers
Pasta compatible vanilla proofs
GPU code generic over a field F

DrPeterVanNostrand · 2022-02-28T17:30:31Z

.circleci/config.yml

-            cargo +$(cat rust-toolchain) test -p storage-proofs-update --features isolated-testing --release test_empty_sector_update_circuit_8kib
-            cargo +$(cat rust-toolchain) test -p storage-proofs-update --features isolated-testing --release test_empty_sector_update_circuit_16kib
-            cargo +$(cat rust-toolchain) test -p storage-proofs-update --features isolated-testing --release test_empty_sector_update_circuit_32kib
+            cargo +$(cat rust-toolchain) test -p storage-proofs-update --features isolated-testing --release circuit_1kib


Note: this change was necessary because I changed the ESU and ESU-Poseidon test names.

DrPeterVanNostrand · 2022-03-01T18:07:46Z

fil-proofs-param/src/bin/paramcache.rs

+fn cache_porep_params<Tree>(porep_config: PoRepConfig)
+where
+    Tree: 'static + MerkleTreeTrait,
+    <Tree::Hasher as Hasher>::Domain: Domain<Field = Fr>,


Note: Groth16 params can only be generated for the field Fr.

DrPeterVanNostrand · 2022-03-01T18:08:37Z

fil-proofs-param/src/bin/paramcache.rs

-) {
+fn cache_empty_sector_update_params<Tree>(porep_config: PoRepConfig)
+where
+    Tree: 'static + MerkleTreeTrait<Hasher = TreeRHasher<Fr>>,


ESU can only use TreeRs whose hasher is TreeRHasher.

DrPeterVanNostrand · 2022-03-01T18:11:21Z

fil-proofs-tooling/src/bin/benchy/window_post.rs

+) -> anyhow::Result<((u64, u64), (u64, u64), (u64, u64))>
+where
+    Tree: 'static + MerkleTreeTrait,
+    <Tree::Hasher as Hasher>::Domain: Domain<Field = Fr>,


Note: for now, only bench using the field Fr.

DrPeterVanNostrand · 2022-03-01T18:11:49Z

fil-proofs-tooling/src/bin/circuitinfo/main.rs

+fn get_porep_info<Tree>(porep_config: PoRepConfig) -> CircuitInfo
+where
+    Tree: 'static + MerkleTreeTrait,
+    <Tree::Hasher as Hasher>::Domain: Domain<Field = Fr>,


Note: Groth16 circuits can only be created using the field Fr.

DrPeterVanNostrand · 2022-03-01T18:13:04Z

fil-proofs-tooling/src/bin/gen_graph_cache/main.rs

+) -> Result<()>
+where
+    Tree: 'static + MerkleTreeTrait,
+    <Tree::Hasher as Hasher>::Domain: Domain<Field = Fr>,


Note: for now, only generate parent cache for the production field Fr.

DrPeterVanNostrand · 2022-03-01T18:14:04Z

filecoin-hashers/Cargo.toml

@@ -32,7 +34,7 @@ opencl = ["bellperson/opencl", "neptune/opencl"]

 # available hashers
 blake2s = ["blake2s_simd"]
-poseidon = ["neptune", "lazy_static"]
+poseidon = ["neptune", "lazy_static", "typemap"]


Note: typemap allows us to lookup the Poseidon constants for a field and arity.

DrPeterVanNostrand · 2022-03-01T18:16:00Z

filecoin-hashers/src/poseidon_types.rs

@@ -26,6 +30,60 @@ lazy_static! {
    pub static ref POSEIDON_CONSTANTS_11: PoseidonConstants::<Fr, U11> = PoseidonConstants::new();
    pub static ref POSEIDON_MD_CONSTANTS: PoseidonConstants::<Fr, PoseidonMDArity> =
        PoseidonConstants::new();
+    pub static ref POSEIDON_CONSTANTS_2_PALLAS: PoseidonConstants::<Fp, U2> =


Note: create Poseidon constants for the three hasher fields; put the constants into a lookup table keyed by (field, arity)

DrPeterVanNostrand · 2022-03-01T18:17:21Z

filecoin-hashers/src/types.rs

@@ -27,21 +28,49 @@ pub trait Domain:
    + Eq
    + Send
    + Sync
+    // TODO (halo): remove once we have Pasta GPU support.


Note: Domain: From<Fr> + Into<Fr> is used as a stopgap to cause GPU code to panic when using Pasta fields; these bounds should be removed once the GPU work has finished.

DrPeterVanNostrand · 2022-03-03T18:20:30Z

fr32/Cargo.toml

@@ -20,6 +20,8 @@ blstrs = "0.4.0"
 bitvec = "0.17"
 criterion = "0.3"
 itertools = "0.9"
+# pasta_curves = "0.3.0"


TODO: change before merging

cryptonemo · 2022-03-09T12:57:22Z

filecoin-hashers/Cargo.toml

+# neptune = { version = "5.1.0", optional = true, features = ["arity2", "arity4", "arity8", "arity11", "arity16", "arity24", "arity36"] }
+# pasta_curves = "0.3.0"
+neptune = { git = "https://github.com/filecoin-project/neptune", branch = "wip-fr-as-trait", optional = true, features = ["arity2", "arity4", "arity8", "arity11", "arity16", "arity24", "arity36"] }
+pasta_curves = { git = "https://github.com/vmx/pasta_curves", branch = "ec-gpu", features = ["gpu"] }


What is status on dep branches here? I saw this one updated (https://github.com/filecoin-project/neptune/pull/135/files)?

storojs72 · 2022-03-09T15:08:15Z

I'm not enough qualified to judge on the essence of the introduced changes... But this PR seems to be too monstrous (+4,912 −2,710). If this huge change is going to be merged, probably it is better to split this single PR on multiply PRs and introduce / review / merge them incrementally. WDYT, @DrPeterVanNostrand?

DrPeterVanNostrand · 2022-03-09T19:37:29Z

@storojs72 You're right, it's an ugly PR (which I'm also not happy about). I'm totally in favor of small precise PRs, just this one is kind of difficult to merge incrementally.

The reason why it hasn't been split into multiple features/PRs is because this PR essentially adds a single feature to rust-fil-proofs (i.e. to make our hash functions compatible with both the BLS12-381 and Pasta scalar fields). Almost everything in rust-fil-proofs is downstream from the hash functions; making a small change to the hash function interface requires updating the interface of everything that uses it.

Also, much of this PR repeats the same 2 or 3 interface changes. For example, adding this one additional trait bound probably accounts for 500 additional lines in the diff.

impl<Tree> SomeType<Tree>
where
    <Tree::Hasher as Hasher>::Domain: Domain<Field = Fr>,

cryptonemo · 2022-03-09T21:15:39Z

I'm not enough qualified to judge on the essence of the introduced changes... But this PR seems to be too monstrous (+4,912 −2,710). If this huge change is going to be merged, probably it is better to split this single PR on multiply PRs and introduce / review / merge them incrementally. WDYT, @DrPeterVanNostrand?

@storojs72 Appreciate the feedback, and thanks for taking a look! This particular PR we knew was going to be a monster and the team had a sync review session for calibration on it not that long ago. These types of PRs don't happen necessarily all that often (thankfully), but we are in the middle of a sort of large re-factor for pending future work.

cryptonemo · 2022-03-09T21:17:07Z

@DrPeterVanNostrand Still reviewing, but I expect to be complete by tomorrow (just FYI)

cryptonemo

This generally all looks good to me. The only concern is how we're handling the dependencies that haven't been merged/released yet.

DrPeterVanNostrand · 2022-03-24T18:41:39Z

filecoin-hashers/Cargo.toml

 ff = "0.11.0"
 anyhow = "1.0.34"
 serde = "1.0.117"
 rand = "0.8.0"
-
-neptune = { version = "5.1.0", optional = true, features = ["arity2", "arity4", "arity8", "arity11", "arity16", "arity24", "arity36"] }
+neptune = { git = "https://github.com/filecoin-project/neptune", branch = "fr-as-trait", optional = true, features = ["bls", "pasta", "arity2", "arity4", "arity8", "arity11", "arity16", "arity24", "arity36"] }


TODO: change neptune dependencies once this PR is merged.

We decided to keep that branch for now until there's a proper pasta_curves release.

DrPeterVanNostrand changed the title ~~[wip] feat(filecoin-hashsers): add Halo2 compatible Sha256 and Poseidon Hashers~~ [wip] feat(filecoin-hashers): add Halo2 compatible Sha256 and Poseidon Hashers Feb 3, 2022

DrPeterVanNostrand mentioned this pull request Feb 3, 2022

feat(filecoin-hashers): Pasta hashers #1491

Closed

3 tasks

DrPeterVanNostrand changed the title ~~[wip] feat(filecoin-hashers): add Halo2 compatible Sha256 and Poseidon Hashers~~ [wip] feat(filecoin-hashers): add Halo2 compatible Sha256 and Poseidon (vanilla) Hashers Feb 3, 2022

DrPeterVanNostrand force-pushed the halo-hashers branch 7 times, most recently from 137c053 to ad5955e Compare February 11, 2022 14:34

DrPeterVanNostrand force-pushed the halo-hashers branch from ad5955e to 1cbb33f Compare February 14, 2022 19:41

DrPeterVanNostrand force-pushed the halo-hashers branch 5 times, most recently from f1af449 to a8be7e8 Compare February 25, 2022 18:25

DrPeterVanNostrand changed the title ~~[wip] feat(filecoin-hashers): add Halo2 compatible Sha256 and Poseidon (vanilla) Hashers~~ [wip] feat(filecoin-hashers): Halo2 compatible (non-circuit) Hashers Feb 25, 2022

DrPeterVanNostrand force-pushed the halo-hashers branch from a8be7e8 to a088c29 Compare February 28, 2022 17:27

DrPeterVanNostrand commented Feb 28, 2022

View reviewed changes

DrPeterVanNostrand force-pushed the halo-hashers branch 3 times, most recently from f992e1b to 220308f Compare March 1, 2022 18:06

DrPeterVanNostrand commented Mar 1, 2022

View reviewed changes

DrPeterVanNostrand force-pushed the halo-hashers branch from 4003a25 to 952341e Compare March 3, 2022 18:18

DrPeterVanNostrand commented Mar 3, 2022

View reviewed changes

DrPeterVanNostrand force-pushed the halo-hashers branch 10 times, most recently from 3f4d23b to ab090d7 Compare March 7, 2022 19:39

cryptonemo reviewed Mar 9, 2022

View reviewed changes

DrPeterVanNostrand force-pushed the halo-hashers branch 2 times, most recently from 069954d to e769deb Compare March 9, 2022 19:04

DrPeterVanNostrand force-pushed the halo-hashers branch 2 times, most recently from 2fd142d to 84496a9 Compare March 10, 2022 19:19

cryptonemo previously approved these changes Mar 10, 2022

View reviewed changes

feat: add Pasta compatible hashers

6c28dd0

DrPeterVanNostrand dismissed cryptonemo’s stale review via 93e8ec5 March 16, 2022 12:40

DrPeterVanNostrand force-pushed the halo-hashers branch from 84496a9 to 93e8ec5 Compare March 16, 2022 12:40

feat: pasta gpu support

7edf855

DrPeterVanNostrand force-pushed the halo-hashers branch from 93e8ec5 to 7edf855 Compare March 16, 2022 12:43

build: use pasta_curves version implementing ec_gpu::GpuField

ee726db

DrPeterVanNostrand commented Mar 24, 2022

View reviewed changes

feat: upgrade bellperson (and neptune)

020b406

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[wip] feat: Halo2 compatible vanilla Hashers #1560

[wip] feat: Halo2 compatible vanilla Hashers #1560

DrPeterVanNostrand commented Feb 3, 2022 •

edited

DrPeterVanNostrand Feb 28, 2022 •

edited

DrPeterVanNostrand Mar 1, 2022

DrPeterVanNostrand Mar 1, 2022

DrPeterVanNostrand Mar 1, 2022

DrPeterVanNostrand Mar 1, 2022

DrPeterVanNostrand Mar 1, 2022

DrPeterVanNostrand Mar 1, 2022

DrPeterVanNostrand Mar 1, 2022

DrPeterVanNostrand Mar 1, 2022

DrPeterVanNostrand Mar 3, 2022

cryptonemo Mar 9, 2022

storojs72 commented Mar 9, 2022 •

edited

DrPeterVanNostrand commented Mar 9, 2022

cryptonemo commented Mar 9, 2022

cryptonemo commented Mar 9, 2022

cryptonemo left a comment

DrPeterVanNostrand Mar 24, 2022

vmx Apr 4, 2022

[wip] feat: Halo2 compatible vanilla Hashers #1560

Are you sure you want to change the base?

[wip] feat: Halo2 compatible vanilla Hashers #1560

Conversation

DrPeterVanNostrand commented Feb 3, 2022 • edited

DrPeterVanNostrand Feb 28, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

storojs72 commented Mar 9, 2022 • edited

DrPeterVanNostrand commented Mar 9, 2022

cryptonemo commented Mar 9, 2022

cryptonemo commented Mar 9, 2022

cryptonemo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

DrPeterVanNostrand commented Feb 3, 2022 •

edited

DrPeterVanNostrand Feb 28, 2022 •

edited

storojs72 commented Mar 9, 2022 •

edited