The progress of convolutional layer verification is abnormally slow

[ExcellentHH]

Question

Hi, I recently used ezkl to verify a single convolutional layer and found that the verification speed is very slow. I would like to ask if this is normal. The details are as follows:

The model：

Get setting.json through ezkl calibrate-settings:

{"run_args":{"tolerance":{"val":0.0,"scale":1.0},"input_scale":7,"param_scale":7,"scale_rebase_multiplier":10,"bits":1,"logrows":22,"variables":[["batch_size",1]],"input_visibility":"Private","output_visibility":"Public","param_visibility":"Private"},"num_constraints":3395864,"total_const_size":0,"model_instance_shapes":[[1,1,224,224]],"model_output_scales":[14],"module_sizes":{"poseidon":[0,[0]],"elgamal":[0,[0]]},"required_lookups":[],"check_mode":"UNSAFE","version":"0.0.0"}

Note that the num_constraints is 3395864

The execution of ezkl mock took 26894s:

[*] [0s, ezkl] - 
 |  
 |          ███████╗███████╗██╗  ██╗██╗
 |          ██╔════╝╚══███╔╝██║ ██╔╝██║
 |          █████╗    ███╔╝ █████╔╝ ██║
 |          ██╔══╝   ███╔╝  ██╔═██╗ ██║
 |          ███████╗███████╗██║  ██╗███████╗
 |          ╚══════╝╚══════╝╚═╝  ╚═╝╚══════╝
 |  
 |          -----------------------------------------------------------
 |          Easy Zero Knowledge for the Lyrical.
 |          -----------------------------------------------------------
 |  
 |          
[*] [0s, ezkl] - command: 
 |   {
 |    "command": {
 |      "Mock": {
 |        "model": "model.ezkl",
 |        "settings_path": "settings.json",
 |        "witness": "witness.json"
 |      }
 |    }
 |  }
[*] [0s, ezkl::graph] - public inputs lengths: [50176]
[*] [0s, ezkl::execute] - Mock proof
[*] [0s, ezkl::graph::model] - configuring model
[*] [0s, ezkl::graph::model] - model layout...
[*] [6s, ezkl::graph::model] - model has 3094862 assigned rows
[*] [26889s, ezkl::graph::model] - model has 3094862 assigned rows
[*] [26894s, ezkl] - succeeded

The execution of ezkl setup took 55207s:

[*] [0s, ezkl] - 
 |  
 |          ███████╗███████╗██╗  ██╗██╗
 |          ██╔════╝╚══███╔╝██║ ██╔╝██║
 |          █████╗    ███╔╝ █████╔╝ ██║
 |          ██╔══╝   ███╔╝  ██╔═██╗ ██║
 |          ███████╗███████╗██║  ██╗███████╗
 |          ╚══════╝╚══════╝╚═╝  ╚═╝╚══════╝
 |  
 |          -----------------------------------------------------------
 |          Easy Zero Knowledge for the Laconic.
 |          -----------------------------------------------------------
 |  
 |          
[*] [0s, ezkl] - command: 
 |   {
 |    "command": {
 |      "Setup": {
 |        "compiled_model": "model.ezkl",
 |        "pk_path": "pk.key",
 |        "settings_path": "settings.json",
 |        "srs_path": "perpetual-powers-of-tau-raw-22",
 |        "vk_path": "vk.key"
 |      }
 |    }
 |  }
[*] [0s, ezkl::pfsys::srs] - loading srs from "perpetual-powers-of-tau-raw-22"
[*] [0s, ezkl::execute] - downsizing params to 22 logrows
[*] [0s, ezkl::graph::model] - configuring model
[*] [1s, ezkl::graph::model] - model layout...
[*] [7s, ezkl::graph::model] - model has 3094862 assigned rows
[*] [26801s, ezkl::graph::model] - model has 3094862 assigned rows
[*] [26837s, ezkl::pfsys] - VK took 26836.289
[*] [26837s, ezkl::graph::model] - configuring model
[*] [26837s, ezkl::graph::model] - model layout...
[*] [26843s, ezkl::graph::model] - model has 3094862 assigned rows
[*] [55121s, ezkl::graph::model] - model has 3094862 assigned rows
[*] [55189s, ezkl::pfsys] - PK took 28352.770
[*] [55189s, ezkl::pfsys] - saving verification key 💾
[*] [55189s, ezkl::pfsys] - saving proving key 💾
[*] [55207s, ezkl] - succeeded

The execution of ezkl prove took 153.11 and the execution of ezkl verify took 0.430.

I'm wondering why the setup stage is taking such a long time (previously when I was testing constraints of the same or even larger scale using ezkl, it took less time than this model). Is it because convolutional operations occupy more columns, or because of the larger input/output sizes or some other reason?

Is there any solution or trick to improve efficiency?

Thank you very much!

[alexander-camuto]

can you send a sample of the model ? will take a look :)
Just send a zipped version of the model. Also can you confirm what github tag / commit you are using ?

[ExcellentHH]

@alexander-camuto Hi, I updated the local code to the version tagged 1.18.0(commit: 2229625) and re-tested the first experiment. The abnormally slow setup process occurred again - I waited for ten minutes but the setup was still incomplete, so I killed the setup process. Afterward, I replaced the halo2_proofs = { git = "https://github.com/zkonduit/halo2", branch= "ac/parallel-lookup-permute", default_features = false, features = ["thread-safe-region"]} with halo2_proofs = { git = "https://github.com/zkonduit/halo2", branch= "ac/parallel-lookup-permute", default_features = false } in the Cargo.toml, and the setup was able to complete in 147s.

If possible, I'd like to ask what might be the cause of this type of issue occurring. Thank you in advance for your guidance!

[alexander-camuto]

Ok sweet thanks for confirming ! Will investigate and try and give you a clear answer :)

[ExcellentHH]

Looking forward to your reply, and thanks again！

[alexander-camuto]

so we have an algo for creating copy constraints which is very memory efficient -- however, because it needs to keep track of ordering the constraints were created in, for a very large number of copy constraints (this is the case with conv layers things), things can get unbearably slow. So it seems like that for this very particular instance -- that algo is no good and the default algorithm in Halo2 is the better choice

ty very much for catching this

[ExcellentHH]

Got it. Thank you for your patient explanation!

[alexander-camuto]

I've swapped it such that low memory targets (eg. wasm etc...) use the memory efficient algo -- else the default one gets used

[ExcellentHH]

Sounds great!