Notebook tests are flaky?

[fdmalone]

The notebook tests in the CI seem to randomly fail every so often. Rerunning them seems to work. Any idea what's going on? Heres's a recent traceback:

[PosixPath('Adjoint.ipynb'), PosixPath('Autodoc.ipynb'), PosixPath('Controlled.ipynb'), PosixPath('Protocols.ipynb'), PosixPath('_infra/Bloqs-Tutorial.ipynb'), PosixPath('_infra/composite_bloq.ipynb'), PosixPath('_infra/gate_with_registers.ipynb'), PosixPath('bloqs/arithmetic/addition.ipynb'), PosixPath('bloqs/arithmetic/comparison.ipynb'), PosixPath('bloqs/arithmetic/conversions.ipynb'), PosixPath('bloqs/arithmetic/error_analysis_for_fxp_arithmetic.ipynb'), PosixPath('bloqs/arithmetic/multiplication.ipynb'), PosixPath('bloqs/arithmetic/sorting.ipynb'), PosixPath('bloqs/arithmetic/t_complexity_of_comparison_gates.ipynb'), PosixPath('bloqs/basic_gates/cnot.ipynb'), PosixPath('bloqs/basic_gates/global_phase.ipynb'), PosixPath('bloqs/basic_gates/hadamard.ipynb'), PosixPath('bloqs/basic_gates/rotation.ipynb'), PosixPath('bloqs/basic_gates/s_gate.ipynb'), PosixPath('bloqs/basic_gates/states_and_effects.ipynb'), PosixPath('bloqs/basic_gates/su2_rotation.ipynb'), PosixPath('bloqs/basic_gates/t_gate.ipynb'), PosixPath('bloqs/basic_gates/toffoli.ipynb'), PosixPath('bloqs/block_encoding.ipynb'), PosixPath('bloqs/chemistry/df/double_factorization.ipynb'), PosixPath('bloqs/chemistry/pbc/first_quantization/first_quantization.ipynb'), PosixPath('bloqs/chemistry/pbc/first_quantization/projectile/projectile.ipynb'), PosixPath('bloqs/chemistry/resource_estimation.ipynb'), PosixPath('bloqs/chemistry/sf/single_factorization.ipynb'), PosixPath('bloqs/chemistry/sparse/sparse.ipynb'), PosixPath('bloqs/chemistry/thc/thc.ipynb'), PosixPath('bloqs/chemistry/trotter/grid_ham/trotter.ipynb'), PosixPath('bloqs/chemistry/trotter/grid_ham/trotter_costs.ipynb'), PosixPath('bloqs/chemistry/trotter/hubbard/hubbard.ipynb'), PosixPath('bloqs/chemistry/trotter/ising/ising.ipynb'), PosixPath('bloqs/chemistry/trotter/trotterized_unitary.ipynb'), PosixPath('bloqs/chemistry/writing_algorithms.ipynb'), PosixPath('bloqs/data_loading/qrom.ipynb'), PosixPath('bloqs/factoring/factoring-via-modexp.ipynb'), PosixPath('bloqs/factoring/mod_exp.ipynb'), PosixPath('bloqs/factoring/mod_mul.ipynb'), PosixPath('bloqs/hamiltonian_simulation/hamiltonian_simulation_by_gqsp.ipynb'), PosixPath('bloqs/hubbard_model.ipynb'), PosixPath('bloqs/mcmt/and_bloq.ipynb'), PosixPath('bloqs/mcmt/multi_control_multi_target_pauli.ipynb'), PosixPath('bloqs/multiplexers/apply_gate_to_lth_target.ipynb'), PosixPath('bloqs/multiplexers/select_pauli_lcu.ipynb'), PosixPath('bloqs/multiplexers/unary_iteration.ipynb'), PosixPath('bloqs/phase_estimation/lp_resource_state.ipynb'), PosixPath('bloqs/phase_estimation/text_book_qpe.ipynb'), PosixPath('bloqs/phase_estimation_of_quantum_walk.ipynb'), PosixPath('bloqs/qft/approximate_qft.ipynb'), PosixPath('bloqs/qft/two_bit_ffft.ipynb'), PosixPath('bloqs/qsp/generalized_qsp.ipynb'), PosixPath('bloqs/qubitization_walk_operator.ipynb'), PosixPath('bloqs/reflection.ipynb'), PosixPath('bloqs/rotations/phasing_via_cost_function.ipynb'), PosixPath('bloqs/rotations/quantum_variable_rotation.ipynb'), PosixPath('bloqs/state_preparation/prepare_uniform_superposition.ipynb'), PosixPath('bloqs/state_preparation/state_preparation_alias_sampling.ipynb'), PosixPath('bloqs/state_preparation/state_preparation_via_rotation.ipynb'), PosixPath('bloqs/state_preparation/state_preparation_via_rotation_tutorial.ipynb'), PosixPath('bloqs/swap_network/swap_network.ipynb'), PosixPath('bloqs/util_bloqs.ipynb'), PosixPath('cirq_interop/cirq_interop.ipynb'), PosixPath('cirq_interop/t_complexity.ipynb'), PosixPath('drawing/drawing_call_graph.ipynb'), PosixPath('drawing/graphviz.ipynb'), PosixPath('drawing/musical_score.ipynb'), PosixPath('resource_counting/bloq_counts.ipynb'), PosixPath('simulation/classical_sim.ipynb'), PosixPath('simulation/tensor.ipynb'), PosixPath('simulation/xcheck_classical_quimb.ipynb'), PosixPath('surface_code/azure_cost_model.ipynb'), PosixPath('surface_code/msft_resource_estimator_interop.ipynb'), PosixPath('surface_code/thc_compilation.ipynb')]
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/ipykernel_launcher.py", line 18, in <module>
    app.launch_new_instance()
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/traitlets/config/application.py", line 1074, in launch_instance
    app.initialize(argv)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/traitlets/config/application.py", line 118, in inner
    return method(app, *args, **kwargs)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 692, in initialize
    self.init_sockets()
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 346, in init_sockets
    self.init_iopub(context)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 375, in init_iopub
    self.iopub_port = self._bind_socket(self.iopub_socket, self.iopub_port)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 253, in _bind_socket
    return self._try_bind_socket(s, port)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 229, in _try_bind_socket
    s.bind("tcp://%s:%i" % (self.ip, port))
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/zmq/sugar/socket.py", line 311, in bind
    super().bind(addr)
  File "_zmq.py", line 898, in zmq.backend.cython._zmq.Socket.bind
  File "_zmq.py", line 160, in zmq.backend.cython._zmq._check_rc
zmq.error.ZMQError: Address already in use (addr='tcp://[12](https://github.com/quantumlib/Qualtran/actions/runs/8899652503/job/24439489256#step:6:13)7.0.0.1:34607')
Exception ignored in atexit callback: <bound method IPKernelApp.close of <ipykernel.kernelapp.IPKernelApp object at 0x7f587f95b3a0>>
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.[14](https://github.com/quantumlib/Qualtran/actions/runs/8899652503/job/24439489256#step:6:15)/x64/lib/python3.10/site-packages/ipykernel/kernelapp.py", line 422, in close
    self.context.term()
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/zmq/sugar/context.py", line 266, in term
    super().term()
  File "_zmq.py", line 545, in zmq.backend.cython._zmq.Context.term
  File "_zmq.py", line 141, in zmq.backend.cython._zmq._check_rc
KeyboardInterrupt: 
Executing global_phase
Exported bloqs/basic_gates/global_phase.ipynb
Executing hadamard
Exported bloqs/basic_gates/hadamard.ipynb
Executing rotation
Exported bloqs/basic_gates/rotation.ipynb
Executing s_gate
Exported bloqs/basic_gates/s_gate.ipynb
Executing states_and_effects
Exported bloqs/basic_gates/states_and_effects.ipynb
Executing first_quantization
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/multiprocessing/pool.py", line 48, in mapstar
    return list(map(*args))
  File "/home/runner/work/Qualtran/Qualtran/dev_tools/qualtran_dev_tools/notebook_execution.py", line 200, in __call__
    err = execute_and_export_notebook(paths)
  File "/home/runner/work/Qualtran/Qualtran/dev_tools/qualtran_dev_tools/notebook_execution.py", line [15](https://github.com/quantumlib/Qualtran/actions/runs/8899652503/job/24439489256#step:6:16)6, in execute_and_export_notebook
    asyncio.run(executor.async_start_new_kernel_client())
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/asyncio/runners.py", line 44, in run
    return loop.run_until_complete(main)
Exported bloqs/chemistry/pbc/first_quantization/first_quantization.ipynb
Executing projectile
Exported bloqs/chemistry/pbc/first_quantization/projectile/projectile.ipynb
Executing resource_estimation
Exported bloqs/chemistry/resource_estimation.ipynb
Executing single_factorization
Exported bloqs/chemistry/sf/single_factorization.ipynb
Executing sparse
Exported bloqs/chemistry/sparse/sparse.ipynb
Executing reflection
Exported bloqs/reflection.ipynb
Executing phasing_via_cost_function
Exported bloqs/rotations/phasing_via_cost_function.ipynb
Executing quantum_variable_rotation
Exported bloqs/rotations/quantum_variable_rotation.ipynb
Executing prepare_uniform_superposition
Exported bloqs/state_preparation/prepare_uniform_superposition.ipynb
Executing state_preparation_alias_sampling
Exported bloqs/state_preparation/state_preparation_alias_sampling.ipynb
Executing error_analysis_for_fxp_arithmetic
Exported bloqs/arithmetic/error_analysis_for_fxp_arithmetic.ipynb
Executing multiplication
Exported bloqs/arithmetic/multiplication.ipynb
Executing sorting
Exported bloqs/arithmetic/sorting.ipynb
Executing t_complexity_of_comparison_gates
Exported bloqs/arithmetic/t_complexity_of_comparison_gates.ipynb
Executing cnot
Exported bloqs/basic_gates/cnot.ipynb
Executing trotterized_unitary
Exported bloqs/chemistry/trotter/trotterized_unitary.ipynb
Executing writing_algorithms
Exported bloqs/chemistry/writing_algorithms.ipynb
Executing qrom
Exported bloqs/data_loading/qrom.ipynb
Executing factoring-via-modexp
Exported bloqs/factoring/factoring-via-modexp.ipynb
Executing mod_exp
Exported bloqs/factoring/mod_exp.ipynb
Executing phase_estimation_of_quantum_walk
Exported bloqs/phase_estimation_of_quantum_walk.ipynb
Executing approximate_qft
Exported bloqs/qft/approximate_qft.ipynb
Executing two_bit_ffft
Exported bloqs/qft/two_bit_ffft.ipynb
Executing generalized_qsp
Exported bloqs/qsp/generalized_qsp.ipynb
Executing qubitization_walk_operator
Exported bloqs/qubitization_walk_operator.ipynb
Executing thc_compilation
Exported surface_code/thc_compilation.ipynb
Executing Adjoint
Exported Adjoint.ipynb
Executing Autodoc
Exported Autodoc.ipynb
Executing Controlled
Exported Controlled.ipynb
Executing su2_rotation
Exported bloqs/basic_gates/su2_rotation.ipynb
Executing t_gate
Exported bloqs/basic_gates/t_gate.ipynb
Executing toffoli
Exported bloqs/basic_gates/toffoli.ipynb
Executing block_encoding
Exported bloqs/block_encoding.ipynb
Executing double_factorization
Exported bloqs/chemistry/df/double_factorization.ipynb
Executing mod_mul
Exported bloqs/factoring/mod_mul.ipynb
Executing hamiltonian_simulation_by_gqsp
Exported bloqs/hamiltonian_simulation/hamiltonian_simulation_by_gqsp.ipynb
Executing hubbard_model
Exported bloqs/hubbard_model.ipynb
Executing and_bloq
Exported bloqs/mcmt/and_bloq.ipynb
Executing multi_control_multi_target_pauli
Exported bloqs/mcmt/multi_control_multi_target_pauli.ipynb
Executing t_complexity
Exported cirq_interop/t_complexity.ipynb
Executing drawing_call_graph
Exported drawing/drawing_call_graph.ipynb
Executing graphviz
Exported drawing/graphviz.ipynb
Executing musical_score
Exported drawing/musical_score.ipynb
Executing bloq_counts
Exported resource_counting/bloq_counts.ipynb
Executing composite_bloq
Exported _infra/composite_bloq.ipynb
Executing gate_with_registers
Exported _infra/gate_with_registers.ipynb
Executing addition
Exported bloqs/arithmetic/addition.ipynb
Executing comparison
Exported bloqs/arithmetic/comparison.ipynb
Executing conversions
Exported bloqs/arithmetic/conversions.ipynb
Executing thc
Exported bloqs/chemistry/thc/thc.ipynb
Executing trotter
Exported bloqs/chemistry/trotter/grid_ham/trotter.ipynb
Executing trotter_costs
Exported bloqs/chemistry/trotter/grid_ham/trotter_costs.ipynb
Executing hubbard
Exported bloqs/chemistry/trotter/hubbard/hubbard.ipynb
Executing ising
Exported bloqs/chemistry/trotter/ising/ising.ipynb
Executing apply_gate_to_lth_target
Exported bloqs/multiplexers/apply_gate_to_lth_target.ipynb
Executing select_pauli_lcu
Exported bloqs/multiplexers/select_pauli_lcu.ipynb
Executing unary_iteration
Exported bloqs/multiplexers/unary_iteration.ipynb
Executing lp_resource_state
Exported bloqs/phase_estimation/lp_resource_state.ipynb
Executing text_book_qpe
Exported bloqs/phase_estimation/text_book_qpe.ipynb
Executing state_preparation_via_rotation
Exported bloqs/state_preparation/state_preparation_via_rotation.ipynb
Executing state_preparation_via_rotation_tutorial
Exported bloqs/state_preparation/state_preparation_via_rotation_tutorial.ipynb
Executing swap_network
Exported bloqs/swap_network/swap_network.ipynb
Executing util_bloqs
Exported bloqs/util_bloqs.ipynb
Executing cirq_interop
Exported cirq_interop/cirq_interop.ipynb
Executing classical_sim
Exported simulation/classical_sim.ipynb
Executing tensor
Exported simulation/tensor.ipynb
Executing xcheck_classical_quimb
Exported simulation/xcheck_classical_quimb.ipynb
Executing azure_cost_model
Exported surface_code/azure_cost_model.ipynb
Executing msft_resource_estimator_interop
Exported surface_code/msft_resource_estimator_interop.ipynb
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/asyncio/base_events.py", line 649, in run_until_complete
    return future.result()
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/nbclient/client.py", line 566, in async_start_new_kernel_client
    await ensure_async(self.kc.wait_for_ready(timeout=self.startup_timeout))
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/jupyter_core/utils/__init__.py", line [19](https://github.com/quantumlib/Qualtran/actions/runs/8899652503/job/24439489256#step:6:20)8, in ensure_async
    result = await obj
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/site-packages/jupyter_client/client.py", line [20](https://github.com/quantumlib/Qualtran/actions/runs/8899652503/job/24439489256#step:6:21)7, in _async_wait_for_ready
    raise RuntimeError("Kernel didn't respond in %d seconds" % timeout)
RuntimeError: Kernel didn't respond in 60 seconds
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/runner/work/Qualtran/Qualtran/dev_tools/execute-notebooks.py", line 34, in <module>
    parse_args()
  File "/home/runner/work/Qualtran/Qualtran/dev_tools/execute-notebooks.py", line 26, in parse_args
    execute_and_export_notebooks(
  File "/home/runner/work/Qualtran/Qualtran/dev_tools/qualtran_dev_tools/notebook_execution.py", line [22](https://github.com/quantumlib/Qualtran/actions/runs/8899652503/job/24439489256#step:6:23)6, in execute_and_export_notebooks
    results = pool.map(func, nb_rel_paths)
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/multiprocessing/pool.py", line 367, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/hostedtoolcache/Python/3.10.14/x64/lib/python3.10/multiprocessing/pool.py", line 774, in get
    raise self._value
RuntimeError: Kernel didn't respond in 60 seconds

[mpharrigan]

This is a consequence of #673 where notebooks are executed in parallel. We could try to disable this on the CI but it's useful locally where notebook execution can take a decent amount of time