Remove aqua from benchmark #1189

hhorii · 2021-03-20T09:45:22Z

Summary

This PR removes aqua from benchmark in test

Details and comments

Aqua is now being refactored and VQE and UCCSD will be in qiskit-nature.
However, their new versions are under development.
Until they are released, benchmarks do not include VQE applications.

mtreinish · 2021-03-20T17:10:38Z

I would say lets keep the benchmarks in place as is until qiskit-nature is released, aqua will continue to work for several months as is, and we can replace the benchmarks at some point after the release with a qiskit-nature based versions (and just bump the version of the benchmark then). That way we don't lose any coverage, especially right up until the release.

hhorii · 2021-03-22T09:24:51Z

@mtreinish OK. Please let me leave this as a placeholder for future change after qiskit-nature is released.

…nchmark

hhorii · 2021-11-12T11:14:05Z

https://hhorii.github.io/qiskit-aer/html/#/

@mtreinish qiskit_nature was released and now it is good timing to work this again.

test/asv.linux.conf.json

test/asv.linux.cuda.conf.json

test/benchmark/simulator_benchmark.py

test/benchmark/vqe_application.py

test/asv.linux.conf.json

test/benchmark/simulator_benchmark.py

Previous benchmarks tracked only C++ implementation. However, in recent releases, functions in Python were increased to transpile, assemble, and support control-flow. With this change, benchmarks additionally track python implementation.

test/benchmark/default_simulator.py

…nchmark

mtreinish

This LGTM, just a few small inline questions/comments but otherwise I'm fine with this.

mtreinish · 2022-05-10T13:21:07Z

test/asv.linux.conf.json


    // The number of characters to retain in the commit hashes.
    // "hash_length": 8,

    // `asv` will cache results of the recent builds in each
    // environment, making them faster to install next time.  This is
    // the number of builds to keep, per environment.
-    // "build_cache_size": 2,
+    "build_cache_size": 100000


Heh, that's a lot of builds

good catch :-)

mtreinish · 2022-05-10T13:22:41Z

test/asv.linux.conf.json

@@ -33,27 +33,19 @@
    "install_command": [
        "python -c \"import shutil; shutil.rmtree('{build_dir}/qiskit', True)\"",
        "python -c \"import shutil; shutil.rmtree('{build_dir}/qiskit_aer.egg-info', True)\"",
-        "pip install git+https://github.com/Qiskit/qiskit-terra",
-        "pip install git+https://github.com/Qiskit/qiskit-aqua",
+        "pip install -U qiskit-terra",


I'm wondering if we want to use matrix here to list qiskit-terra at a specific version so the benchmark results are tied to the qiskit-terra version (so if we bump the terra version in the matrix it shows as different from earlier runs).

mtreinish · 2022-05-10T13:23:17Z

test/benchmark/default_simulator.py

+        try:
+            from qiskit.providers.aer import AerSimulator
+            self._simulator = AerSimulator()
+        except ImportError:
+            from qiskit.providers.aer import QasmSimulator
+            self._simulator = QasmSimulator()
+        return self._simulator


Are you still supporting comparisons from the releases prior to AerSimulator?

mtreinish · 2022-05-10T13:24:25Z

test/benchmark/default_simulator.py

+        self._simulator = None
+        self.qubits = qubits
+        self.params = (qubits)
+        self.param_names = ["qubits"]


Suggested change

self.param_names = ["qubits"]

self.param_names = ["Number of qubits"]

Earlier comments addressed

hhorii requested review from chriseclectic, mtreinish and vvilpas as code owners March 20, 2021 09:45

remove aqua from benchmark

fdb25e0

hhorii force-pushed the remove_aqua_from_benchmark branch from 5d87bd7 to fdb25e0 Compare March 20, 2021 09:58

hhorii changed the title ~~remove aqua from benchmark~~ [WIP] remove aqua from benchmark Mar 22, 2021

hhorii added 2 commits November 11, 2021 11:53

Merge remote-tracking branch 'upstream/main' into remove_aqua_from_be…

56a6a64

…nchmark

disable truncation

b2948af

hhorii force-pushed the remove_aqua_from_benchmark branch from 2c249cb to 9cff2dc Compare November 11, 2021 06:16

use aer_simulator correctly in benchmark

e6b9940

hhorii force-pushed the remove_aqua_from_benchmark branch from 9cff2dc to e6b9940 Compare November 11, 2021 06:35

enable vqe application

ba52df4

hhorii changed the title ~~[WIP] remove aqua from benchmark~~ Remove aqua from benchmark Nov 12, 2021

change asv files

aa87fdb

hhorii force-pushed the remove_aqua_from_benchmark branch from 06621a3 to aa87fdb Compare November 12, 2021 11:13

mtreinish previously requested changes Nov 15, 2021

View reviewed changes

fix typo and asv conf bugs

4d5b29c

jakelishman reviewed Nov 23, 2021

View reviewed changes

test/asv.linux.conf.json Outdated Show resolved Hide resolved

test/asv.linux.conf.json Outdated Show resolved Hide resolved

test/benchmark/simulator_benchmark.py Outdated Show resolved Hide resolved

test/benchmark/simulator_benchmark.py Outdated Show resolved Hide resolved

remove unnecessary fileds from benchmarks

63a05ef

hhorii force-pushed the remove_aqua_from_benchmark branch from 5c8c12c to 6544b00 Compare December 10, 2021 07:33

support recent change related to noise model

a27314e

hhorii force-pushed the remove_aqua_from_benchmark branch from 6544b00 to a27314e Compare December 10, 2021 07:35

hhorii changed the title ~~Remove aqua from benchmark~~ [WIP] Remove aqua from benchmark Jan 4, 2022

hhorii force-pushed the remove_aqua_from_benchmark branch 4 times, most recently from d6eb2ee to 9e66f3d Compare January 14, 2022 00:52

renewed benchmarks

03999bd

Previous benchmarks tracked only C++ implementation. However, in recent releases, functions in Python were increased to transpile, assemble, and support control-flow. With this change, benchmarks additionally track python implementation.

hhorii force-pushed the remove_aqua_from_benchmark branch from 9e66f3d to 03999bd Compare January 14, 2022 16:15

hhorii changed the title ~~[WIP] Remove aqua from benchmark~~ Remove aqua from benchmark Jan 18, 2022

jakelishman reviewed Jan 21, 2022

View reviewed changes

test/benchmark/default_simulator.py Outdated Show resolved Hide resolved

test/benchmark/default_simulator.py Outdated Show resolved Hide resolved

hhorii added 2 commits March 15, 2022 10:31

Merge remote-tracking branch 'upstream/main' into remove_aqua_from_be…

90e860d

…nchmark

update benchmark document and correct units of time

ba603f7

hhorii force-pushed the remove_aqua_from_benchmark branch from 2e20fb0 to 3c4f77d Compare March 15, 2022 11:52

use time methods of asv

43a1b20

hhorii force-pushed the remove_aqua_from_benchmark branch from 3c4f77d to 43a1b20 Compare March 15, 2022 12:37

hhorii added 2 commits March 29, 2022 17:53

Merge branch 'main' into remove_aqua_from_benchmark

429a6bf

Merge branch 'main' into remove_aqua_from_benchmark

c8ffe4a

hhorii requested a review from mtreinish March 30, 2022 01:21

Merge branch 'main' into remove_aqua_from_benchmark

d3d282d

mtreinish reviewed May 10, 2022

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove aqua from benchmark #1189

Remove aqua from benchmark #1189

hhorii commented Mar 20, 2021

mtreinish commented Mar 20, 2021 •

edited

hhorii commented Mar 22, 2021

hhorii commented Nov 12, 2021 •

edited

mtreinish left a comment

mtreinish May 10, 2022

hhorii May 11, 2022

mtreinish May 10, 2022

mtreinish May 10, 2022

mtreinish May 10, 2022

	self.param_names = ["qubits"]
	self.param_names = ["Number of qubits"]

Remove aqua from benchmark #1189

Are you sure you want to change the base?

Remove aqua from benchmark #1189

Conversation

hhorii commented Mar 20, 2021

Summary

Details and comments

mtreinish commented Mar 20, 2021 • edited

hhorii commented Mar 22, 2021

hhorii commented Nov 12, 2021 • edited

mtreinish left a comment

Choose a reason for hiding this comment

mtreinish May 10, 2022

Choose a reason for hiding this comment

hhorii May 11, 2022

Choose a reason for hiding this comment

mtreinish May 10, 2022

Choose a reason for hiding this comment

mtreinish May 10, 2022

Choose a reason for hiding this comment

mtreinish May 10, 2022

Choose a reason for hiding this comment

mtreinish commented Mar 20, 2021 •

edited

hhorii commented Nov 12, 2021 •

edited