We found Qiskit to be the highest-performing quantum software development kit for building and transpiling quantum circuits, based on the results of our benchmarking using over 1,000 tests developed by leading universities, national labs, and IBM®.
Today, we’re releasing the results of extensive testing performed by IBM to compare Qiskit against other leading quantum software development kits. These tests benchmark some of the most important tasks for a quantum SDK, which must be feature-complete enough to build and manipulate circuits while transpiling circuits with the best combination of speed and resulting circuit quality. These tests compared Qiskit to BQSKit, Braket, Cirq, Staq, and TKET.
The results? Qiskit SDK was capable of completing more of the tests than any other SDK. On average, it did so either faster, with fewer two-qubit gates, or both. It was the highest-performing quantum SDK that we tested—13x faster at transpiling and producing circuits with 24% fewer two-qubit gates than TKET, the second highest-performing SDK.
Alongside a paper summarizing the results, we are releasing the suite of tests today as part of an open-source package named Benchpress. We built Benchpress to help industry stakeholders examine and expand upon this analysis in order to glean meaningful information about various quantum SDKs. By releasing Benchpress as open source, we aim to create an open framework for performance comparisons, allowing users to make assessments themselves.
We hope that Benchpress will give users the opportunity to monitor the continuing software development progress of quantum SDKs, and to make informed decisions about which quantum SDK to use in quantum computing. We will continue adding additional tests to the Benchpress package, and we encourage you to check out and contribute to Benchpress yourself.
Not all quantum software development kits are capable of both creating/manipulating circuits and transpiling circuits, as shown in the above diagram.
The number of tests passed out of 1,066 tests* in the Benchpress suite. The difference in pass rate is predominantly indicative of the feature sets of each SDK. In particular, Staq does not have significant support for circuit manipulation, while Cirq and Braket lack features for circuit synthesis and transpilation. *Twenty-two tests were skipped across all SDKs due to target device size.
Leveling the playing field
Qiskit’s performance improvements began earlier this year, when we released the first major version of the Qiskit software development kit, Qiskit SDK v1.0. Initial results showed that Qiskit v1.0 was a high-performance quantum SDK, but we weren’t prepared to say that Qiskit was the world’s highest-performing quantum SDK without running a formal comparison. So, we assembled a suite of rigorous tests to evaluate the ability of a quantum SDK to undertake its most important tasks: manipulating, creating, synthesizing, and transpiling circuits.
Circuit building and manipulation gauge how well a quantum SDK can create and apply transformations to quantum circuits. Some quantum SDKs offer tools that automatically build and manipulate circuits, while others require coding circuits and circuit transformations by hand. A performant quantum SDK should be able to generate and manipulate circuits quickly so that the code itself does not serve as a bottleneck to receiving results quickly. Circuit building and manipulation tasks comprise 12 of the tests, and success on these tests means successfully building and manipulating the circuits within an hour.
Synthesis and transpilation is the ability of a quantum SDK to take a circuit as input and translate it into a form capable of running on quantum hardware—both abstract quantum hardware with arbitrary qubit layouts and real quantum hardware, such as the IBM QuantumTM Heron chip. Success on these tests means being able to transpile the circuits within an hour, and to do so with the fewest number of two-qubit gates in the shortest amount of time.
Most of these tests — approximately 90% — are standardized benchmarking tests from open-source libraries and used by other members of the quantum community. For example, Benchpress compares SDKs’ abilities to generate QASMBench circuits developed by Pacific Northwest National Labs, Feynman circuits developed by Matthew Amy at the University of Waterloo, and Hamiltonian circuits developed by a team of physicists from around the world. The rest are tests we already use internally to benchmark Qiskit.
Taking SDKs to the gym
We ran circuit building and manipulation tests on Braket, BQSKit, Cirq, Qiskit and TKET, and transpilation tests on BQSKit, Qiskit, Qiskit Transpiler Service, Staq and TKET, based on their capabilities. We opted not to benchmark software development kits whose feature sets don’t provide sufficient functionality to run the tests, such as CUDA-Q.
We broadly compared SDKs’ abilities to perform these tests using two key metrics: speed and quality. We measured how long it takes the SDK to create and manipulate the circuit, and separately, how long it takes to synthesize and transpile the circuit. We also measured the quality of both the created circuits and the transpiled circuits, based on the depth and number of two-qubit gates. For each test, we flagged whether the SDK passed, failed, lacked the functionality to complete the test, or was expected to fail the test.
Qiskit passed more tests than any other quantum SDK that was tested — and passed all of the tests that were possible for it to pass. On average, Qiskit was 13x faster at transpiling and produced circuits with 24% fewer two-qubit gates than the second highest-performing SDK, TKET. Where the other SDKs failed, they either could not complete the test within a one-hour time limit, or had bugs that prevented them from passing.
In short, Qiskit is the highest-performing quantum SDK.
Why is performance so important? Well, a quantum computer can’t run computations without a quantum software development kit. And if it takes too long to build, manipulate, and transpile circuits, that could mean the difference between running circuits quickly versus waiting several days to run circuits, enough to deter researchers from studying problems on real quantum systems.
We ran these tests in good faith and to the best of our abilities using the best tools offered by each SDK. We recognize, however, that we are not experts in other quantum SDKs. Therefore, we've made Benchpress open source so that anyone interested can review and run these tests on their own. We hope to grow the capabilities of Benchpress through open-source efforts beyond this initial release to solidify the package’s role as a trusted benchmarking suite for quantum SDKs.
New features for a performant Qiskit
We continue to add features and upgrades to Qiskit that push its performance forward. Many of those performance gains derive from our refactoring much of the code into the Rust programming language. Rust includes security and memory management features to help build higher-quality, more efficient software. It’s much beloved by Qiskit developers.
Furthermore, our team will soon publish a paper detailing an algorithm called LightSABRE, an enhancement of the SABRE algorithm that Qiskit uses to transpile circuits — that is, to map and route circuits onto physical devices while obeying the hardware’s connectivity constraints, all while aiming for a minimum number of gates and circuit layers. The original SABRE algorithm was considered a high-quality solution for this task when it debuted in 2018, but its runtime has turned into a bottleneck in the era of larger utility-scale quantum processors. LightSABRE is an upgrade written in Rust that scales more efficiently, delivering an 18.9% decrease in costly SWAP gates for the same circuits versus SABRE.
Finally, in our benchmarking tests, only the Qiskit Transpiler Service (QTS) demonstrated a consistent ability to producer shallower, higher-quality circuits than the standard Qiskit transpiler. Thanks to its AI-powered transpiler passes, we were able to achieve significant improvements in circuit quality, particularly for larger circuits. Compared to standard Qiskit SDK transpilation, AI transpiler passes delivered a further 24% reduction on average in 2-qubit gates, and a 36% reduction on average in circuit depth for utility-scale circuits (i.e. 100+ qubits) when transpiling to the heavy hex topology of IBM Quantum hardware.
Circuit depth for circuits transpiled with Qiskit+AI, relative to Qiskit SDK, for the Benchpress circuits (lower is better).
Building a performant software stack
Benchpress measures a quantum SDK's ability to build and transpile circuits, however — it must also run circuits on real hardware, potentially representing different quantum computing architectures. At THINK 2024, we redefined Qiskit to represent the full-stack software for quantum at IBM, including the Qiskit SDK, Qiskit Functions, Qiskit Transpiler Service, Qiskit Serverless, Qiskit Runtime Primitives, and the Qiskit Code Assistant, a generative AI code assistant tool. Together, these tools ensure a high-performance experience across the software stack.
IBM tracks a number of metrics to benchmark Qiskit’s ability to run circuits on quantum hardware, which inform how well we perform in the execution phase of a workflow. These metrics include CLOPS, which measures speed as the number of circuit layer operations per second that Qiskit Runtime + hardware can run. They also include error per layered gate, the average error for each 2-qubit gate in a layered circuit.
We are exploring ways to augment these metrics to measure Primitive quality using mirror circuits and other measures. We haven’t yet formalized these new metrics, but we already have tools allowing users to explore these benchmarks in the qiskit-device-benchmarking repository. By publishing these metrics, we demonstrate the importance of hardware and software working together to deliver a high-performance experience—and challenge other quantum computer vendors to present similar data of their own.
Together, these tests demonstrate that Qiskit is more than just a tool for learning quantum — it should be the go-to quantum SDK for researchers and developers interested in performance. We encourage you to get started with Qiskit here.
