Publication
Guidelines for the Quality Assessment of Energy-Aware NAS Benchmarks
Nick Kocher; Christian Wassermann; Leona Hennig; Jonas Seng; Holger H. Hoos; Kristian Kersting; Marius Lindauer; Matthias S. Müller
In: Computing Research Repository eprint Journal (CoRR), Vol. abs/2505.15631, Pages 1-10, Computing Research Repository, 2025.
Abstract
Neural Architecture Search (NAS) accelerates
progress in deep learning through systematic refinement of
model architectures. The downside is increasingly large en-
ergy consumption during the search process. Surrogate-based
benchmarking mitigates the cost of full training by querying
a pre-trained surrogate to obtain an estimate for the quality
of the model. Specifically, energy-aware benchmarking aims to
make it possible for NAS to favourably trade off model energy
consumption against accuracy. Towards this end, we propose
three design principles for such energy-aware benchmarks: (i)
reliable power measurements, (ii) a wide range of GPU usage,
and (iii) holistic cost reporting. We analyse EA-HAS-Bench
based on these principles and find that the choice of GPU
measurement API has a large impact on the quality of results.
Using the Nvidia System Management Interface (SMI) on top
of its underlying library influences the sampling rate during the
initial data collection, returning faulty low-power estimations.
This results in poor correlation with accurate measurements
obtained from an external power meter. With this study, we bring
to attention several key considerations when performing energy-
aware surrogate-based benchmarking and derive first guidelines
that can help design novel benchmarks. We show a narrow
usage range of the four GPUs attached to our device, ranging
from 146 W to 305 W in a single-GPU setting, and narrowing
down even further when using all four GPUs. To improve
holistic energy reporting, we propose calibration experiments
over assumptions made in popular tools, such as Code Carbon,
thus achieving reductions in the maximum inaccuracy from
10.3 % to 8.9 % without and to 6.6 % with prior estimation of
the expected load on the device.
Index Terms—Neural Architecture Search, Green AutoML,
Energy reporting, Energy estimation