Publication
Scalable Evaluation Pipeline of CNN-based perception for Robotic Sensor Data under different Environment Conditions
Naeem Iqbal; Mark Niemeyer; Jan Christoph Krause; Joachim Hertzberg
In: 2023 European Conference on Mobile Robots. European Conference on Mobile Robots (ECMR-2023), September 4-7, Coimbra, Portugal, IEEE, 2023.
Abstract
Deep learning impacted a wide variety of perception applications for autonomous mobile robots. In classic computer vision benchmark tests, new algorithms keep appearing that outperform each other. However, these benchmark tests cannot be generalized, so that the specific application must be considered for the selection of sensors and algorithms. Especially in the agricultural domain, environmental conditions like weather and vegetation significantly influence the reliability of sensor systems. Therefore, it is essential to test different sensor modalities and algorithms in the operational design domain. This motivates the need for an evaluation framework which has the flexibility to compare and validate various perception algorithms, sensors suites, and data samples with a focus on different conditions.
This paper proposes a pipeline combining a test environment (AI-TEST-FIELD), a semantic environment representation (SEEREP), and an inference server (Triton) for an automatic evaluation of different CNN-based perception algorithms under various environment conditions. Recurring and comparable recordings of sensor raw data with identical scenarios and objects can be performed on the test field, with the only difference being the environmental conditions. The inference results are inferred once and stored alongside the sensor data in SEEREP. Thus, they can be queried efficiently based on the environment conditions to generate (partially overlapping) subsets of the whole dataset. It is demonstrated how this pipeline can be used to apply the CNN-inference just once on the data, and how the queried subsets can subsequently be used to evaluate the performance in different environment conditions.