Over the last decade, research on cyber-physical systems (CPS) and Internet of Things (IoT) has led to smart systems at different scales and environments, from smart homes to smart cities and smart factories. Significant progress has been made through contributions in areas as diverse as control, embedded and real-time systems, wireless communication, and networking. Despite these advances, it is difficult to measure and compare the utility of these results due to a lack of standard evaluation criteria and methodologies. This problem concerns the evaluation against the state of the art in an individual area, the comparability of different integrated designs that span multiple areas (e.g., control and networking), and the applicability of tested scenarios to the present and future real-world CPS and IoT applications and deployments. This state of affairs is alarming as it may significantly hinder further progress in CPS and IoT research.
The 2nd Workshop on Benchmarking Cyber-Physical Systems and Internet of Things (CPS-IoTBench) brings together researchers from the different sub-communities to engage in a lively debate on all facets of rigorously evaluating and comparing CPS and IoT solutions. CPS-IoTBench provides a venue for learning about each other’s challenges and evaluation methodologies and for debating future research agendas to jointly define the performance metrics and benchmarking scenarios that matter from an overall system’s perspective.
We invite researchers and practitioners from academia and industry to submit short papers. We particularly encourage submissions that focus on one of the following:
- identify fundamental challenges and open questions in rigorous benchmarking and evaluation of CPS and IoT solutions;
- offer a constructive critique on the current practice and state of experimental comparison;
- report on success stories or failures with using standard evaluation criteria;
- describe efforts to replicate or reproduce experimental results from published research;
- present example benchmark systems and approaches from any of the relevant communities (embedded systems, real-time systems, networking, wireless communication and control, etc.);
- benchmark industrial standardized solutions against each other and against academic solutions, and discuss their strengths and weakness for different application use-cases and industry verticals;
- propose new research directions, methodologies, or tools to increase the level of reproducibility and comparability of evaluation results.
Well-reasoned arguments or preliminary evaluations are sufficient to support a paper’s claims.
Accepted papers will be published in both IEEE Xplore and the ACM Digital Library as part of the CPS-IoT Week proceedings. Authors of accepted papers are expected to present their work at the workshop
Detailed submission instructions are available here.