RTPTorrent: An Open-source Dataset for Evaluating Regression Test Prioritization
MSR - Technical Paper
The software engineering practice of automated testing helps programmers find defects earlier during development. With growing software projects and longer-running test suites, frequency and immediacy of feedback decline, thereby making defects harder to repair. Regression test prioritization (RTP) is concerned with running relevant tests earlier to lower the costs of defect localization and to improve feedback.
Finding representative data to evaluate RTP techniques is non-trivial, as most software is published without failing tests. In this work, we systematically survey a wide range of RTP literature regarding whether their dataset uses real or synthetic defects or tests, whether they are publicly available, and whether datasets are reused. We observed that some datasets are reused, however, many projects study only few projects and these rarely resemble real-world development activity.
In light of these threats to ecological validity, we describe the construction and characteristics of a new dataset based on 20 open-source Java programs.
Our dataset allows researchers to evaluate prioritization heuristics based on version control meta-data, source code, and test results from fine-grained, automated builds over 9 years of development history. We provide reproducible baselines for initial comparisons and make all data publicly available.
We see this as a step towards better reproducibility, ecological validity, and long-term availability of studied software in the field of test prioritization.
Mon 29 Jun Times are displayed in time zone: (UTC) Coordinated Universal Time change
16:30 - 17:00: Platforms & DatasetsTechnical Papers / Registered Reports / Keynote / MSR Awards / FOSS Award / Education / Data Showcase / Mining Challenge / MSR Challenge Proposals / Ask Me Anything at MSR:Zoom2
Chair(s): Moritz BellerFacebook, USA
Q/A & Discussion of Session Papers over Zoom (Joining info available on Slack)
|16:30 - 16:37|
RTPTorrent: An Open-source Dataset for Evaluating Regression Test PrioritizationMSR - Technical Paper
Toni MattisHasso Plattner Institute, University of Potsdam, Patrick ReinHasso Plattner Institute, Falco Dürsch, Robert HirschfeldHasso-Plattner-Institut (HPI), GermanyDOI Pre-print Media Attached
|16:37 - 16:45|
Konstantinos Barmpis , Patrick NeubauerUniversity of York, UK, Jonathan Co, Dimitris KolovosUniversity of York, Nicholas Matragkas, Richard PaigeMcMaster UniversityMedia Attached
|16:45 - 16:52|
Technical PapersPre-print Media Attached
|16:52 - 17:00|
A: Antoine PietriInria, A: Guillaume RousseauUniversité de Paris and Inria, A: Stefano ZacchiroliUniversité de Paris and InriaPre-print Media Attached