OneMoreTest: A Learning-Based Approach to Generating and Selecting Fault-Revealing Unit Tests

Developers often manually design a few unit tests for a given method under development. After passing such manually designed tests, however, they usually have to turn to automated test case generation tools like EvoSuite and Randoop for more thorough testing. Although the automatically generated tests may achieve a high coverage, they rarely identify hard-to-detect defects automatically because of the well-known test oracle problem: It is challenging to tell whether the output is correct or incorrect without explicit test oracle (expected output). Consequently, developers should manually select and verify a few suspicious test cases to identify hard-to-detect defects. To this end, in this paper, we propose a novel approach, calledOneMoreTest, to generating and selecting the most suspicious tests for manual verification. Based on a manually designed passed test,OneMoreTestautomatically generates millions of input-output pairs for the method under test (MUT) with mutationbased fuzzing. It then trains an automatically generated neural network to simulate the MUT’s behavior. For new tests automatically generated for the same MUT,OneMoreTestsuggests developers with the topkmost suspicious tests that have the greatest distances between their actual output and estimated output (i.e., network’s output). Our evaluation on real-world faulty methods suggests thatOneMoreTestis accurate. On 70.79% of the involved 178 real-world faulty methods, we can identify the defects by manually verifying only a SINGLE test for each of the methods according toOneMoreTest’s suggestions. Compared against the state of the art,OneMoreTestimproved the precision from 46.63% to 72.62%, and recall from 46.63% to 70.79%.