Speaker:Alexis Marsh

Using Software Testing Techniques for Bioinformatics Tools

While scientific software has become increasingly important to research, the ability to fully understand the behavior of these tools has remained limited. This is, in part, due to the difficulties in developing tests for these tools. Consider computational biology software. Not only do we often lack an oracle, there are multiple ways results deviating from the expected can arise. This includes bugs, issues with the input, or misunderstandings on the user end. Further, it is often the case that there are many open source tools available to perform the same task and it is difficult for users to determine what tools to use. Despite the existence of data-sharing principles such as FAIR, little attention has been paid to ensuring computational tools provide a platform to compare experimental results and data across different studies. Tools often are lightly documented, non-extensible, and provide different levels of accuracy in their representation. This lack of standardization and rigor reduces the potential power for new insight and discovery and makes it more difficult for biologists to experiment, compare, and trust results between different studies. To address these issues, we focused on a set of  four flux balance analysis tools, which are commonly used in both computational biology and metabolic engineering, and four models. We used both metamorphic and differential testing including tests focusing both on basic functionality and ones requiring domain knowledge.

We found not all tests could be run on all tools. Of the 3,738 metamorphic tests ran, we observed 1,185 failures.  We found nine different types of faults.  Further, even when using metamorphic-differential testing, we still saw high rates of failures (overall fail rate of 54.92\%). Finally, we had multiple faults confirmed by developers.

Next, we evaluated the tools from a biologist (user) perspective using a set of seven models with a focus on the comparability between tools and a goal of developing a set of principles similar to FAIR (which are data sharing principles). In our study, we noted a range of differences including in the workflows, network metrics, and biological behavior. Using what we observed, we developed a set of principles we called CORE (Comparable, Open, Reliable and Extensible). Our goal is that these principles will make the tools more user-friendly (decreasing the number of tools that are re-made) as well as more biologically accurate.

Committee: Myra Cohen (major professor), Robyn Lutz, and Andrew Miner