Mentor:
- Abhik Roychoudhury (National University of Singapore)
Leader:
- Tamás Gergely (University of Szeged, Hungary)
Members:
- Casandra Holotescu
- Pooyan Jamshidi
- Salman Mirghasemi
- György Orbán
- Hoang Duong Thien Nguyen
Read and comment in the topic forum. (authorization required)
CONTEXT:
Programmers do not write programs entirely from scratch. Rather, over time, a program gradually evolves with more features possibly being added. Validation of such evolving programs (say, to address possible bugs introduced via program changes) remains a huge problem in terms of program development.
AIM:
The goal of this working group is to investigate methods for uncovering and localizing of regression errors. Regression errors are a common cause of concern in large-scale software development projects. Unless handled properly, regression errors can significantly delay a software project’s timeline.
METHOD:
The study will be conducted by contacting experts of this field-possibly from both academia and industry-during the ESEC-FSE conference and interviewing them to collect information aimed at addressing our goal.
For this purpose, the working group should identify a set of questions to be asked, for example:
1. Do you have any specific regression error patterns in mind? Any common error types that get introduced when changes are made?
a) Branch condition errors or assignment statement errors.
b) Wrong usage of interfaces (e.g. wrong parameters).
c) Wrong constant usage/handling.
d) Language specific errors.
x) other ...
2. How do you feel regression errors can be effectively discovered?
a) Coverage based testing is enough in most cases.
b) Specific change-stressing techniques for test-suite augmentation are needed.
c) Re-execution of functional tests discover the important errors.
x) other ...
3. What kind of programming environments / practices would the developers like to help them quickly detect regression errors?
a) To have a programming environment where whenever a change is made, a change-stressing test is synthesized and highlighted to the developer.
b) To have a programming environment where whenever a change is made in a unit, all adjacent (e.g. by impact analysis) units are marked as "to be reviewed".
c) Pair programming / regular code reviews would help much to detect regression errors.
d) Test driven development is the best way to avoid regression errors.
x) other ...
4. Apart from better programming environments, how effective do you think can programming practices such as pair-programming be for detecting / avoiding regression errors?
a) Programming practices has no effect on regression errors.
b) As pair programming reduces the chance of any errors, it can reduce regression errors too.
x) other ...
5. Once an observable regression error is identified, what kind of techniques can developers employ in practice to quickly localize the error cause?
a) Simply eyeball a "diff" of the program versions and locate the error cause manually?
x) other ...
6. How effective the above techniques can be?
- open question
7. In your opinion/experience, what are the common issues/reasons when an application program crashes when running in another OS?
a) different libraries
b) different configurations
x) other ...
REFERENCES:
[1] Yesterday, my program worked. Today, it does not. Why? ESEC 1999 by Andreas Zeller. Oscar Nierstrasz, M. Lemoine (Ed.). Proceedings of the ESEC/FSE'99, 7th European Software Engineering Conference, Pages 253-267, Lecture Notes in Computer Science, Volume 1687, Springer, September 1999.
[2] DARWIN: An Approach for Debugging Evolving Programs. Dawei Qi, Abhik Roychoudhury, Zhenkai Liang, Kapil Vaswani. Joint meeting of ESEC and ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE), ESEC-FSE 2009.
[3] Test-suite Augmentation for Evolving Software. Raul Andres Santelices, Pavan Kumar Chittimalli, Taweesup Apiwattanapong, Alessandro Orso, Mary Jean Harrold. IEEE/ACM International Conference on Automated Software Engineering (ASE 2008), L'Aquila, Italy, September 2008, pp. 218-227.
[4] Test Generation to Expose Changes in Evolving Programs. Dawei Qi, Abhik Roychoudhury, Zhenkai Liang. 25nd IEEE/ACM International Conference on Automated Software Engineering (ASE), 2010.
[5] Directed Test Suite Augmentation: Techniques and Tradeoffs. Z. Xu, Y. Kim, M. Kim, G. Rothermel, M. Cohen. Proceedings of the ACM International Symposium on Foundations of Software Engineering, November, 2010, pages 257-266.
[6] Golden Implementation Driven Software Debugging. Ansuman Banerjee, Abhik Roychoudhury, Johannes A. Harlie, Zhenkai Liang. ACM SIGSOFT Symposium on Foundations of Software Engineering (FSE), 2010.
[7] BERT: A Tool for Behavioral Regression Testing. W. Jin, A. Orso, and T. Xie. Proceedings of the 18th European Software Engineering Conference and ACM SIGSOFT Symposium on the Foundations of Software Engineering (ESEC/FSE 2010) -- Formal research demonstration.
[8] eXpress: Guided Path Exploration for Efficient Regression Test Generation. Kunal Taneja, Tao Xie, Nikolai Tillmann, and Jonathan de Halleux. In Proceedings of the 2011 International Symposium on Software Testing and Analysis (ISSTA 2011), Toronto, Canada, July 2011.
[9] Path Exploration based on Symbolic Output. Dawei Qi, Hoang D.T. Nguyen, Abhik Roychoudhury. Joint meeting of ESEC and ACM SIGSOFT Symposium on the Foundations of Software Engineering (FSE), ESEC-FSE 2011.