BENEVOL 2019

Abstract

Test cases form the first line of defense against the introduction of software faults. Unfortunately, even tests may be affected by defects that can preclude their effective testing of software systems. Test flakiness is among the most harmful and investigated types of test defects: It affects tests that exhibit a seemingly random outcome (pass or fail) when run against the same, identical code.
In this talk, I will provide an overview of the research around the topic of flaky tests, focusing on: (1) The various forms in which flakiness can manifest itself in practice and what are the problems connected to its identification; (2) The solutions, based on dynamic and static analysis, available so far; and (3) The open issues and challenges for the research community.

Biography

Fabio Palomba is an Assistant Professor of Software Engineering at the University of Salerno, Italy. He received the European PhD degree in Management & Information Technology in 2017. His PhD Thesis was the recipient of the 2017 IEEE Computer Society Best PhD Thesis Award (Italy section). His research interests include software maintenance and evolution, software testing, empirical software engineering, source code quality, and mining software repositories.
He was the recipient of two ACM/SIGSOFT and one IEEE/TCSE Distinguished Paper Awards at ASE'13, ICSE'15, and ICSME'17, respectively, and Best Paper Awards at CSCW'18 and SANER'18. In 2019 he was the recipient of an SNSF Ambizione grant, one of the most prestigious individual research grants in Europe.
He serves and has served as a program committee member of various international conferences (e.g., MSR, ICPC, ICSME), and as referee for various international journals (e.g., TSE, EMSE, JSS) in the field of software engineering. Fabio will be the program co-chair of ICPC 2021, which will be held in Madrid, Spain. Since 2016 he is Review Board Member of EMSE and, since 2019, Editorial Board Member of TOSEM, JSS, and SCICO. He was the recipient of five Distinguished/Outstanding Reviewer Awards for his reviewing activities conducted for EMSE, IST, and JSS between 2015 and 2019.

Abstract

The world of legacy software exists.
Despite numerous prognoses about certain languages going out of fashion and disappearing, many languages resist this natural process and continue to be used extensively across billions of lines of code, literally decades after going out of fashion. The world of legacy software evolves and grows, absorbing new languages as years go by. Only a handful of legacy systems are suitable for complete redesign and rewrite, while the bulk of software systems in finance, banking, insurance, logistics, booking, as well as in a number of other domains, are too large, too complex and too expensive to be replaced in simplistic ways. Hence, we need to develop methods and tools to analyse such software to aid in this never-ending battle by providing compilers, analysers, transformers, refactorers, testers, linters, validators.
During this keynote, you will be confronted with a carefully selected subset of features encountered by industrial compiler engineers in legacy code. Condensed by language design magic into one deliberately small language, this collection is meant to represent the challenge of processing legacy code to the software evolution community. You will witness the first public exposure of this meeting point between the industrial community that is creating tools for handling real legacy languages (each demanding several human-years of professional investment and therefore completely unsuitable for research prototyping) and the academic community which consists of many comfort bubbles of idealised toy languages (with consistent design that rewards emergence of glorified techniques that optimistically assume overly rational language features).

Biography

Dr. Vadim Zaytsev, also known in the social media as @grammarware, is the Chief Science Officer of Raincode, a Belgian company specialising in modernisation of software legacy systems, as well as of Raincode Labs, the largest independent company in the world providing compiler services and consulting.
He has acquired PhD in 2010 at the Vrije Universiteit Amsterdam in the field of software language engineering with a focus on grammar(ware) technology. He has done research in that direction at Universität Koblenz-Landau in Germany (2008–2010), Centrum Wiskunde & Informatica in the Netherlands (2010–2013), as well as at University of Amsterdam in the Netherlands (2013–2016) where he also was extensively involved in teaching many graduate-level courses, supervising student projects, maintaining and expanding the network of university relations with the industry.
Besides software evolution, his interests and research activities tend to invade such topics as software language engineering, software quality assessment, source code analysis and transformation, modelling, metamodelling and megamodelling.
More information on his publications can be found at http://grammarware.net and on his projects at http://grammarware.github.io.

Abstract

Research on empirical software engineering has increasingly used the data that is made available in online repositories, specifically Free/Libre/Open Source projects (FLOSS). The latest trends for researchers is to gather "as much data as possible" to (i) prevent bias in the representation of a small sample, (ii) work with a sample as close as the population itself, and (iii) showcase the performance of existing or new tools in treating vast amount of data.
The effects of harvesting enormous amounts of data have been only marginally considered so far: data could be corrupted; repositories could be forked; and developer identities could be duplicated. In this paper we posit that there is a fundamental flaw in harvesting large amounts of data, and when generalising the conclusions: the application domain, or context, of the analysed systems must be the primary factor for the cluster sampling of FLOSS projects.
This paper presents two contributions: first, we analyse all the BENEVOL papers that appeared showing whether (and how much) FLOSS data has been harvested, and how many times the authors flagged an issue in their different application domains. Second, we discuss the implications of using "application domain" as the clustering factor in the sampling of FLOSS data, and the generalisations within and outside the clusters.

Abstract

Program synthesis is the task of enabling a computer system to automatically write program code based on user intent. Test amplification on the other hand is an emerging research area, where the goal is to generate new test cases from manually written ones. From this perspective, test amplification is awfully similar to program synthesis. Therefore, in this short paper, we explore the benefits of using program synthesis for test amplification.

Abstract

Compilers play a crucial role in software development, and demands for their correctness are high. Testing them is a challenging process, involving many layers of complexity and many interwoven methodologies. One of the popular techniques that is known to increase the effectiveness of manually written test cases, is fuzzing: seeded with an initial test case, a fuzzer can mutate it to automatically infer multitudes of mutated test cases. Yet, not all is well when you try to apply fuzzing to test an industrial compiler of a legacy assembler-level language, since most research on fuzzing was done on C, and such results are not easy to generalise.
In this paper, we explain the complications and show a number of mutation strategies that we successfully applied to fuzz HLASM code. So far we have tested 24000 mutated files and have been able to flag 621 of them as triggering a bug, and found and fixed at least one unique bug in Raincode Assembler Compiler.

Abstract

Duplication in source code is often seen as one of the most harmful types of technical debt as it increases the size of the codebase and creates implicit dependencies between fragments of code. Detecting such problems can provide valuable insight into the quality of systems and help to improve the source code. To correctly identify cloned code, contextual information should be considered, such as the type of variables and called methods.
Comparing code fragments including their contextual information introduces an optimization problem, as this information may be hard to retrieve. It can be ambiguous where contextual information resides and tracking it down may require to follow cross-file references. For large codebases, it could become time-consuming due to the sheer number of referenced symbols.
We propose a method to efficiently detect clones taking into account contextual information. We introduce a tool that uses an AST-parsing library named JavaParser to detect clones and retrieve contextual information. Our method parses the Abstract Syntax Tree retrieved from JavaParser into a graph structure, which is used to find clones. This graph maps the following relations for each statement in the codebase: the next statement, the previous statement, and the previous cloned statement.
We find that, when taking into account contextual information in our clone detection, 11% fewer clones are found. Manually inspecting a sample of the difference, we find that they are less relevant for refactoring.

Abstract

We compare two approaches to discover recurrent fragments in source code: clone detection and frequent subtree mining. We apply both approaches to a medium-sized Java case and compare qualitatively and quantitatively their results in terms of what types of code fragments are detected, as well as their size, relevance, coverage, and level of detail. We conclude that both approaches are complementary, while existing overlap may be used for cross-validation.

Abstract

CodeCity is a software analysis tool with the goal of visualizing software systems as interactive, navigable 3D cities. They rely on the city metaphor, that uses the layout of the city in order to visualize different metrics about software systems. There were other implementations of CodeCity among the years but there are no modern options that contemplate the representation of time evolution of the city. We propose a solution to this time evolution, developing a web version of CodeCity that works with any device that has a web browser. Then, we analyze different projects and show the time evolution of their cities. Finally, we conclude with some needed features and drawbacks that are going to be considered in future works.

Abstract

Information systems rely on decision knowledge during their execution. A recently introduced standard, the Decision Model and Notation (DMN), has been adopted in both industry and academia as a suitable method for modelling decision knowledge. However, this decision knowledge is not static and may undergo changes after system deployment. DMN change patterns have not yet been studied in the literature. This paper fills this gap by presenting an initial set of DMN change patterns. The patterns presented in this paper will not only facilitate the understanding of decision change management, but can also be capitalised on for, among other things, adapting decision management systems to be more flexible, consistency checking of decision models, and developing modelling tools that facilitate those changes.

Abstract

Continuously attracting contributors is key to the health of open source software projects. The appearance of badges in online collaborative development platforms affords maintainers the opportunity to advertise the quality of their project to potential contributors. We speculate that contributors rely on those badges to evaluate which projects they should contribute to. In this preliminary research, we measure how prevalent badges are in open source projects, which badges are used, when and how they are introduced, and which combinations of badges co-occur.
We find that the most widespread badges convey static information or relay information about the build status of a project. %, as determined by an external tool.
Those badges are typically added early in projects and prior to or at the same time as other badges.

Abstract

Open-source software (OSS) relies on contributions from different types of contributors. Online collaborative development platforms, such as GitHub, usually provide explicit support for these contributions through the mechanism of pull requests, allowing project members and external contributors to discuss and evaluate the submitted code. These discussions can play an important role in the decision-making process leading to the acceptance or rejection of a pull request. We empirically examine in this paper 184K pull requests and their discussions, for almost 4800 GitHub repositories for the Cargo ecosystem. We investigate the prevalence of such discussions, their participants and their size in terms of messages and durations, and we show how these aspects relate to the acceptance and rejection of pull requests.

Abstract

Communities of software practice are increasingly playing a central role in the development, operation, maintenance, and evolution of good-quality software, as well as DevOps pipelines, lean Organizations, and Global Software Development. However, it is still unknown the structures and characteristics behind such communities. For this reason, in this paper, we tried to explore the organizational secret of communities, trying to offer a few practical extracts of (1) what we know and is known, (2) what we know to be unknown, and (3) what we know to be tentatively discoverable in the near future from an empirical research point of view. Moreover, the paper provides a number of recommendations for practitioners to help and be helped in their community endeavors.

Abstract

Most of existing quality models rely on aggregation as weighted linear combination of software metrics, where weights are determined manually. Aggregation using such approach results in subjective quality models that are hard to interpret. To address this challenge, we propose an automated aggregation based on joint software metrics distributions to allow sound aggregation and results that are easier to interpret. To evaluate the effectiveness of our approach, we conduct an empirical study on maintainability assessment for ca. 5000 classes from open source software systems written in Java, and compare our approach with classical weighted linear combination approach. The results show that the approaches are in agreement, but also that our approach is more interpretable, sensitive, and actionable.