ICSME 2014

This page gathers all the information necessary to reproduce the experiment made for an article submitted to ICSME 2014.


Functional testing requires executing particular sequences of user actions. Test automation tools enable scripting user actions such that they can be repeated more easily. Selenium, for instance, enables testing web applications through scripts that interact with a web browser and assert properties about its observable state. However, little is known about how common such tests are in practice. We therefore present a cross-sectional quantitative study of the prevalence of \selenium{}-based tests among open-source web applications, and of the extent to which such tests are used within individual applications. Automating functional tests also brings about the problem of maintaining test scripts. As the system under test evolves, its test scripts are bound to break. Even less is known about the way test scripts change over time. We therefore also present a longitudinal quantitative study of whether and for how long test scripts are maintained, as well as a longitudinal qualitative study of the kind of changes they undergo. To the former's end, we propose two new metrics based on whether a commit to the application's version repository touches a test file. To the latter's end, we propose to categorize the changes within each commit based on the elements of the test upon which they operate. As such, we are able to identify the elements of a test that are most prone to change.

Scrapping GitHub

We use a ruby script based on Selenium to scrap GitHub. This first script stores the hits onto an sqlite database [143927 hits, 70 MB] which contain basic information for each hit such as repository url, file url, keyword and targeted language. Our second script uses a wrapper of the github search api for collecting repository-wise information such as size, creation date, number of contributors, etc. Those results are stored onto an other sqlite database [7977 repositories, 200 MB].

Scrapping GitHub GitHub scrap.rb scrap.sqlite github.js coarse.js coarse.sqlite

Mining and Catergorizing Git Clones

Independently from the language, 580 of the best git repositories from the coarse database were cloned. A JS script and its dependencies counts the number of Java file/loc as well as the number of Java Selenium file/loc and stores these information mine.sqlite. On the initial 580, 287 do have at least one Java selenium file. Note that it would have been more efficient to discard repositories that do not contain any Java hit in scrap.sqlite. According to their documentation, the 287 repositories were manually classified into five categories: (1) web services, (2) web frameworks, (3) code example, (4) selenium extension, (5) miscellaneous. This categorization is stored in a csv file. A R script has been used to generate the scatter plots present in the article.

Mining and Catergorizing Git Clones Git mine.js mine.sqlite dep.zip www mine_cat.csv plot_cat.r plot_cat.zip

Expensive Metrics on Git Clones

Computationally expensive metrics are computed on the best 8 projects of the 287 from the previous section. The first JS script detects commits that affect selenium files ; it uses the same dependencies as mine.js. The first sqlite database contains information relative to selenium commits such as commit date and blob oids. The second JS script detects renames (merge a pair addition - deletion into a modification) and regenerations (split a modification into a pair addition - deletion). The second sqlite database is the corrected version of metric.sqlite and is used to compute both the commit and the life metrics.

The first JS script of the commit branch simply summarizes information contained into metric2.sqlite onto a new sqlite database. The second JS script of the commit branch merges successive Selenium and non-Selenium commits to create Selenium and non-Selenium spans. This can be found in a sqlite database.

The second metric is centered around Selenium tests instead. The JS script of the life branch computes chains of related blobs that account as the life of a selenium test. Each selenium life can be found in a sqlite database.

Layer 1 Git metric.js metric.sqlite metric2.js metric2.sqlite metric_commit.js metric_commit.sqlite metric_commit_span.js metric_commit_span.sqlite metric_life.js metric_life.sqlite dep.zip


The author cannot be responsible to any harm that the content of this page could have cause to your computer. Moreover, the author does not guarantee that scripts present in this web-page will run as such. Due to hazards during the experiment, scripts have been slightly modified to process only parts of the corpora or alternative databases not present in this web-page. The following environments need to be installed on your machine to utilize the content of this page: (i) node.js with sqlite3, (ii) ruby, (iii) R. If you encounter difficulties to set things up, you can contact the author at lachrist@vub.ac.be.