Full description of the investigated open issues

#bluebird - petkaantonov/bluebird#1449

Bluebird is a promise library with innovative features and performance; it is used by >3.2M projects in GitHub, and has more than 13.7M weekly downloads in npm. The issue is related to an extension that allows to set up a timeout for a promise to be fulfilled; if the promise is not resolved after the defined delay, the promise is rejected with a TimeoutError. The combination of this extension with setTimeout has brought an inconsistent behavior, i.e., the TimeoutError was not intuitively expected (Open issue \#1449, described in petkaantonov/bluebird#1417). The test provided in the issue reproduces the issue in almost all execution using vanilla Node.js. In this case, we used NodeRacer to show that a correct output may be produced for the test (28 out of 100 times) and it is a race bug. 

While a solution was initially dismissed due to the inconsistent behavior of setTimeout, a fix was proposed (petkaantonov/bluebird#1449). With the fix, the test did not fail anymore using vanilla Node.js or NodeRacer. During the analysis of happens-before and traces, we noticed that the race arises when a timeout occurs. So, we slightly modified the test so that the timeout is now expected.  NodeRacer caused this test to fail in 34 out of 100 runs. Therefore, programmers may still be mislead by such use case of Bluebird promise's timeout and native setTimeout. One may argue that such remaining inconsistency is less harmful as the unexpected behavior is not an error but the fullfillment of a promise. 

#express - expressjs/express#3536

Express is a minimalist Web framework to implement RESTful APIs; it is used by >4.8M projects in GitHub, and has more than 10M weekly downloads in npm. The issue is about returning a prettified JSON response for a request that has a certain parameter. The code snippet for the first proposed solution is claimed to be subjected to race conditions.  Based on this snippet and descriptions of the issue, we designed a small express application using also a MongoDB database. Our test sent two trivial requests and one request asking for a prettified JSON, checking the format of the three responses. Using vanilla Node.js, the test was not able to hit the race even after 100 executions.  We then applied NodeRacer to the test, hitting the race bug 62 out of 100 runs. This confirmed that this first solution was subjected to races.

We notice that the race is not actually in the library, but it may be introduced by programmers using express. The proposed solution involved a change per request in the global variable representing the express application. While this race only affects the output format of a request, races may appear in similar scenarios that modify such global variable. Besides this issue, there is also an open pull on making this global settings available locally per request (expressjs/express#2935). We then adopted NodeRacer to test an alternative solution that formats the output locally per request. After 100 runs and no errors, NodeRacer provided extra evidence that this solution is not subjected to similar races.

#get-port - sindresorhus/get-port#23

Get-port is a library that returns an available TCP port; it is used by >130K projects in GitHub, and has more than 1.5M weekly downloads in npm. The issues reports that several tests are run in parallel and a race condition occurs(sindresorhus/get-port#23). The race actually happens between the time get-port returns an available port and a new server (using such port) is started. So, two or more servers may try to start using the same port but only one will succeed. Based on code snippets in the issue discussion, we came up with a concrete and simple test case that would exercise the race. Using vanilla Node.js, the test was not able to hit the race even after several executions. We then applied NodeRacer to the test, hitting the race 17 out of 100 runs. So, we could confirm with a concrete example and sample runs the presence of the race, and potential failures. 

In the issue discussion, someone suggested using an asynchronous mutex (https://github.com/DirtyHairy/async-mutex) to avoid the race. We modified our test to include the mutex and applied NodeRacer again. This time, the expected error did not show up even after 100 runs. In this case, NodeRacer provided extra evidence that the proposed fix is good. Other observation is that NodeRacer worked in a scenario with a specific synchronization mechanism (locks), without errors or false alarms. Finally, the traces and happens-before relations provided us means to opine in a future enhancement of the library. In particular, we observed that a retention timeout of the returned available port could be implemented without locks and races.

We used a transpiled version due to async-await and generators.

#live-server - tapio/live-server#262

Live-server is a development server with live reload capabilities; it is used by >34K projects in GitHub, and has more than 100K weekly downloads in npm. The issues reports a potential race that may happen between changes in a given file served; if the race does occur, this client would be looking at an outdated version of the file (tapio/live-server#262). As no test was provided, we set up a test that starts Live-Server, modifies an HTML file twice, and uses Puppeteer (https://github.com/GoogleChrome/puppeteer) to observed the file served from the client-side perspective. Using NodeRacer, the test did not fail even after 100 runs. 

After providing our feedback that rejected the aforementioned case, a user pointed out to a specific scenario where an entire directory was deleted and recreated. So, we added two new tests in which a file and a directory are deleted and recreated, respectively. We could confirm that these tests revealed the reported bug and the proposed fix is good for trivial runs. Nevertheless, such tests failed when we applied NodeRacer. As guessed by a user, the fix would not deal with the race. While using NodeRacer, we could observe that the race arised from callbacks of the Chokidar library.

#socket-io - socketio/socket.io#3358

Socket.io-client is the client library of real-time framework socket.io; it is used by >1.1M projects in GitHub, and has more than 3.2M weekly downloads in npm. The issue reports a case where a client does not try to reconnect when two sockets are asynchronously open from the same pool (socketio/socket.io#3358).

The test initially provided reproduced the bug in trivial runs using vanilla Node.js. In this case, we intended to show that the issue was related to some race among the callbacks. Using NodeRacer, the test passed in 48 out of 100 runs; this gave evidence that the bug was actually due to some race condition. We then tested the submitted fix (socketio/socket.io-client#1253); the tests passed in all 100 runs.