Rest2LDAP requests hang indefinitely if an async request thread encounters a RuntimeException

[GuyPaddock]

Summary

If a background request thread encounters a RuntimeException (NullPointerException, IllegalStateException, etc), that thread silently dies off, leaving the main request thread blocking idefinitely on an Object.wait() call.

Affected Version

master (4.0.0-SNAPSHOT)

Steps to Reproduce

One possible way to reproduce this was mentioned in my post on the FR forums:
https://forum.forgerock.com/topic/possible-bug-in-crest-when-using-clientnaming/#post-17428

A slightly easier way is to simply modify the read() method of JsonConstantPropertyMapper to throw a NullPointerException inside the promsise, as follows:

    Promise<JsonValue, ResourceException> read(final Context context, final Resource resource,
                                               final JsonPointer path, final Entry entry) {
        final Promise<JsonValue, ResourceException> resultPromise =
            newResultPromise(new JsonValue(null));

        return resultPromise.then(
          new Function<JsonValue, JsonValue, ResourceException>() {
              @Override
              public JsonValue apply(JsonValue jsonValue) throws ResourceException {
                  throw new NullPointerException("Test");
              }
          });
    }

Then run the BasicRequestsTest.

Expected Results

A request that unexpectedly encounters a RuntimeException while assembling a response returns an appropriate message along with a 500 Internal Server error.

Actual Results

The request hangs indefinitely. Multiple subsequent requests that each result in a runtime exception eventually cause the application server to become unresponsive because there are no more threads available in the thread pool. The only remedy is to restart the application server.

The main thread stack looks something like this:

	  at java.lang.Object.wait(Object.java:-1)
	  at java.lang.Object.wait(Object.java:502)
	  at org.forgerock.util.promise.PromiseImpl.await(PromiseImpl.java:618)
	  at org.forgerock.util.promise.PromiseImpl.getOrThrow(PromiseImpl.java:144)
	  at org.forgerock.json.resource.AbstractAsynchronousConnection.query(AbstractAsynchronousConnection.java:80)
	  at org.forgerock.json.resource.AbstractAsynchronousConnection.query(AbstractAsynchronousConnection.java:89)
	  at org.forgerock.opendj.rest2ldap.BasicRequestsTest.testQueryAll

The cause of this behavior appears to be that the asynchronous promise-based method Rest2LDAP uses to obtain results has no top-level exception handler on the result threads.

As a potential way to mitigate this issue, it may make sense to modify org.forgerock.json.resource.AbstractAsynchronousConnection.query() to use a sensible timeout (5 mins?). Note that this code changed substantially between json-resource-21.0.0-alpha-6 (which is what Wren has in source control) and json-resource-21.0.0-alpha-17 (which is only available in Wren's FR archive).

[Kortanul]

@GuyPaddock good write-up. let's use labels instead of 'bug' in the title, though.

[GuyPaddock]

Understood, @Kortanul! It doesn't give me the option to add a label, though. I think that's admin-only.