Feature request: Better unit testing of plugins

[DraTeots]

Lets imagine I have beagle_reader plugin which, as the name says, opens and reads "BeAGLE" events file.

Now, having sample BeAGLE file with several events, I would like to write some tests for it. Probably it will not be a unit tests, but more like integration tests. How to do it?

I have several ideas, but all of them aren't pretty:

1. Just compile BeagleEventSource.cc into 'unit_tests" executable, and check GetEvent function. Works, but ugly and with limited functionality.
2. Use tested "writer" plugin (like csv writer). Use python unit tests and write something like:

   subprocess.popen("ejana -pPlugins=beagle_reader,csv_writer sample.txt")
   # open csv and check values

   Too high level and even not an integration, but smoke tests.
3. For each test deploy a JApplication, programmatically open the plugin and the right source. Probably requires C++ test infrastructure written for that. Should tests then be a JProcessors?
4. Use InitPlugin with mocked JApplication to load library and mock unit called. Probably another way to go, but seems too perverse.
5. Have InitTests function in plugins and self unit testing infrastructure...

Soo... I'm out of ideas.

[nathanwbrei]

I like the idea of putting the testing code as close to the plugin being tested as possible, e.g. in the JEventProcessor itself or in an adjacent one) The reason is that otherwise the tests always fall out of date and become an albatross around our necks, plus (at least for me) the parameters of the testing environment are constantly changing.

At some point we'll definitely need to generate csv files out of fairly arbitrary JObjects, and we should talk about how to do this in a clean way. I'd be tempted to not even bother writing a test harness to compare generated csv files, but simply put the files in the same repository so that any changes show up in the diff. (Of course, we'd have to sort them by event number, and there might always small differences due to numerical instabilities)

[DraTeots]

For now THIS considered as an example of how to test plugins

[nathanwbrei]

The latest jana-generate.py generates test case stubs inside the plugin skeleton, though I haven't documented this yet.

[nathanwbrei]

Take a look at examples/UnitTestingExample on v2.0.7.

[nathanwbrei]

Trying to break this down into actionable steps so we can close this issue someday:

• jana-generate.py creates a skeleton for unit tests, but this skeleton is incomplete and undocumented
• UnitTestingExample manually creates a factory and injects it into a dummy JEvent. This is not the recommended approach for users any more, because it has the user call methods that ideally would be internal and private.
• Most built-in plugins don't have unit tests side-by-side, even though that is what jana-generate.py produces. Instead, unit tests live under src/programs/unit_tests and link against plugins directly. Either jana-generate.py should be reworked to match this pattern, or the CMake tooling should be extended so that any unit tests that are scattered among the various plugins can still be linked into a single unit testing executable.
• Either way, JANA2 is in dire need of dedicated CMake abstractions for plugins and their unit tests. These have already been implemented for EICrecon and previously escalate. However, they should be upstreamed and incorporated in jana-generate.py.
• For users' unit tests to work, they need access to catch.hpp. However, catch.hpp shouldn't be installed in the system include directory because it may conflict with an existing catch install (See issue Non-standard installation of plugins to <prefix>/plugins may lead to conflicts #136).
• catch2 has evolved away from being a drop-in header file and now is a full library with deep CMake integration. We need to understand and adopt the new idiomatic way of using catch2.

[nathanwbrei]

Addressed 3 of the 6 actionables in PR #330.

[DraTeots]

• Add testing chapter to the documentation. Describe suggested ways to test elements. If users can use JTest and other built-it infrastructure, describe it.