Hi all.
After noticing some tests still misclassified as functional (jalview.ws.jws2 service tests), I was stumped as to the distinction between a 'Network' test and an 'External' test.
1. Our current count of External tests seem to only include Charles' PDB client and UI tests, and the majority of those are also marked as being in the Network group. Are there any tests that are 'External' but *not* network ?
2. We also need tests to ensure Jalview's responses to failure of third party servers work correctly, to ensure Jalview can gracefully fail when third party services are unavailable. These are Functional tests that require a specially configured runtime environment where the server's failure is mocked. Do we have any of these ?
More generally, I'm not convinced we are using TestNG's capabilities effectively (see below), and it needs to be looked at again after the 2.9 release. It will then be essential to have all the issues with the test suite procedure clarified when the new build system is deployed, since it will support a wider variety of test platforms.
tl;dr
We really need to think about how to use the Suites/Groups most effectively here. Code that works with network resources should be covered by both Functional and Network tests.
Network tests clearly depend on the availability of an external resource - the external server which is being connected to. However, a server may be local to the test environment, or it may be truly a third party resource. Local servers should be started prior to test group launch, and are thus reliable unless the test environment has been misconfigured, whereas third party servers may not always be available. This constitutes two distinct test environments - 'Functional' (because we can rely on the local environment being configured to provide the necessary network environment), and 'External Network' (because the same code needs to work correctly with the external resource, and relies on the resource being available as well as being compatible with the code).
It's not clear to me if we are employing TestNG's groups in the most effective way to achieve these distinct test environments. As far as I can see, a group provides method level control of test execution, but it is impossible to create different setup/teardown methods for a particular test class, so any special environment configuration (like configuring network or starting a local instance of a server) needs to be done within the method, which makes the test method more complex. We've already seen this with properties & preferences - and I'm concerned that messing with singleton states within test methods may lead to unexpected interactions since those methods run concurrently.
Of course, we can just create more test classes - one for each instance, to get around the setup/teardown problem, but that means duplication of test method code. Duplication can be avoided by using static library methods (as I've done in the past), but the result is a more complex set of tests, and greater technical debt. How do we avoid that ?
...j.