How to allow package access, without breaking the classes

[LDAP]

Describe the bug
I would like to allow only java.lang and java.io. But when students try to read console input (like thy should) the test freezes and runs into the timeout.

I used:

@BlacklistPackage("**")
@WhitelistPackage("java.lang**")
@WhitelistPackage("java.io**")

This may be caused by java.io accessing a restricted package.

Expected behavior
The access of a blacklisted package through a whitelisted package should be allowed.

Additional context
Tested locally.

[MaisiKoleni]

Trying

@Public
@StrictTimeout(1)
@BlacklistPackage("**")
@WhitelistPackage({ "java.lang", "java.nio**", "java.io**" })
class BasicExampleTest {

	@Test
	void testExample(IOTester io) {
		io.in().addLinesToInput("Christian");

		BasicExample.main(new String[0]);

		String actualOutput = io.out().getOutputAsString();
		if (!"""
				Enter your name:
				Hello Christian!""".equals(actualOutput))
			fail("'Wrong output!");
	}
}

and

import java.io.BufferedReader;
import java.io.IOException;
import java.io.InputStreamReader;

public class BasicExample {

	public static void main(String[] args) {
		try (var bufferedReader = new BufferedReader(new InputStreamReader(System.in))) {
			System.out.println("Enter your name:");
			var name = bufferedReader.readLine();
			System.out.println("Hello " + name + "!");
		} catch (IOException e) {
			e.printStackTrace();
		}
	}
}

worked for me.

But you are right that java.nio is needed here, although it is not directly used. I can have a look at that. But I never ran into timeout problems.

[LDAP]

Thanks! I'll try that. Although, I would prefer a solution where only the "first" accessed class is restricted and if the call goes over a whitelisted class the call is permitted.

[MaisiKoleni]

I can have a look at that, but I doubt I can fix that easily (the solution that comes to my mind would not protect against reflective access, as Class/ClassLoader are whitelisted).

Also keep in mind that @BlacklistPackage is experimental, there are some situations where it does not work for some reason related to how Java is loading classes and checking package access (e.g. it can not be allowed for one test and disabled for another).

[LDAP]

I am still running into timeout issues when I add the annotations. The debug output does not really help:

[INFO] [main] Request install with ASC-Impl [testMethod=Optional[public java.util.stream.Stream edu.kit.informatik.ProtocolTest.generateTests() throws java.io.IOException,java.lang.IllegalAccessException,java.lang.IllegalArgumentException,java.lang.reflect.InvocationTargetException], executionPath=D:\Local_Repositories\Studium\Tutor\Test-Framework\Examples\ProtocolTest2, whitelistedPaths=Optional[[]], blacklistedPaths=[], allowedLocalPorts=[], trustedPackages=[PackageRule["edu.kit.kastel.sdq.artemis**" in WHITELIST]]]
[INFO] [main] Request whitelisting: Thread[ajts-to-4,5,Test-Threadgroup] []
[INFO] [main] Thread whitelisted: Thread[ajts-to-4,5,Test-Threadgroup]
[INFO] [ajts-to-4] Request whitelisting: Thread[Thread-3,5,Test-Threadgroup] [Thread[ajts-to-4,5,Test-Threadgroup]]
[INFO] [ajts-to-4] Thread whitelisted: Thread[Thread-3,5,Test-Threadgroup]
[DEBUG] [main] State WAITING after interrupt and join of Thread[ajts-to-4,5,Test-Threadgroup]
[DEBUG] [main] State TERMINATED after interrupt and join of Thread[Thread-3,5,]
[DEBUG] [main] Try 1 to stop Thread[ajts-to-4,5,Test-Threadgroup], state: WAITING
[ERROR] Tests run: 3, Failures: 3, Errors: 0, Skipped: 0, Time elapsed: 46.056 s <<< FAILURE! - in edu.kit.informatik.ProtocolTest
[ERROR] generateTests().Private Test: Test extended functionality  Time elapsed: 15.283 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: execution timed out after 15 s

[ERROR] generateTests().Example Interaction 1  Time elapsed: 15.275 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: execution timed out after 15 s

[ERROR] generateTests().Example Interaction 2  Time elapsed: 15.27 s  <<< FAILURE!
org.opentest4j.AssertionFailedError: execution timed out after 15 s

Maybe it's because the student submission runs in its own thread and is called by reflection?

[MaisiKoleni]

Yes, very likely, especially when the thread is started outside the test and you are providing input in the test.

[LDAP]

Is this a limit of AJTS or did I something wrong? I used @AddTrustedPackage for the package where the Thread gets created and and called requestThreadWhitelisting(Thread t) before starting the thread.

Unrelated: Could you briefly explain what invokeBlacklisted and privilegedThrow do?

[MaisiKoleni]

If you want the students to wait for input and then react and always provide input after the student code requests the input, this will not work with Ares as it is now.

The input/output testing in Ares was created such that all read operations on the input stream either directly succeed or fail (because we wanted to avoid any timeouts).

[LDAP]

I do not use AJTS Input/Output Testing. I use my own thing. Without the Blacklist/Whilelist that does work like a charm. But with, the timeouts occur.

[MaisiKoleni]

Is this a limit of AJTS or did I something wrong? I used @AddTrustedPackage for the package where the Thread gets created and and called requestThreadWhitelisting(Thread t) before starting the thread.

If the package of AddTrustedPackage is a package the students can edit this effectively disables the security. Otherwise, it is fine. I guess the latter is the case if you have access to the thread object.

Unrelated: Could you briefly explain what invokeBlacklisted and privilegedThrow do?

invokeBlacklisted will execute the passed function on the same level as student code. This is normally not needed unless you are handling classes that students could have declared to be in trusted packages (also see #15). I use this in Ares itself often e.g. when invoking methods on exceptions and other non-final classes.

privilegedThrow is for privileged exceptions and allows the tests to throw exceptions that pass through @PrivilegedExceptionsOnly. For more details, see the Javadoc of PrivilegedExceptionsOnly, there is an example and many explanations.

[MaisiKoleni]

I do not use AJTS Input/Output Testing. I use my own thing. Without the Blacklist/Whilelist that does work like a charm. But with, the timeouts occur.

I assume "Without the Blacklist/Whilelist" means with Ares but just without those annotations. Then it depends on how this custom solution works. My guess is that the custom solution somehow catches the package access security exception and then uses a different strategy after that or just works differently. Or hangs in some other way.

[LDAP]

Is this a limit of AJTS or did I something wrong? I used @AddTrustedPackage for the package where the Thread gets created and and called requestThreadWhitelisting(Thread t) before starting the thread.

If the package of AddTrustedPackage is a package the students can edit this effectively disables the security. Otherwise, it is fine. I guess the latter is the case if you have access to the thread object.

Unrelated: Could you briefly explain what invokeBlacklisted and privilegedThrow do?

invokeBlacklisted will execute the passed function on the same level as student code. This is normally not needed unless you are handling classes that students could have declared to be in trusted packages (also see #15). I use this in Ares itself often e.g. when invoking methods on exceptions and other non-final classes.

privilegedThrow is for privileged exceptions and allows the tests to throw exceptions that pass through @PrivilegedExceptionsOnly. For more details, see the Javadoc of PrivilegedExceptionsOnly, there is an example and many explanations.

Thanks for your explanation!

I do not use AJTS Input/Output Testing. I use my own thing. Without the Blacklist/Whilelist that does work like a charm. But with, the timeouts occur.

I assume "Without the Blacklist/Whilelist" means with Ares but just without those annotations. Then it depends on how this custom solution works. My guess is that the custom solution somehow catches the package access security exception and then uses a different strategy after that or just works differently. Or hangs in some other way.

Yes! Using Ares but without any additional annotations. Just for my understanding: When a student accesses a blacklisted package then a SecurityException is thrown?

[MaisiKoleni]

Yes, for example inserting

Class.forName("java.util.List").isInterface();

in my given example will yield:

java.lang.SecurityException: access to package java.util denied in line 14 in BasicExample.java
/// potential problem location: ajts.example.BasicExample.main(BasicExample.java:14) ///

[LDAP]

Is the @AddTrustedPackage annotation transitive? That means, can a trusted package declare trusted packages itself? Or is there another method that should be used in this case?

[MaisiKoleni]

Is the @AddTrustedPackage annotation transitive? That means, can a trusted package declare trusted packages itself? Or is there another method that should be used in this case?

All annotations are only relevant in a test context. So only the test class and its members can use the annotations.

[MaisiKoleni]

I will make a discussion out of this because this is/became one. Tell me if there are still problems related to this (I guess there is already a related issue #105 for that). Please select which of my answers helped (if they did, but there were not further messages here, so I guess one did) to resolve this discussion.

[MaisiKoleni]

Marked this as resolved due to inactivity, I have to assume one of my answers was sufficient.

[LDAP]

We did switch away from Ares since it did not meet our requirements anymore.

[MaisiKoleni]

May I ask what you use now, or what those requirements were?

[LDAP]

Sure.

We have a quite complex interactive IO-testing setup that did not work reliably with Ares, we run into issues like this one or #105.
Generally, we needed more control over what happens in a test run and the output that is displayed to students.

We now use a custom implementation which is not publicly available.

[MaisiKoleni]

Ok, thank you for the information.