Ftps proxy error

README_REPRODUCE.md

@@ -0,0 +1,150 @@
+# How To Reproduce 
+
+This reproducer refers to [this ticket](https://issues.apache.org/jira/browse/NET-718) which does report an "Unsupported or unrecognized SSL message" error. This error surfaced when we bumped from 2.8 to 2.9 [here](https://github.com/akka/alpakka/pull/2945).
+
+While making the reproducer, a different error happens:
+
+```
+javax.net.ssl.SSLException: Connection reset
+```
+
+The scenario is when running an FTPS connection via proxy, as shown in the reproducer spec [FTPSProxyClientTest.java](./src/test/java/org/apache/commons/net/ftp/FTPSProxyClientTest.java). 
+
+Before you run this spec, you have to have a proxy (squid) and an FTP server running.
+
+## Squid + FTP on Linux
+
+```
+docker-compose up
+```
+
+## Squid + FTP on Mac
+
+The squid part of docker-compose depends on `network_mode: host` which does not work on Mac. You have to run squid locally and only start ftp via docker-compose.
+
+```
+brew install squid
+```
+
+Once done, put the [content of the squid config file](src/test/resources/squid.conf) into `/usr/local/etc/squid.conf`.
+
+```
+brew services restart squid
+```
+
+Now start ftp via `docker-compose up ftp`.
+
+---
+
+Now, if you run the spec, you should see:
+
+```
+javax.net.ssl.SSLException: Connection reset
+
+	at java.base/sun.security.ssl.Alert.createSSLException(Alert.java:127)
+	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:326)
+	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269)
+	at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:264)
+	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:137)
+	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1144)
+	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1055)
+	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:395)
+	at org.apache.commons.net.ftp.FTPSClient._openDataConnection_(FTPSClient.java:278)
+	at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:639)
+	at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:1989)
+	at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2085)
+	at org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:2283)
+	at org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:2249)
+	at org.apache.commons.net.ftp.FTPSProxyClientTest.testListFiles(FTPSProxyClientTest.java:94)
+	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
+	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
+	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
+	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
+	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
+	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
+	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
+	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
+	at java.base/java.lang.Thread.run(Thread.java:834)
+	Suppressed: java.net.SocketException: Broken pipe (Write failed)
+		at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
+		at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
+		at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
+		at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81)
+		at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:357)
+		... 25 more
+Caused by: java.net.SocketException: Connection reset
+	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:186)
+	at java.base/java.net.SocketInputStream.read(SocketInputStream.java:140)
+	at java.base/sun.security.ssl.SSLSocketInputRecord.read(SSLSocketInputRecord.java:448)
+	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:165)
+	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
+	... 22 more
+
+
+javax.net.ssl.SSLHandshakeException: Remote host terminated the handshake
+
+	at java.base/sun.security.ssl.SSLSocketImpl.handleEOF(SSLSocketImpl.java:1313)
+	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1152)
+	at java.base/sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1055)
+	at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:395)
+	at org.apache.commons.net.ftp.FTPSClient._openDataConnection_(FTPSClient.java:278)
+	at org.apache.commons.net.ftp.FTPClient._openDataConnection_(FTPClient.java:639)
+	at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:1989)
+	at org.apache.commons.net.ftp.FTPClient.initiateListParsing(FTPClient.java:2085)
+	at org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:2283)
+	at org.apache.commons.net.ftp.FTPClient.listFiles(FTPClient.java:2249)
+	at org.apache.commons.net.ftp.FTPSProxyClientTest.testListFiles(FTPSProxyClientTest.java:94)
+	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
+	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
+	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
+	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
+	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
+	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
+	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
+	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
+	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
+	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
+	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
+	at java.base/java.lang.Thread.run(Thread.java:834)
+	Suppressed: java.net.SocketException: Broken pipe (Write failed)
+		at java.base/java.net.SocketOutputStream.socketWrite0(Native Method)
+		at java.base/java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:110)
+		at java.base/java.net.SocketOutputStream.write(SocketOutputStream.java:150)
+		at java.base/sun.security.ssl.SSLSocketOutputRecord.encodeAlert(SSLSocketOutputRecord.java:81)
+		at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:357)
+		at java.base/sun.security.ssl.TransportContext.fatal(TransportContext.java:269)
+		at java.base/sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:398)
+		... 19 more
+Caused by: java.io.EOFException: SSL peer shut down incorrectly
+	at java.base/sun.security.ssl.SSLSocketInputRecord.decode(SSLSocketInputRecord.java:167)
+	at java.base/sun.security.ssl.SSLTransport.decode(SSLTransport.java:108)
+	at java.base/sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1144)
+	... 21 more
+
+SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
+SLF4J: Defaulting to no-operation (NOP) logger implementation
+SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
+Loading file:/Users/sebalf/dev/sebastian-alfers/commons-net/target/test-classes/org/apache/commons/net/ftpsserver/ftpserver.jks
+
+Process finished with exit code 255
+
+```
+
+---
+
+## Possible Solution a
+
+When I uncomment [this lines](https://github.com/apache/commons-net/blob/master/src/main/java/org/apache/commons/net/ftp/FTPSClient.java#L794-L796) the test passes:
+
+```
+if (getProxy() != null) {
+    sslSocket = context.getSocketFactory().createSocket(socket, getPassiveHost(), getPassivePort(), true);
+}
+```
+
+## Possible Solution b
+
+Re-inroduce the call to `super._openDataConnection_(command, arg);` made [here](https://github.com/apache/commons-net/pull/90/files#diff-b4292a5bd3e39f502d24bce1eb934384a951a120080c870cdc68c0585a78c6e9R269).

docker-compose.yml

@@ -0,0 +1,24 @@
+version: '2.2'
+services:
+  ftp:
+    image: stilliard/pure-ftpd@sha256:bfdfe0d725bee473b6c06ec0c39d6956dcaa2a504a1e33825ecf41242f05b436
+    ports:
+      - "21000:21"
+      - "30000-30009:30000-30009"
+    volumes:
+      - ./ftp/tmp/home:/home/username/
+      - ./src/test/resources/ftpd/:/etc/ssl/private/
+    environment:
+      PUBLICHOST: "localhost"
+      FTP_USER_NAME: username
+      FTP_USER_PASS: userpass
+      FTP_USER_HOME: /home/username
+      # https://docs.docker.com/compose/environment-variables/
+      FTP_USER_UID: ${FTP_USER_UID:-2000}
+      FTP_USER_GID: ${FTP_USER_GID:-2000}
+      ADDED_FLAGS: "--tls=1"
+  squid:
+    network_mode: host # required for route back to localhost
+    image: ubuntu/squid
+    volumes:
+      - ./src/test/resources/squid.conf:/etc/squid/squid.conf

src/test/java/org/apache/commons/net/ftp/FTPSProxyClientTest.java

@@ -0,0 +1,103 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *      http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.commons.net.ftp;
+
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+import org.junit.runners.Parameterized.Parameters;
+
+import java.io.IOException;
+import java.net.InetAddress;
+import java.net.InetSocketAddress;
+import java.net.Proxy;
+import java.net.SocketException;
+import java.time.Instant;
+import java.util.Calendar;
+
+import static org.junit.Assert.*;
+
+/**
+ * Tests {@link FTPSClient}.
+ * <p>
+ * To get our test cert to work on Java 11, this test must be run with:
+ * </p>
+ *
+ * <pre>
+ * -Djdk.tls.client.protocols="TLSv1.1"
+ * </pre>
+ * <p>
+ * This test does the above programmatically.
+ * </p>
+ */
+@RunWith(Parameterized.class)
+public class FTPSProxyClientTest extends AbstractFtpsTest {
+
+    private static final String USER_PROPS_RES = "org/apache/commons/net/ftpsserver/users.properties";
+
+    private static final String SERVER_JKS_RES = "org/apache/commons/net/ftpsserver/ftpserver.jks";
+
+    @BeforeClass
+    public static void setupServer() throws Exception {
+        setupServer(IMPLICIT, USER_PROPS_RES, SERVER_JKS_RES, "target/test-classes/org/apache/commons/net/test-data");
+    }
+
+
+    @Parameters(name = "endpointCheckingEnabled={0}")
+    public static Boolean[] testConstructurData() {
+        return new Boolean[] { Boolean.FALSE, Boolean.TRUE };
+    }
+
+    public FTPSProxyClientTest(final boolean endpointCheckingEnabled) {
+        super(endpointCheckingEnabled, null, null);
+    }
+
+    public final String HOSTNAME = "localhost";
+    public final int PORT = 21000;
+    private final Integer PROXYPORT = 3128;
+    private final Proxy PROXY =
+            new Proxy(Proxy.Type.HTTP, new InetSocketAddress(HOSTNAME, PROXYPORT));
+
+    @Test(timeout = TEST_TIMEOUT)
+    public void testListFiles() throws SocketException, IOException {
+        //final FTPSClient client = loginClient();
+        FTPSClient client = new FTPSClient();
+        try {
+
+            client.setProxy(PROXY);
+            client.connect(HOSTNAME, PORT);
+
+            client.login("username", "userpass");
+
+            if (client.getReplyCode() == 530) {
+                trace(">>unable to login");
+            }
+
+            client.enterLocalPassiveMode();
+
+            FTPFile[] files = client.listFiles();
+
+            System.out.println(files.length);
+
+        } finally {
+            client.disconnect();
+        }
+    }
+
+}

src/test/resources/ftpd/pure-ftpd.pem

@@ -0,0 +1,48 @@
+-----BEGIN PRIVATE KEY-----
+MIIEvwIBADANBgkqhkiG9w0BAQEFAASCBKkwggSlAgEAAoIBAQCx/i/7dZ5rKFUD
+PsFb3/xtZF5qSftKj14vPGrTTb85guP1VjAzmFuZE4jFWsr4edJMzOzy5NvBrE5t
+zrefRKbqUc5cwfPT9v1xawn/pj9h7+JMo3VUdqL0eVm8IF0q3dp+KQOrZrcRhsIz
+1JsOqXJmEKagj0X62VYv39jeIWTndDG6h7+54Q5GMzyd/LorEkM69XrnFEtfCF5w
+BVe+mz2UIAO+xDtxbwBTzkOcGDCYEiXTManlJvDCKciQPsfLF0VwmxIkb+xgZ2zB
+2Ckao20twD/LKjZzqlJODde8FMAIQNbgpM8DdZUp5kFhmHj2redw5fdarOr7XE6X
+HizOejcBAgMBAAECggEAX5mnK+iArg17m8KZGD+11QuTqoMR9XoLnFNXDSc1Di6/
+QYaJXSz4Bb/4NggN/GdyDM4EdnF1lxB1D4V4GpNFE5XcwPVrgO2oMsLLHASvBmtR
+oNgqrLML00NabzDy2ZNPR3PABM+mhENdC3mlzro0N5254YMSkykooY4ZbTWCfM2F
+VlqoloLrwwCQGXMCBmMZUKKmofEac03KmG8ouNSSqsEfMtoGfL+M7La1n/lQ51UF
+wK2HgLi4flU34TLUXrGxYC7F1y1+tmm4hZG2WCxJS6uvJ+GekERnIXyWhbIQuMUW
+sOBft3A75BPNFt/1r8ew2nhvjcjRaohIWQscmf6gAQKBgQDtFf1djsA7HNJGpKUg
+ahUlYIt+cucEEesfovufENBOhpNTqWn0JLFVpJorqehSsWvF3taxr8Hf7DyVvlFv
+MO+VD8TSme+xrASHpNwd7rpc2n5QM02IKxgsTh3xCO8sAk7yxfJ9Ue4s6H/6tFsy
+2WC935HHHhgigaZKA/pD2ALCGQKBgQDAMVcw8CoeDoaGE2KSWsjBUSJeoZ5pA77F
++F7kJAmvwYlbdUka6nxuyMbtbLnAVwOacae1xxSXasW0wGULNZKgbD1NdhztpLt0
+mEFtE+uc83Q+HNwrIMDBpsaIqU9OW+U2GOZUUCIHwjdfgDpjygEOUqhc4ALecZyD
+ZsfTSDBJKQKBgQCs5Tlk7gJv2V/bVpx5HAOZw2NW7pJcHHkcFC5tXMScT3XHMCft
+fIi6TRSFPR4ImAxhO1XUNLktBElWZnlanhRJ3zsI3mu3ZRvUk7xWM89CgbBV6mPj
+JpI1VS5upbZNoM1ULFjfXU1VAKS7/qT3WyE6tnzH+cFeALB1D5uFuFSimQKBgQCY
+VsPA46zOtD7HCZiJX3JfYRs4HS0+GhzeONemSDZxXJuupdGzhwfonDapvROjNJWD
+lvETceCNgLGDazjsYKN/iywwOR4G9Bst+P1rI24Psx2Bmkid2tFO7g3SBzn4Z3jQ
+n336eKXwtm5DwZUGwfiCTNxs+ZbskOs5cH+VplO3uQKBgQCkVfpt6eufT85ArjwY
+EzNI4T/6DXc/b/v6d+7Tk7LafxPIjBkM8bnKPSHqsBA3HJ7630xLg880YSN0ew6d
+tTeCIFFRodYcslFNfnNOqkA2x4q4pEmjAk+ChHg+s9jvarBAHFLTSPgkcMiQru11
+TXpsVPrNzl+eQObY/dNouIvuYw==
+-----END PRIVATE KEY-----
+-----BEGIN CERTIFICATE-----
+MIIDQDCCAiigAwIBAgIJAI/KIXjP2G2BMA0GCSqGSIb3DQEBCwUAMDUxEjAQBgNV
+BAMMCWxvY2FsaG9zdDESMBAGA1UECgwJbG9jYWxob3N0MQswCQYDVQQGEwJVUzAe
+Fw0xOTA0MjUwNjM2NTZaFw0yNDA0MjQwNjM2NTZaMDUxEjAQBgNVBAMMCWxvY2Fs
+aG9zdDESMBAGA1UECgwJbG9jYWxob3N0MQswCQYDVQQGEwJVUzCCASIwDQYJKoZI
+hvcNAQEBBQADggEPADCCAQoCggEBALH+L/t1nmsoVQM+wVvf/G1kXmpJ+0qPXi88
+atNNvzmC4/VWMDOYW5kTiMVayvh50kzM7PLk28GsTm3Ot59EpupRzlzB89P2/XFr
+Cf+mP2Hv4kyjdVR2ovR5WbwgXSrd2n4pA6tmtxGGwjPUmw6pcmYQpqCPRfrZVi/f
+2N4hZOd0MbqHv7nhDkYzPJ38uisSQzr1eucUS18IXnAFV76bPZQgA77EO3FvAFPO
+Q5wYMJgSJdMxqeUm8MIpyJA+x8sXRXCbEiRv7GBnbMHYKRqjbS3AP8sqNnOqUk4N
+17wUwAhA1uCkzwN1lSnmQWGYePat53Dl91qs6vtcTpceLM56NwECAwEAAaNTMFEw
+HQYDVR0OBBYEFP/3oItwaZDztL0GoWiM1YMcyb8WMB8GA1UdIwQYMBaAFP/3oItw
+aZDztL0GoWiM1YMcyb8WMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQAD
+ggEBADO6cRZPsqAj/6JZSw177TPlGSq90G2PRdMW+ET0ljqO1c8AHOId7y9nGTgJ
+xEeAaR2ZZ6wLQShRFNF73KGo4vc+jbp/WcxhUCWEDoK4AMKqch6XDjtxid9UH1wj
+wgm2LB223hMzBU4Ub7c3G8b1wpksRYQyNjv/i6DsJIJf+B+p3OoxdVUmbeyen0gA
+rnyEo7IsJLTjf2aWpNIqkL7y9GZeTTPKx4k6uIn+RB3d7u6AFXQ7FdH6Bqu3vslg
+NrOoUSnl1pa5Np1DMzJj/oiJ/8vLN/dVYzl9QDZ3heIHdEROvt1/5anLmhfO0AEH
+X24nJrGi6nZbhh6A/6Kcg9W2Ehg=
+-----END CERTIFICATE-----

src/test/resources/squid.conf

@@ -0,0 +1,7 @@
+debug_options ALL,2
+
+http_port 3128
+acl http_ports port 443 80
+acl ftp_ports port 21 21000 1025-65353
+acl sftp_ports port 22 2222
+http_access deny !http_ports !ftp_ports !sftp_ports