Fix an issue when LD_LIBRARY_PATH is overriden in a Conda environment (…

…#1540)

In a Conda environment, it is sometimes necessary to set LD_LIBRARY_PATH to
first include the Conda's environment libraries before anything else.

This, however, causes problems with the escape hatch which uses this same
modified LD_LIBRARY_PATH *outside* of the Conda environment. This patch
fixes this issue by using the convention that, if a Conda environment
modifies LD_LIBRARY_PATH, it will save the original LD_LIBRARY_PATH
in MF_ORIG_LD_LIBRARY_PATH.

metaflow/plugins/env_escape/client.py

@@ -75,6 +75,15 @@ def inner_init(self, python_executable, pythonpath, max_pickle_version, config_d
             raise RuntimeError("Existing socket: %s" % self._socket_path)
         env = os.environ.copy()
         env["PYTHONPATH"] = pythonpath
+        # When coming from a conda environment, LD_LIBRARY_PATH may be set to
+        # first include the Conda environment's library. When breaking out to
+        # the underlying python, we need to reset it to the original LD_LIBRARY_PATH
+        ld_lib_path = env.get("LD_LIBRARY_PATH")
+        orig_ld_lib_path = env.get("MF_ORIG_LD_LIBRARY_PATH")
+        if ld_lib_path is not None and orig_ld_lib_path is not None:
+            env["LD_LIBRARY_PATH"] = orig_ld_lib_path
+            if orig_ld_lib_path is not None:
+                del env["MF_ORIG_LD_LIBRARY_PATH"]
         self._server_process = Popen(
             [
                 python_executable,