[Bug] initialize() and compose() do not work on Databricks 13.3 LTS

[jamie-cho]

🐛 Bug

Description

Calling initialize() followed by compose() does not work on Databricks 13.3 LTS. compose() fails with:

MissingConfigException: Primary config directory not found.
Check that the config directory '/root/.ipykernel/config' exists and readable

Checklist

• I checked on the latest version of Hydra
• I created a minimal repro (See this for tips).

To reproduce

Create a notebook in Databricks 13.3 with a cell that contains:

%pip install hydra-core==1.3.2
from hydra import initialize, compose
from hydra.core.global_hydra import GlobalHydra

GlobalHydra.instance().clear() 
initialize(config_path='../config')
cfg = compose(config_name="hydra_test")

** Stack trace/error message **

---------------------------------------------------------------------------
MissingConfigException                    Traceback (most recent call last)
File <command-2822350390308282>, line 7
      5 GlobalHydra.instance().clear() 
      6 initialize(config_path='../config')
----> 7 cfg = compose(config_name="hydra_test")

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-97cc073d-b7de-4e72-a2ab-bbd7de56f65f/lib/python3.10/site-packages/hydra/compose.py:38, in compose(config_name, overrides, return_hydra_config, strict)
     36 gh = GlobalHydra.instance()
     37 assert gh.hydra is not None
---> 38 cfg = gh.hydra.compose_config(
     39     config_name=config_name,
     40     overrides=overrides,
     41     run_mode=RunMode.RUN,
     42     from_shell=False,
     43     with_log_configuration=False,
     44 )
     45 assert isinstance(cfg, DictConfig)
     47 if not return_hydra_config:

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-97cc073d-b7de-4e72-a2ab-bbd7de56f65f/lib/python3.10/site-packages/hydra/_internal/hydra.py:594, in Hydra.compose_config(self, config_name, overrides, run_mode, with_log_configuration, from_shell, validate_sweep_overrides)
    576 def compose_config(
    577     self,
    578     config_name: Optional[str],
   (...)
    583     validate_sweep_overrides: bool = True,
    584 ) -> DictConfig:
    585     """
    586     :param config_name:
    587     :param overrides:
   (...)
    591     :return:
    592     """
--> 594     cfg = self.config_loader.load_configuration(
    595         config_name=config_name,
    596         overrides=overrides,
    597         run_mode=run_mode,
    598         from_shell=from_shell,
    599         validate_sweep_overrides=validate_sweep_overrides,
    600     )
    601     if with_log_configuration:
    602         configure_log(cfg.hydra.hydra_logging, cfg.hydra.verbose)

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-97cc073d-b7de-4e72-a2ab-bbd7de56f65f/lib/python3.10/site-packages/hydra/_internal/config_loader_impl.py:142, in ConfigLoaderImpl.load_configuration(self, config_name, overrides, run_mode, from_shell, validate_sweep_overrides)
    133 def load_configuration(
    134     self,
    135     config_name: Optional[str],
   (...)
    139     validate_sweep_overrides: bool = True,
    140 ) -> DictConfig:
    141     try:
--> 142         return self._load_configuration_impl(
    143             config_name=config_name,
    144             overrides=overrides,
    145             run_mode=run_mode,
    146             from_shell=from_shell,
    147             validate_sweep_overrides=validate_sweep_overrides,
    148         )
    149     except OmegaConfBaseException as e:
    150         raise ConfigCompositionException().with_traceback(sys.exc_info()[2]) from e

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-97cc073d-b7de-4e72-a2ab-bbd7de56f65f/lib/python3.10/site-packages/hydra/_internal/config_loader_impl.py:243, in ConfigLoaderImpl._load_configuration_impl(self, config_name, overrides, run_mode, from_shell, validate_sweep_overrides)
    233 def _load_configuration_impl(
    234     self,
    235     config_name: Optional[str],
   (...)
    239     validate_sweep_overrides: bool = True,
    240 ) -> DictConfig:
    241     from hydra import __version__, version
--> 243     self.ensure_main_config_source_available()
    244     parsed_overrides, caching_repo = self._parse_overrides_and_create_caching_repo(
    245         config_name, overrides
    246     )
    248     if validate_sweep_overrides:

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-97cc073d-b7de-4e72-a2ab-bbd7de56f65f/lib/python3.10/site-packages/hydra/_internal/config_loader_impl.py:129, in ConfigLoaderImpl.ensure_main_config_source_available(self)
    123 else:
    124     msg = (
    125         "Primary config directory not found.\nCheck that the"
    126         f" config directory '{source.path}' exists and readable"
    127     )
--> 129 self._missing_config_error(
    130     config_name=None, msg=msg, with_search_path=False
    131 )

File /local_disk0/.ephemeral_nfs/envs/pythonEnv-97cc073d-b7de-4e72-a2ab-bbd7de56f65f/lib/python3.10/site-packages/hydra/_internal/config_loader_impl.py:102, in ConfigLoaderImpl._missing_config_error(self, config_name, msg, with_search_path)
     99     else:
    100         return msg
--> 102 raise MissingConfigException(
    103     missing_cfg_file=config_name, message=add_search_path()
    104 )

MissingConfigException: Primary config directory not found.
Check that the config directory '/root/.ipykernel/config' exists and readable

Expected Behavior

The cell to successfully execute and set cfg.

System information

• Hydra Version : 1.3.2
• Python version : 3.10
• Virtual environment type and version :
• Operating system : Linux

Additional context

• This issue does not occur in Databricks 12.2
• This issue arrises in Databricks 13.3 LTS due to a change in behavior for returning an absolute path in its stack frames
  • It might look like /root/.ipykernel/2652/command-2258121748915565-587514257
• Further is_notebook() returns False because it is not checking for DatabricksShell
• A fix would involve creating a special case for Databricks, adding a dependency on its sdk to get the notebook entry location.

[omry]

Hi Jamie,
Feel free to submit a PR that addresses your issue without introducing a real dependency on DatabricksShell. (You can check for it's presence dynamically).