Merge branch 'main' into feature/watchfiles

Signed-off-by: Jitendra Gundaniya <jitendra_gundaniya@mckinsey.com>

RELEASE.md

@@ -16,7 +16,9 @@ Please follow the established format:
 
 - Improve `kedro viz build` usage documentation (#2126)
 - Fix unserializable parameters value (#2122)
-- Replace `watchgod` library with `watchfiles` (#2134)
+- Replace `watchgod` library with `watchfiles` and improve autoreload file watching filter (#2134)
+- Display full dataset type with library prefix in metadata panel (#2136)
+- Enable SQLite WAL mode for Azure ML to fix database locking issues (#2131)
 
 
 # Release 10.0.0

docs/source/kedro-viz_visualisation.md

@@ -9,6 +9,16 @@ If you haven't installed Kedro {doc}`follow the documentation to get set up<kedr
 We recommend that you use the same version of Kedro that was most recently used to test this tutorial (0.19.0). To check the version installed, type `kedro -V` in your terminal window.
 ```
 
+Once you have installed Kedro, you can install Kedro-viz
+
+```bash
+pip install kedro-viz
+```
+
+```{note}
+[Kedro Viz](https://github.com/kedro-org/kedro-viz) supports all Python versions that are actively maintained by the CPython core team. When a [Python version reaches end of life](https://devguide.python.org/versions/#versions), support for that version is dropped from Kedro-viz. This is not considered a breaking change.
+```
+
 In your terminal window, navigate to the folder you want to store the project. Generate the spaceflights tutorial project with all the code in place by using the [Kedro starter for the spaceflights tutorial](https://github.com/kedro-org/kedro-starters/tree/main/spaceflights-pandas):
 
 
@@ -29,25 +39,15 @@ The next step is optional, but useful to check that all is working. Run the full
 ```bash
 kedro run
 ```
-Kedro-Viz provides stats related to datasets under the metadata panel
-
-![](./images/pipeline_dataset_stats.png)
-
-These stats are generated by hooks. If you have Kedro-Viz installed and execute `kedro run`, the hooks will generate the stats by default. To disable this, you can disable Kedro-Viz hooks in the settings file of your project.
-
-```{note}
-Starting from Kedro-Viz 9.2.0, the dataset stats file `stats.json` will be moved to `.viz` directory at the root of your Kedro project in-case you have `kedro-viz` installed and had not disabled hooks for `kedro-viz` plugin.
-```
 
 To start Kedro-Viz, type the following into your terminal from the project directory:
 
 ```bash
 kedro viz run
 ```
 
-```{important}
-The former `kedro viz` command used here is now deprecated with the release of Kedro-Viz 7.0.0. 
-`kedro viz run` is now the new way to run the tool.
+```{note}
+You can also type `kedro viz` to start Kedro-viz
 ```
 
 The command opens a browser tab to serve the visualisation at `http://127.0.0.1:4141/`.
@@ -63,6 +63,16 @@ If a visualisation panel opens up and a pipeline is not visible, refresh the vie
 
 To exit the visualisation, close the browser tab. To regain control of the terminal, enter `^+c` on Mac or `Ctrl+c` on Windows or Linux machines.
 
+Kedro-Viz provides stats related to datasets under the metadata panel
+
+![](./images/pipeline_dataset_stats.png)
+
+These stats are generated by hooks. If you have Kedro-Viz installed and execute `kedro run`, the hooks will generate the stats by default. To disable this, you can disable Kedro-Viz hooks in the settings file of your project.
+
+```{note}
+Starting from Kedro-Viz 9.2.0, the dataset stats file `stats.json` will be moved to `.viz` directory at the root of your Kedro project in-case you have `kedro-viz` installed and had not disabled hooks for `kedro-viz` plugin.
+```
+
 ## Visualise a Kedro project without installing project dependencies
 
 You can use the experimental `--lite` flag to visualise your pipelines without installing Kedro project dependencies. Add the flag to the command you use to start Kedro-Viz:

package/README.md

@@ -283,17 +283,16 @@ For more information on how to use Kedro-Viz in Visual Studio Code, follow this
 
 ## Feature Flags
 
-Kedro-Viz uses features flags to roll out some experimental features. The following flags are currently in use:
-
-| Flag               | Description                                                                             |
-| ------------------ | --------------------------------------------------------------------------------------- |
-| sizewarning        | From release v3.9.1. Show a warning before rendering very large graphs (default `true`) |
-| expandAllPipelines | From release v4.3.2. Expand all modular pipelines on first load (default `false`)       |
+Kedro-Viz uses features flags to roll out some experimental features. No feature flags are currently in use.
 
 To enable or disable a flag, click on the settings icon in the toolbar and toggle the flag on/off.
 
 Kedro-Viz also logs a message in your browser's [developer console](https://developer.mozilla.org/en-US/docs/Learn/Common_questions/What_are_browser_developer_tools#The_JavaScript_console) to show the available flags and their values as currently set on your machine.
 
+
+## Python version support policy
+Kedro Viz supports all Python versions that are actively maintained by the CPython core team. When a [Python version reaches end of life](https://devguide.python.org/versions/#versions), support for that version is dropped from Kedro-viz. This is not considered a breaking change.
+
 ## Maintainers
 
 Kedro-Viz is maintained by the [Kedro team](https://docs.kedro.org/en/stable/contribution/technical_steering_committee.html#kedro-maintainers) and a number of [contributors from across the world](https://github.com/kedro-org/Kedro-Viz/contributors).

package/kedro_viz/database.py

@@ -1,18 +1,41 @@
 """Database management layer based on SQLAlchemy"""
 
-from sqlalchemy import create_engine
+import os
+
+from sqlalchemy import create_engine, text
 from sqlalchemy.orm import sessionmaker
 
 from kedro_viz.models.experiment_tracking import Base
 
 
+def configure_wal_for_azure(engine):
+    """Applies WAL mode to SQLite if running in an Azure ML environment."""
+    is_azure_ml = any(
+        var in os.environ
+        for var in [
+            "AZUREML_ARM_SUBSCRIPTION",
+            "AZUREML_ARM_RESOURCEGROUP",
+            "AZUREML_RUN_ID",
+        ]
+    )
+    if is_azure_ml:
+        with engine.connect() as conn:
+            conn.execute(text("PRAGMA journal_mode=WAL;"))
+
+
 def make_db_session_factory(session_store_location: str) -> sessionmaker:
     """SQLAlchemy connection to a SQLite DB"""
     database_url = f"sqlite:///{session_store_location}"
     engine = create_engine(database_url, connect_args={"check_same_thread": False})
-    session_class = sessionmaker(engine)
     # TODO: making db session factory shouldn't depend on models.
     # So want to move the table creation elsewhere ideally.
     # But this means returning engine as well as session class.
+
+    # Check if we are running in an Azure ML environment if so enable WAL mode.
+    configure_wal_for_azure(engine)
+
+    # Create the database tables if they do not exist.
     Base.metadata.create_all(bind=engine)
-    return session_class
+
+    # Return a session factory bound to the engine.
+    return sessionmaker(bind=engine)

package/tests/test_integrations/test_sqlite_store.py

@@ -8,7 +8,7 @@
 import boto3
 import pytest
 from moto import mock_aws
-from sqlalchemy import create_engine, func, select
+from sqlalchemy import create_engine, func, select, text
 from sqlalchemy.orm import sessionmaker
 
 from kedro_viz.database import make_db_session_factory
@@ -372,3 +372,22 @@ def test_sync_with_merge_error(self, mocker, store_path, remote_path, caplog):
         mock_merge.assert_called_once()
         mock_upload.assert_called_once()
         assert "Merge failed on sync: Merge failed" in caplog.text
+
+    def test_make_db_session_factory_with_azure_env_var(self, mocker, tmp_path):
+        """Test that WAL mode is enabled when running in an Azure environment."""
+        mocker.patch.dict(
+            os.environ,
+            {
+                "AZUREML_ARM_SUBSCRIPTION": "dummy_value",
+                "AZUREML_ARM_RESOURCEGROUP": "dummy_value",
+            },
+        )
+        db_location = str(tmp_path / "test_session_store.db")
+        session_class = make_db_session_factory(db_location)
+
+        # Ensure that the session can be created without issues.
+        with session_class() as session:
+            assert session is not None
+            # Check if the database is using WAL mode by querying the PRAGMA
+            result = session.execute(text("PRAGMA journal_mode;")).scalar()
+            assert result == "wal"

src/components/metadata/metadata.js

@@ -121,10 +121,17 @@ const MetaData = ({
 
   const shortenDatasetType = (value) => {
     const isList = Array.isArray(value);
+    // Extract the library (first part) and the dataset type (last part)
+    const getQualifier = (val) => {
+      if (typeof val === 'string' && val.includes('.')) {
+        const parts = val.split('.');
+        return `${parts[0]}.${parts.pop()}`;
+      }
+      // If val is not a string or does not include a dot return as is
+      return val;
+    };
 
-    return isList
-      ? value.map((val) => val.split('.').pop())
-      : value?.split('.').pop();
+    return isList ? value.map(getQualifier) : getQualifier(value);
   };
 
   return (

src/components/metadata/metadata.test.js

@@ -303,7 +303,7 @@ describe('MetaData', () => {
         mockMetadata: nodeData,
       });
       const row = rowByLabel(wrapper, 'Dataset Type:');
-      expect(textOf(rowValue(row))).toEqual(['CSVDataset']);
+      expect(textOf(rowValue(row))).toEqual(['pandas.CSVDataset']);
     });
 
     it('shows the node filepath', () => {
@@ -402,7 +402,7 @@ describe('MetaData', () => {
           mockMetadata: nodeTranscodedData,
         });
         const row = rowByLabel(wrapper, 'Original Type:');
-        expect(textOf(rowValue(row))).toEqual(['SparkDataset']);
+        expect(textOf(rowValue(row))).toEqual(['spark.SparkDataset']);
       });
 
       it('shows the node transcoded type', () => {
@@ -411,7 +411,7 @@ describe('MetaData', () => {
           mockMetadata: nodeTranscodedData,
         });
         const row = rowByLabel(wrapper, 'Transcoded Types:');
-        expect(textOf(rowValue(row))).toEqual(['ParquetDataset']);
+        expect(textOf(rowValue(row))).toEqual(['pandas.ParquetDataset']);
       });
     });
     describe('Metrics dataset nodes', () => {