Kedro use cases for Machine Learning (ML) on Databricks

[nthanhtung]

Databricks has native features that support end to end ML :

• Features: AutoML, Managed ML Flow (Model Tracking, Registry, Serving), Feature Store, Repos (Built-in CICD), integration with Delta Lakehouse features (data governance, ETL)
• Demo: https://youtu.be/zQEiwJqqeeA , https://youtu.be/K9DIKk_XFFM
• What will be the use cases for Kedro in this scenario?

---

Btw, still waiting for answers in this topic: #1009

[datajoely]

Hi @nthanhtung - Kedro isn't a substitute for databricks it can fit in as a lightweight mechanism for building out ML pipelines in all of these platform solutions be it Databricks or Sagemaker or similar.

[nthanhtung]

Hi @datajoely , what is the best practice to leverage both Databricks native ML features and Kedro features?
In which use case of bulding ML pipelines we should:

• only use Kedro features
• only use Databricks native ML features
• use both Kedro & Databricks

Databricks has native features that support end to end ML :

• Features: AutoML, Managed ML Flow (Model Tracking, Registry, Serving), Feature Store, Repos (Built-in CICD), integration with Delta Lakehouse features (data governance, ETL)
• Demo: https://youtu.be/zQEiwJqqeeA , https://youtu.be/K9DIKk_XFFM

[datajoely]

It's the final point - I'd really encourage you to follow the spaceflights tutorial here to better understand what Kedro is and isn't.

[nthanhtung]

@datajoely , I have installed Kedro & followed the above tutorial to create a workflow: transform, merge data, then do ML, view linage

But I can also do similar things using Databricks delta live table ETL (SQL transformation, linage, data quality, dependency), then auto ML

So I still haven't got an answer for my above questions