[ADAP-970] [Feature] Incremental updates should update table description

[andreas-gjensidige]

Is this your first time submitting a feature request?

• I have read the expectations for open source contributors
• I have searched the existing issues, and I could not find an existing issue for this feature
• I am requesting a straightforward extension of existing dbt-spark functionality, rather than a Big Idea better suited to a discussion

Describe the feature

When running an incremental model the table comment should be updated if the model description has changed.

I think this should be pretty straight forward to implement. dbt-core has a alter_relation_comment-macro which can update the model description. This is then called in the persist_docs-macro. So to implement this in dbt-spark we should only need to create the spark__alter_relation_comment-macro, which can use something like ALTER TABLE table SET TBLPROPERTIES ('comment' = MODEL_DESCRIPTION), and then add a call to the spark__alter_relation_comment-macro in the spark__persist_docs-macro.

I think ALTER TABLE table SET TBLPROPERTIES ('comment' = MODEL_DESCRIPTION) should work generally in Spark, but I have only been able to test this on Databricks with Unity Catalog.

Describe alternatives you've considered

Running a full-refresh after updating the model-description. But I see no reason why doing that should be needed. That just feels like wasted computational resources.

Manually updating the table description, but that's something that will manually have to be kept in sync, which will be forgotten eventually.

Who will this benefit?

People using incremental models who use the persist_docs feature and has model-descriptions.

Are you interested in contributing this feature?

Yes, but I need to clarify with my employers legal department if I'm allowed to or not...

Anything else?

No response

[andreas-gjensidige]

I think I might have messed up this workflow (https://github.com/dbt-labs/dbt-spark/actions/runs/6690653166) when editing the title... I accidentally removed the Iíssue title before submitting, and when updating it straight after submitting I accidentally removed the "[ADAP 970]" part which was automatically added.

[github-actions]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[github-actions]

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

[npsables]

Cmon guys, this is such a good feature.