DBT ingestion does not delete all tags

[hjribeiro-moj]

We believe we found a unexpected behaviour caused by this line, where if an update has no tags it does not update tags.

We have replicated the issue on our datahub instances.

[jjoyce0510]

Hi there! This behavior is intentional, if not misleading. The reason for the defensive check on this code is to prevent overwriting tag changes that were made by users on the DataHub UI inadvertently.

What is the impact for your organization?

[LavMatt]

Hi,

I think there may be some unintended behaviour happening too though as there are scenarios where the state of datahub will not match the state of a dbt project.

e.g.

If you have a dbt model with one or more tags and run the ingestion the entity is ingested and the tags are created and associated with that entity.

If you then remove all these tags from your model in your dbt project, the manifest.json file will have an empty list for tags for the model but any ingestion will fail to remove the tags

This is impacting us because we have a lot of entities that had a single tag that has now been removed but we cannot remove it in datahub via the dbt source ingestion.

As a workaround we are tagging every entity with an arbitrary tag so we can ensure the state of tags in datahub reflects the state in our dbt project

[jjoyce0510]

This makes sense. We are internally tracking this as a feature request for "owners" of tags based on the platform where the tag originates. Will update here with any planned work to address this more wholistically.

Thanks