Ephemerals are missing columns

[TeckelBrownie]

Thank you for this handy data-lineage tool,

When testing Colibri I found out that my ephemeral models are missing columns. View, incremental and table materializations are having columns. I'm running on databricks.

Regards

[curlyz]

this is possible related to ephemeral → dbt docs generate catalog.json won't have that model → columns not mapped correctly. I think this can be fix @b-ned , do you need a PR for this ?

[b-ned]

Hi @curlyz,

If you can make a PR that would be amazing!

Yeah the ephemeral model will likely not show up with columns in catalog;
https://docs.getdbt.com/docs/build/materializations#ephemeral
To get the columns, I think we might need to parse them from the SQL using SQLGlot ourselves,

To ensure we fix it properly, we need to recreate the manifest.json and catalog.json
(I normally use this jaffle shop repo as a base, to generate the test data: https://github.com/b-ned/jaffle-shop-colibri)

[curlyz]

Hi @b-ned , in my case, I will have effort to fix 2 bug in this.

• the ephemeral one
• and also, handling of select * → sqlglot won't have lineage, we can fallback to manifest.json column lineage

[b-ned]

Hi @curlyz,

Great!

For the second issue, could you make another github issue to describe the issue? And we would need some context there to recreate the issue you are experiencing,

Colibri can actually handle select * quite well:

Example from https://demo.colibri-data.com/

If the select * is a confirmed bug (and we can recreate), let's solve it in a separate branch and PR.