If a Dataset symlink is created afterwards with a DatasetEvent, the link is not created in the lineage about marquez HOT 3 OPEN

@dkt-sophie-ly, with PR #2641, we should expect the lineage graph to begin using the symlink in the DatasetEvent. But, after looking into how the lineage graph is built:

1. Call LineageDao.getLineage() to get the job node data
2. Then, LineageDao.getDatasetData for the dataset node data

I don't think we've invested heavily on building out symlink support for our lineage graph. @pawel-big-lebowski let me know if that's not the case.

from marquez.

Hi @wslulciuc ! Thanks for your reply :)

With this PR #2736 the lineage graph should be able to see the lineage of symlink dataset but only if the symlinks is built beforehand like that.
ex:

{"input1":
"symlinks": 
"identifiers" [{"namespace": "ns2", "name": "input2"}]
}

Here input1 and input2 can be linked together because they have the same dataset uuid in datasets_view.

If the symlink is created afterwards (both dataset created separately with 2 different runs and then a dataset event add a symlink between these 2 lineage) the lineage won't be linked because they already have a different dataset uuid.

I don't know if it will be possible but it could be great if a symlinks is created afterwards with a DatasetEvent the dataset uuid change accordingly (ex: change input2 dataset uuid to be the same as input1).

from marquez.

Hi @wslulciuc Just a kind reminder in this issue :)

from marquez.