[NEOS-381] make system transformers an enum in proto so they're statically typed about neosync HOT 5 CLOSED

Hey @sibis thanks for wanting to contribute and sorry for the delay in getting back to you!

Yes, we want to make the system transformers into an enum so that they're typed. We have sections in our codebase where we are hardcoding a system transformer and we want to type those fields. For example, here,

	for _, col := range cols {
		if col.Transformer != nil && col.Transformer.Source != "" && col.Transformer.Source != "passthrough" && col.Transformer.Source != "generate_default" {

we want to update this:

col.Transformer.Source != "passthrough"

to something like:

col.Transformer.Source != systemTransformer.Passthrough

There are a few other instances (particularly in our tests) where this occurs. I think the majority of them will be the "Passthrough" transformer but there could be others as well.

So this will require creating an enum in the protos file and then updating any instances where we're hardcoding those system transformers with the reference to the enum.

@nickzelei anything else to add in here?

from neosync.

Hi @sibis - sorry I missed this thread here.

I think the property is called source now, for the jobmappingtransformer message
https://github.com/nucleuscloud/neosync/blob/main/backend/protos/mgmt/v1alpha1/job.proto#L131

This is just a string, but my thinking is that it can be made into a proto enum.

This (at least today) corresponds 1:1 with the source key for SystemTransformer https://github.com/nucleuscloud/neosync/blob/main/backend/protos/mgmt/v1alpha1/transformer.proto#L82

The worker process that processes the transformers when it turns them into a sync config uses a big switch statement and cases off of those string values:
https://github.com/nucleuscloud/neosync/blob/main/worker/pkg/workflows/datasync/activities/activities.go#L624

Ideally, I'd like this to be more statically typed instead of just strings so that we're not passing around string values everywhere.

If the JobMappingTransformer is user defined (meaning: custom) the source is custom today.

from neosync.

Awesome! Will check it out.

from neosync.

Hello @evisdrenova 👋

I'd like to take this up, but I need some clarification on what exactly needs to be changed.

Could you please provide more details on the specific changes expected? For instance, are we looking to create an enum for different types of system transformers, or is there something else that needs to be addressed in the proto file?

Any additional information or examples would be of great help.

from neosync.

Thanks a lot for the detailed explanation, @nickzelei and @evisdrenova. I have made changes to the PR.

from neosync.