tensorflow=2.10.0 to 2.11.1 are incompatible with latest keras-tuner=1.3.4 (python 3.10) [protobuf version limit] about keras-tuner-feedstock HOT 17 CLOSED

issue transfered to keras-tuner feedstock since it seems that the constraint is coming from here.

from keras-tuner-feedstock.

Sorry, no can you make a new PR and remove the limit, please? And increase the Conda recipe version, too.

done. I increased build number from 0 to 1

from keras-tuner-feedstock.

Would you please merge it? it passed the tests

Sorry, no can you make a new PR and remove the limit, please? And increase the Conda recipe version, too.

from keras-tuner-feedstock.

The limit here is wrong. Or perhaps only "wrong", as in: not suitable for what we do in conda-forge (using shared libraries, of which there can only be one per environment, which therefore needs to match for all packages, which means those packages need to be migrated for each new version as we do).

We need to remove it (perhaps even patch the source), because it is unworkable with all of the above.

This is against a common rule ("reflect upstream dependency constraints accurately"), but protobuf is special. Its ABI impact (combined with PyPI's lack of infrastructure & metadata for distributing non-python artefacts) makes many projects very defensive to changing anything about the protobuf version, and will pin hard to the latest one they've tested.

Despite protobuf's versioning shenanigans, their API is actually very stable and once one can handle the recompilation aspect for new protobuf minor versions (as we do), it's a non-issue to raise that bound.

from keras-tuner-feedstock.

The limit here is wrong. Or perhaps only "wrong", as in: not suitable for what we do in conda-forge (using shared libraries, of which there can only be one per environment, which therefore needs to match for all packages, which means those packages need to be migrated for each new version as we do).

We need to remove it (perhaps even patch the source), because it is unworkable with all of the above.

This is against a common rule ("reflect upstream dependency constraints accurately"), but protobuf is special. Its ABI impact (combined with PyPI's lack of infrastructure & metadata for distributing non-python artefacts) makes many projects very defensive to changing anything about the protobuf version, and will pin hard to the latest one they've tested.

Despite protobuf's versioning shenanigans, their API is actually very stable and once one can handle the recompilation aspect for new protobuf minor versions (as we do), it's a non-issue to raise that bound.

FYI, here is a pull request on upstream, which removes the limit on protobuf.
keras-team/keras-tuner#892

from keras-tuner-feedstock.

Upstream may not want to remove it (because they care about distribution on PyPI, where this can go wrong). Of course, upstream changes are better (no patches), but I wouldn't be surprised if they reject this, and then we should still change it, but only on the feedstock.

from keras-tuner-feedstock.

KerasTuner will not be compatible with TF 2.11 and below if you are using parallel tuning. But single-node usages are not affected. The incompatibility is because TF 2.12+ will install protobuf 4, and KerasTuner use protobuf 4 to regenerated its grpc files, which is not compatible with protobuf 3 which is used by TF 2.11 and below.

Are you using parallel tuning? @sh-shahrokhi

from keras-tuner-feedstock.

KerasTuner will not be compatible with TF 2.11 and below if you are using parallel tuning. But single-node usages are not affected. The incompatibility is because TF 2.12+ will install protobuf 4, and KerasTuner use protobuf 4 to regenerated its grpc files, which is not compatible with protobuf 3 which is used by TF 2.11 and below.

Are you using parallel tuning? @sh-shahrokhi

Thanks for clarification. I do not use parallel tuning.
I think it would better to remove the limit here on this feedstock. @h-vetinari

from keras-tuner-feedstock.

Thanks for the input @haifeng-jin!

Indeed it's possible that protobuf- or grpc-generated files are not compatible between versions. The important thing here is that we still keep the tensorflow builds separate per given version, and we could do the same for kerastuner (e.g. rebuild for protobuf 3.20 & 3.21), which would then enable the solver to choose one of our compatible tensorflow builds (whether that's 2.9, 2.10, 2.11 etc.)

from keras-tuner-feedstock.

We have generated grpc files using both protobuf v3 and v4, they will be dynamically loaded based on the situation. So it should be compatible for both TF 2.11 and TF 2.12 now.
keras-team/keras-tuner#898

from keras-tuner-feedstock.

Thanks @haifeng-jin, though this wouldn't have been necessary for conda-forge. In fact, I think it would be better if we do the grpc-generation ourselves, because we often migrate grpc ahead of other packages as well.

We can handle the fact that there would be different kerastuner builds, because we have infrastructure to correctly track dependence on non-python libraries

from keras-tuner-feedstock.

We have generated grpc files using both protobuf v3 and v4, they will be dynamically loaded based on the situation. So it should be compatible for both TF 2.11 and TF 2.12 now.
keras-team/keras-tuner#898

See also: https://github.com/keras-team/keras-tuner/releases/tag/v1.3.5

from keras-tuner-feedstock.

@sh-shahrokhi, based on #22 (comment) can you check it, please?

from keras-tuner-feedstock.

@sh-shahrokhi, based on #22 (comment) can you check it, please?

Hi, in the 1.3.5 recipe protobuf is limited to 3.20.3
Is this limit necessary?

from keras-tuner-feedstock.

Sorry, no can you make a new PR and remove the limit, please? And increase the Conda recipe version, too.

from keras-tuner-feedstock.

it works. thank you

from keras-tuner-feedstock.

it works. thank you

thank you for figuring It out 👍

from keras-tuner-feedstock.