How to determine if field was set (not oneof)? about python-betterproto HOT 11 CLOSED

Oh, I first thought this had to do with oneof fields not working well, hence the reference from commit, but I see you need something else.

from python-betterproto.

👉 https://stackoverflow.com/questions/42622015/how-to-define-an-optional-field-in-protobuf-3

Actually, turns out in Protobuf 3 all fields are optional, and primitive values such as integers, bools will always contain their default value if they were not explicitly set.

In other words, there is no way to check which field was set.

Please let us know if that answers your question~

from python-betterproto.

Sure, but would it be possible to track which fields are set? Maybe an optional feature of it significantly impacts performance. Or another option would be to set fields to None by default?

from python-betterproto.

@nyurik
Concerning having each instance track which fields have been set (unrelated to what is serialized in the end), this isn't something dataclasses are very well suited to, but it's possbile.

I think this would require modifying the Message base class to:

• include set of names of fields that have been set per instance (logically this just needs to be a bitvector with one bit per field, would have to see how efficiently this can be done in python with reasonable complexity)
• add a setattr method thus set adds the target field to the updated set whenever it is set.

There would certainly be some performance cost to this, the cost/benefit isn't obvious to be. Neither is how this could be made optional.

from python-betterproto.

Sure, but would it be possible to track which fields are set? Maybe an optional feature of it significantly impacts performance. Or another option would be to set fields to None by default?

Just to be clear, do you mean, whether they were set by the python client (user of this library)? or whether they were set in the message that came in over the wire

from python-betterproto.

@nyurik
Concerning having each instance track which fields have been set (unrelated to what is serialized in the end), this isn't something dataclasses are very well suited to, but it's possbile.

Technically, it would certainly be possible to keep track of set-actions. However, if it means we introduce a method like .is_set(), it will only work when clients have set it, and does not tell whether it was set on the wire.

I'm concerned that this will confuse consumers of the library, who will expect to be able to inspect set/unset fields on all messages.

Alternatively, having None values by default might solve that issue, which sounds pretty intuitive to me. Although this also ties in with the question, what do we do with incoming messages with oneof fields. Do we also ignore the default values for unused fields and set them all to None? Normally, within oneof, unused int32 fields will be 0, and not None.

Out of curiosity, why do you need to keep track of the set fields, if they are set in the client code?

from python-betterproto.

For future reference, the above request is not satisfied by the feature betterproto.serialized_on_wire, which only detects fields that are messages, whereas the example concerns messages with primitive fields.

https://github.com/danielgtaylor/python-betterproto#determining-if-a-message-was-sent

from python-betterproto.

Technically, this proto file is in proto 2 format. Proto 3 does not have optionals.

Superficially, that would mean we would not support this case, however, it is the case that clients/servers may send messages across the wire and omit some fields.

This could mean that:

• they didn't know about the field
• or that they wish to transmit the default value (which clients are required to return when a field is not present)
• or that it was a proto2 message that was optional and unset
• or that the server has updated their version of the message so that the field was wrapped with oneof
• or that only part of the message was sent. protobuf is designed so that messages can be split up and re-assembled.

We can probably keep track of the information you want, but we must be careful interpreting it.

from python-betterproto.

If anyone else needs a quick fix, you can simply change the defaults for primitive types to None this way:

def init_default_gen_patch(self):
    default_gen = {}

    for field in dataclasses.fields(self.cls):
        meta = FieldMetadata.get(field)
        if meta.proto_type == TYPE_MESSAGE:
            default_gen[field.name] = self.cls._get_field_default_gen(field, meta)
        else:
            default_gen[field.name] = lambda: None

    self.default_gen = default_gen

betterproto.ProtoClassMetadata.init_default_gen = init_default_gen_patch

from python-betterproto.

This is fixed in 2.0.0b5 with Message.is_set()

from python-betterproto.

If anyone else needs a quick fix, you can simply change the defaults for primitive types to None this way:

def init_default_gen_patch(self):
    default_gen = {}

    for field in dataclasses.fields(self.cls):
        meta = FieldMetadata.get(field)
        if meta.proto_type == TYPE_MESSAGE:
            default_gen[field.name] = self.cls._get_field_default_gen(field, meta)
        else:
            default_gen[field.name] = lambda: None

    self.default_gen = default_gen

betterproto.ProtoClassMetadata.init_default_gen = init_default_gen_patch

@vidklopcic Do you have a guide on where to place that snippet when using plain betterprot, e.g.
python -m grpc_tools.protoc -I . --python_betterproto_out=. echo.proto?

from python-betterproto.