Seeing

E           AssertionError: Pattern 'Unknown field garbage.' not found in 'Protocol message SimpleProto3 has no field garbage.'

When attempting to access the DESCRIPTOR attribute on a protobuf module, i get a Module has no attribute "DESCRIPTOR" error. From looking at the code, it seems we generate a DESCRIPTOR attributer on messages and enums, but not one for the module itself.

This should probably be generated, since it is valid syntax and is the correct way to access the FileDescriptor for the file that corresponds to the current pb2.py file.

Hi,

Thank you for this wonderful project. I'm integrating mypy with a codebase that uses protobufs and feel that this will be quite helpful in doing so!

I'm able to generate the *.pyi type stub files along with the other definitions by invoking the following command:

$ python -m grpc_tools.protoc -I=github.com/gogo/protobuf/gogoproto -I=. \
    --python_out=generated/python \
    --grpc_python_out=generated/python \
    --mypy_out=generated/python

The generated *.pyi files look correct, however when I try to run mypy over modules that import these definitions, I receive error messages like the following (names/line #s scrubbed):

generated/python/foo_pb2.pyi:123: error: Function "Foo" is not valid as a type
generated/python/foo_pb2.pyi:123: note: Perhaps you need "Callable[...]" or a callback protocol?

Inside of foo_pb2.py, Foo is defined like follows:

Foo = _reflection.GeneratedProtocolMessageType('Foo', (_message.Message,), {
  'DESCRIPTOR' : _FOO,
  '__module__' : 'path.to.foo'
  # @@protoc_insertion_point(class_scope:Foo)
  })
_sym_db.RegisterMessage(Foo)

Is there something obvious I'm doing wrong? I'm using the following versions for my dependencies:

• mypy 0.761
• mypy-protobuf 1.18
• Python 3.6.5
• python-protobuf 3.11.3

$ python -m grpc_tools.protoc --version
libprotoc 3.8.0

Thanks!

Currently, the repeated check for mapping support crashes if the message is an enum with a KeyError

Generate some example protos and verify the output of the protoc-gen-mypy script

• Verify output matches expected
• Verify output runs through mypy (syntactically correct)

What will it take to add support for nested imports? This seems to be supported in the Go version but not in Python.

Hence, adding a service currently will result in mypy errors.

I'm working on a workaround which sets these as Any in the meantime

This tool generates .pyi files with syntax error when parsing a proto-file with a field named from. E.g. foo.proto:

message FooRequest {
	int64 from = 5;
}

Running mypy on a project with this kind of autogenerated files results in syntax error:

$ mypy
package/schemas/foo_pb2.pyi:356: error: invalid syntax

This is because the foo_pb2.pyi file has this in it:

class FooRequest(google___protobuf___message___Message):
    from = ... # type: int

The from = ... is invalid syntax, and so we have a problem.

Just to say I've set up my mypy to ignore these files, mypy.ini:

[mypy]

[mypy-package.schemas.*]
ignore_errors = True

But this doesn't ignore syntax errors. As per python/mypy#6897 mypy can't ignore syntax errors. It would be favorable if this project did not generate syntax-invalid output.

I'm using versions:

$ python --version Python 3.6.7
mypy==0.701
mypy-protobuf==1.10

And here is a gist with the entire contents of the .proto file, the generated .py file (which does not have syntax errors and the problematic and unexcludable .pyi file:
https://gist.github.com/gaggle/8aef6102d314936c9eb33724194682b0

We started using mypy-protobuf for a bit larger codebase and hit two problems. For the first I have PR #106, the second is harder.

mypy-protobuf create this nice Literal union to validate the parameter to HasField when it's a literal. However, when it's dynamically generated, it fails the type check (e.g., foo.HasField(arg + "_timestamp")). I don't know - is it possible to still validate the literal strings while allowing for non-literal strings as well?

Eg

package test;

import test2";

message Inner {}
message Msg {
    Inner x = 1;
    test2.Inner y = 2;
}

Solution would be to use fully qualified paths

Thanks to @smessmer for the report
Currently, mypy-protobuf uses absolute imports. If the protobuf stubs aren't a top level package, they won't be found.

Fixing mypy-protobuf would involve either

• Using relative imports everywhere (including in/out of nested packages)
• Prepending the "python_out" path from the protoc invocation (this is not provided to us through CodeGeneratorRequest, so it may need to be an additional argument)

Workarounds in the meantime might include

• rejigger the protoc invocation to set python_out="." (top level)
• Adding the sub path of the stubs to PYTHONPATH

Writing mypy to oracle_pb2.pyi
Writing mypy to types_pb2.pyi
Traceback (most recent call last):
  File "/home/lpp/go/src/github.com/nmiculinic/x/.venv/bin/protoc-gen-mypy", line 313, in <module>
    main()
  File "/home/lpp/go/src/github.com/nmiculinic/x/.venv/bin/protoc-gen-mypy", line 310, in main
    sys.stdout.write(output)
TypeError: write() argument must be str, not bytes
--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.

python is mapped to: Python 3.6.4

Generates:

from google.protobuf.timestamp import (
    Timestamp,
)

and it should be:

from google.protobuf.timestamp_pb2 import (
    Timestamp,
)

similarly for other standard types:

from google.protobuf.empty_pb2 import (
    Empty,
)

The generated stubs in the Python version emit maps as RepeatedScalarFieldContainer types rather than as MutableMapping types as the Go version does. The latter approach (Go) seems to be correct and working.

The annotated __init__ method generated for a protobuf message accepts positional arguments, but the underlying implementation accepts only named arguments. The first parameter after self should be a nameless * parameter. This will tell mypy and other type checkers that positional arguments are not allowed when calling the constructor.

For example, a protobuf message with a string: name field will generate the following annotated stub:

class MyMessage(google___protobuf___message___Message):
    name = ... # type: typing___Text

    def __init__(self,
        name : typing___Optional[typing___Text] = None) -> None: ...

If callers attempt to construct this message by calling MyMessage('bob'), a runtime error occurs indicating that positional arguments are not allowed for this call.

To match the underlying implementation, it should be:

    def __init__(self, *,
        name : typing___Optional[typing___Text] = None) -> None: ...

Hey guys, great work! This is already really helpful, thanks :).

I am having an issue with the generated code for repeated fields, I see you are using RepeatedScalarFieldContainer from mypy, but with version 0.560, I can't find it. Which version of mypy were you using for this to work?

For the record, in myproto_pb2.pyi, I am talking about the import

from mypy import (
    RepeatedScalarFieldContainer,
)

Thank you

Starting from some version of the protoc compiler, gRPC service stubs are no longer generated into *_pb2.py files. Now they are generated into separate *_pb2_grpc.py files. So to properly support grpcio library *_pb2_grpc.pyi files should be generated for service stubs.

And it would be great to generate gRPC stubs optionally, this would be possible if two separate CLI options (plugins) for protoc were created:

• --mypy_out for messages
• --mypy_gprc_out for gRPC stubs

I'm asking for this feature because I'm the author of the grpclib library, which is an alternative gRPC implementation for asyncio, and users of this library don't need gRPC stubs for grpcio.

From #18 (thanks @jcppkkk)

This PR cause mypy-protobuf generate error pyi files that fail mypy check.

For example, in https://developers.google.com/protocol-buffers/docs/reference/python-generated there is a example command:

protoc --proto_path=src --python_out=build/gen src/foo.proto src/bar/baz.proto
What if baz.proto imports foo.proto?

import "foo.proto";
It generate

from .foo_pb2 import (
and fails mypy check: cannot find module named foo_pb2; The same error happens to our project after update to 1.4.

This change need to calculate relative filepath between current file to target file.

As I understand, the typeshed definition is used for the definition of the base Message class and a commit in October ([1]) changed the argument of FromString from "Any" to "bytes" in Python3 and a Union of "bytes", "buffer" and "unicode"(strange) in Python 2. mypy-protobuf emits bytes as the type of the argument of FromString, thus in Python2 we get lots of errors about the argument type of the subclass being more narrow that the superclass.

The code in the attachment seems to make it work. Will you want to include such a fix?

mypy-diff.txt

[1] python/typeshed@036abc7#diff-2492d636b6705e307c9cb32d169f3614

See Discussion in #56

I actually think we're able to get best of both worlds here by using the mypy Literal type. This way, the typing will be indeed an int, but the int is constrained to only the legitimate values.

https://mypy.readthedocs.io/en/latest/literal_types.html

First attempt at a diff was #79
Discussion after landing:

@Kami

First thanks for adding enum literal support.

This indeed seems to fix direct enum literal notation, but it breaks the Value() method scenario which takes a string (enum name) and returns an enum value.

For example, with the following definition:

enum ExecutionStatus {
    REQUESTED = 0;
    SCHEDULED = 1;
    RUNNING = 2;
}

This used to work:

from messages_pb2 import ExecutionStatus

status_str = 'requested'
obj.status = ExecutionStatus.Value(status_str.upper())  # this should return enum value (aka Enum type / int)

Now it returns the following error:

error: Incompatible return value type (got "Union[Literal[0], Literal[1], Literal[2]]", expected "ExecutionStatus")

It works fine with just the change from #81, but not with this change.

It may also brake Name() and other enum class methods, but I didn't have time to dig in yet.

@nipunn1313

I poked around a bit and I believe that the issue here is that obj.status is typed to ExecutionStatus, but it would now need to be typed to ExecutionStatus.ClosedValueType

since ExecutionStatus is the enum itself and ExecutionStatus.ClosedValueType is the type of the Values in the enum (and ExecutionStatus.ClosedKeyType is the type of the Keys in the enum)

The merged PR was non-backward-compatible in that it changed the return value of Typing to literals, and move the int-y-ness of the enum to the ClosedValueType member.

This is more accurate based on this example I found

enum UserFSWAction {
   UNKNOWN = 0;
   ALLOW = 1;
}

>>> type(ALLOW)
<class 'int'>
>>> ALLOW
1
>>> UserFSWAction.Name(1)
'ALLOW'
>>> UserFSWAction.Value('ALLOW')
1
>>> type(UserFSWAction.Value('ALLOW'))
<class 'int'>

@Kami - does that make sense directionally?

I'll hold off on publishing to pypi until we have some alignment here. I'm ok with making a non-backward compatible change, but I'll want to have some clear changelist and bump the big version number.

Couple more considerations for proposal

change typing of the constants to be the ClosedValueType for better error messages

- ALLOW: typing_extensions___Literal[1]
+ ALLOW: UserFSWAction.ClosedValueType

change ClosedValueType to just ValueType to simplify notation required by callers.

eg

import dropbox.proto.nucleus.types_pb2
from dropbox.proto.nucleus.types_pb2 import FileId
reveal_type(FileId)
reveal_type(dropbox.proto.nucleus.types_pb2.FileId)

results in

dropbox/proto/nucleus/app_service_pb2.pyi:17: error: Revealed type is 'def (id: Union[builtins.str, None] =) -> dropbox.proto.nucleus.types_pb2.FileId'
dropbox/proto/nucleus/app_service_pb2.pyi:18: error: Revealed type is 'Any'

Proposed workaround is:

from dropbox.proto.nucleus.types_pb2 import FileId as dropbox___proto___nucleus___types_pb2___FileId

Until mypy gains support for module imports like this (see python/mypy#5583).

Hi!

We currently use mypy-protobuf in a typechecking CI step for buildgrid. With the release of mypy 0.780, we began to see the following error in the CI.

buildgrid/_protos/build/bazel/semver/semver_pb2.pyi: error: Source file found twice under different module names: 'buildgrid._protos.build.bazel.semver.semver_pb2' and 'build.bazel.semver.semver_pb2'
Found 1 error in 1 file (checked 144 source files)

We tried regenerating the .pyi files with the newer version of mypy, but the issue persisted, so we've had to roll back our mypy version.

Some additional information:

• We noticed that the new version of mypy shipped with a change in its import following logic. The blogpost says this was for the mypy daemon, but perhaps it's relevant for the CLI tool as well?
• Our mypy.ini. Notably, buildgrid/_protos is in the import path.
• Our type-checking CI step is pretty simple; it just installs some dependencies and runs mypy on the main source folder: https://gitlab.com/BuildGrid/buildgrid/-/blob/master/.gitlab-ci.yml#L62-63

I'm not sure if this is an issue with mypy, mypy-protobuf, or our configuration, so please let us know if we should post this elsewhere!

Hey team🙌
I know dropbox is using Bazel and wanted to know if you can publish a BUILD file with mypy-protobuf as a dependency (go impl in better)

Thanks and really helpfull project!

Running mypy-protobuf on the following proto definition

syntax = "proto3";

enum Enum {
  OK = 0;
  NOT_OK = 1;
}

message Message {
  Enum enum = 1;
}

Produces a .pyi file with

class Enum(builtin___int):
    DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ...
    @classmethod
    def Name(cls, number: builtin___int) -> builtin___str: ...
    @classmethod
    def Value(cls, name: builtin___str) -> 'Enum': ...
    @classmethod
    def keys(cls) -> typing___List[builtin___str]: ...
    @classmethod
    def values(cls) -> typing___List['Enum']: ...
    @classmethod
    def items(cls) -> typing___List[typing___Tuple[builtin___str, 'Enum']]: ...
    OK = typing___cast('Enum', 0)
    NOT_OK = typing___cast('Enum', 1)
OK = typing___cast('Enum', 0)
NOT_OK = typing___cast('Enum', 1)
global___Enum = Enum

class Message(google___protobuf___message___Message):
    DESCRIPTOR: google___protobuf___descriptor___Descriptor = ...
    enum = ... # type: global___Enum

which suggests to mypy that the enum attributes exist on the enum object e.g. that Message().enum.OK is valid. In reality the generated code (using grpc_tools) has type(Message().enum) == int, which obviously doesn't have the OK attributes (or keys, values etc) and Message().enum.OK fails at runtime

Hi,
It would be great if json_format definitions are also created while compiling the proto file.

Thanks,
Dilip Vamsi.

Currently it appears that this plugin doesn't support all type of notations for accessing Enum values.

Example protobuf definition (execution.proto):

enum ExecutionStatus {
    REQUESTED = 0;
    SCHEDULED = 1;
    RUNNING = 2;
    SUCCEEDED = 3;
    FAILED = 4;
    TIMED_OUT = 5;
    CANCELED = 6;
}

Right now, the following ways to refer to the enum value work and don't result in a mypy error:

import execution_pb2

execution_pb2.REQUESTED
execution_pb2.Value('REQUESTED')
...

The problem is that the following more verbose approach (where you access the value via enum type class) returns an error:

execution_pb2.ExecutionStatus.REQUESTED

foo.py: error: "Type[ExecutionStatus]" has no attribute "SUCCEEDED"

This approach is supported by the protobuf generated Python code. I also personally prefer it since it's more explicit and obvious when there are multiple top-level enum types defined in a single proto file.

Using Value('') method is also not desired since it doesn't provide any static typing guarantees - if you pass in an invalid / unsupported value, this will only be detected during run-time.

I checked the generated annotations file and it looks like it only writes type hints for accessing enum values via global module level variables.

From the generated type hints code:

class ExecutionStatus(int):
    DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ...
    @classmethod
    def Name(cls, number: int) -> str: ...
    @classmethod
    def Value(cls, name: str) -> ExecutionStatus: ...
    @classmethod
    def keys(cls) -> typing___List[str]: ...
    @classmethod
    def values(cls) -> typing___List[ExecutionStatus]: ...
    @classmethod
    def items(cls) -> typing___List[typing___Tuple[str, ExecutionStatus]]: ...
REQUESTED = typing___cast(ExecutionStatus, 0)
SCHEDULED = typing___cast(ExecutionStatus, 1)
RUNNING = typing___cast(ExecutionStatus, 2)
SUCCEEDED = typing___cast(ExecutionStatus, 3)
FAILED = typing___cast(ExecutionStatus, 4)
TIMED_OUT = typing___cast(ExecutionStatus, 5)
CANCELED = typing___cast(ExecutionStatus, 6)

I propose to update it so it looks like this:

class ExecutionStatus(int):
    DESCRIPTOR: google___protobuf___descriptor___EnumDescriptor = ...
    @classmethod
    def Name(cls, number: int) -> str: ...
    @classmethod
    def Value(cls, name: str) -> ExecutionStatus: ...
    @classmethod
    def keys(cls) -> typing___List[str]: ...
    @classmethod
    def values(cls) -> typing___List[ExecutionStatus]: ...
    @classmethod
    def items(cls) -> typing___List[typing___Tuple[str, ExecutionStatus]]: ...
    REQUESTED = typing___cast(ExecutionStatus, 0)
    SCHEDULED = typing___cast(ExecutionStatus, 1)
    RUNNING = typing___cast(ExecutionStatus, 2)
    SUCCEEDED = typing___cast(ExecutionStatus, 3)
    FAILED = typing___cast(ExecutionStatus, 4)
    TIMED_OUT = typing___cast(ExecutionStatus, 5)
    CANCELED = typing___cast(ExecutionStatus, 6)

REQUESTED = typing___cast(ExecutionStatus, 0)
SCHEDULED = typing___cast(ExecutionStatus, 1)
RUNNING = typing___cast(ExecutionStatus, 2)
SUCCEEDED = typing___cast(ExecutionStatus, 3)
FAILED = typing___cast(ExecutionStatus, 4)
TIMED_OUT = typing___cast(ExecutionStatus, 5)
CANCELED = typing___cast(ExecutionStatus, 6)

This way those values can be access using both supported way - directly via top level variable or via enum class level variable.

Somewhat related issue #56.

When using mypy-protobuf, the plugin expects pypi/protobuf to be installed.
If it's not, the build fails:

Traceback (most recent call last):
  File "/Users/gtalarico/dev/repos/gtalarico/schema/.venv/bin/protoc-gen-mypy", line 13, in <module>
    import google.protobuf.descriptor_pb2 as d
ModuleNotFoundError: No module named 'google'
--mypy_out: protoc-gen-mypy: Plugin failed with status code 1.

If protobuf pkg is required, seems to me we could just add it to setup.py so the dependency is automatically when mypy-protobuf is installed

Thoughts?

# setup.py
setup(
    install_requires=["protobuf"],
    # ...
)

We should gate the generation of generic service stubs in *_pb2.py to only appear when option py_generic_services = true; is set

See
https://developers.google.com/protocol-buffers/docs/reference/python-generated#service

Note that this is different than gRPC stubs which is covered in #46

Is there a possibility of upstreaming this and including it in protoc's Python plugin? I don't know how difficult it would be to integrate, but the protobuf team is at least accepting contributions to generate type hints: protocolbuffers/protobuf#2638 (comment)

Hello, sorry for offtopic, but as PyCharm supports pyi files internally, are there known way to configure PyCharm to understand mypy-protobuf generated files? I tried to configure it, but failed, no completion achieved.

Thanks.

Just curious if this repository is going to be actively maintained going forward, since it's gone somewhat stagnant. We definitely appreciate having this utility around (big quality of life improvement when using Python and gRPC/protobuf).

Wondering whether we should either (a.) invite in a few more active maintainers for this repo (i.e., to help review/merge PR's, etc.) or (b.) fork it and move on.

Proto

# rules.proto
message Result {
  string name = 1;
}

message ResultList {
  repeated Result results = 2;
}

Code

import rules_pb2
r = Results(name="x")
result_list = ResultList()
result_list.results.append(r)
# Append works as expected, mypy raises: 
# RepeatedCompositeFieldContainer[Results]" has no attribute "append" mypy(error)

Mypy stub

class Results(google___protobuf___message___Message):

    @property
    def results(self) -> google___protobuf___internal___containers___RepeatedCompositeFieldContainer[ResultList]: ...

For example, let's assume a Money message (containing amount and currency) and a Money.Currency enum.

This should be valid:

some_money = Money()
some_money.amount = 200
some_money.currency = Money.USD

and

some_money.currency = Money.Currency.Value('CAD')

However, this should be invalid (but is currently valid):

cad = some_money.currency.Value('CAD')

Similarly, this is a false positive:

Money.Currency('CAD')

since it finds int.__init__ although in runtime it would result in:

TypeError: 'EnumTypeWrapper' object is not callable

One way to solve it would be introduce a CurrencyValue:

class Money(google___protobuf___message___Message):
  CurrencyValue = NewType('int')
  class Currency:
    DESCRIPTOR: ...
    ...
  USD = typing___cast(CurrencyValue, 1)
  ...

This would be a breaking change since it'll require changing any signature accepting Money.Currency. Also, since it exists only in the type stubs, you'd have to quote it, i.e.

def do_something(currency: 'Money.CurrencyValue'):

Alternatively, there might be some (new?) mypy construct that allows to designate fields and functions as class-access only -- basically the same as its built-in (hardcoded, from what I could tell) support for enum.

Before importing a type, the current code checks whether it's in a package and then do not import it if it's not. I'll send a fix shortly.

On trying to use this with proto 2.5.0 and python 2.7 I am getting the below error

  [exec] Traceback (most recent call last):
     [exec]   File "/Users/dmanna/Documents/fresh_code/github/vnera/main/main/venv/bin/protoc-gen-mypy", line 364, in <module>
     [exec]     main()
     [exec]   File "/Users/dmanna/Documents/fresh_code/github/vnera/main/main/venv/bin/protoc-gen-mypy", line 351, in main
     [exec]     generate_mypy_stubs(Descriptors(request), response, "quiet" in request.parameter)
     [exec]   File "/Users/dmanna/Documents/fresh_code/github/vnera/main/main/venv/bin/protoc-gen-mypy", line 293, in generate_mypy_stubs
     [exec]     pkg_writer.write_messages(fd.message_type, "")
     [exec]   File "/Users/dmanna/Documents/fresh_code/github/vnera/main/main/venv/bin/protoc-gen-mypy", line 169, in write_messages
     [exec]     if msg.options.map_entry:
     [exec] AttributeError: 'MessageOptions' object has no attribute 'map_entry'
     [exec] --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
     [exec] Result: 1

• Proto Version - 2.5.0
• Python Version - 2.7.6

With mypy-protobuf 1.19, mypy passed my code. Upgrading to mypy-protobuf 1.20 changes the output to add some global___ prefixes to types, breaking the type check. If I pin my dependency to 1.19, or just strip out these prefixes, mypy passes my code again. Is there another alternative? Perhaps an option to disable the name mangling?

Changes made in #120 are breaking typechecks when compiled types are used with dataclasses, or in general when enums are used as parameters or return values.

Quick example:
proto:

syntax = "proto3";

package bikes;
    
enum EngineType {
    TWO_STROKE = 0;
    FOUR_STROKE = 1;
}

message Motorcycle {
    string name = 1;
    EngineType engine_type = 2;
}

python:

from dataclasses import dataclass
from motorcycles_pb2 import Engine, EngineType, Motorcycle

@dataclass
class DemoEngine:
    engine_type: EngineType
    name: str

    def __init__(self, engine_type: EngineType, name: str):
        self.engine_type = engine_type
        self.name = name

d = DemoEngine(EngineType.TWO_STROKE, "demo")

type check failure:

motorcycles/__main__.py:17: error: Argument 1 to "DemoEngine" has incompatible type "EngineTypeValue"; expected "EngineType"

Protos compiled with protoc 3.12.2 using python -m grpc_tools.protoc

Hey! I've started using your plugin, and it's working to an extent that the type hints are picked up by my IDE, which is helpful.

When trying to run mypy on the command line I get this error however:

com/fish/time/time_range_pb2.pyi:45: error: invalid syntax

The relevant lines:

 42 class TimeRange(google___protobuf___message___Message):
 43 
 44     @property
 45     def from(self) -> google___protobuf___timestamp_pb2___Timestamp: ...
 46 

I'm using this command line to check the types:

mypy --ignore-missing-imports --python-version 3.7 --namespace-packages --check-untyped-defs -p fish

where the com and fish directories reside inside the same directory, and fish is importing protobufs from com.

Before adding the --namespace-packages mypy did finish, but it didn't seem like it took the types of the protobufs into account.

The protobufs are generated by

python3 -m grpc_tools.protoc -I ./proto --python_out=. --mypy_out=. --grpc_python_out=. ./proto/com/fish/*/*.proto

mypy version 0.701, Python 3.7.3

Any ideas?

I have the following code:

from proto.example_pb2 import SomeMessage
from google.protobuf.message import Message


def request(...) -> Message:
	...

result: SomeMessage = request(
    ...
)

Request will return a protobuf message which can vary depending on the input parameters. This is giving the following mypy error:

error: Incompatible types in assignment (expression has type "Message", variable has type "SomeMessage")

Now, I understand why this is happening: Message is not assignable to SomeMessage. It would be valid the other way around (SomeMessage -> Message) because SomeMessage is a subclass of Message.

The reasoning behind this code is that while request can return any message, when I use it I already know which specific message it will return, and I want to be able to type it like this. I just don't know how to go about it.

The only solution I thought about is to use @overload with some Literal types to set the return of every message type, as it really only depends on the value a string parameter, but it seems a bit overkill. I have some other situations in my code where this happens (even without protobuf) so what I'm really asking is for help on what pattern I should be using in these kind of situations, because I'm all out of ideas.

Can anyone help me figure this out?

Thanks :)

Make it easier to ensure contributions pass tests / mypy / formatting!

The plugin runs fine for pure protobuf files, but it seems to trip up when using gRPC:

(venv) python [master ?] -> protoc -I../protos --python_out=. --grpc_python_out=. --mypy_out=. ../protos/pingpong.proto
protoc-gen-grpc_python: program not found or is not executable
--grpc_python_out: protoc-gen-grpc_python: Plugin failed with status code 1.

Hi,

I was wondering when a new version would be released on PyPI. We're very much looking forward to some of the changes introduced since last release, but have had some trouble specifying a git SHA in our requirements.txt to point at a non-PyPI verison. Thanks so much!

For existing places in which they are not equivalent, discuss/determine which behavior makes sense.

Protoc replaces dashes with underscore when creating python modules (i.e. my-file.proto will generate my_file_pb2.py). However, mypy-protobuf still creates my-file_pb2.pyi. It should probably also generate my_file_pb2.pyi.

Generating this file with protoc --python_out=. --mypy_out=. tetration_network_policy.proto on python 3.7. Python 3.6 tested with same results.

The resulting file is full of Unresolved reference 'TetrationNetworkPolicyProto' errors and auto-completions does not work. You can check the screenshot here:

When I simply delete all the TetrationNetworkPolicyProto. occurrences in the file, tada! All errors are gone and completions throughout the project work great. I'm not versed enough with the mechanics of python typing yet so I've had no insights for where the problem could be coming from.

Thanks Kami for the idea
ba6b43e#commitcomment-35412280

I noticed mypy check fails on enum and nested types usage. I tweaked the test_generated_mypy.py to minimally reproduce the issue.

https://github.com/dropbox/mypy-protobuf/blob/master/test/test_generated_mypy.py#L52

Passing an enum instead:

s = Simple1(a_enum=1)

The example runs fine though, but mypy complains of incompatible type. Please see error below:

mypy --py2 test/test_generated_mypy.py

test/test_generated_mypy.py:70: error: Argument "a_enum" to "Simple1" has incompatible type "int"; expected "Optional[OuterEnum]"

Please see the type hint here:
https://github.com/dropbox/mypy-protobuf/blob/master/test/proto/test_pb2.pyi.expected#L126

Should the type hint say int (or str?) instead of the enum type (OuterEnum here)? Or am I missing something here?

def ClearField(self, field_name: Text) -> None: ...
def WhichOneof(self, oneof_group) -> Optional[Text]: ...

We could type these more strongly in mypy-protobuf using Literal, which would only allow valid field names.

Eg.

Union[Literal["field1"], Literal["field2"]]

Currently, if we try this in mypy-protobuf, we get errors of the format

dropbox/proto/nucleus/types_pb2.pyi:40: error: Argument 1 of "HasField" incompatible with supertype "Message"

however, I confirmed if we either

1. remove HasField from the supertype
2. mark the supertype as Any rather than Text

then mypy-protobuf can provide better typing on the specific instances of Message

I'll propose (1) to typeshed under the argument that you can't usefully call HasField on a Message without some additional knowledge of the real type.

This proposal would prevent a duck-typey use case

Message1 {
   a,
   b,
}
Message2 {
   b,
   c,
}

if m.HasField("b"):
    quack(b)

But if a caller is trying to ducktype, they could work around this by requiring m to be typed with a more specific protocol type.

For most cases, forcing HasField to be a literal will be better.

I'm curious about (a) what the decisionmaking process / motivation was behind the creation of each implementation (including motivation for language selection), and (b) how you see the overall tech investment/support for each of these playing out over time. Will you be making an effort to keep one implementation functionally and qualitatively equivalent to the other?

If it's possible, could you also please characterize how each of these implementations is used inside dropbox?

@nipunn1313 In your PR you're only using comments for types. As such all you use are strings (they are never considered as actual Python object). What I use is full typing annotations but using strings:

e: 'test_pb2.OuterEnumValue' = test_pb2.BAR

Previously I was able to use without the quotes but not anymore

e: test_pb2.OuterEnum = test_pb2.BAR

Originally posted by @pcorpet in #126 (comment)