bufbuild / protovalidate-python Goto Github PK
View Code? Open in Web Editor NEWProtocol Buffer Validation for Python.
Home Page: https://pypi.org/project/protovalidate
License: Apache License 2.0
Protocol Buffer Validation for Python.
Home Page: https://pypi.org/project/protovalidate
License: Apache License 2.0
The following import line is included in the generated python code:
from buf.validate import validate_pb2 as buf_dot_validate_dot_validate__pb2
However, this code leads to runtime errors since there is no module named buf
ModuleNotFoundError: No module named 'buf'
I have the python package protovalidate
installed as dependency in my project.
.proto
file that utilises protovalidate
syntax = "proto3";
package examplev1;
import "buf/validate/validate.proto";
service ExampleService {
....
}
message ExampleRequest {
string device_id = 1 [(buf.validate.field).string.pattern = "^([0-9A-Fa-f]{2}[:]){5}([0-9A-Fa-f]{2})$"];
}
buf generate
The buf.gen.yaml
file I'm using
version: v1
managed:
enabled: true
go_package_prefix:
default: example/proto/gen/go
except:
- buf.build/bufbuild/protovalidate
plugins:
- plugin: buf.build/protocolbuffers/go
out: proto/gen/go
opt: paths=source_relative
- plugin: buf.build/bufbuild/connect-go
out: proto/gen/go
opt: paths=source_relative
- plugin: buf.build/grpc/go
out: proto/gen/go
opt: paths=source_relative
- plugin: buf.build/grpc/python
out: proto/gen/python
- plugin: buf.build/protocolbuffers/python
out: proto/gen/python
opt: pyi_out
the buf.yaml
I'm using
version: v1
name: buf.build/mystuff/core
deps:
- buf.build/bufbuild/protovalidate
breaking:
use:
- FILE
lint:
ignore:
- validate
use:
- DEFAULT
- COMMENTS
- UNARY_RPC
- PACKAGE_NO_IMPORT_CYCLE
I've made sure that I've ran buf mod update
and that my buf.lock
contains protovalidate
as a dep.
buf generate command I'm executing: buf generate --template buf.gen.yaml
*_pb2.py
and *_pb2.pyi
filesfrom buf.validate import validate_pb2 as buf_dot_validate_dot_validate__pb2
There is no module named buf
hence a runtime error is encountered
ModuleNotFoundError: No module named 'buf'
I expected the validate imports in the generated python code to not lead to a runtime error
The generated python code attempts to utilises a module that does not exists/is not present in the project dependencies (despite installed the protovalidate
python package)
N/A
After pip installing protovalidate==0.3.1
and trying to import it I get the error ModuleNotFoundError: No module named 'buf'
. This was using Python 3.11
import protovalidate
Be able to import with out error. The reference to buf
is internal to protovalidate
but it doesn't seem to be in the package
Import errors out with ModuleNotFoundError: No module named 'buf'
>>> import protovalidate
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/venv/lib/python3.11/site-packages/protovalidate/__init__.py", line 15, in <module>
from protovalidate import validator
File "/venv/lib/python3.11/site-packages/protovalidate/validator.py", line 19, in <module>
from buf.validate import expression_pb2 # type: ignore
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ModuleNotFoundError: No module named 'buf'
Per the readme: I've done:
pip install protovalidate
And created the example .proto (and named it transaction.proto)
Then try protoc --python_out=. transaction.proto
Unsurprisingly:
buf/validate/validate.proto: File not found.
transaction.proto:6:1: Import "buf/validate/validate.proto" was not found or had errors.
Seems like the .proto it's looking for is in the main repo, but I've been stuck to trying to get the import to work.
Does this library support proto-plus defined messages?
https://proto-plus-python.readthedocs.io/en/latest/reference/message.html
If not, please do so. Proto-plus classes have a .pb() method that returns the descriptor.
python3.10 -m venv venv
source ./venv/bin/activate
python --version
# Python 3.10.13
pip install protovalidate --no-cache
# Collecting protovalidate
# Using cached protovalidate-0.3.1-py3-none-any.whl.metadata (19 kB)
# Collecting cel-python (from protovalidate)
# Using cached cel_python-0.1.5-py3-none-any.whl (87 kB)
# Collecting protobuf (from protovalidate)
# Using cached protobuf-4.25.1-cp37-abi3-macosx_10_9_universal2.whl.metadata (541 bytes)
# Collecting jmespath>=0.10.0 (from cel-python->protovalidate)
# Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
# Collecting lark-parser>=0.10.1 (from cel-python->protovalidate)
# Using cached lark_parser-0.12.0-py2.py3-none-any.whl (103 kB)
# Collecting python-dateutil>=2.8.1 (from cel-python->protovalidate)
# Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
# Collecting pyyaml>=5.4.1 (from cel-python->protovalidate)
# Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (2.1 kB)
# Collecting requests>=2.25.1 (from cel-python->protovalidate)
# Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
# Collecting urllib3>=1.26.4 (from cel-python->protovalidate)
# Using cached urllib3-2.1.0-py3-none-any.whl.metadata (6.4 kB)
# Collecting babel>=2.9.0 (from cel-python->protovalidate)
# Using cached Babel-2.14.0-py3-none-any.whl.metadata (1.6 kB)
# Collecting six>=1.5 (from python-dateutil>=2.8.1->cel-python->protovalidate)
# Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
# Collecting charset-normalizer<4,>=2 (from requests>=2.25.1->cel-python->protovalidate)
# Using cached charset_normalizer-3.3.2-cp310-cp310-macosx_11_0_arm64.whl.metadata (33 kB)
# Collecting idna<4,>=2.5 (from requests>=2.25.1->cel-python->protovalidate)
# Using cached idna-3.6-py3-none-any.whl.metadata (9.9 kB)
# Collecting certifi>=2017.4.17 (from requests>=2.25.1->cel-python->protovalidate)
# Using cached certifi-2023.11.17-py3-none-any.whl.metadata (2.2 kB)
# Using cached protovalidate-0.3.1-py3-none-any.whl (22 kB)
# Using cached protobuf-4.25.1-cp37-abi3-macosx_10_9_universal2.whl (394 kB)
# Using cached Babel-2.14.0-py3-none-any.whl (11.0 MB)
# Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl (169 kB)
# Using cached requests-2.31.0-py3-none-any.whl (62 kB)
# Using cached urllib3-2.1.0-py3-none-any.whl (104 kB)
# Using cached certifi-2023.11.17-py3-none-any.whl (162 kB)
# Using cached charset_normalizer-3.3.2-cp310-cp310-macosx_11_0_arm64.whl (120 kB)
# Using cached idna-3.6-py3-none-any.whl (61 kB)
# Installing collected packages: lark-parser, urllib3, six, pyyaml, protobuf, jmespath, idna, charset-normalizer, certifi, babel, requests, python-dateutil, cel-python, protovalidate
# Successfully installed babel-2.14.0 cel-python-0.1.5 certifi-2023.11.17 charset-normalizer-3.3.2 idna-3.6 jmespath-1.0.1 lark-parser-0.12.0 protobuf-4.25.1 protovalidate-0.3.1 python-dateutil-2.8.2 pyyaml-6.0.1 requests-2.31.0 six-1.16.0 urllib3-2.1.0
#
# [notice] A new release of pip is available: 23.3.1 -> 23.3.2
# [notice] To update, run: pip install --upgrade pip
python
# Python 3.10.13 (main, Aug 24 2023, 12:59:26) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
# Type "help", "copyright", "credits" or "license" for more information.
# >>> import protovalidate
# Traceback (most recent call last):
# File "<stdin>", line 1, in <module>
# File "/Users/r/tmp/2/venv/lib/python3.10/site-packages/protovalidate/__init__.py", line 15, in <module>
# from protovalidate import validator
# File "/Users/r/tmp/2/venv/lib/python3.10/site-packages/protovalidate/validator.py", line 19, in <module>
# from buf.validate import expression_pb2 # type: ignore
# ModuleNotFoundError: No module named 'buf'
# >>>
here is an example in google colab (https://colab.research.google.com/drive/1QuCI88VCUmkIT7-AfrqxPU50OuoePRXO?usp=sharing)
import is successful
ModuleNotFoundError: No module named 'buf'
It seems to have started recently (past couple of days, way working fine before). Not sure what changed
(moved issue to bufbuild/protovalidate repository)
I would like to figure out why version 3.0.0 has dropped support for python <3.11 (#89) ?
For us updating python version was not a huge pain, but it seems like this restriction might hurt adoption a lot. Python 3.11 is only one year old, i don't think it has reached the critical mass. Even 3.8 is not yet end-of-life
What am i missing?
Feature description:
celpy which protovalidate uses does not support RE2-style regular expressions, which is required according to the CEL specification. Consider using a CEL evaluator that does support RE2-style regular expressions.
Problem it solves or use case:
protovalidate implementations should behave similarly across languages but for rules relying on pattern matching, protovalidate-python will consider common patterns like ^foo$
to match a string like foo\n
, while other protovalidate implementations would correctly reject it.
The re2 specification indicates that $
should only match end of text unless multiline mode is enabled, and multiline mode is not the default:
$ at end of text (like \z not \Z) or line (m=true)
Additionally, some protovalidate rules are implemented in terms of patterns, e.g., (buf.validate.field).string.uuid
which is implemented as the CEL "this == \'\' || this.matches(\'^[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}$\')"
which means it can match a string of 37 characters (the normal 36 plus newline).
Examples or references:
echo '"\n"' | python -m celpy 'string(this).matches("^$")'
# Outputs true
which makes sense because all celpy does is call re.search
and
import re
assert re.search("^$", "\n") is not None
The cel-go REPL (correctly) does not match this pattern:
$ git clone https://github.com/google/cel-go
$ cd cel-go/repl/main
$ go run .
cel-repl> %let this = '\n'
cel-repl> this.matches('^$')
false : bool
one proto file gen go and python code. go run success, but python run get
ModuleNotFoundError: No module named 'buf'
syntax = "proto3";
package my.package;
import "google/protobuf/timestamp.proto";
import "buf/validate/validate.proto";
message Transaction {
uint64 id = 1 [(buf.validate.field).uint64.gt = 999];
google.protobuf.Timestamp purchase_date = 2;
google.protobuf.Timestamp delivery_date = 3;
string price = 4 [(buf.validate.field).cel = {
id: "transaction.price",
message: "price must be positive and include a valid currency symbol ($ or £)",
expression: "(this.startswith('$') or this.startswith('£')) and float(this[1:]) > 0"
}];
option (buf.validate.message).cel = {
id: "transaction.delivery_date",
message: "delivery date must be after purchase date",
expression: "this.delivery_date > this.purchase_date"
};
}
go use protoc to gen code and run sucess.
python use
python3 -m grpc_tools.protoc
to gen code and run falied
python run success.
ModuleNotFoundError: No module named 'buf'
if we generate the python sources with buf and --include-imports flag, the protovalidate tries to import buf locally, but if we want to make generated sources a part of other module, the protovalidate raises ImportError on buf/validate import, so we need modify the protovalidate module sources to set where we want to import buf/validate
After pip installing protovalidate I get a ModuleNotFoundError: No module named 'buf'
from the import line in validatory.py from buf.validate import expression_pb2 # type: ignore
Unit test executes and raises ValidationError exception
Import error of protovalidate (by way of buf import)
============================= test session starts ==============================
collecting ...
test_sdk.py:None (test_sdk.py)
ImportError while importing test module '/Users/stewartb/PycharmProjects/pythonProject3/tests/test_sdk.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../Library/Caches/pypoetry/virtualenvs/pythonproject3-RFspHNkf-py3.11/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
../../../Library/Caches/pypoetry/virtualenvs/pythonproject3-RFspHNkf-py3.11/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
importlib.import_module(module_name)
/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py:126: in import_module
return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
???
<frozen importlib._bootstrap>:1176: in _find_and_load
???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
???
<frozen importlib._bootstrap>:690: in _load_unlocked
???
../../../Library/Caches/pypoetry/virtualenvs/pythonproject3-RFspHNkf-py3.11/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:186: in exec_module
exec(co, module.__dict__)
test_sdk.py:2: in <module>
import protovalidate
../../../Library/Caches/pypoetry/virtualenvs/pythonproject3-RFspHNkf-py3.11/lib/python3.11/site-packages/protovalidate/__init__.py:15: in <module>
from protovalidate import validator
../../../Library/Caches/pypoetry/virtualenvs/pythonproject3-RFspHNkf-py3.11/lib/python3.11/site-packages/protovalidate/validator.py:17: in <module>
from buf.validate import expression_pb2 # type: ignore
E ModuleNotFoundError: No module named 'buf'
When looking through the dependency, it seemed that the import path should have been from gen.buf.validate import expression_pb2
instead of buf.validate import experssion_pb2
I'm sure this is likely user error, on my part, but hopefully the authors can help me out.
protovalidate-python/pyproject.toml
Line 11 in bc0279d
In pyproject.toml, required python version is ">=3.7". However, Python 3.7 does not support PEP 604 and causes an error when using protovalidate.
Please clarify the required Python version for protovalidate-python, maybe 3.11?
Line 18 in bc0279d
Python files not generated.
plugins:
- plugin: buf.build/protocolbuffers/python:v26.1
out: proto/python_pb2
version: v1
deps:
- buf.build/googleapis/googleapis
- buf.build/envoyproxy/protoc-gen-validate
- buf.build/bufbuild/protovalidate
buf generate --include-imports
Generated python files should have buf/validate/validate.py files and other files generated .
googleapis and protoc-gen-validate have files generated but not for buf protovalidate.
It works if I run buf generate --include-imports buf.build/bufbuild/protovalidate
. Looks like the generate command without specifying dependency has some issue.
N/A
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.