Giter Club home page Giter Club logo

protovalidate-python's Introduction

protovalidate-python

CI Conformance PyPI version

protovalidate-python is the Python implementation of protovalidate, designed to validate Protobuf messages at runtime based on user-defined validation constraints. Powered by Google's Common Expression Language (CEL), it provides a flexible and efficient foundation for defining and evaluating custom validation rules. The primary goal of protovalidate is to help developers ensure data consistency and integrity across the network without requiring generated code.

The protovalidate project

Head over to the core protovalidate repository for:

Other protovalidate runtime implementations include:

And others coming soon:

  • TypeScript: protovalidate-ts

Installation

To install the package, use pip:

pip install protovalidate

Make sure you have the latest version of protovalidate-python by checking the project's PyPI page.

Usage

Implementing validation constraints

Validation constraints are defined directly within .proto files. Documentation for adding constraints can be found in the protovalidate project README and its comprehensive docs.

syntax = "proto3";

package my.package;

import "google/protobuf/timestamp.proto";
import "buf/validate/validate.proto";

message Transaction {
  uint64 id = 1 [(buf.validate.field).uint64.gt = 999];
  google.protobuf.Timestamp purchase_date = 2;
  google.protobuf.Timestamp delivery_date = 3;

  string price = 4 [(buf.validate.field).cel = {
    id: "transaction.price",
    message: "price must be positive and include a valid currency symbol ($ or £)",
    expression: "(this.startswith('$') or this.startswith('£')) and float(this[1:]) > 0"
  }];

  option (buf.validate.message).cel = {
    id: "transaction.delivery_date",
    message: "delivery date must be after purchase date",
    expression: "this.delivery_date > this.purchase_date"
  };
}

Generating Code with buf

When using the runtime library after installing it with pip, it's necessary to generate the Python code for the core buf.protovalidate Protobuf package. buf provides an efficient method for this:

  1. Initialize a New Configuration File:

    buf mod init

    This initializes the buf.yaml configuration file at the root of the Protobuf source files.

  2. Module Configuration and Dependencies:

    # buf.yaml
    version: v1
    deps: 
      - buf.build/bufbuild/protovalidate

    Ensure your dependencies are up-to-date with:

    buf mod update
  3. Setup Code Generation:

    # buf.gen.yaml
    version: v1
    plugins:
      - plugin: buf.build/protocolbuffers/python:v23.4
        out: gen
  4. Generate Code: To generate the required Python code:

    buf generate --include-imports

If your goal is to generate code specifically for the buf.protovalidate Protobuf package, run:

buf generate buf.build/bufbuild/protovalidate

Note: For users familiar with protoc, while it's an alternative to buf, it is recommended to use tooling or frameworks like Bazel for direct code generation, as it provides an encapsulated environment for such tasks.

Example

import protovalidate
from google.protobuf.timestamp_pb2 import Timestamp
from my.package import Transaction

transaction = Transaction()
transaction.id = 1234
transaction.price = "$5.67"
transaction.purchase_date.CopyFrom(Timestamp())
transaction.delivery_date.CopyFrom(Timestamp())

try:
    protovalidate.validate(transaction)
except protovalidate.ValidationError as e:
# Report the violations

Ecosystem

Legal

Offered under the Apache 2 license.

protovalidate-python's People

Contributors

akshayjshah avatar alfus avatar buildbreaker avatar chrispine avatar dependabot[bot] avatar elliotmjackson avatar higebu avatar kyleconroy avatar marekbuild avatar rodaine avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

protovalidate-python's Issues

[BUG] ImportError on import buf/validate

Description

if we generate the python sources with buf and --include-imports flag, the protovalidate tries to import buf locally, but if we want to make generated sources a part of other module, the protovalidate raises ImportError on buf/validate import, so we need modify the protovalidate module sources to set where we want to import buf/validate

[Question] How to install?

Per the readme: I've done:
pip install protovalidate

And created the example .proto (and named it transaction.proto)
Then try protoc --python_out=. transaction.proto

Unsurprisingly:

buf/validate/validate.proto: File not found.
transaction.proto:6:1: Import "buf/validate/validate.proto" was not found or had errors.

Seems like the .proto it's looking for is in the main repo, but I've been stuck to trying to get the import to work.

Support for python <3.11

I would like to figure out why version 3.0.0 has dropped support for python <3.11 (#89) ?

  1. PEP-604 seems to be supported starting with python 3.10, not 3.11. So why 3.11?
  2. Why use a feature that has not yet necessarily reached wide-spread adoption in a library?

For us updating python version was not a huge pain, but it seems like this restriction might hurt adoption a lot. Python 3.11 is only one year old, i don't think it has reached the critical mass. Even 3.8 is not yet end-of-life

What am i missing?

[BUG]: On import of protovalidate

Description

After pip installing protovalidate I get a ModuleNotFoundError: No module named 'buf' from the import line in validatory.py from buf.validate import expression_pb2 # type: ignore

Steps to Reproduce

Expected Behavior

Unit test executes and raises ValidationError exception

Actual Behavior

Import error of protovalidate (by way of buf import)

Screenshots/Logs

============================= test session starts ==============================
collecting ... 
test_sdk.py:None (test_sdk.py)
ImportError while importing test module '/Users/stewartb/PycharmProjects/pythonProject3/tests/test_sdk.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
../../../Library/Caches/pypoetry/virtualenvs/pythonproject3-RFspHNkf-py3.11/lib/python3.11/site-packages/_pytest/python.py:617: in _importtestmodule
    mod = import_path(self.path, mode=importmode, root=self.config.rootpath)
../../../Library/Caches/pypoetry/virtualenvs/pythonproject3-RFspHNkf-py3.11/lib/python3.11/site-packages/_pytest/pathlib.py:567: in import_path
    importlib.import_module(module_name)
/opt/homebrew/Cellar/[email protected]/3.11.6_1/Frameworks/Python.framework/Versions/3.11/lib/python3.11/importlib/__init__.py:126: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
<frozen importlib._bootstrap>:1204: in _gcd_import
    ???
<frozen importlib._bootstrap>:1176: in _find_and_load
    ???
<frozen importlib._bootstrap>:1147: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:690: in _load_unlocked
    ???
../../../Library/Caches/pypoetry/virtualenvs/pythonproject3-RFspHNkf-py3.11/lib/python3.11/site-packages/_pytest/assertion/rewrite.py:186: in exec_module
    exec(co, module.__dict__)
test_sdk.py:2: in <module>
    import protovalidate
../../../Library/Caches/pypoetry/virtualenvs/pythonproject3-RFspHNkf-py3.11/lib/python3.11/site-packages/protovalidate/__init__.py:15: in <module>
    from protovalidate import validator
../../../Library/Caches/pypoetry/virtualenvs/pythonproject3-RFspHNkf-py3.11/lib/python3.11/site-packages/protovalidate/validator.py:17: in <module>
    from buf.validate import expression_pb2  # type: ignore
E   ModuleNotFoundError: No module named 'buf'

Environment

  • Operating System: macOS
  • Version: Sonoma 14.1.1
  • Compiler/Toolchain: Python 3.11.6
  • Protobuf Compiler & Version: buf 1.28.1
  • Protoc-gen-validate Version:
  • Protovalidate Version: 0.3.0

Possible Solution

When looking through the dependency, it seemed that the import path should have been from gen.buf.validate import expression_pb2 instead of buf.validate import experssion_pb2

Additional Context

I'm sure this is likely user error, on my part, but hopefully the authors can help me out.

[BUG] Generated Python code leads to runtime errors

Description

The following import line is included in the generated python code:

from buf.validate import validate_pb2 as buf_dot_validate_dot_validate__pb2

However, this code leads to runtime errors since there is no module named buf

ModuleNotFoundError: No module named 'buf'

I have the python package protovalidate installed as dependency in my project.

Steps to Reproduce

  1. Write a .proto file that utilises protovalidate
syntax = "proto3";

package examplev1;

import "buf/validate/validate.proto";

service ExampleService {
  ....
}

message ExampleRequest {
  string device_id = 1 [(buf.validate.field).string.pattern = "^([0-9A-Fa-f]{2}[:]){5}([0-9A-Fa-f]{2})$"];
}
  1. Generate code with buf generate

The buf.gen.yaml file I'm using

version: v1
managed:
  enabled: true
  go_package_prefix:
    default: example/proto/gen/go
    except:
      - buf.build/bufbuild/protovalidate
plugins:
  - plugin: buf.build/protocolbuffers/go
    out: proto/gen/go
    opt: paths=source_relative
  - plugin: buf.build/bufbuild/connect-go
    out: proto/gen/go
    opt: paths=source_relative
  - plugin: buf.build/grpc/go
    out: proto/gen/go
    opt: paths=source_relative
  - plugin: buf.build/grpc/python
    out: proto/gen/python
  - plugin: buf.build/protocolbuffers/python
    out: proto/gen/python
    opt: pyi_out

the buf.yaml I'm using

version: v1
name: buf.build/mystuff/core
deps:
  - buf.build/bufbuild/protovalidate
breaking:
  use:
    - FILE
lint:
  ignore:
    - validate
  use:
    - DEFAULT
    - COMMENTS
    - UNARY_RPC
    - PACKAGE_NO_IMPORT_CYCLE

I've made sure that I've ran buf mod update and that my buf.lock contains protovalidate as a dep.

buf generate command I'm executing: buf generate --template buf.gen.yaml

  1. Inspect generated python code: the following line is contained in the *_pb2.py and *_pb2.pyi files
from buf.validate import validate_pb2 as buf_dot_validate_dot_validate__pb2

There is no module named buf hence a runtime error is encountered

ModuleNotFoundError: No module named 'buf'

Expected Behavior

I expected the validate imports in the generated python code to not lead to a runtime error

Actual Behavior

The generated python code attempts to utilises a module that does not exists/is not present in the project dependencies (despite installed the protovalidate python package)

Screenshots/Logs

N/A

Environment

  • Operating System: macOS
  • Version: macOS Ventura 13.4.1 (c)
  • Compiler/Toolchain:
    Apple clang version 14.0.3 (clang-1403.0.22.14.1)
    Target: x86_64-apple-darwin22.5.0
    Thread model: posix
    InstalledDir: /Library/Developer/CommandLineTools/usr/bin
  • Protobuf Compiler & Version: buf v1.25.0, protoc 3.19.4
  • Protoc-gen-validate Version: N/A
  • Protovalidate Version: 0.1.0

Possible Solution

Additional Context

Cannot import protovalidate `ModuleNotFoundError: No module named 'buf'`

Description

Steps to Reproduce

python3.10 -m venv venv
source ./venv/bin/activate
python --version
# Python 3.10.13
pip install protovalidate --no-cache
# Collecting protovalidate
#   Using cached protovalidate-0.3.1-py3-none-any.whl.metadata (19 kB)
# Collecting cel-python (from protovalidate)
#   Using cached cel_python-0.1.5-py3-none-any.whl (87 kB)
# Collecting protobuf (from protovalidate)
#   Using cached protobuf-4.25.1-cp37-abi3-macosx_10_9_universal2.whl.metadata (541 bytes)
# Collecting jmespath>=0.10.0 (from cel-python->protovalidate)
#   Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
# Collecting lark-parser>=0.10.1 (from cel-python->protovalidate)
#   Using cached lark_parser-0.12.0-py2.py3-none-any.whl (103 kB)
# Collecting python-dateutil>=2.8.1 (from cel-python->protovalidate)
#   Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
# Collecting pyyaml>=5.4.1 (from cel-python->protovalidate)
#   Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (2.1 kB)
# Collecting requests>=2.25.1 (from cel-python->protovalidate)
#   Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
# Collecting urllib3>=1.26.4 (from cel-python->protovalidate)
#   Using cached urllib3-2.1.0-py3-none-any.whl.metadata (6.4 kB)
# Collecting babel>=2.9.0 (from cel-python->protovalidate)
#   Using cached Babel-2.14.0-py3-none-any.whl.metadata (1.6 kB)
# Collecting six>=1.5 (from python-dateutil>=2.8.1->cel-python->protovalidate)
#   Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
# Collecting charset-normalizer<4,>=2 (from requests>=2.25.1->cel-python->protovalidate)
#   Using cached charset_normalizer-3.3.2-cp310-cp310-macosx_11_0_arm64.whl.metadata (33 kB)
# Collecting idna<4,>=2.5 (from requests>=2.25.1->cel-python->protovalidate)
#   Using cached idna-3.6-py3-none-any.whl.metadata (9.9 kB)
# Collecting certifi>=2017.4.17 (from requests>=2.25.1->cel-python->protovalidate)
#   Using cached certifi-2023.11.17-py3-none-any.whl.metadata (2.2 kB)
# Using cached protovalidate-0.3.1-py3-none-any.whl (22 kB)
# Using cached protobuf-4.25.1-cp37-abi3-macosx_10_9_universal2.whl (394 kB)
# Using cached Babel-2.14.0-py3-none-any.whl (11.0 MB)
# Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl (169 kB)
# Using cached requests-2.31.0-py3-none-any.whl (62 kB)
# Using cached urllib3-2.1.0-py3-none-any.whl (104 kB)
# Using cached certifi-2023.11.17-py3-none-any.whl (162 kB)
# Using cached charset_normalizer-3.3.2-cp310-cp310-macosx_11_0_arm64.whl (120 kB)
# Using cached idna-3.6-py3-none-any.whl (61 kB)
# Installing collected packages: lark-parser, urllib3, six, pyyaml, protobuf, jmespath, idna, charset-normalizer, certifi, babel, requests, python-dateutil, cel-python, protovalidate
# Successfully installed babel-2.14.0 cel-python-0.1.5 certifi-2023.11.17 charset-normalizer-3.3.2 idna-3.6 jmespath-1.0.1 lark-parser-0.12.0 protobuf-4.25.1 protovalidate-0.3.1 python-dateutil-2.8.2 pyyaml-6.0.1 requests-2.31.0 six-1.16.0 urllib3-2.1.0
# 
# [notice] A new release of pip is available: 23.3.1 -> 23.3.2
# [notice] To update, run: pip install --upgrade pip
python
# Python 3.10.13 (main, Aug 24 2023, 12:59:26) [Clang 15.0.0 (clang-1500.0.40.1)] on darwin
# Type "help", "copyright", "credits" or "license" for more information.
# >>> import protovalidate
# Traceback (most recent call last):
#   File "<stdin>", line 1, in <module>
#   File "/Users/r/tmp/2/venv/lib/python3.10/site-packages/protovalidate/__init__.py", line 15, in <module>
#     from protovalidate import validator
#   File "/Users/r/tmp/2/venv/lib/python3.10/site-packages/protovalidate/validator.py", line 19, in <module>
#     from buf.validate import expression_pb2  # type: ignore
# ModuleNotFoundError: No module named 'buf'
# >>>

here is an example in google colab (https://colab.research.google.com/drive/1QuCI88VCUmkIT7-AfrqxPU50OuoePRXO?usp=sharing)

Expected Behavior

import is successful

ModuleNotFoundError: No module named 'buf'

Environment

  • Operating System: macos
  • Version: 14.0 (23A344)
  • Protovalidate Version: 0.3.1, 0.2.1 (didn't try with 0.3.0)

Additional Context

It seems to have started recently (past couple of days, way working fine before). Not sure what changed

[BUG] requires-python = ">=3.7" in pyproject.toml is inaccurate

Description

requires-python = ">=3.7"

In pyproject.toml, required python version is ">=3.7". However, Python 3.7 does not support PEP 604 and causes an error when using protovalidate.

Steps to Reproduce

Expected Behavior

Please clarify the required Python version for protovalidate-python, maybe 3.11?

python_version = "3.11"

Actual Behavior

Screenshots/Logs

Environment

  • Operating System:
  • Version:
  • Compiler/Toolchain:
  • Protobuf Compiler & Version:
  • Protoc-gen-validate Version:
  • Protovalidate Version:

Possible Solution

Additional Context

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.