pyvalid is a Python data validation tool.
Documentation: uzumaxy.github.io/pyvalid.
Note that this project is distributed under the MIT License.
:white_check_mark: Python values validator
Home Page: http://eterey.github.io/pyvalid/
License: BSD 3-Clause "New" or "Revised" License
pyvalid is a Python data validation tool.
Documentation: uzumaxy.github.io/pyvalid.
Note that this project is distributed under the MIT License.
Provide an ability to validate objects against the schema
I am putting a pull request together that will integrate pyvalid with Travis CI and codecov.io. The build failed on python 2.7:
https://travis-ci.org/jeking3/pyvalid/jobs/547753779
platform linux2 -- Python 2.7.14, pytest-4.6.3, py-1.8.0, pluggy-0.12.0
cachedir: .tox/coverage/.pytest_cache
rootdir: /home/travis/build/jeking3/pyvalid
collected 23 items
tests/test_accepts.py F...... [ 30%]
tests/test_number_validator.py ..... [ 52%]
tests/test_returns.py .. [ 60%]
tests/test_string_validator.py ...... [ 86%]
tests/test_switch.py ... [100%]
=================================== FAILURES ===================================
___________________ AcceptsDecoratorTestCase.test_args_dict1 ___________________
self = <tests.test_accepts.AcceptsDecoratorTestCase testMethod=test_args_dict1>
def test_args_dict1(self):
# Send all arguments.
arg1 = str()
kwargs = {
'arg2': int(), 'arg3': True, 'arg4': True
}
result = self.func3(arg1, **kwargs)
self.assertEqual((arg1, kwargs), result)
> result = self.func4(arg1, **kwargs)
tests/test_accepts.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
pyvalid/__accepts.py:39: in decorator_wrapper
self.__validate_args(func.__name__, func_args, func_kwargs)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
self = <pyvalid.__accepts.Accepts object at 0x7f22f88ba990>, func_name = 'func4'
args = ('',), kwargs = {'arg2': 0, 'arg3': True, 'arg4': True}
def __validate_args(self, func_name, args, kwargs):
"""Compare value of each required argument with list of
accepted values.
Args:
func_name (str): Function name.
args (list): Collection of the position arguments.
kwargs (dict): Collection of the keyword arguments.
Raises:
InvalidArgumentNumberError: When position or count of the arguments
is incorrect.
ArgumentValidationError: When encountered unexpected argument
value.
"""
from pyvalid.validators import Validator
for i, (arg_name, accepted_values) in enumerate(self.accepted_args):
if i < len(args):
value = args[i]
else:
if arg_name in kwargs:
value = kwargs[arg_name]
elif i in self.optional_args:
continue
else:
raise InvalidArgumentNumberError(func_name)
is_valid = False
for accepted_val in accepted_values:
is_validator = (
isinstance(accepted_val, Validator) or
(
isinstance(accepted_val, MethodType) and
hasattr(accepted_val, '__func__') and
isinstance(accepted_val.__func__, Validator)
)
)
if is_validator:
is_valid = accepted_val(value)
elif isinstance(accepted_val, type):
is_valid = isinstance(value, accepted_val)
else:
is_valid = value == accepted_val
if is_valid:
break
if not is_valid:
ord_num = self.__ordinal(i + 1)
raise ArgumentValidationError(
ord_num,
func_name,
value,
> accepted_values
)
E ArgumentValidationError: The 1st argument of func4() is and not in a [<type 'bool'>]
pyvalid/__accepts.py:144: ArgumentValidationError
===================== 1 failed, 22 passed in 0.24 seconds ======================
ERROR: InvocationError for command /home/travis/build/jeking3/pyvalid/.tox/coverage/bin/coverage run --branch -m pytest (exited with code 1)
___________________________________ summary ____________________________________
Remove the .travis.yml
since we're using Github Actions now
The following usage of deprecated collections syntax will cause pyvalid to fail with python3.8:
/home/jking/pyvalid/pyvalid/__accepts.py:1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Callable
pyvalid/validators.py:5
pyvalid/validators.py:5
/home/jking/pyvalid/pyvalid/validators.py:5: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated, and in 3.8 it will stop working
from collections import Iterable, Container, Callable
Provide a possibility to turn on the "logs only" mode, which will work in the following way:
It may be helpful for production mode, when any runtime errors are unwelcome.
`seva@spin:~$ /tmp/venv/bin/pip install pyvalid==0.9.2
Collecting pyvalid==0.9.2
Downloading https://files.pythonhosted.org/packages/f9/37/5a6aafd4cf2d6db43cbf6f79f211d69b48107e2045cdeca906f2eacac5a6/pyvalid-0.9.2.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-FtnEbh/pyvalid/setup.py", line 81, in
main()
File "/tmp/pip-install-FtnEbh/pyvalid/setup.py", line 12, in main
with open('requirements.txt') as fd:
IOError: [Errno 2] No such file or directory: 'requirements.txt'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-FtnEbh/pyvalid/
seva@spin:~$ /tmp/venv/bin/pip install pyvalid==0.9.1
Collecting pyvalid==0.9.1
Downloading https://files.pythonhosted.org/packages/04/e8/ee039c1b990b6ca2c8af439e05e1bba681eeaf360a1a14e3565fd4914f46/pyvalid-0.9.1.tar.gz
Complete output from command python setup.py egg_info:
Traceback (most recent call last):
File "", line 1, in
File "/tmp/pip-install-tAmnTm/pyvalid/setup.py", line 81, in
main()
File "/tmp/pip-install-tAmnTm/pyvalid/setup.py", line 12, in main
with open('requirements.txt') as fd:
IOError: [Errno 2] No such file or directory: 'requirements.txt'
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-tAmnTm/pyvalid/
`
We need to improve unit tests coverage in order to reach the value close to 100%
How to:
coverage
packagecoverage
package and find the code not covered by testsProvide an ability to check the type of the input value automatically and trigger the _check
function after that.
Right now we have pretty similar code in all the validators:
# NumberValidator (pyvalid/validators/__number.py)
def __call__(self, val):
valid = isinstance(val, NumberValidator.number_types) and self._check(val)
return valid
# StringValidator (pyvalid/validators/__string.py)
def __call__(self, val):
valid = isinstance(val, str) and self._check(val)
return valid
# IterableValidator (pyvalid/validators/__iterable.py)
def __call__(self, val):
is_iterable = isinstance(val, IterableValidator.iterable_types)
valid = is_iterable and self._check(val)
return valid
# ...
I want to remove this code from the validators and move this logic in the AbstractValidator
since it's the same for all of them.
So it won't be needed to implement the __call__
method all the time, but only in some rare cases.
from pyvalid import accepts
import pandas as pd
from pyvalid.validators import NumberValidator
class Mention:
@accepts(object,
overlap_percentage=NumberValidator(min_val=0.0, max_val=1.0)
)
def __init__(self, df: pd.DataFrame, presence_col: str, overlap_percentage=1.2) -> None:
pass
Mention(None,"abc",0.9)
Traceback (most recent call last):
File "example.py", line 14, in
Mention(None,"abc",0.9)
File "C:\Users\40102447\oss\pyvalid\pyvalid__accepts.py", line 88, in decorator_wrapper
self.__validate_args(func, func_args, func_kwargs)
File "C:\Users\40102447\oss\pyvalid\pyvalid__accepts.py", line 196, in __validate_args
raise ArgumentValidationError(func, ord_num, value, allowed_values)
pyvalid.__exceptions.ArgumentValidationError: The 2nd argument of the "init(self, df: pandas.core.frame.DataFrame, presence_col: str, overlap_percentage=1.2) -> None" functi
on is "None" of the "<class 'NoneType'>" type, while expected values are: "[<pyvalid.validators.__number.NumberValidator object at 0x000001B9AC892730>]".
Add regex checker into StringValidator class.
Decorator accepts
does not works correctly with function, which has list or dictionary of arguments.
For example:
@accepts(int, object, float, str)
def func(arg1, *args, *kwargs): pass
func(1) # Will cause IndexError exception.
Provide an ability to turn off/on particular decorators. For example, it should be possible to turn off the @accepts
decorator and keep the @returns
decorator enabled.
It should be done through the pyvalid.switch
module.
Transform "Validators" (right now it's validators.py
) into a folder with one class per file as it will increase readability and make it easy to add more validators in the future.
This does not seem to work:
from pyvalid import returns
@returns((dict, [str]))
def good():
return ({"foo": "bar"}, ["42"])
@returns((dict, [str]))
def good2():
return (None, ["42"])
@returns((dict, [str]))
def good3():
return (None, None)
good()
good2()
good3()
root@3794c8d02dc5:/data/blueprints# python3 /tmp/foo.py
Traceback (most recent call last):
File "/tmp/foo.py", line 15, in <module>
good()
File "/usr/local/lib/python3.7/dist-packages/pyvalid/__returns.py", line 39, in decorator_wrapper
raise InvalidReturnType(type(returns_val), func.__name__)
pyvalid.__exceptions.InvalidReturnType: Invalid return type <class 'tuple'> for good()
Affected package:
pyvalid-0.9
Environment
Python 3.6.3 (v3.6.3:2c5fed8, Oct 3 2017, 18:11:49) [MSC v.1900 64 bit (AMD64)] on win32
Issue:
Using Returns with a function destroys the docstring of that function.
docstring is an important feature of python and using the Returns decorator should not remove it.
Demonstration:
from pyvalid import returns
def foo():
""" Simple function to demonstrate docstring/returns bug """
return 42
@returns(int)
def fooWithReturn():
""" Simple function to demonstrate docstring/returns bug """
return 42
print(foo.doc)
print(fooWithReturn.doc)
Output of the demonstration program:
Simple function to demonstrate docstring/returns bug
None
Solution:
In package pyvalid file __returns.py change the following:
"
def call(self, func):
# Include the following line to fix the bug.
@functools.wraps(func)
def decorator_wrapper(*func_args, **func_kwargs):
"
Allow to disable and enable all existed validators manually.
@accepts(list, IterableValidator(empty_allowed=True, elements_type=str)
def example([]):
pass
Iterable Validator fails in the above shown scenario. If we are allowing empty list the user may either provide empty or non-empty list, and the type check should not throw exception when the list is empty.
Decorator accepts
does not works correctly with a function which accepts only keyword arguments.
For example this code:
@accepts(arg1=[str, int], arg2=['val1', 'val2'])
def func(**kwargs):
return kwargs
func(arg1=float(), arg2='val3')
Will not cause ArgumentValidationError
and return following value:
{'arg1': 0.0, 'arg2': 'val3'}
Create chainable shorthands for pyvalid's validators. It may be useful for cases, when we just need to quickly verify some value without preparing comprehensive validators.
Examples:
from pyvalid import check
check('Some_Str0ng_P@ssword!').all().min_length(6).max_length(64).contains_special_symbols(2).contains_numbers(1).exec()
# Returns `True`
check('WeakPass').all().min_length(6).max_length(64).contains_special_symbols(2).contains_numbers(1).exec()
# Returns `False`
check('12345').is_postal_code('gb').exec()
# Returns `False`
check('SW1W 0NY').is_postal_code('gb').exec()
# Returns `True`
The naming and style can be changed thought.
Traceback (most recent call last):
File "C:/Users/40102447/oss/pyvalid/example.py", line 61, in
register_users(new_user, "Nandan")
File "C:\Users\40102447\oss\pyvalid\pyvalid__accepts.py", line 86, in decorator_wrapper
self.__validate_args(func, func_args, func_kwargs)
File "C:\Users\40102447\oss\pyvalid\pyvalid__accepts.py", line 181, in __validate_args
is_valid = allowed_val(value)
File "C:\Users\40102447\oss\pyvalid\pyvalid\validators__schema.py", line 79, in call
valid_args = self._check(val)
File "C:\Users\40102447\oss\pyvalid\pyvalid\validators__schema.py", line 94, in _check
valid = checker_func(value, *checker_args)
File "C:\Users\40102447\oss\pyvalid\pyvalid\validators__string.py", line 19, in max_len_checker
return len(val) <= max_len
TypeError: object of type 'NoneType' has no len()
Provide the ability to validate input/output values base on type hints. It's possible to use inspect.signature
, typing.get_origin()
, typing.get_args()
, etc for retrieving information about type hints.
We'll need to create decorator, which analyzes type hints, generates the set of rules base in the hints and validates input/output data agaisnt these rules.
Provide the set of the validation tools for checking such data as: email address, postal code, phone number, credit card number, etc
We need to make the pyvalid package lightweight and easy to install. Therefore it would be great to try to avoid too heavy deps such as torch
(which is about 750 MB) and check if it's possible to implement existing functionaly without these deps.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.