suned / pfun Goto Github PK

This is caused by python/mypy#5485

Minimal example:

from typing import TypeVar, Any
from pfun.effect import Try, combine, catch

A = TypeVar("A")

def test(*effects: Try[Exception,A]) -> Try[Exception,None]:
    return combine(*effects)(lambda *a: None)

Blows up with a mypy internal error:

It doesn't matter if I move that lambda to a function.

(Probably a mypy issue, as I tried with the latest dev version of mypy (0.920dev) and didn't get the error. But note I had to disable the pfun mypy_plugin on the dev version, got a different error.)

Things to check for:

from typing import TypeVar, Generic
from pfun import Immutable


A = TypeVar('A')


class C(Immutable, Generic[A]):
    a: str
    b: int
    c: A

c = C('', 1, '')  # inferred type is C[str]
c.clone(d='')   # type error: Class "C" does not have attribute "d"
c.clone(a=1)  # type error: Expected "str" for argument "a" got "int"
c.clone(c=1)  # inferred type should be: C[int]

In this example I just want to transform Right to Data and Left to Error

@dataclass
class Error:
    msg: str

@dataclass
class Data:
    data: str


def i_can_fail2(s: str) -> Either[str, str]:
    # do something 
    return Right(f"{s}!")


def i_can_fail1(s: str) -> Either[str, str]:
    if s == 'illegal value':
        return Left("illegal value")
    return Right('Ok')


def func1() -> Either[Error, Data]:
    # what about a map but only for Left value?
    return i_can_fail1('illegal value').and_then(i_can_fail2).map(lambda x: Data(data=x))

I can do it with few lines of code, but I want to known if there is a proper way to do this.
Thanks

The documentation still has a few phrases that are left from a previous version of the Effect api. While not strictly incorrect, they can be made more clear.

Moreover, the pytkdocs wheel used to collect api documentation doesn't collect accurate api documentation in all cases for the cython code. A cython mkdocstrings handler should be implemented that collects accurate information.

Something inspired by this http://www.valuedlessons.com/2008/01/monads-in-python-with-nice-syntax.html

The main issues involve typing it correctly. Consider:

from typing import Generator, Any, TypeVar

from pfun.maybe import Maybe
from pfun.monad import Monad

A = TypeVar('A')
M = TypeVar('M', bound=Monad)

def do(f):
    ...

Do = Generator[M, Any, A]   # This type could of course be part of pfun

@do
def f() -> Do[Maybe, int]:
    a_str = yield produce_a_str()    # we somehow have to tell mypy that even though
                                     # the return type of 'produce_a_str' is Maybe[str], 'a_str' is actually 'str'
    an_int = yield produce_an_int(a_str)
    return an_int

I'm totally new of functional programming, I'm trying to use a reader monad to get rid of passing a lot of function parameter in my code, but I don't understand how to use it properly. Here a very simple use case.

def process(item, c):
    # how to get Connection?
    print(item, c)


def process_all() -> Reader[Connection, str]:
    def _(c):
        for item in range(5):
            process(item, c)
        return value("result")

    return ask().and_then(_)


def main():
    connection = Connection(URI='testuri')
    result = process_all().run(connection)
    print(result)

how can I get Connection in the process function without explicitly passing it? If I must pass it then I don't understand the advantage to directly pass Connection through functions without using a reader monad. What I'm missing?

Thanks.

E.g

def f(a: int) -> str:
    pass

def g(a: float) -> int:
    pass

def h(a: object) -> float:
    pass

compose(f, g, *[h])  # Error: too many arguments for compose

It might be useful for users of pfun if they could reuse the testing strategies implemented for the pfun test suite. Suggested solution is to add an optional test group that would install hypothesis with pip install pfun[test], and then provide the hypothesis strategies in pfun.hypothesis_strategies

Currently the Curry type expects an argument of Callable, which means that you have to write

Curry[Callable[[int], int]]

It would be nicer to simply write

Curry[[int, int], int]

Its unclear whether there is any meaning in using an argument list or whether to just use Curry[int, int, int]. One argument is to make it congruent with Callable

Also it might make sense to be able to write Curry[..., int] for cases where you can't give the exact arguments but you know the return type. Calling a value of that type would have the return type Curry[..., int] | int.

Similar to https://github.com/vlasovskikh/funcparserlib (or other monadic parsing libraries) but with a focus on static type checking.

The test suite takes so long to run that I probably wouldn't run it very frequently. I'm not sure if this is because of tox or if it's the tests themselves, I haven't looked into it in detail but it makes writing tests quite tedious.

E.g this should be ok:

from pfun.maybe import Just
def f():
     return Just(1)

Just(None).and_then(f)

I think the only way to provide correct typing for this is to handle it using the mypy plugin.

With all likelihood caused by python/mypy#7735

Use mypy plugin to enable static checking of case classes (see pfun/case.py for a start).

Things that can be checked:

• That all all cases are covered
• That all cases return same value
• That case arguments are of the correct type (t must correspond to match type, must be case subclasses, must all have the same base case)
• That arguments to when and then are of the correct type

Other things that plugin must (should) do:

• Handle CaseClass calls to dataclass to enable __init__ generation (maybe possible to use mypy dataclass plugin)
• Make sure that each Case subclass has exactly one base case class.

Minimal example:

from pfun.maybe import Maybe, Just, Nothing

def test_and_then() -> Maybe[int]:
   nope: Maybe[int] = Nothing()
   return nope.and_then(lambda x: Just(x + 1))

def test_map() -> Maybe[int]:
   nope: Maybe[int] = Nothing()
   return nope.map(lambda x: x + 1)

The first one results in:
Cannot infer type argument 1 of "and_then" of "Nothing"

The second:
Returning Any from function declared to return int

The second one seems to be because it is picking up the signature of the ABC: Maybe_.map.

I am using the pfun.mypy_plugin.

Thanks for this very promising library!

The current implementation of the type analyzer for Curry types and curry calls only adds the uncurried function type as the type of __call__. This means that the following call produces a false negative:

f: Curry[Callable[[int, int], int]]
g: Callable[[Callable[[int], Callable[[int], int]]], int]

g(f)

As a first solution I suggest to add an overloaded signature to Curry.__call__ that is curried in the required arguments only

E.g

from pfun.maybe import with_effect, Maybe
from typing import Generator, Any, TypeVar

A = TypeVar('A')
Maybes = Generator[Maybe[Any], Any, A]
def g() -> Maybe[int]:
    ...

@with_effect
def f() -> Maybes[int]:
    a = yield g()
    reveal_type(a)  # should be int

E.g

from typing import TypeVar
from pfun import compose

A = TypeVar('A')

def f(a: A) -> A:
    pass

def g(a: int) -> str:
    pass

compose(f, g)  # error: Cannot infer argument 1 of compose

I think this is a current limitation of the mypy inference, since this also does not work:

R1 = TypeVar('R1')
R2 = TypeVar('R2')
R3 = TypeVar('R3')

def compose(f: Callable[[R2], R1], g: Callable[[R3], R2]) -> Callable[[R3], R1]:
    pass

A = TypeVar('A')

def f(a: A) -> A:
    pass

def g(a: int) -> str:
    pass

compose(f, g)  # error: Cannot infer type of argument 1 of compose

Expected Behavior

Run my code without problem since I'm not using proccess

Current Behavior

Fail with message:

[Errno 38] Function not implemented: OSError

Possible Solution

Add option to disable ProccessPool or ThreadPool when isn't required

Steps to Reproduce

1. Write some AWS lambda with using effects
2. Try call your handler

Workaround:

For anyone with the same problem

from asyncio import get_event_loop
from concurrent.futures import ThreadPoolExecutor
from contextlib import AsyncExitStack

from pfun.effect import RuntimeEnv, CSuccess


async def call(self, r, max_threads=None):
    stack = AsyncExitStack()
    thread_executor = ThreadPoolExecutor(max_workers=max_threads)
    async with stack:
        stack.enter_context(thread_executor)
        env = RuntimeEnv(r, stack, None, thread_executor)
        effect = await self.do(env)
        if isinstance(effect, CSuccess):
            return effect.result
        if isinstance(effect.reason, Exception):
            raise effect.reason
        raise RuntimeError(effect.reason)


def run(effect, r):
    result = call(effect, r)
    return get_event_loop().run_until_complete(result)

Then change

main(event).run(env) by run(main(event), env)

Context (Environment)

Isn't a problem of pfun that it fails, but Proccess Pool isn't required either

https://stackoverflow.com/questions/34005930/multiprocessing-semlock-is-not-implemented-when-running-on-aws-lambda

Creating a long chain of and_then calls to the same monad can in theory overflow the stack. The common solution to this problem is trampolines.

I'm not sure how big a of a problem this will be in practice, since I think combining imperative style with functional style can in many cases alleviate the problem (see e.g the implementation of util.sequence_.)

Edit:
It will be a big problem.

From the doc string:

List(range(3)).append(3)

this generates a TypeError because append only takes an Iterable. Am I missing something?

from pfun.reader import Reader
from typing import TypeVar

A = TypeVar('A')
Action = Reader[str, A]
reveal_type(Action(lambda _: True))  # Revealed type is 'pfun.reader.Reader[builtins.str*, Any]'

This is probably due to a bug in mypy. At least the following also fails:

from typing import TypeVar, Generic, Callable

A = TypeVar('A')
B = TypeVar('B')
class C(Generic[A, B]):
    def __init__(self, f: Callable[[A], B]):
        pass
C = TypeVar('C')
Alias = C[str, C]
reveal_type(Alias(lambda _: True))  # Revealed type is 'C[builtins.str*, Any]'

At the moment, the mypy plugin supports only one call signature for curried function

from pfun import curry

@curry
def f(a: int, b: int, c: int) -> int:
    pass

reveal_type(f)  # int -> int -> int -> int

This means that calling f as e.g f(1, 1) does not type correctly, although this works fine at runtime.

The mypy plugin should be extended to support all eligible function signatures for curry

https://en.wikipedia.org/wiki/Monad_transformer

:)

i'm sure it's a typo, should be 'rb'

First off, cool library and nice docs and examples 👍

However, I am having problems making it work with mypy. It might be something with my setup or my lack of knowledge about mypy, -in that case the outcome might be some more value to add to the readme for other potential users :).

I am simply trying to run mypy on a file mytest.py which contains following lines

from pfun.io import get_line, put_line, IO

def hello(foo: str) ->IO[None]:
    return put_line('hello ' + foo)

hello('sir').run()

I get following error output when I try to run mypy on it

$ mypy mytest.py 
mytest.py:1: error: Cannot find module named 'pfun.io'
mytest.py:1: note: See https://mypy.readthedocs.io/en/latest/running_mypy.html#missing-imports
Found 1 error in 1 file (checked 1 source file)

I have a mypy.ini file in the same directory as mytest.py, and it contains:

[mypy]
plugins = pfun.mypy_plugin

Besides that, I am running NixOs and I am using a python virtualenv which contains following packages:

$ pip freeze
appdirs==1.4.3
astor==0.8.0
beautifulsoup4==4.8.1
bs4==0.0.1
certifi==2019.9.11
chardet==3.0.4
colorama==0.4.1
funcparserlib==0.3.6
hy==0.17.0+100.gf8d3826
idna==2.8
mypy==0.740
mypy-extensions==0.4.3
pfun==0.5.1
pythonping==1.0.4
requests==2.22.0
rply==0.7.7
soupsieve==1.9.4
toolz==0.10.0
typed-ast==1.4.0
typing-extensions==3.7.4.1
urllib3==1.25.6
youtube-dl==2019.7.16

and python version

$ python --version
Python 3.7.3

I have tried various things after reading a bit in the mypy docs, but nothing really helped.
At first I suspected that mypy wasn't picking up on the mypy.ini file, but then I tried adding ignore_missing_imports = True and then it reported success for mytest.py (because it was happily ignoring pfun). I have also verified that there is indeed a file mypy_plugin.py inside pfun in the site-packages of my virtualenv.

Is there something obvious that I am missing?.. Or could the problem be something with my setup? (for example some environment variable pointing to something that doesn't exist in NixOs...)

The experience of pfun using Pycharm's own type-checker is currently somewhat subpar to using mypy (because of the pfun mypy plugin). I suspect that it may be possible to provide an equally good experience by modifying the type checking of code that uses pfun by hooking into the pycharm inspections. (I haven't looked closely into it though).

from pfun import Immutable

class C(Immutable):
    pass

class B(Immutable):
    c: C

error:

Traceback (most recent call last):
  File "mypy/semanal.py", line 4626, in accept
  File "mypy/nodes.py", line 927, in accept
  File "mypy/semanal.py", line 1022, in visit_class_def
  File "mypy/semanal.py", line 1093, in analyze_class
  File "mypy/semanal.py", line 1102, in analyze_class_body_common
  File "mypy/semanal.py", line 1162, in apply_class_plugin_hooks
  File "/Users/sune/pfun/pfun/mypy_plugin.py", line 207, in _immutable_hook
    transformer.transform()
  File "mypy/plugins/dataclasses.py", line 90, in transform
  File "mypy/semanal.py", line 4360, in defer
AssertionError: Must not defer during final iteration

See https://github.com/suned/pfun/actions/runs/715062592

Maybe the solution is simply to fetch the gh-pages branch first?

Trampoline is used extensively to build stack safe, recursive structures, e.g in reader and state. When interpreting these structures, a significant amount of time is spent in the main loop of Trampoline.run. Consider this:

In [13]: from pfun.reader import value, sequence

In [14]: r = sequence([value(v) for v in range(100000)])

In [15]: %prun r.run(None)
         9799985 function calls in 18.876 seconds

   Ordered by: internal time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   100000    7.918    0.000    8.201    0.000 monad.py:31(<lambda>)
        1    5.780    5.780   19.344   19.344 trampoline.py:51(run)
  1599995    1.396    0.000    1.616    0.000 typing.py:806(__new__)
   599996    0.557    0.000    1.350    0.000 trampoline.py:32(and_then)
   200000    0.406    0.000    1.588    0.000 reader.py:45(<lambda>)
   799994    0.402    0.000    0.402    0.000 <string>:1(__init__)
   399996    0.261    0.000   11.689    0.000 trampoline.py:85(_handle_cont)
   600001    0.226    0.000    0.472    0.000 {built-in method builtins.isinstance}
   599996    0.221    0.000   12.953    0.000 trampoline.py:105(_resume)
  1599995    0.220    0.000    0.220    0.000 {built-in method __new__ of type object at 0x107597830}
   199998    0.184    0.000    0.462    0.000 trampoline.py:108(and_then)
   200000    0.180    0.000    0.459    0.000 reader.py:46(<lambda>)
   200000    0.151    0.000    0.588    0.000 reader.py:44(<lambda>)
   200001    0.143    0.000    0.376    0.000 reader.py:95(<lambda>)
   600001    0.139    0.000    0.611    0.000 trampoline.py:28(_is_done)
   199998    0.134    0.000    0.975    0.000 trampoline.py:113(<lambda>)
   600000    0.128    0.000    0.128    0.000 {built-in method _abc._abc_instancecheck}
   600000    0.119    0.000    0.246    0.000 abc.py:137(__instancecheck__)
   100000    0.108    0.000    0.283    0.000 reader.py:84(value)
   100000    0.079    0.000    0.204    0.000 reader.py:26(and_then)
   200000    0.068    0.000    1.043    0.000 trampoline.py:74(_handle_cont)
   100000    0.058    0.000    0.262    0.000 monad.py:30(<lambda>)
        1    0.000    0.000   19.344   19.344 <string>:1(<module>)
        3    0.000    0.000    0.000    0.000 future.py:47(__del__)
        1    0.000    0.000   19.344   19.344 {built-in method builtins.exec}
        1    0.000    0.000   19.344   19.344 reader.py:67(run)
        1    0.000    0.000    0.000    0.000 typing.py:245(inner)
        1    0.000    0.000    0.000    0.000 typing.py:890(cast)
        4    0.000    0.000    0.000    0.000 trampoline.py:89(_resume)
        1    0.000    0.000    0.000    0.000 {method 'disable' of '_lsprof.Profiler' objects}

Most of the execution time is spent in Trampoline.run. Optimising this loop is therefore a reasonable investment since it will speed up every usage of trampolines.

This could also be used in an eventual effect rotation "god monad" ala ZIO

from pfun.reader import Reader, ask, with_effect
from typing import TypeVar, Generator, Any

A = TypeVar('A')
R = TypeVar('R')
Readers = Generator[Reader[R, Any], Any, A]


@with_effect
def f() -> Readers[str, int]:
    env = yield ask()
    reveal_type(env)  # should be str

A mappable type is something that implemets __map__(self, f).
A chainable type is something that implements __chain__(self, f).

A type T that implements both can be used in do expressions as

fullname = do user_id('sad') as uid, first_name(uid) as first, last_name(uid) as last in f'{first} {last}'

Which will translate to

fullname = user_id('sad').__chain__(lambda uid: first_name(uid).__chain__(lambda first: last_name(uid).__map__(lambda last: f'{first} {last}')))

Is it a Python environment (like cpython)? Is it a linter (like mypy / pylint / flake8)? Is it a learning project?

Who should use it? In which context is it useful? Why was it created? Are there similar projects?

There are currently multiple ways to produce RecursionError when working with IO.

Some IO classes hold a "continuation" function to pass the result to, once the IO action is run (such as ReadFile). When composing a large number of ReadFiles with and_then calling run will unfold this large stack of recursive calls to the lambda created in and_then and cause a RecursionError:

from pfun.io import read_file, sequence

action = sequence([read_file('test.txt') for _ in range(5000)])
action.run()  # RecursionError

Other instances hold references to other IO actions (such as Put). Composing a large number of these instances will cause a RecursionError while building the structure because it calls and_then recursively.

from pfun.io import put_line, sequence

sequence([put_line('test') for _ in range(5000)])  # RecursionError

Ideally both types of calls should never cause a RecursionError. This should be achievable by trampolining appropriately (see pfun.trampoline and e.g pfun.reader for a usage example). Ideally this should be solved without changing the api.

Probably something I'm doing wrong (ie not informing types arguments of Either), but...

mypy MethodContext.default_return_type.args may return a tuple instead of a list.

Raising ''TypeError: 'tuple' object does not support item assignment" for code above

return_type = context.default_return_type
return_type_args: return_type.args
...
return_type_args[0]: _combine_environments(r1, r2)

We could expect that if isn't a list dev made something wrong - like I did - but this and similar assignments may cause unintended side effects 🤔

The lens mypy plugin hook currently doesn't work with generic types, eg in:

from typing import List

[0] | lens(List)[0] << 1  #  error: Unsupported operand types for << ("Lens[List[_T], _T]" and "int")

One possible solution approach might be to add the type variable definitions and types of the generic type to the signature of __call__, __ror__, and __lshift__ of the Lens instance.

Currently, curry instances returned from decorating methods with curry doesn't eliminate the self argument, which issues a type error when trying to call a curried method:

from pfun import curry

class C:
    @curry
    def f(x: int, y: int) -> int: ...

reveal_type(C().f)  #  note: Revealed type is 'pfun.functions.Curry[def (self: test_types.C, x: builtins.int, y: builtins.int) -> builtins.int]'
C().f(1)  # error: Argument 1 to "__call__" of "Curry" has incompatible type "int"; expected "C"

The main issue in fixing this is finding a way to distinguish this call to curry with a call with an instance that has a __call__ method in e.g:

class C:
     def __call__(self, x: int, y: int) -> int: ...

curry(C())