jakeret / hope Goto Github PK
View Code? Open in Web Editor NEWHOPE: A Python Just-In-Time compiler for astrophysical computations
License: GNU General Public License v3.0
HOPE: A Python Just-In-Time compiler for astrophysical computations
License: GNU General Public License v3.0
as the title says. Thanks
I found an issue when using the function numpy.interp within a Python function compiled using HOPE. The issue is that interpolation produces different results depending on whether HOPE is used or not. The following example illustrates this.
import numpy as np
import hope
import pickle
def interpolation(x, xp, fp):
intp = np.interp(x, xp, fp)
return intp
@hope.jit
def interpolation_hope(x, xp, fp):
intp = np.interp(x, xp, fp)
return intp
with open('np.interp_github_issue.pkl', 'r') as f:
x, xp, fp = pickle.load(f)
x = x.astype(np.float32)
xp = xp.astype(np.float32)
fp = fp.astype(np.float32)
intp = interpolation(x, xp, fp)
intp_hope = interpolation_hope(x, xp, fp)
np.allclose(intp, intp_hope)
For some reason, this project is interesting to me. Could you/your team give more examples so people can quickly learn what this library can do and cannot do. Thank you!
When adding an np array to the test class and trying to access said array from a jitted function, it crashes while generating the CPP:
Traceback (most recent call last):
File "examples/hope_class.py", line 56, in <module>
example()
File "examples/hope_class.py", line 47, in example
r = dc.run(2)
File "/home/wolfv/Programs/hope/hope/_wrapper.py", line 71, in _hope_callback
return self(*args) if self.cache is None else self.cache(*args)
File "/home/wolfv/Programs/hope/hope/_wrapper.py", line 77, in __call__
ASTTransformer(self.modtoken).module_visit(self.fkt, args)
File "/home/wolfv/Programs/hope/hope/_transformer.py", line 706, in module_visit
parsed = self.visit(fktast)
File "/usr/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/home/wolfv/Programs/hope/hope/_transformer.py", line 644, in visit_FunctionDef
AllocateVisitor(self.variables).visit(self.token.body)
File "/home/wolfv/Programs/hope/hope/_ast.py", line 387, in visit
return getattr(self, 'visit_{0}'.format(type(node).__name__))(node)
File "/home/wolfv/Programs/hope/hope/_transformer.py", line 152, in visit_Body
start = check_blocks(start)
File "/home/wolfv/Programs/hope/hope/_transformer.py", line 141, in check_blocks
if self.variables[symbol].scope == "block":
KeyError: u'self.c'
Source:
# Copyright (C) 2014 ETH Zurich, Institute for Astronomy
'''
A simple example to demonstrate the supported functionality with class methods.
Note: this should be used with precaution as unboxing member variables causes some overhead
Created on Oct 23, 2014
author: jakeret
'''
from __future__ import print_function, division, absolute_import, unicode_literals
import os
import shutil
import hope
import numpy as np
prefix = "hope_class"
hope.config.keeptemp = True
hope.config.verbose = True
hope.config.prefix = prefix
if os.path.exists(prefix):
shutil.rmtree(prefix)
@hope.jit
def external(a, b):
return a + b
class DemoClass(object):
def __init__(self):
self.a = 1
self.c = np.array([1, 2, 3])
@hope.jit
def run(self, b):
return self.a + b + self.c
def run_external(self, b):
return external(self.a, b)
def example():
dc = DemoClass()
#jit compile a method of a class
r = dc.run(2)
assert r == 3
#call jitted function from a class
r = dc.run_external(2)
assert r == 3
if __name__ == '__main__':
example()
Hi I'm trying to get a function that iterates over xrange and wants to extract a row from a numpy array (then do an expression).
But I can't seem to get it to access the row of the array?
i get this exception:
raise Exception("Extends of variable and subscript do not match")
Exception: Extends of variable and subscript do not match
in line c = t[i]
in body c = t[i]
These are what the types are:
ds = 1227, <type 'numpy.int32'>
dt 400 <type 'int'>
as 69 <type 'numpy.int32'>
mt 2147483647 <type 'int'>
ns 2680 <type 'int'>
248256 <type 'int'>
[[ 21 22 1 7 48650 0]
[ 26 27 1 3 84823 1]
[ 64 65 1 11 77351 2]
...,
[ 68 69 2948 2970 53287 248253]
[ 67 68 2952 2964 53498 248254]
[ 68 312 2980 3034 53498 248255]]
<type 'numpy.ndarray'> shape (248256, 6) dtype int32
And the code:
@jit
def hope_loop(self, ds, dt, as, mt, ns, nr, t):
earliest_arrival = np.zeros(ns, dtype=np.int32)
earliest_arrival[:] = mt
earliest_arrival[ds] = dt
# Set up array of all stations to track back
in_connection = np.zeros(ns, dtype=np.int32)
in_connection[:] = -1
for i in xrange(nr):
c = t[i]
# if c[2] >= earliest_arrival[c[0]] and c[3] < earliest_arrival[c[1]]:
# continue
return earliest_arrival
When I just tried to import hope (without actually using any function), I got the following error. What is the fix for this? Thank you.
Traceback (most recent call last):
File "testHope.py", line 2, in
import hope
File "/Users/username/anaconda/lib/python2.7/site-packages/hope/init.py", line 11, in
from hope import config
File "/Users/username/anaconda/lib/python2.7/site-packages/hope/config.py", line 9, in
cxxflags = get_cxxflags()
File "/Users/username/anaconda/lib/python2.7/site-packages/hope/options.py", line 67, in get_cxxflags
_check_version(compiler_name, compiler.compiler[0])
File "/Users/username/anaconda/lib/python2.7/site-packages/hope/options.py", line 83, in _check_version
SUPPORTED_VERSIONS[compiler_name]))
hope.exceptions.UnsupportedCompilerException: Compiler 'gcc' with version '4.2.1' is not supported. Minimum version is '4.7.0'
I tried to see the performance, but it doesn't even compile. Maybe I'm doing something wrong?
Here's my code:
import time
from hope import jit
def normsum(n):
return sum((i for i in xrange(n)))
@jit
def hopesum(n):
return sum([i for i in range(n)])
st = time.time()
s = normsum(100000000)
print "Simple sum time: ", time.time()-st
print s
st = time.time()
hopesum(100000000)
print "Hope sum time: ", time.time()-st
print s
This is the output:
Simple sum time: 3.57045102119
4999999950000000
Traceback (most recent call last):
File "hope_test.py", line 21, in <module>
hopesum(100000000)
File "/usr/local/lib/python2.7/dist-packages/hope-0.3.0-py2.7.egg/hope/_wrapper.py", line 66, in _hope_callback
return self(*args) if self.cache is None else self.cache(*args)
File "/usr/local/lib/python2.7/dist-packages/hope-0.3.0-py2.7.egg/hope/_wrapper.py", line 72, in __call__
ASTTransformer(self.modtoken).module_visit(self.fkt, args)
File "/usr/local/lib/python2.7/dist-packages/hope-0.3.0-py2.7.egg/hope/_transformer.py", line 633, in module_visit
parsed = self.visit(fktast)
File "/usr/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/usr/local/lib/python2.7/dist-packages/hope-0.3.0-py2.7.egg/hope/_transformer.py", line 570, in visit_FunctionDef
self.token.body = self.body_visit(node.body)
File "/usr/local/lib/python2.7/dist-packages/hope-0.3.0-py2.7.egg/hope/_transformer.py", line 583, in body_visit
self.exprs.append(self.visit(stmt))
File "/usr/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/usr/local/lib/python2.7/dist-packages/hope-0.3.0-py2.7.egg/hope/_transformer.py", line 537, in visit_Return
value = self.visit(node.value)
File "/usr/lib/python2.7/ast.py", line 241, in visit
return visitor(node)
File "/usr/local/lib/python2.7/dist-packages/hope-0.3.0-py2.7.egg/hope/_transformer.py", line 450, in visit_Call
raise Exception("Function '{0}' not accessible form global scope of function '{1}': ({2})".format(node.func.id, self.fkt.__name__, ast.dump(node)))
Exception: Function 'sum' not accessible form global scope of function 'hopesum': (Call(func=Name(id='sum', ctx=Load()), args=[ListComp(elt=Name(id='i', ctx=Load()), generators=[comprehension(target=Name(id='i', ctx=Store()), iter=Call(func=Name(id='range', ctx=Load()), args=[Name(id='n', ctx=Load())], keywords=[], starargs=None, kwargs=None), ifs=[])])], keywords=[], starargs=None, kwargs=None))
in body return sum([i for i in range(n)])
On OSX 10.9 I get the following error when trying to import hope
:
hope.exceptions.UnsupportedCompilerException: Compiler 'gcc' with version '4.2.1
' is not supported. Minimum version is '4.7.0'
4.2.1
is indeed what the default gcc
(a proxy for clang
) reports. Are there any reasons it's not supported? And if there are, is there any way to point hope
to a separately installed GCC, say, the one from homebrew
?
Hello!
As I understand it, the project was abandoned about 6 years ago. Won't it be difficult for you to republish it under a different name in pypi?
I am just writing a Python framework and I really like the name, I would really appreciate it! The project will be ready for open source in a few days, now I can only tell you the details, if you're interested.
Thank you for your time.
The following code works in Python 2.7 on in macOS 10.13:
y=np.ones([n,n])
x=2*np.ones([n,n])
z=1
out=np.ones_like(x)
@hope.jit
def hope_exp(X,Y,Z,A):
A[:] = hope.exp((X**2+Y**2)/Z)
hope_exp(x,y,z,out)
but in Python 3.6.1 it returns the following error:
AttributeError Traceback (most recent call last)
/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_transformer.py in visit_Assign(self, node)
286
--> 287 target, value = self.visit(node.targets[0]), self.visit(node.value)
288/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/ast.py in visit(self, node)
252 visitor = getattr(self, method, self.generic_visit)
--> 253 return visitor(node)
254/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_transformer.py in visit_Call(self, node)
577 elif isinstance(node.func, ast.Attribute):
--> 578 if not node.starargs is None or not node.kwargs is None:
579 raise UnsupportedFeatureException("Only arguments without default values are supported in calls")AttributeError: 'Call' object has no attribute 'starargs'
During handling of the above exception, another exception occurred:
AttributeError Traceback (most recent call last)
/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_transformer.py in body_visit(self, stmts)
641 try:
--> 642 self.exprs.append(self.visit(stmt))
643 except Exception as ex:/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/ast.py in visit(self, node)
252 visitor = getattr(self, method, self.generic_visit)
--> 253 return visitor(node)
254/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_transformer.py in visit_Assign(self, node)
318 from hope._tosource import tosource
--> 319 ex.args = ((ex.args[0] if ex.args else "") + "\nin line " + tosource(node),) + ex.args[1:]
320 raise/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_tosource.py in tosource(node, indent_with, add_line_information)
77 generator = SourceGenerator(indent_with, add_line_information)
---> 78 generator.visit(node)
79/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/ast.py in visit(self, node)
252 visitor = getattr(self, method, self.generic_visit)
--> 253 return visitor(node)
254/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_tosource.py in visit_Assign(self, node)
173 self.write(' = ')
--> 174 self.visit(node.value)
175/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/ast.py in visit(self, node)
252 visitor = getattr(self, method, self.generic_visit)
--> 253 return visitor(node)
254/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_tosource.py in visit_Call(self, node)
440 self.visit(keyword.value)
--> 441 if node.starargs is not None:
442 write_comma()AttributeError: 'Call' object has no attribute 'starargs'
During handling of the above exception, another exception occurred:
AttributeError Traceback (most recent call last)
in ()
9 def hope_exp(X,Y,Z,A):
10 A[:] = hope.exp((X2+Y2)/Z)
---> 11 hope_exp(x,y,z,out)
12 np.exp((x2 + y2)/z)/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_wrapper.py in _hope_callback(*args)
69
70 def _hope_callback(*args):
---> 71 return self(*args) if self.cache is None else self.cache(*args)
72 self.callback = _hope_callback
73 setattr(cache, str(id(self.callback)), fkt)/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_wrapper.py in call(self, *args)
75
76 def call(self, *args):
---> 77 ASTTransformer(self.modtoken).module_visit(self.fkt, args)
78
79 if config.optimize:/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_transformer.py in module_visit(self, fkt, args, argid)
690 if argid is None or not argid in self.modtoken.functions[fkt.name]:
691 #Create the HOPE AST from the builtin AST
--> 692 parsed = self.visit(fktast)
693 if not parsed.getId() in [arg.getId() for arg in self.modtoken.functions[fkt.name]]:
694 self.modtoken.functions[fkt.name].append(parsed)/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/ast.py in visit(self, node)
251 method = 'visit_' + node.class.name
252 visitor = getattr(self, method, self.generic_visit)
--> 253 return visitor(node)
254
255 def generic_visit(self, node):/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_transformer.py in visit_FunctionDef(self, node)
627
628 # TODO: how do we parse recursive subfuncitons?
--> 629 self.token.body = self.body_visit(node.body)
630 AllocateVisitor(self.variables).visit(self.token.body)
631 merged = {}/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_transformer.py in body_visit(self, stmts)
644 if "\nin body " not in (ex.args[0] if ex.args else ""):
645 from hope._tosource import tosource
--> 646 ex.args = ((ex.args[0] if ex.args else "") + "\nin body " + tosource(stmt),) + ex.args[1:]
647 raise
648 for expr in self.exprs:/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_tosource.py in tosource(node, indent_with, add_line_information)
76
77 generator = SourceGenerator(indent_with, add_line_information)
---> 78 generator.visit(node)
79
80 return ''.join(generator.result)/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/ast.py in visit(self, node)
251 method = 'visit_' + node.class.name
252 visitor = getattr(self, method, self.generic_visit)
--> 253 return visitor(node)
254
255 def generic_visit(self, node):/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_tosource.py in visit_Assign(self, node)
172 self.visit(target)
173 self.write(' = ')
--> 174 self.visit(node.value)
175
176 def visit_AugAssign(self, node):/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/ast.py in visit(self, node)
251 method = 'visit_' + node.class.name
252 visitor = getattr(self, method, self.generic_visit)
--> 253 return visitor(node)
254
255 def generic_visit(self, node):/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_tosource.py in visit_Call(self, node)
439 self.write(keyword.arg + '=')
440 self.visit(keyword.value)
--> 441 if node.starargs is not None:
442 write_comma()
443 self.write('*')AttributeError: 'Call' object has no attribute 'starargs'
Additionally, the standard sum example from the readme gives this error in 3.6.1 but works in 2.7:
TypeError Traceback (most recent call last)
in ()
4 def sums(x, y):
5 return x + y
----> 6 sums(x,y)/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_wrapper.py in _hope_callback(*args)
69
70 def _hope_callback(*args):
---> 71 return self(*args) if self.cache is None else self.cache(*args)
72 self.callback = _hope_callback
73 setattr(cache, str(id(self.callback)), fkt)/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_wrapper.py in call(self, *args)
102 fp.write(code)
103
--> 104 so_filename = _compile(tempfolder, localfilename, self.fkt.name)
105
106 self._store_state(tempfolder, localfilename)/Users/edouglas/anaconda/envs/astroconda/lib/python3.6/site-packages/hope/_wrapper.py in _compile(target, localfilename, fkt_name)
271 try:
272 #trying to encode utf-8 to support AstroPy
--> 273 warnings.warn("A warning has been issued during compilation:\n{0}".format(out).encode('utf-8'))
274 except UnicodeError:
275 #encoding fails on LinuxTypeError: cannot use a string pattern on a bytes-like object
Hi,
I would like to contribute a variant on ln_python benchmark.
Instead of using numpy slice, we directly access to array elements inside a loop. This is of course terribly slow for interpreted Python but seems to be faster with compiled code like generated with HOPE and numba. In my tests, it happens to be fairly faster than the numpy slice version.
In the following code, I had to add a N
parameter beacuse HOPE doesn't support np.ndarray.size
attribute. I also use a Horner schema for the polynomial evaluation.
def loop_ln_python(X,Y, N):
for i in xrange(N):
xm1 = X[i] - 1.
tmp = xm1 * (1./9.) - 1./8.
tmp = tmp * xm1 + 1./7.
tmp = tmp * xm1 - 1./6.
tmp = tmp * xm1 + 1./5.
tmp = tmp * xm1 - 1./4.
tmp = tmp * xm1 + 1./3.
tmp = tmp * xm1 - 1./2.
tmp = tmp * xm1 + 1.
Y[i] = tmp * xm1
loop_ln_hope = hope.jit(loop_ln_python)
loop_ln_numba = numba.autojit(loop_ln_python)
I test with the following:
X = np.random.random(10000).astype(np.float64)
Y = np.ones_like(X)
loop_ln_hope(X, Y, len(X))
loop_ln_numba(X, Y, len(X))
%timeit loop_ln_python(X, Y, len(X))
%timeit loop_ln_hope(X, Y, len(X))
%timeit loop_ln_numba(X, Y, len(X))
On my laptop, I get the following timings:
100 loops, best of 3: 15.1 ms per loop
10000 loops, best of 3: 46.1 µs per loop
100000 loops, best of 3: 12.7 µs per loop
To be compared to the ln_python_exp()
timings:
1000 loops, best of 3: 257 µs per loop
10000 loops, best of 3: 76.8 µs per loop
10000 loops, best of 3: 110 µs per loop
I hope it will be of some interest for you.
For the sake of completeness, we can write a Horner schema version that doesn't add much to the compiled performances but which is more fair for numpy interpreted code:
def ln_python_horner(X, Y):
Xm1 = X - 1
Y[:] = Xm1 / 9. - 1./8.
Y[:] = Y * Xm1 + 1./7.
Y[:] = Y * Xm1 - 1./6.
Y[:] = Y * Xm1 + 1./5.
Y[:] = Y * Xm1 - 1./4.
Y[:] = Y * Xm1 + 1./3.
Y[:] = Y * Xm1 - 1./2.
Y[:] = Y * Xm1 + 1.
Y[:] = Y * Xm1
And with the timings:
10000 loops, best of 3: 169 µs per loop
10000 loops, best of 3: 104 µs per loop
10000 loops, best of 3: 119 µs per loop
Anyway, thanks for this nice package.
Cheers,
Pierre
I was trying to run the basic sum example, but seems the compiler does not get detected.
(test)sf@sf-MacBookPro:~/Projects/test/hope_test$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=13.10
DISTRIB_CODENAME=saucy
DISTRIB_DESCRIPTION="Ubuntu 13.10"
#!/usr/bin/env python
from hope import jit
@jit
def sum(x, y):
return x + y
(test)sf@sf-MacBookPro:~/Projects/test/hope_test$ python test.py
Traceback (most recent call last):
File "test.py", line 3, in <module>
from hope import jit
File "/home/sf/.virtualenvs/test/local/lib/python2.7/site-packages/hope/__init__.py", line 11, in <module>
from hope import config
File "/home/sf/.virtualenvs/test/local/lib/python2.7/site-packages/hope/config.py", line 9, in <module>
cxxflags = get_cxxflags()
File "/home/sf/.virtualenvs/test/local/lib/python2.7/site-packages/hope/options.py", line 72, in get_cxxflags
raise UnsupportedCompilerException("Unknown compiler: {0}".format(compiler_name))
hope.exceptions.UnsupportedCompilerException: Unknown compiler: x86_64-linux-gnu-gcc
(test)sf@sf-MacBookPro:~/Projects/test/hope_test$ python -c "from distutils.ccompiler import new_compiler; print new_compiler().compiler[0].split('/')[-1]"
cc
While testing the library, I found that doubles are printed with less precision.
For example, np.pi is printed in the CPP with only 15 decimal places (it is defined with 35 in the numpy math header).
I tried to add {:.35} to the format string, which works but prints the number in pythons internal representation (ie. it's not the same as the one defined in the headers because of floating point issues, I guess).
Another alternative (for np.pi) would be to include numpy/npy_math.h
and use NPY_PI as constant. However, that includes to add a String type to the ast.
Best,
Wolf
Right now, the fibbonachi example (visible at http://nbviewer.ipython.org/url/refreweb.phys.ethz.ch/hope/notebooks/julialang.org.ipynb ... for some reason this doesn't match this repository?) reads:
cimport cython
@cython.boundscheck(False)
@cython.wraparound(False)
def cython_fib(int n):
if n<2:
return n
return cython_fib(n-1)+cython_fib(n-2)
This is a mis-use of Cython, however. If you change it instead to
cimport cython
@cython.boundscheck(False)
@cython.wraparound(False)
cpdef int cython_fib(int n):
if n<2:
return n
return cython_fib(n-1)+cython_fib(n-2)
That yields a drastic speedup, making the Cython example equivalent to native code (and hope).
The point is that using "def" in tight loops is a mis-use of Cython: it makes Cython functions act just like regular Python functions, rather than using it as a c function with an int return type, which is what makes the recursion here go fast.
I installed hope from the git and run the following:
import numpy as np
import hope
@hope.jit
def pdf(density, dims, center, w2D, r50, b, a):
for x in range(dims[0]):
for y in range(dims[1]):
dr = np.sqrt((x - center[0]) ** 2 + (y - center[1]) ** 2)
density[x, y] = np.sum(w2D * 2 * (b - 1) / (2 * np.pi * (r50 * a)**2) * (1 + (dr / (r50 * a))**2)**(-b))
return density
with:
python -m timeit -s 'import numpy as np; b = 3.5; a = 1. / np.sqrt(2. ** (1. / (b - 1.)) - 1.) ; r50=20;center = np.array([10.141, 10.414]);dims = np.array([20, 20]) ; x1D = np.array([ 0.5 - 0.9491079123427585245262 / 2 , 0.5 - 0.7415311855993944398639 / 2 , 0.5 - 0.4058451513773971669066 / 2 , 0.5 , 0.5 + 0.4058451513773971669066 / 2 , 0.5 + 0.7415311855993944398639 / 2 , 0.5 + 0.9491079123427585245262 / 2 ], dtype=np.float32) ; w1D = np.array([ 0.1294849661688696932706 / 2 , 0.2797053914892766679015 / 2 , 0.38183005050511894495 / 2 , 0.4179591836734693877551 / 2 , 0.38183005050511894495 / 2 , 0.2797053914892766679015 / 2 , 0.1294849661688696932706 / 2 ], dtype=np.float32) ; w2D = np.outer(w1D, w1D) ; from pdf import pdf; density = np.zeros(dims, dtype=np.float32)' 'pdf(density, dims, center, w2D, r50, b, a)'
and the output is rather slow compared to the expected result. C++ module runs at the expected speed, so what did I do wrong?
I'm using hope 0.3.1 (upgraded today), anaconda python2.7 on Mac 10.9.5 with clang 3.4.
When I try to follow the very simple example of the sum function using hope in ipython 2.2.0, I get this error.
In [1]: from hope import jit
In [2]: @jit
...: def jitsum(x,y):
...: return x + y
...:
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-2-cc998918d205> in <module>()
----> 1 @jit
2 def jitsum(x,y):
3 return x + y
4
/astro/grads/arh5361/code/anaconda/envs/py27/lib/python2.7/site-packages/hope/jit.pyc in jit(fkt)
87
88 try:
---> 89 state = serialization.unserialize(filename)
90 if not state is None:
91 _check_state(fkt, state)
/astro/grads/arh5361/code/anaconda/envs/py27/lib/python2.7/site-packages/hope/serialization.pyc in unserialize(name)
40
41 with open(os.path.join(config.prefix, "{0}.pck".format(name)), "rb") as fp:
---> 42 return pickle.load(fp)
/astro/grads/arh5361/code/anaconda/envs/py27/lib/python2.7/pickle.pyc in load(file)
1376
1377 def load(file):
-> 1378 return Unpickler(file).load()
1379
1380 def loads(str):
/astro/grads/arh5361/code/anaconda/envs/py27/lib/python2.7/pickle.pyc in load(self)
856 while 1:
857 key = read(1)
--> 858 dispatch[key](self)
859 except _Stop, stopinst:
860 return stopinst.value
/astro/grads/arh5361/code/anaconda/envs/py27/lib/python2.7/pickle.pyc in load_proto(self)
884 proto = ord(self.read(1))
885 if not 0 <= proto <= 2:
--> 886 raise ValueError, "unsupported pickle protocol: %d" % proto
887 dispatch[PROTO] = load_proto
888
ValueError: unsupported pickle protocol: 3
For example:
https://travis-ci.org/cosmo-ethz/hope/jobs/38583739
The commands module was removed from the python3 standard library. We should replace commands with subprocess.
https://docs.python.org/2/library/commands.html
I'm working on this.
Hi Joel, did not find an email address of you so I write here, maybe we can follow up per email.
I'm working on the documentation and the benchmarks folder lacks a file util.py
which is used within the notebooks. Any idea where to find it ?
Best
Uwe
requirements.txt
and setup.py
do not specify some packages as a dependencies. They should because the makefile uses them. The following packages should be development dependencies:
On OS X Yosemite (10.10), I ran py.test test/test_functions.py
.
I get failures which are variations on the following:
E Exception: Error compiling function fkt (compiled to /var/folders/m1/78tf18750v38gm6y6y3dsm700000gn/T/hopemOTFNd)
hope/_wrapper.py:267: Exception
------------------------------------------------ Captured stdout call ------------------------------------------------
/var/folders/m1/78tf18750v38gm6y6y3dsm700000gn/T/hopemOTFNd/fkt_c90e045da53c2665412e2510a0e01b41d3f281dfe94b6581c8cb1895_0.cpp:67:52: warning: implicit conversion loses integer precision: 'std::size_t' (aka 'unsigned long') to 'int' [-Wshorten-64-to-32]
char ** symbols = backtrace_symbols(stack, depth);
~~~~~~~~~~~~~~~~~ ^~~~~
1 warning generated.
/var/folders/m1/78tf18750v38gm6y6y3dsm700000gn/T/hopemOTFNd/fkt_c90e045da53c2665412e2510a0e01b41d3f281dfe94b6581c8cb1895_0.cpp:38:24: error: non-constant-expression cannot be narrowed from type 'npy_int64' (aka 'long long') to 'npy_intp' (aka 'long') in initializer list [-Wc++11-narrowing]
npy_intp d__ret0[] = {cshape};
^~~~~~
/var/folders/m1/78tf18750v38gm6y6y3dsm700000gn/T/hopemOTFNd/fkt_c90e045da53c2665412e2510a0e01b41d3f281dfe94b6581c8cb1895_0.cpp:38:24: note: override this message by inserting an explicit cast
npy_intp d__ret0[] = {cshape};
^~~~~~
static_cast<npy_intp>( )
/var/folders/m1/78tf18750v38gm6y6y3dsm700000gn/T/hopemOTFNd/fkt_c90e045da53c2665412e2510a0e01b41d3f281dfe94b6581c8cb1895_0.cpp:38:24: warning: implicit conversion loses integer precision: 'npy_int64' (aka 'long long') to 'npy_intp' (aka 'long') [-Wshorten-64-to-32]
npy_intp d__ret0[] = {cshape};
~^~~~~~
1 warning and 1 error generated.
running build_ext
building 'fkt_c90e045da53c2665412e2510a0e01b41d3f281dfe94b6581c8cb1895_0' extension
C compiler: cc -fno-strict-aliasing -fno-common -dynamic -arch x86_64 -arch i386 -g -Os -pipe -fno-common -fno-strict-aliasing -fwrapv -DENABLE_DTRACE -DMACOSX -DNDEBUG -Wall -Wshorten-64-to-32 -DNDEBUG -g -fwrapv -Os -Wall -DENABLE_DTRACE -arch x86_64 -arch i386 -pipe
I'll look at fixing this tonight.
Hi,
on my box a basic example under python3 (3.4.2) does not work:
In [1]: from hope import jit
4.9.1
In [2]: @jit
def sum(x, y):
return x + y
...:
In [3]: sum(4,8)
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
running build_ext
building 'sum_900a4b7729fa81ded8e12217d97f52982df0978d23240cb0daa79505_0' extension
C compiler: gcc -pthread -Wno-unused-result -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fPIC
compile options: '-I/usr/lib/python3.4/site-packages/numpy/core/include -I/usr/include/python3.4m -c'
extra options: '-Wall -Wno-unused-variable -std=c++11'
gcc: /tmp/hopexz2jayp3/sum_900a4b7729fa81ded8e12217d97f52982df0978d23240cb0daa79505_0.cpp
g++ -pthread -shared -Wl,-O1,--sort-common,--as-needed,-z,relro /tmp/hopexz2jayp3/sum_900a4b7729fa81ded8e12217d97f52982df0978d23240cb0daa79505_0.o -L/usr/lib -lpython3.4m -o /tmp/hopexz2jayp3/sum_900a4b7729fa81ded8e12217d97f52982df0978d23240cb0daa79505_0.cpython-34m.so
---------------------------------------------------------------------------
Exception Traceback (most recent call last)
<ipython-input-3-2315bad740f1> in <module>()
----> 1 sum(4,8)
/home/xantares/projects/hope/hope/_wrapper.py in _hope_callback(*args)
64
65 def _hope_callback(*args):
---> 66 return self(*args) if self.cache is None else self.cache(*args)
67 self.callback = _hope_callback
68 setattr(cache, str(id(self.callback)), fkt)
/home/xantares/projects/hope/hope/_wrapper.py in __call__(self, *args)
97 fp.write(code)
98
---> 99 _compile(tempfolder, localfilename, self.fkt.__name__)
100
101 self._store_state(tempfolder, localfilename)
/home/xantares/projects/hope/hope/_wrapper.py in _compile(target, localfilename, fkt_name)
248 if not os.path.isfile(os.path.join(target, "{0}.so".format(localfilename))) or out.find("error:") > -1:
249 print(out)
--> 250 raise Exception("Error compiling function {0} (compiled to {1})".format(fkt_name, target))
251
252 #TODO: add test case
Exception: Error compiling function sum (compiled to /tmp/hopexz2jayp3)
In [4]:
Do you really want to exit ([y]/n)?
xantares@archlinux:~/projects/hope$ g++ -pthread -shared -Wl,-O1,--sort-common,--as-needed,-z,relro /tmp/hopexz2jayp3/sum_900a4b7729fa81ded8e12217d97f52982df0978d23240cb0daa79505_0.o -L/usr/lib -lpython3.4m -o /tmp/hopexz2jayp3/sum_900a4b7729fa81ded8e12217d97f52982df0978d23240cb0daa79505_0.cpython-34m.so
g++: error: /tmp/hopexz2jayp3/sum_900a4b7729fa81ded8e12217d97f52982df0978d23240cb0daa79505_0.o: No such file or directory
I'm using python 3.3, hope 0.3.1, on Mac 10.9.5 with clang 3.4. When I try the basic sum example, I get this error from the six library. I have the most recent version of six (1.8.0) installed and I'm uncertain if this is an issue with hope or truly an issues with six.
In [1]: from hope import jit
In [2]: @jit
...: def jitsum(x,y):
...: return x + y
...:
In [3]: jitsum(1,2)
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-3-64161ff6ad22> in <module>()
----> 1 jitsum(1,2)
/astro/grads/arh5361/code/anaconda/lib/python3.3/site-packages/hope/_wrapper.py in _hope_callback(*args)
69
70 def _hope_callback(*args):
---> 71 return self(*args) if self.cache is None else self.cache(*args)
72 self.callback = _hope_callback
73 setattr(cache, str(id(self.callback)), fkt)
/astro/grads/arh5361/code/anaconda/lib/python3.3/site-packages/hope/_wrapper.py in __call__(self, *args)
121
122 for name in list(sys.modules.keys()):
--> 123 if hasattr(sys.modules[name], self.fkt.__name__) and getattr(sys.modules[name], self.fkt.__name__) is self.callback:
124 setattr(sys.modules[name], self.fkt.__name__, module.run)
125 setattr(cache, str(id(module.run)), self.fkt)
/astro/grads/arh5361/code/anaconda/lib/python3.3/site-packages/astropy/extern/bundled/six.py in __getattr__(self, attr)
114 if attr in ("__file__", "__name__") and self.mod not in sys.modules:
115 raise AttributeError
--> 116 _module = self._resolve()
117 value = getattr(_module, attr)
118 setattr(self, attr, value)
/astro/grads/arh5361/code/anaconda/lib/python3.3/site-packages/astropy/extern/bundled/six.py in _resolve(self)
103
104 def _resolve(self):
--> 105 return _import_module(self.mod)
106
107 def __getattr__(self, attr):
/astro/grads/arh5361/code/anaconda/lib/python3.3/site-packages/astropy/extern/bundled/six.py in _import_module(name)
74 def _import_module(name):
75 """Import module, returning the module after the last dot."""
---> 76 __import__(name)
77 return sys.modules[name]
78
ImportError: No module named 'winreg'
It is convenient to have data packed into structures. For example, if a
calculation requires a large number of pieces of information, it is preferable
to have the following ( I realize this is a bit of a contrived example)
def func(sarray):
for i=0,range(sarray.size):
x = sarray['a'][i] + sarray['b'][i] + ... sarray['z'][i]
# do something with x
as opposed to
def func(a, b, c, d, ......, z):
for i in xrange(a.size):
x = a[i] + b[i] ... + z[i];
# do something with x
This could be solved by accepting structured arrays for input
sarray = zeros(n, dtype=[('a','f8'),('b','f8'),....('z','f8')])
res=func(sarray)
(edited for bugs)
The pycosmo test fails on my machine, with python 2.7.8 and python 3.4.2, I also have numpy 1.9.0, gcc 4.9.1.
It fails in test_pycosmo_1_opt, but only if I run test_pycosmo_1 first:
test_pycosmo_1()
test_pycosmo_1_opt()
If I set hope.config.optimize = False in test_pycosmo_1_opt it passes too.
Here is the output with hope.config.verbose=True :
$ python2 test/test_pycosmo.py
[...]/hope-0.3.1-py2.7.egg/hope/jit.py:113: UserWarning: Recompiling... Reason: State is inconsistent with config. Inconsistent state key: [keeptemp].
warnings.warn("Recompiling... Reason: {0}".format(le))
Compiling following functions:
pycosmo(float32^1 y, float64 lna, float64 k, float64 eta, float64 hubble_a, float64 tdot, float64 omega_gam, float64 omega_neu, float64 omega_dm_0, float64 omega_b_0, float64 ha, float64 rh, float64 r_bph_a, int64 xc_damp)
running build_ext
building 'pycosmo_595aba8b1f769fe4802a2e41fd8168d3a643953f6adc1192807b2249_0' extension
C compiler: gcc -pthread -fno-strict-aliasing -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -DNDEBUG -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fPIC
compile options: '-I/usr/lib/python2.7/site-packages/numpy/core/include -I/usr/include/python2.7 -c'
extra options: '-Wall -Wno-unused-variable -std=c++11'
gcc: /tmp/hopeIYR0lp/pycosmo_595aba8b1f769fe4802a2e41fd8168d3a643953f6adc1192807b2249_0.cpp
g++ -pthread -shared -Wl,-O1,--sort-common,--as-needed,-z,relro /tmp/hopeIYR0lp/pycosmo_595aba8b1f769fe4802a2e41fd8168d3a643953f6adc1192807b2249_0.o -L/usr/lib -lpython2.7 -o /tmp/hopeIYR0lp/pycosmo_595aba8b1f769fe4802a2e41fd8168d3a643953f6adc1192807b2249_0.so
[...]/hope-0.3.1-py2.7.egg/hope/jit.py:113: UserWarning: Recompiling... Reason: State is inconsistent with config. Inconsistent state key: [optimize].
warnings.warn("Recompiling... Reason: {0}".format(le))
pycosmo(float32^1 y, float64 lna, float64 k, float64 eta, float64 hubble_a, float64 tdot, float64 omega_gam, float64 omega_neu, float64 omega_dm_0, float64 omega_b_0, float64 ha, float64 rh, float64 r_bph_a, int64 xc_damp)
new a.D
a.D = numpy.exp(lna.D)
new psi.d
psi.d = (-y.f[0.J] - ((12.0.D / (((rh.D * k.D) * a.D) ** 2.J)) * ((omega_gam.D * y.f[9.J]) + (omega_neu.D * y.f[((6.J + (2.J * 10.J)) + 3.J)]))))
new dphidlna.d
dphidlna.d = ((psi.d - (((k.D ** 2.J) / ((3.0.D * (a.D ** 2.J)) * (ha.D ** 2.J))) * y.f[0.J])) + ((0.5.D / ((ha.D * rh.D) ** 2.J)) * (((((omega_dm_0.D * (a.D ** -3.J)) * y.f[1.J]) + ((omega_b_0.D * (a.D ** -3.J)) * y.f[3.J])) + (((4.0.D * omega_gam.D) * (a.D ** -4.J)) * y.f[5.J])) + (((4.0.D * omega_neu.D) * (a.D ** -4.J)) * y.f[((6.J + (2.J * 10.J)) + 1.J)]))))
new ppi.f
ppi.f = ((y.f[9.J] + y.f[6.J]) + y.f[10.J])
new n_y.J
n_y.J = (8.J + (3.J * 10.J))
new dydlna.f[:n_y.J]
(:n_y.J) {
dydlna.f[:n_y.J] = numpy.zeros(n_y.J, dtype=numpy.float32)
}
dydlna.f[0.J] = dphidlna.d
dydlna.f[1.J] = (((-k.D / (a.D * ha.D)) * y.f[2.J]) - (3.0.D * dphidlna.d))
dydlna.f[2.J] = (-y.f[2.J] + ((k.D / (a.D * ha.D)) * psi.d))
dydlna.f[3.J] = (((-k.D / (a.D * ha.D)) * y.f[4.J]) - (3.0.D * dphidlna.d))
dydlna.f[4.J] = ((-y.f[4.J] + ((k.D / (a.D * ha.D)) * psi.d)) + (((tdot.D / r_bph_a.D) / (a.D * ha.D)) * (y.f[4.J] - (3.0.D * y.f[7.J]))))
dydlna.f[5.J] = (((-k.D / (a.D * ha.D)) * y.f[7.J]) - dphidlna.d)
dydlna.f[6.J] = (((k.D / (a.D * ha.D)) * -y.f[8.J]) + ((tdot.D / (a.D * ha.D)) * (y.f[6.J] - (ppi.f / 2.0.D))))
dydlna.f[7.J] = (((k.D / ((3.0.D * a.D) * ha.D)) * ((y.f[5.J] - y.f[9.J]) + psi.d)) + ((tdot.D / (a.D * ha.D)) * (y.f[7.J] - (y.f[4.J] / 3.0.D))))
dydlna.f[8.J] = ((((k.D / (a.D * ha.D)) / 3.0.D) * (y.f[6.J] - (2.0.D * y.f[10.J]))) + ((tdot.D / (a.D * ha.D)) * y.f[8.J]))
dydlna.f[9.J] = (((k.D / ((5.0.D * a.D) * ha.D)) * ((2.0.D * y.f[7.J]) - (3.0.D * y.f[11.J]))) + ((tdot.D / (a.D * ha.D)) * (y.f[9.J] - (ppi.f / 10.0.D))))
dydlna.f[10.J] = ((((k.D / (a.D * ha.D)) / 5.0.D) * ((2.0.D * y.f[8.J]) - (3.0.D * y.f[12.J]))) + ((tdot.D / (a.D * ha.D)) * (y.f[10.J] - (ppi.f / 5.0.D))))
for i.l in (3.J:10.J) {
dydlna.f[(5.J + (2.J * i.l))] = ((((k.D / (a.D * ha.D)) / ((2.0.D * i.l) + 1.0.D)) * ((i.l * y.f[(5.J + (2.J * (i.l - 1.J)))]) - ((i.l + 1.0.D) * y.f[(5.J + (2.J * (i.l + 1.J)))]))) + ((tdot.D / (a.D * ha.D)) * y.f[(5.J + (2.J * i.l))]))
dydlna.f[(6.J + (2.J * i.l))] = ((((k.D / (a.D * ha.D)) / ((2.0.D * i.l) + 1.0.D)) * ((i.l * y.f[(6.J + (2.J * (i.l - 1.J)))]) - ((i.l + 1.0.D) * y.f[(6.J + (2.J * (i.l + 1.J)))]))) + ((tdot.D / (a.D * ha.D)) * y.f[(6.J + (2.J * i.l))]))
}
dydlna.f[(5.J + (2.J * 10.J))] = ((1.0.D / (a.D * ha.D)) * ((k.D * y.f[(5.J + (2.J * (10.J - 1.J)))]) - ((((10.J + 1.0.D) / eta.D) - tdot.D) * y.f[(5.J + (2.J * 10.J))])))
dydlna.f[(6.J + (2.J * 10.J))] = ((1.0.D / (a.D * ha.D)) * ((k.D * y.f[(6.J + (2.J * (10.J - 1.J)))]) - ((((10.J + 1.0.D) / eta.D) - tdot.D) * y.f[(6.J + (2.J * 10.J))])))
dydlna.f[((6.J + (2.J * 10.J)) + 1.J)] = (((-k.D / (a.D * ha.D)) * y.f[((6.J + (2.J * 10.J)) + 2.J)]) - dphidlna.d)
dydlna.f[((6.J + (2.J * 10.J)) + 2.J)] = ((k.D / ((3.0.D * a.D) * ha.D)) * ((y.f[((6.J + (2.J * 10.J)) + 1.J)] - y.f[((6.J + (2.J * 10.J)) + 3.J)]) + psi.d))
for j.l in (2.J:10.J) {
dydlna.f[(((6.J + (2.J * 10.J)) + 1.J) + j.l)] = (((k.D / (a.D * ha.D)) / ((2.0.D * j.l) + 1.0.D)) * ((j.l * y.f[((((6.J + (2.J * 10.J)) + 1.J) + j.l) - 1.J)]) - ((j.l + 1.0.D) * y.f[((((6.J + (2.J * 10.J)) + 1.J) + j.l) + 1.J)])))
}
dydlna.f[(((6.J + (2.J * 10.J)) + 1.J) + 10.J)] = ((1.0.D / (a.D * ha.D)) * ((k.D * y.f[((((6.J + (2.J * 10.J)) + 1.J) + 10.J) - 1.J)]) - (((10.J + 1.0.D) / eta.D) * y.f[(((6.J + (2.J * 10.J)) + 1.J) + 10.J)])))
if (xc_damp.J > 0.J) {
(5.J:(n_y.J - 1.J)) {
dydlna.f[5.J:(n_y.J - 1.J)] = ((y.f[5.J:(n_y.J - 1.J)] * 0.5.D) * (1.0.D - numpy.tanh((((k.D * eta.D) - xc_damp.J) / 50.0.D))))
}
}
return dydlna.f[:n_y.J]
Compiling following functions:
pycosmo(float32^1 y, float64 lna, float64 k, float64 eta, float64 hubble_a, float64 tdot, float64 omega_gam, float64 omega_neu, float64 omega_dm_0, float64 omega_b_0, float64 ha, float64 rh, float64 r_bph_a, int64 xc_damp)
running build_ext
building 'pycosmo_595aba8b1f769fe4802a2e41fd8168d3a643953f6adc1192807b2249_0' extension
C compiler: gcc -pthread -fno-strict-aliasing -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -DNDEBUG -march=x86-64 -mtune=generic -O2 -pipe -fstack-protector-strong --param=ssp-buffer-size=4 -fPIC
compile options: '-I/usr/lib/python2.7/site-packages/numpy/core/include -I/usr/include/python2.7 -c'
extra options: '-Wall -Wno-unused-variable -std=c++11'
gcc: /tmp/hopeJkZHki/pycosmo_595aba8b1f769fe4802a2e41fd8168d3a643953f6adc1192807b2249_0.cpp
g++ -pthread -shared -Wl,-O1,--sort-common,--as-needed,-z,relro /tmp/hopeJkZHki/pycosmo_595aba8b1f769fe4802a2e41fd8168d3a643953f6adc1192807b2249_0.o -L/usr/lib -lpython2.7 -o /tmp/hopeJkZHki/pycosmo_595aba8b1f769fe4802a2e41fd8168d3a643953f6adc1192807b2249_0.so
Segmentation fault (core dumped)
[x] Bug (Typo)
unneccesary
, however expect to see unnecessary
.simpify
, however expect to see simplify
.retuns
, however expect to see returns
.reasonamble
, however expect to see reasonable
.purpos
, however expect to see purpose
.propageted
, however expect to see propagated
.proplem
, however expect to see problem
.multipications
, however expect to see multiplications
.Semi-automated issue generated by
https://github.com/timgates42/meticulous/blob/master/docs/NOTE.md
To avoid wasting CI processing resources a branch with the fix has been
prepared but a pull request has not yet been created. A pull request fixing
the issue can be prepared from the link below, feel free to create it or
request @timgates42 create the PR. Alternatively if the fix is undesired please
close the issue with a small comment about the reasoning.
https://github.com/timgates42/hope/pull/new/bugfix_typos
Thanks.
Is it possible to relax the license to a MIT or BSD like license - at least for the runtime code that is included into the users python programm and extensions?
After fixing a typo in the .travis.yml the build are being kicked of on a commit.
Using @xantares pull request #11 will install gcc 4.8 which will allow to compile the tests. Pull request #7 will fix the version detection with py3.
Unfortunately it looks like that the build is taking to long and is being killed after 50 mins.
It's a large number of tests and all the tests are required to ensure the numerical accuracy up to type precision. Any clues on this?
If this package is run with Python 3.12, there is an issue because of the numpy.distutils dependency, as this is discontinued. Is there any prospects of removing this dependency?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.