pydata / patsy Goto Github PK
View Code? Open in Web Editor NEWDescribing statistical models in Python using symbolic formulas
License: Other
Describing statistical models in Python using symbolic formulas
License: Other
Two possible improvements in how we work with contrasts:
linear_constraint
but just the linear part, not the = constant
part.Syntax for this will be tricky. Possibly just an array of weights + a list of data dicts?
I need to brush up on the estimable contrast literature...
I have a columns whose unique looks like:
array([nan, 'CONFERENCE', 'ANALYST', 'FORUM', 'SEMINAR'], dtype=object)
I would expect that adding C(col_name)
to the formula would create 4 dummy variables (5 values-1), bu in fact it only adds 3.
When I tried to explicitly set control to be nan
, i get an exception:
C(col_name, Treatment(reference=nan))
PatsyError: specified level nan not found
From what I can tell, this project requires 2to3 to create a Python 3 build, but this is not stated in the README. Please clarify this, preferably including an example coversion like:
2to3 --output-dir=patsy3 -W -n patsy
Thanks!
Patsy automatically remove redundant columns (linearly dependent) so that the final matrix is not overdetermined. is there an option to turn off the removal? I would like to use patsy formulas for regularized linear regression and for that I need all the columns, even if they repeat.
I'm running a logistic regression and having trouble using Patsy's API to prepare the data when it is bigger than a small sample.
Using the dmatrices
function directly on a DataFrame, I am left with this abrupt error ( please note, I spun up an EC2 with 300GB of RAM after encountering this on my laptop, and got the same error ):
Traceback (most recent call last):
File "My_File.py", line 22, in <module>
df, return_type="dataframe")
File "/root/anaconda/lib/python2.7/site-packages/patsy/highlevel.py", line 297, in dmatrices
NA_action, return_type)
File "/root/anaconda/lib/python2.7/site-packages/patsy/highlevel.py", line 156, in do_highlevel_design
return_type=return_type)
File "/root/anaconda/lib/python2.7/site-packages/patsy/build.py", line 989, in build_design_matrices
results.append(builder._build(evaluator_to_values, dtype))
File "/root/anaconda/lib/python2.7/site-packages/patsy/build.py", line 821, in _build
m = DesignMatrix(np.empty((num_rows, self.total_columns), dtype=dtype),
MemoryError
So, I combed through Patsy's docs and found this gem:
patsy.incr_dbuilder(formula_like, data_iter_maker, eval_env=0)
Construct a design matrix builder incrementally from a large data set.
However, the method is sparsely documented, and the source code is largely uncommented.
I have arrived at this code:
def iter_maker():
with open("test.tsv", "r") as f:
reader = csv.DictReader(f, delimiter="\t")
for row in reader:
yield(row)
y, dta = incr_dbuilders("s ~ C(x) + C(y):C(rgh) + \
C(z):C(f) + C(r):C(p) + C(q):C(w) + \
C(zr):C(rt) + C(ff):C(djjj) + C(hh):C(tt) + \
C(bb):lat + C(jj):lng + C(ee):C(bb) + C(qq):C(uu)",
iter_maker)
df = dmatrix(dta, {}, 0, "drop", return_type="dataframe")
but I receive PatsyError: Error evaluating factor: NameError: name 'ff' is not defined
This is being thrown because _try_incr_builders (called from dmatrix) is returning None on line 151 of highlevel.py
What is the correct way to use these Patsy functions to prepare my data? Any examples or guidance you may have will be helpful.
Doing this works fine:
foo = pd.DataFrame(dict(a=[1,2.0], b=[4,5.0]))
patsy.dmatrices('a ~ b', foo)
However, if the formula is given by a unicode string:
patsy.dmatrices(u'a ~ b', foo)
An exception is raised:
ValueError: design matrix must be real-valued floating point
The exception is very misleading about the nature of the problem. I thought an inf or nan had crept into the dataframe, but no bad values were there.
One possible fix is to send the first argument of dmatrices through the str() function:
patsy.dmatrices(str(u'a ~ b'), foo)
Another possible fix is to raise print a different error message so that the user has some clear idea of the cause of the problem.
I'm using patsy version 0.3.0
Thanks!
Listwise deletion is an extremely common operation. It needs to be applied after selecting variables (done by patsy
when it interprets the formula), but before splitting exogenous and endogenous variables in separate matrices (also done by patsy
).
R's model.matrix
function applies list-wise deletion by default, and I expect that statsmodels
will also want to have that be the default behavior. Ohterwise, many of the estimation procedures will bork and we will need to remind users to handle their NAs.
Thanks!
R lets you regress a target against all the variables in the dataframe by simply typing "target ~ ."
This is particularly useful when we have lots of columns and don't want to specify each predictor. I don't think patsy supports this yet. Any chance if it can be added? Thanks.
Is this just me or are the stylesheets not working / uploaded?
Hi, COPYING was renamed to LICENSE.txt in 625317c but the header of *.py files still mention it
It would be nice if this behaved as expected:
y ~ x**2
When trying to build & package for Ubuntu, I get the following error:
Copying patsy.egg-info to /build/buildd/patsy-0.1.0~ppa1~revno/debian/python-patsy/usr/lib/python2.7/dist-packages/patsy-0.1.0_dev.egg-info
running install_scripts
dh_install -O--buildsystem=python_distutils
dh_installdocs -O--buildsystem=python_distutils
cp: cannot stat `LICENSE.txt': No such file or directory
dh_installdocs: cp -a LICENSE.txt debian/python-patsy/usr/share/doc/python-patsy returned exit code 1
make: *** [binary] Error 2
Would it be possible to rename
https://github.com/pydata/patsy/blob/master/COPYING
into
https://github.com/pydata/patsy/blob/master/LICENSE.txt
?
I was wondering if using ~
instead of =
in a formula string is to be compatible with the R formula syntax, or because using the latter will break something. If it won't break anything, could we optionally use =
in the place of ~
?
I know it is not a big deal, but I think a model statement with =
would look more natural.
I have a workflow where I want to construct a design matrix from some training data that happens to be stored in a pandas dataframe. I then want to be able to take a new pandas dataframe and apply the same transformation, so that I can have a pure numpy representation of test dataset with the same category indices. I suppose an error would be raised if my training dataset has categorical values that were missing in the original dmatrix.
I can't seem to figure out the most idiomatic way to achieve this.
My big picture workflow looks something like this:
What I need is a clean way to transform / featurize my test dataset. I suppose I'd also love for the API to look something like the sklearn model.transform()
api.
Does this exist? Can someone point me in the right direction?
Do we really need, say, the reference level in the Treatment contrast? I'm not sure it adds enough information vs. the complexity it adds to the names to warrant inclusion. Thoughts? AFAICT, it only appears if you specify a reference level. If you specify one, then surely you know what you specified.
[~/]
[7]: dmatrix('~C(A, Treatment)', data=pd.DataFrame([['some really long name'], ['other name'], ['other name']], columns=['A']))
[7]:
DesignMatrix with shape (3, 2)
Intercept C(A, Treatment)[T.some really long name]
1 1
1 0
1 0
Terms:
'Intercept' (column 0)
'C(A, Treatment)' (column 1)
[~/]
[8]: dmatrix("~C(A, Treatment('some really long name'))", data=pd.DataFrame([['some really long name'], ['other name'], ['other name']], columns=['A']))
[8]:
DesignMatrix with shape (3, 2)
Intercept C(A, Treatment('some really long name'))[T.other name]
1 0
1 1
1 1
Terms:
'Intercept' (column 0)
"C(A, Treatment('some really long name'))" (column 1)
With NumPy 1.9.1, Pandas 0.15.2, and Patsy 0.3.0:
$ nosetests
............./usr/local/lib/python2.7/dist-packages/pandas-0.15.2_182_gbb9c311-py2.7-linux-x86_64.egg/pandas/core/c
ategorical.py:462: FutureWarning: Accessing 'levels' is deprecated, use 'categories'
warn("Accessing 'levels' is deprecated, use 'categories'", FutureWarning)
/usr/local/lib/python2.7/dist-packages/pandas-0.15.2_182_gbb9c311-py2.7-linux-x86_64.egg/pandas/core/categorical.py
:414: FutureWarning: 'labels' is deprecated. Use 'codes' instead
warnings.warn("'labels' is deprecated. Use 'codes' instead", FutureWarning)
...FF.................
======================================================================
FAIL: patsy.test_highlevel.test_formula_likes
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/whelchel/patsy-0.3.0/patsy/test_highlevel.py", line 202, in test_formula_likes
[[1], [2], [3]], ["x"])
File "/home/whelchel/patsy-0.3.0/patsy/test_highlevel.py", line 104, in t
expected_lhs_values, expected_lhs_names)
File "/home/whelchel/patsy-0.3.0/patsy/test_highlevel.py", line 32, in check_result
assert rhs.design_info.column_names == expected_rhs_names
AssertionError
======================================================================
FAIL: patsy.test_highlevel.test_return_pandas
----------------------------------------------------------------------
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/home/whelchel/patsy-0.3.0/patsy/test_highlevel.py", line 348, in test_return_pandas
assert np.array_equal(df4.columns, ["AA"])
AssertionError
----------------------------------------------------------------------
Ran 35 tests in 54.269s
FAILED (failures=2)
I would like to use bivariate splines similarly to the way you have patsy.spline working already, e.g.
z ~ 1 + bispline(x, y, knots_x, knots_y)
Since scipy includes scipy.interpolate.bisplrep
and scipy.interpolate.bisplev
, you have already shown the way, and I can try to follow it and send a pull request if it goes well. Would this be of interest?
Is there an implementation of bivariate splines in R that you would like patsy to follow in notation?
Python code byte-compiled with -OO has doc-strings stripped out.
This creates problems when compiling different packages which changes the doc-strings by doing something like this:
doc += "additional text"
(when the docstring is 'None', this will fail).
mgcv_cubic_splines.py does this and thus creates problems when trying to bytecompile it with the -OO option using cx_freeze/py2exe.
In order to fix this, I would propose to change all the lines which say:
doc += CubicRegressionSpline.common_doc
with
doc =doc +CubicRegressionSpline.common_doc if doc else ''
See also:
http://bugs.python.org/issue23189#
http://stackoverflow.com/questions/22299532/unsupported-operand-types-for-nonetype-and-str-winappdbg-error-after-c
First I want to thank the developer team for their excellent work.
Well, I feel that the "predictors" names when using for instaqnce C( predictor, Treatment(5)) ar too long and somehow confusing. When you make interactions between predictors, you get things like:
C(trimcod, Treatment(4))[T.3]:C(flota, Treatment(11))[T.10]
It would be nice to be able to assign an alias or just forget all the stuff apart from the predictor name to get something like:
[trimcod][T.3]
or just [trimcod 3]
I've playng with the MyTreat example but I can't get any positive results
Thank you very much
Jorge Tornero
Hi -- me again!
In my project, I'm planning to use pandas to do initial manipulation of the data. Patsy works great with pandas DataFrames, but when I build the design matrices, they lose their index. It's definitely something I can work around -- but is there a reasonable way to preserve the index?
This would be useful, for example, when I really want to be able to use a design matrix as both a raw numpy array and a pandas dataframe.
I suppose I could specify return_type="dataframe"
and then get the numpy array from df.values, and it's also not hard to build the dataframe from scratch, but this would be particularly handy for interactive use, where it would provide a useful shortcut (e.g., X.to_dataframe().plot()
or X.to_dataframe().head()
).
To do this right, the new method would be factored out of build_design_matrices. Roughly speaking, it would look like this:
def to_dataframe(self):
if not have_pandas:
raise PatsyError("pandas.DataFrame was requested, but "
"pandas is not installed")
di = self.design_info
df = pandas.DataFrame(self, columns=di.column_names,
index=di.pandas_index)
df.design_info = di
return df
The main design change would be that DesignInfo (or DesignMatrix) would need to gain a pandas_index
attribute, which would keep track of any index from the original data.
If this seems reasonable, I could put together a pull request.
I was just looking for a nice example of stat_* in the tests (useable in a tweet...) and found mostly the *bline ones. stat_density and co should also get test cases.
Ideas for testcases: http://docs.ggplot2.org/0.9.3.1/index.html -> Statistics
geom_smooth also needs tests
I have features with [] in their names like "features[1]" and that confuses the parser. Is there any way to escape it?
In [164]: data = demo_data("a", "b", "x1", "x2", "x[5]")
In [165]: dmatrices("a ~ x[5]", data)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-165-1a6534f0f134> in <module>()
----> 1 dmatrices("a ~ x[5]", data)
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/highlevel.pyc in dmatrices(formula_like, data, eval_env, return_type)
281 """
282 eval_env = EvalEnvironment.capture(eval_env, reference=1)
--> 283 (lhs, rhs) = _do_highlevel_design(formula_like, data, eval_env, return_type)
284 if lhs.shape[1] == 0:
285 raise PatsyError("model is missing required outcome variables")
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/highlevel.pyc in _do_highlevel_design(formula_like, data, eval_env, return_type)
145 def data_iter_maker():
146 return iter([data])
--> 147 builders = _try_incr_builders(formula_like, data_iter_maker, eval_env)
148 if builders is not None:
149 return build_design_matrices(builders, data,
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/highlevel.pyc in _try_incr_builders(formula_like, data_iter_maker, eval_env)
59 return design_matrix_builders([formula_like.lhs_termlist,
60 formula_like.rhs_termlist],
---> 61 data_iter_maker)
62 else:
63 return None
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/build.pyc in design_matrix_builders(termlists, data_iter_maker)
691 cat_postprocessors) = _examine_factor_types(all_factors,
692 factor_states,
--> 693 data_iter_maker)
694 # Now we need the factor evaluators, which encapsulate the knowledge of
695 # how to turn any given factor into a chunk of data:
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/build.pyc in _examine_factor_types(factors, factor_states, data_iter_maker)
441 break
442 for factor in list(examine_needed):
--> 443 value = factor.eval(factor_states[factor], data)
444 if isinstance(value, Categorical):
445 postprocessor = CategoricalTransform(levels=value.levels)
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/eval.pyc in eval(self, memorize_state, data)
471 # http://nedbatchelder.com/blog/200711/rethrowing_exceptions_in_python.html
472 def eval(self, memorize_state, data):
--> 473 return self._eval(memorize_state["eval_code"], memorize_state, data)
474
475 def test_EvalFactor_basics():
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/eval.pyc in _eval(self, code, memorize_state, data)
454 def _eval(self, code, memorize_state, data):
455 inner_namespace = VarLookupDict([data, memorize_state["transforms"]])
--> 456 return self._eval_env.eval(code, inner_namespace=inner_namespace)
457
458 def memorize_chunk(self, state, which_pass, data):
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/eval.pyc in eval(self, expr, source_name, inner_namespace)
119 code = compile(expr, source_name, "eval", self.flags, False)
120 return eval(code, {}, VarLookupDict([inner_namespace]
--> 121 + self._namespaces))
122
123 @classmethod
<string> in <module>()
NameError: name 'x' is not defined
I've seen https://groups.google.com/forum/#!topic/pystatsmodels/96cMRgFXBaA, but why doesn't something like this work.
from patsy import dmatrices
data = dict(y=range(1,11), x1=range(21,31), x2=range(11,21))
dmatrices("y ~ x1 + x2**2", data)
or
dmatrices("y ~ x1 + I(x2**2)", data)
This works
dmatrices("y ~ x1 + np.power(x2, 2)", data)
Not sure what's going on here.
In [44]: patsy.dmatrix("C(a)", {'a':['m', 'n', 'o']})
Out[44]:
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-44-24c8bf15ba3b> in <module>()
----> 1 dmatrix("C(a)", {'a':['m', 'n', 'o']})
/Users/mike/venv/sci/lib/python2.7/site-packages/patsy/highlevel.pyc in dmatrix(formula_like, data, eval_env, return_type)
259 """
260 (lhs, rhs) = _do_highlevel_design(formula_like, data, _get_env(eval_env),
--> 261 return_type)
262 if lhs.shape[1] != 0:
263 raise PatsyError("encountered outcome variables for a model "
/Users/mike/venv/sci/lib/python2.7/site-packages/patsy/highlevel.pyc in _do_highlevel_design(formula_like, data, eval_env, return_type)
145 def data_iter_maker():
146 return iter([data])
--> 147 builders = _try_incr_builders(formula_like, data_iter_maker, eval_env)
148 if builders is not None:
149 return build_design_matrices(builders, data,
/Users/mike/venv/sci/lib/python2.7/site-packages/patsy/highlevel.pyc in _try_incr_builders(formula_like, data_iter_maker, eval_env)
59 return design_matrix_builders([formula_like.lhs_termlist,
60 formula_like.rhs_termlist],
---> 61 data_iter_maker)
62 else:
63 return None
/Users/mike/venv/sci/lib/python2.7/site-packages/patsy/build.pyc in design_matrix_builders(termlists, data_iter_maker)
691 cat_postprocessors) = _examine_factor_types(all_factors,
692 factor_states,
--> 693 data_iter_maker)
694 # Now we need the factor evaluators, which encapsulate the knowledge of
695 # how to turn any given factor into a chunk of data:
/Users/mike/venv/sci/lib/python2.7/site-packages/patsy/build.pyc in _examine_factor_types(factors, factor_states, data_iter_maker)
441 break
442 for factor in list(examine_needed):
--> 443 value = factor.eval(factor_states[factor], data)
444 if isinstance(value, Categorical):
445 postprocessor = CategoricalTransform(levels=value.levels)
/Users/mike/venv/sci/lib/python2.7/site-packages/patsy/eval.pyc in eval(self, memorize_state, data)
429 # http://nedbatchelder.com/blog/200711/rethrowing_exceptions_in_python.html
430 def eval(self, memorize_state, data):
--> 431 return self._eval(memorize_state["eval_code"], memorize_state, data)
432
433 def test_EvalFactor_basics():
/Users/mike/venv/sci/lib/python2.7/site-packages/patsy/eval.pyc in _eval(self, code, memorize_state, data)
412 def _eval(self, code, memorize_state, data):
413 inner_namespace = VarLookupDict([data, memorize_state["transforms"]])
--> 414 return self._eval_env.eval(code, inner_namespace=inner_namespace)
415
416 def memorize_chunk(self, state, which_pass, data):
/Users/mike/venv/sci/lib/python2.7/site-packages/patsy/eval.pyc in eval(self, expr, source_name, inner_namespace)
119 code = compile(expr, source_name, "eval", self.flags, False)
120 return eval(code, {}, VarLookupDict([inner_namespace]
--> 121 + self._namespaces))
122
123 @classmethod
<string> in <module>()
TypeError: 'ClassRegistry' object is not callable
Using virtualenv
on a Mac. Python 2.7.3.
Pygments==1.5
cloud==2.6.9
distribute==0.6.31
ipython==0.13.1
matplotlib==1.2.0
nose==1.2.1
numpy==1.6.2
pandas==0.10.0
patsy==0.1.0
python-dateutil==2.1
pytz==2012h
pyzmq==2.2.0.1
readline==6.2.4.1
scipy==0.11.0
six==1.2.0
statsmodels==0.5.0
sympy==0.7.2
tornado==2.4.1
wsgiref==0.1.2
I mentioned this on the ML, but I thought I'd raise an issue. dmatrices is a bit of a misnomer since there is no such thing as a left-hand side design matrix. This is why I preferred the original make_matrices or make_model_matrices.
Physicists / astronomers (and maybe other potential users like engineers, ...?) often use a different vocabulary than statisticians / economists. E.g. I'm an astronomer and I would say that y(x) = a * x + b
is a linear model with parameters a
and b
and that y(x) = exp(- x / s)
is a non-linear model with parameter s
. Now when I read "This is patsy, a Python library for describing statistical models and building design matrices." I was excited that I might be able to specify arbitrary models (being ignorant of what exactly "statistical models" are and thinking that "design matrices" refers to the subset of linear models patsy can handle) and fit them to data, but if I understand correctly patsy only handles linear models, right?
Maybe you could add a sentences or two to the README or to http://patsy.readthedocs.org/en/latest/overview.html to make it even more clear up-front what kinds of "statistical models" patsy can and can't handle?
(I had the same problem when I first saw statsmodels, it took me a while to figure out that it only can fit very specific models, not arbitrary nonlinear models.)
Two more things you could add to the docs:
Thanks for making patsy and writing great documentation!
I'll try to learn how statisticians do regression from the patsy and statsmodels docs.
Error Message in the RTD-rendered documentation http://patsy.readthedocs.org/en/v0.1.0/library-developers.html [1]:
NameError: name 'LM' is not defined
It looks like the example LM least squares regression model class in example_lm.py [2] _ (that is not for reuse, but may be relevant to #3) is not persisted between invocations of the IPython directive [3] in docs/library-developers.rst. [1]
The IPython directive docs [3] say that "[functions] persist across sessions", but I guess execfile does not persist into locals().
[1] https://github.com/pydata/patsy/blob/master/doc/library-developers.rst
[2] https://github.com/pydata/patsy/blob/master/doc/_examples/example_lm.py
[3] http://ipython.org/ipython-doc/dev/development/ipython_directive.html
In DesignInfo
, we should provide:
ContrastMatrix
we use for each (factor, term) pair.This is pretty simple and obvious stuff, and useful to statsmodels
(in particular, knowing what levels exist so you can generate all pairwise tests etc.).
To provide one way for users who definitely want overparametrized dummy coding to just do
C(myfactor, DummyDammit)
or whatever.
[Request from Josef, talked over at PyCon]
This is observed on patsy 0.1.0.
I saw that the design_info object of a design matrix uses "Intercept" as the encoding for the intercept term so I wondered what would happen if a programmer chose this as the name for a feature.
The ideal scenerio is that patsy either:
a. Does some name mangling
b. throws an error telling me exactly what I did wrong if this is not going to be supported
What happens in reality is that an uniformative assertion message is produced:
Traceback (most recent call last): File "failure.py", line 5, in <module> y,X = patsy.dmatrices("sl ~ Intercept",dataFrame) File "build/bdist.macosx-10.8-intel/egg/patsy/highlevel.py", line 283, in dmatrices File "build/bdist.macosx-10.8-intel/egg/patsy/highlevel.py", line 150, in _do_highlevel_design File "build/bdist.macosx-10.8-intel/egg/patsy/build.py", line 860, in build_design_matrices File "build/bdist.macosx-10.8-intel/egg/patsy/build.py", line 776, in _build File "build/bdist.macosx-10.8-intel/egg/patsy/build.py", line 757, in design_info File "build/bdist.macosx-10.8-intel/egg/patsy/design_info.py", line 78, in __init__ AssertionError
Here is the code that produces the error:
import pandas
import patsy
dataFrame = pandas.io.parsers.read_csv("salary2.txt")
y,X = patsy.dmatrices("sl ~ Intercept",dataFrame)
Is this necessary?
>>> import sys
>>> 'IPython' in sys.modules
False
>>> import patsy
>>> 'IPython' in sys.modules
True
cc @changhiskhan. this was causing downstream issues in pandas testing where some tests were skipped when IPython was imported ( to prevent the test suite from messing up an interactive session with plot windows, etc.)
Maybe I'm doing something wrong, but these example fails:
df = pd.DataFrame( np.arange(10), columns = ['test data'] )
patsy.dmatrix('test data', df )
#SyntaxError: unexpected EOF while parsing
patsy.dmatrix('"test data"', df )
#PatsyError: categorical data must be an iterable container
Hi,
Patsy is awesome, and I'm hoping to use it in a web application. However, the fact that it'll eval() arbitrary expressions makes it a big ol' security hole. For example, you can do this:
dmatrices("y ~ open('bad.txt', 'w')")
and while patsy raises an exception that files don't have a len(), it does indeed create the file. With a bit of creativity, I'm pretty sure there are a variety of paths to arbitrary code execution exploits there; see:
http://me.veekun.com/blog/2012/03/24/on-principle/
for a much more devious example.
What might work better would be to actually have a minimal parser and a set of functions it'll be willing to run on data (one could maybe pass in a mapping of function names to functions) instead -- this:
http://effbot.org/zone/simple-iterator-parser.htm
suggests it should be pretty easy.
Arbitrary python code isn't really the right thing here, IMO -- there's no reason to be defining classes or using comprehensions in a formula.
In any case: eval() in this context probably cannot reasonably be made safe, and will probably wind up biting people who are using patsy as part of anything that accepts untrusted strings.
Right now when creating a formula, we capture the namespace itself.
This can pin large variables in memory, and presents an obstacle to serializiing model designs (#26).
What we should do is to figure out which variables from the enclosing namespace are actually used, and then capture only those.
The klugey way to do this is to observe which variables are accessed when evaluating the formula the first time, and then save only those.
The more principle, reliable, and modular way is to use ast
to parse the formula, and then extract all bare variable references. (Or maybe we should just re-use the token-based implementation of this.) Those which are found not in the data, and not in the builtins, but in the environment, should get stashed to use for actual evaluation.
This isn't just an optimization, it does produce a user-visible effect: if some variable name referenced in the formula is rebound after the formula is created, then previously the new value would be used in future predictions, but after this change, the old value will continue to be used.
This is unaesthetic (I think PHP's so-called "closures" work this way?), but in our case it's actually for the best -- ideally we'd save a read-only snapshot of the environment, period, which is not tractable. But this moves us slightly in that direction, so, okay.
Is it currently possible to build a sparse design matrix from a formula? This is desirable for formulas with lot of categorical variables and their interactions, which are naturally represented as a sparse matrix of zeros and a relatively few ones. R sparse.model.matrix produces an sparse matrix but, when the model includes interactions, it fallbacks to the standard model.matrix so a dense matrix is created as an intermediate step. I don't know if patsy includes any sparse matrix support at all; in case it doesn't consider this a request for enhancement.
I'm a bit baffled by this. Is this a local problem?
|16 $ python setup.py develop
/usr/lib/python2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'install_requires'
warnings.warn(msg)
usage: setup.py [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
or: setup.py --help [cmd1 cmd2 ...]
or: setup.py --help-commands
or: setup.py cmd --help
error: invalid command 'develop'
Setuptools version
>>> import setuptools
>>> print(setuptools.__version__)
3.4.4
On
|20 $ python --version
Python 2.7.5+
Parking this here so I don't forget about it. The following error message is a little obscure. I think it could be improved (from the statsmodels perspective) if object -> observation and if it also gave the factor name that caused the problem. E.g.,
Observation with level 4 in C(X) does not match any of the expected levels.
from patsy import build_design_matrices, dmatrix
from patsy import EvalEnvironment
env = EvalEnvironment.capture()
data = {"X" : [0,1,2,3], "Y" : [1,2,3,4]}
formula = "C(X) + Y"
new_data = {"X" : [0,0,1,2,3,3,4], "Y" : [1,2,3,4,5,6,7]}
info = dmatrix(formula, data)
build_design_matrices([info.design_info.builder], new_data)
Finally echo of API change in Ipython 2.0 reached Debian:
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=751045
so I wondered if I should patch or consider taking current master snapshot or just wait for a new release... ?
Use cases:
DesignMatrixBuilder
(and/or DesignInfo
, same issue)center(...)
with different means across the different analyses.The easy part of this is reviewing the inner structure of DesignMatrixBuilder
(column builders and all that) to make sure it's sensible, and similarly for factor state dicts.
The more complicated part is capturing the evaluation environment in a reasonable way.
Precondition: #25
Some things that would be nice:
build_design_matrices([builder], {"x": 1, "y": [1, 2, 3]})
should probably broadcast x
against y
-- very nice for prediction! (But if it were {"x": [1], "y": [1, 2, 3]}
then that should be an error.)
build_design_matrices([builder], {"x": 1, "a": "a0"})
should not be an error. (Right now, scalar numerics get converted up to columns via atleast2d_column_default
, but scalar categoricals are just an error.)
And also, that last one should perhaps return a 1d ndarray or Series
, not a 2d ndarray or DataFrame
. (And this also applies when the data passed in is a Series
, e.g. a row from a DataFrame
.) The motivation is that this would make
pred_x = build_design_matrices([builder], {"x": 1, "a": "a0"})
dot(pred_x, betas)
give you a scalar when betas.ndim == 1
or a 1d vector in the multivariate case where betas.ndim == 2
. But
pred_x = build_design_matrices([builder], {"x": [1], "a": ["a0"]})
dot(pred_x, betas)
would give you a 1d vector when betas.ndim == 1
or a 2d vector in the multivariate case.
This would definitely simplify patsy's prediction code!
A concern is that just starting to return 1d design "matrices" (or Series
with return_type="dataframe"
, oops) might break existing code.
Have no idea what is testing in
cat = pandas.Categorical([1, 0, -1], ("a", "b"))
conv = categorical_to_int(cat, ("a", "b"), NAAction())
assert np.all(conv == [1, 0, -1])
of test_categorical_to_int(). Need to be corrected
I found it awkward to use the syntax for using stateful transforms, as shown by the tutorial example:
>>> build_design_matrices([mat.design_info.builder], new_data)[0]
Two reasons:
[0]
at the end of an expression because it looks like data might be being thrown away. (I suppose one solution to this is to do explicit assignment like new_mat, = ...
.)So instead, I wrote a helper function:
def updated_design_matrix(design_matrix, data, NA_action='drop'):
"""Shortcut to ``build_design_matrices`` with the builder from
``design_matrix.design_info.builder``
"""
if have_pandas and isinstance(design_matrix, pandas.DataFrame):
return_type = 'dataframe'
else:
return_type = 'matrix'
return build_design_matrices([design_matrix.design_info.builder], data,
NA_action, return_type, design_matrix.dtype)[0]
This lets me write this instead:
>>> updated_design_matrix(mat, new_data)
...which looks much closer to the "high level" syntax.
Does something like this belong in core patsy?
Note: we will need a similarly named (but probably not the same) function to handle the "update" syntax of .
(note #28).
P.S. In case it isn't obvious, it has been a pleasure for me to discover and use patsy over the past few weeks :).
The pandas example gives an ImportError:
http://patsy.readthedocs.org/en/latest/expert-model-specification.html
I think you can install numpy and pandas via a pip requirements file on readthedocs (haven't tried), and then it should work:
http://read-the-docs.readthedocs.org/en/latest/faq.html
The least squares example (the LM class) gives a NameError:
http://patsy.readthedocs.org/en/latest/library-developers.html
Does LM stand for LinearModel? Maybe you can mention that in a class docstring?
Hi Nathaniel, can you please remove this line
install_requires=["numpy"],
from setup.py? It isn't a good idea to have this line here, because then if you use pip or easy_install to install patsy, it will try to upgrade numpy under some circumstances. This is rude and should never happen automatically. In my case, I had 1.8.0-dev on my system (in-place build) and pip tried to ignore it and install 1.7.0 into site-packages.
The same was done for statsmodels. There it was replaced with
try:
from numpy.version import short_version as npversion
except ImportError:
raise ImportError("statsmodels requires numpy")
Just exploring patsy seriously for the first time -- instant love affair! Thanks a lot!
Found one little bug (patsy 0.2.0, pandas 0.12) as described in issue title. Minimal example taken from the first line of the docs and slightly adjusted to reproduce the behaviour. Seems like the missing data should be dropped from some index but is not. Or am I missing something?
In [1]: from patsy import dmatrix, demo_data
In [2]: import pandas as pd
In [3]: import numpy as np
In [4]: data = pd.DataFrame(demo_data("a", "b", "x1", "x2", "y"))
In [5]: data['a'].iloc[0] = np.nan
In [6]: dmatrix('a', data, return_type='dataframe')
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-6-3ab0bd40ae1e> in <module>()
----> 1 dmatrix('a', data, return_type='dataframe')
/home/xxx/anaconda/envs/py33/lib/python3.3/site-packages/patsy/highlevel.py in dmatrix(formula_like, data, eval_env, NA_action, return_type)
276 eval_env = EvalEnvironment.capture(eval_env, reference=1)
277 (lhs, rhs) = _do_highlevel_design(formula_like, data, eval_env,
--> 278 NA_action, return_type)
279 if lhs.shape[1] != 0:
280 raise PatsyError("encountered outcome variables for a model "
/home/xxx/anaconda/envs/py33/lib/python3.3/site-packages/patsy/highlevel.py in _do_highlevel_design(formula_like, data, eval_env, NA_action, return_type)
154 return build_design_matrices(builders, data,
155 NA_action=NA_action,
--> 156 return_type=return_type)
157 else:
158 # No builders, but maybe we can still get matrices
/home/xxx/anaconda/envs/py33/lib/python3.3/site-packages/patsy/build.py in build_design_matrices(builders, data, NA_action, return_type, dtype)
954 matrices[i] = pandas.DataFrame(matrix,
955 columns=di.column_names,
--> 956 index=pandas_index)
957 matrices[i].design_info = di
958 return matrices
/home/xxx/anaconda/envs/py33/lib/python3.3/site-packages/pandas/core/frame.py in __init__(self, data, index, columns, dtype, copy)
413 else:
414 mgr = self._init_ndarray(data, index, columns, dtype=dtype,
--> 415 copy=copy)
416 elif isinstance(data, list):
417 if len(data) > 0:
/home/xxx/anaconda/envs/py33/lib/python3.3/site-packages/pandas/core/frame.py in _init_ndarray(self, values, index, columns, dtype, copy)
559 columns = _ensure_index(columns)
560
--> 561 return create_block_manager_from_blocks([ values.T ], [ columns, index ])
562
563 def _wrap_array(self, arr, axes, copy=False):
/home/xxx/anaconda/envs/py33/lib/python3.3/site-packages/pandas/core/internals.py in create_block_manager_from_blocks(blocks, axes)
2233 blocks = [ getattr(b,'values',b) for b in blocks ]
2234 tot_items = sum(b.shape[0] for b in blocks)
-> 2235 construction_error(tot_items,blocks[0].shape[1:],axes)
2236
2237 def create_block_manager_from_arrays(arrays, names, axes):
/home/xxx/anaconda/envs/py33/lib/python3.3/site-packages/pandas/core/internals.py in construction_error(tot_items, block_shape, axes)
2215 raise ValueError("Shape of passed values is %s, indices imply %s" % (
2216 tuple(map(int, [tot_items] + list(block_shape))),
-> 2217 tuple(map(int, [len(ax) for ax in axes]))))
2218
2219
ValueError: Shape of passed values is (0, 8), indices imply (0, 7)
I am working with a dataframe which has 7000 columns and it turns out that once you go beyond 485 terms, patsy throws a recursion error when going from a formula to a design matrix. Is there a better way of doing this?
Thanks!
In [282]: df = pd.DataFrame(dict(('a' + str(i), np.random.randn(5)) for i in xrange(500)))
In [283]: formula = " + ".join(df.columns)
In [284]: dmatrices(formula, df)
....
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/desc.pyc in eval(self, tree, require_evalexpr)
452 "'%s' operator" % (tree.type,),
453 tree.token)
--> 454 result = self._evaluators[key](self, tree)
455 if require_evalexpr and not isinstance(result, IntermediateExpr):
456 if isinstance(result, ModelDesc):
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/desc.pyc in _eval_binary_plus(evaluator, tree)
283
284 def _eval_binary_plus(evaluator, tree):
--> 285 left_expr = evaluator.eval(tree.args[0])
286 if tree.args[1].type == "ZERO":
287 return IntermediateExpr(False, None, True, left_expr.terms)
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/desc.pyc in eval(self, tree, require_evalexpr)
452 "'%s' operator" % (tree.type,),
453 tree.token)
--> 454 result = self._evaluators[key](self, tree)
455 if require_evalexpr and not isinstance(result, IntermediateExpr):
456 if isinstance(result, ModelDesc):
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/desc.pyc in _eval_binary_plus(evaluator, tree)
283
284 def _eval_binary_plus(evaluator, tree):
--> 285 left_expr = evaluator.eval(tree.args[0])
286 if tree.args[1].type == "ZERO":
287 return IntermediateExpr(False, None, True,
left_expr.terms)
/Users/xxx/lib/python2.7/site-packages/patsy-0.1.0_dev-py2.7.egg/patsy/desc.pyc in eval(self, tree, require_evalexpr)
448 assert isinstance(tree, ParseNode)
449 key = (tree.type, len(tree.args))
--> 450 if key not in self._evaluators:
451 raise PatsyError("I don't know how to evaluate this "
452 "'%s' operator" % (tree.type,),
RuntimeError: maximum recursion depth exceeded in cmp
Dealing with just Python 2 for now, I understand that patsy expects strings. But the data containers might not have this design. So what's the recommended way for handling this? Should we be messing with the data keys under the hood, or should patsy? The only way I can think to handle this (other than statsmodels doing it under the hood) is for patsy to accept unicode but also an encoding so the formula and they data keys can both be encoded correctly. E.g., this fails
import pandas as pd
import patsy
data = pd.DataFrame({
u'àèéòù' : np.random.randn(100),
'x' : np.random.randn(100)})
formula = u"Q('àèéòù') ~ x".encode('utf-8')
dmatrices = patsy.dmatrices(formula, data=data)
But if we also encode the data keys, it's fine. So should dmatrices and whatever other entry points also take an encoding? Am I missing something?
https://groups.google.com/forum/#!topic/pystatsmodels/ZvsyZag3xaw
import patsy
import pandas as pd
cps = pd.read_csv("http://www.mosaic-web.org/go/datasets/cps.csv")
patsy.dmatrix("age + educ + married", data=cps)
I use ptsy almost everyday trough the statsmodels library. There is a little catch that hit me here and there, and it's about pandas dataframe with column name that are written with non ascii characters or that are not valid python names.
In the first case I obtain an UnicodeError saying that it can't convert the special character to a proper ascii value.
take for example this code:
bad_df = pd.DataFrame({u'xè':randn(10), u'x2': randn(10)})
dmatrices(u"xè ~ x2 ", bad_df)
This can be due to some internal conversion to a simple string instead of unicode. This looks trivial, but in Europe we use a lot of special characters and this force me to "reshape" all the dataset beforehand in a very error-prone job. I know this is an issue of python 2.7 and probably won't appear in python 3.x, but I cannot do the switch yet.
The second case is a lot more tricky and without an obvious solution. take this dataframe:
bad_df2 = pd.DataFrame({'x 1':randn(10), 'x 2': randn(10)})
The names "x 1"
and "x 2"
cannot be used as it is, forcing again to make a lot of conversion for alla special kind of characters mangling the names of all columns. Would it be possible to implement a special case to pass the name of the columns as a string (inside the formula string) and let patsy evaluate it as a LookupFactor
? With this kind of solution the model could look like this:
" 'x 1' ~ 'x 2' "
It's not beautiful, but shouldn't create any kind of ambiguity and still let us use a more general set of meaningful names
Thank you very much for your hard work!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.