A Python library for creating fast, repeatable and self-documenting data analysis pipelines.
Home Page: http://proof.rtfd.org
License: MIT License
Hi
If there will be a feature that let us add argument to the process would be good. for example:
proof.arguments({'file_path': "path/to/file", "lr": .1})So the arguments will be shared between process. It is just a raw idea. Maybe be a better way exists for this.
Thank you for your project 👍
Formerly wireservice/agate#193
Formerly wireservice/agate#189
I like the idea behind proof, but find the step chaining cumbersome in some cases. I know this isn't very constructive feedback and I'm still working out exactly which cases these are. Rather than feature requesting and making the tool more opinionated about workflow, I'm wondering if the caching logic/API could be factored out into a separate package, allowing the persisting of data and passing it between steps to work using a different task runner, such as invoke.
Factoring this out into a separate package could also parameterize this operation, one day allowing power users to address #11 by serializing to a different format assuming their data is easily serialized as JSON (or something else).
File "/Users/tylerfisher/.virtualenvs/amazon-data/lib/python2.7/site-packages/proof/analysis.py", line 133, in then
_trace=self._trace
File "/Users/tylerfisher/.virtualenvs/amazon-data/lib/python2.7/site-packages/proof/analysis.py", line 92, in __init__
self._cache_path = os.path.join(self._cache_dir, '%s.cache' % self._fingerprint())
File "/Users/tylerfisher/.virtualenvs/amazon-data/lib/python2.7/site-packages/proof/analysis.py", line 108, in _fingerprint
hasher.update(source.encode('utf-8'))
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 106: ordinal not in range(128)
In broca there's support for branching pipelines so you can combine pipelines which have many redundant components - I was thinking of breaking that out into a separate package but it seems like it may be appropriate for proof - what do you think?
This would make it easier to use Proof to load and clean your data, then to switch to exploratory analysis with another tool, such as a Jupyter notebook.
Can I switch to something like JSON and keep it generic? Is there a method I can implement on my agate objects to ensure they serialize correctly?
http://www.benfrederickson.com/dont-pickle-your-data/
https://kovshenin.com/2010/pickle-vs-json-which-is-faster/
http://matthewrocklin.com/blog/work/2015/03/16/Fast-Serialization/
Any reason the library is a package and not a single Python module?
Might make sense as a way of allowing "print blocks"
There are often cases where multiple reporters, with different skillsets, will be working on analysis in parallel. Often the common language is a SQL database. It would be cool to be able to replace the disk-based Pickle cache with a cache that is storing the data to SQL tables. This is outside the scope of proof, but parameterizing the cache would let people write their own cache implementations.
Hello,
Would be great if there was a built-in way to define that a process will be the last one executed. Something like:
analysis = proof.Analysis(initialize_data, last_process=last_process)
analysis.then(process_0)
analysis.then(process_1)
analysis.then(process_2)
analysis.then(last_process)analysis = proof.Analysis(initialize_data, last_process=last_process)
analysis.then(process_0)
analysis.then(process_1)
analysis.then(process_2)
analysis.then(last_process)
analysis.then(process_3)
# Exception(Other process was added after the one specified as the last one.)analysis = proof.Analysis(initialize_data, last_process=last_process)
analysis.then(process_0)
analysis.then(process_1)
analysis.then(process_2)
# Exception(It's missing the last one!)In the last example, instead an exception, the call of the method then (with the last process) could be implicit, but you know... "Explicit is better than implicit."
What do you think ?
I think this is a proof issue?
Sample code:
import agate
import proof
def load_data(data):
data['responses'] = agate.Table.from_csv('responses.csv')
def unweighted_totals(data):
data['first'] = data['responses'].group_by('first')
if __name__ == '__main__':
data_loaded = proof.Analysis(load_data)
data_loaded.then(unweighted_totals)
Result:
File "tally.py", line 31, in <module>
data_loaded.run()
File "/Users/tylerfisher/.virtualenvs/listeners2015/lib/python2.7/site-packages/proof/analysis.py", line 204, in run
analysis.run(refresh=refresh, _parent_cache=self._cache)
File "/Users/tylerfisher/.virtualenvs/listeners2015/lib/python2.7/site-packages/proof/analysis.py", line 192, in run
local_data = _parent_cache.get()
File "/Users/tylerfisher/.virtualenvs/listeners2015/lib/python2.7/site-packages/proof/analysis.py", line 50, in get
return deepcopy(self._data)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 237, in _deepcopy_tuple
y.append(deepcopy(a, memo))
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 298, in _deepcopy_inst
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 190, in deepcopy
y = _reconstruct(x, rv, 1, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 334, in _reconstruct
state = deepcopy(state, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 163, in deepcopy
y = copier(x, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 257, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/copy.py", line 174, in deepcopy
y = copier(memo)
TypeError: cannot deepcopy this pattern object
If I remove proof from the equation:
import agate
data = {}
def load_data():
data['responses'] = agate.Table.from_csv('responses.csv')
def unweighted_totals():
data['first'] = data['responses'].group_by('first')
if __name__ == '__main__':
load_data()
unweighted_totals()
It all works, and I can do things with my grouped data.
Formerly wireservice/agate#184
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.