Comments (6)
answering question to #44 (comment) (which is now closed) here
@yarikoptic So what should #58 do about resolving path attributes when decorating methods?
I am thinking about something along the lines of abstracting away "fingerprinters" so we could generalize to not only path
and not only the first arg of the function, but also to attributes etc. Inspection of signature to figure out which arguments we need to deal with is to be only once so it should not have notable performance hit IMHO.
- Define base class
Fingerprinter
which will be subclassed for File and Directory and where to_get_file_fingerprint
and_get_dir_fingerprint
would fold into along with themodified_in_window
to be called at the end of them to return None if "too early" PathFingerprinter
adapter will switch between file and dir fingerprinter depending on what the value (path) is (directory or not)- generalize
memoize_path
intomemoize_params
(or any other name you like) which would also take an option (so more of functions nesting, uff) e.g.params: dict[str, Fingerprinter]
to specify which fingerprinter to use for the args needing special handling. And add ability to specify access to class instance attributes
params={
"path": PathFingerprinter,
"self.path": PathFingerprinter, # here due to `self.` take corresponding attribute of the bound instance
}
edit 1: or make it more explicit by separate two options: args
and params
of the same kind.
4. arguments (not self.
attributes) which went through "transformation" will be ignore
d in the calls to joblib like we did for the path
. Their Fingerprinter should provide desired value as part of the returned fingerprint (e.g. resolved path as you @jwodder did in #45 IIRC)
from fscacher.
- Why would a user want to specify a fingerprinter other than
PathFingerprinter
? If we want to support fingerprinting of multiple path arguments, wouldn't it be simpler to take a list or lists of argument & attribute names? - So if an instance attribute is a fingerprinted path, are you saying that the unresolved path should continue to be part of the function arguments and thus the cache key, meaning that referring to the same file with two different paths won't give you the benefit of caching?
from fscacher.
- I might be over engineering, but who knows - we might use it for http URLs and fingerprint size/mtime/etag this avoiding redownload. Proceed as you see fit - we could generalize later
- no, we will ignore original path as we do now, and fingerprinter will resolve and add resolved path into the invocation signature. So if different paths (relative or symlinks) point to the same location, it should use cached value since resolved path would be the same. Or did I miss the point?
from fscacher.
no, we will ignore original path as we do now
How? The instance needs to be passed to the instance method, and if we tell joblib to ignore "self", then all non-path attributes will be ignored by joblib as well.
from fscacher.
Ah, good point. Let me think about it.
from fscacher.
Aha, I think it is manageable although fragile: we can also add argument to list attributes to be added to the fingerprint (if needed). Sure we wouldn't capture full state but I expect it to be applied cautiously ;-)
from fscacher.
Related Issues (20)
- 0.1 alpha release? HOT 8
- add inode into the file fingerprint
- FOI: joblib can announce if it takes it too long
- conda-forge feedstock HOT 3
- Q: does joblib memoization works across python versions? HOT 2
- support generator functions HOT 1
- fix in joblib broke fscacher?
- Automagic GC HOT 6
- do not resolve path we pass into decorated function? HOT 7
- add a test and verify that fscacher works ok with git-annex'ed files in direct mode
- fscacher is broken under joblib 1.1.0
- flaky test_memoize_path_dir ? HOT 4
- memoized_path_copy helper to complement @memoize_path HOT 5
- multithread traversal of directories
- from joblib: UserWarning: Persisting input arguments took XXX to run. HOT 6
- Next design, directories oriented?/localized, is needed HOT 2
- Inconsistent behavior on Windows from deprecated `st_ctime_ns` stat
- Can get stuck on windows
- tests started to fail - need troubleshooting/fixing HOT 7
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from fscacher.