Comments (7)
A very wise observation from: rust-lang/rfcs#1317 (comment)
In a strictly on-demand setting (IDE, not a compiler), serialization to disk creates more problems than it solves.
from salsa.
In a strictly on-demand setting (IDE, not a compiler), serialization to disk creates more problems than it solves.
Note that some popular IDEs like Visual Studio actually use a disk database. VS migrated a while ago from a custom format to a SQLite database: https://devblogs.microsoft.com/cppblog/introducing-c-experimental-editor-tools/.
from salsa.
Hi! This would be a desirable feature for me. Is this being worked on?
Not trying to rush you, just trying to evaluate how suitable this library is for my use-case. Thank you. :)
from salsa.
from salsa.
I think serialization should be generally opt-in:
- maybe at
salsa::database
level:
rather coarse, without lazy loading or transparent spilling, useful for "whole session" store/load and short-term-running scenarios - or even per query:
fine-grained, with lazy loading and maybe transparent spilling, useful to reduce RAM usage in long-term-running scenarios
I think I already have a kind of usage scenario ("scenario" as in "salsa is currently not used, but I investigate potential usages") in zs-filecrawler
.
Click to expand
That program first walks through a file list and computes the hash of each file. Then it iterates over the list of hashes, takes the first associated file, and calls a user-defined hook script on that file. It caches the hash list and the progress. It might not really fit the usual `salsa` usage scenario, but the target is similiar: avoid redoing work.QueryGroup 1:
file_content(filepath) <-- hash_data(filepath)
^-[maybe lazy input] --> association [filepath -> hash_of_file_data]
QueryGroup 2:
hash2file(hash) <-- call_hook(hash)
^-[input, from QG1] --> implicit association [hash -> done(hook return value)]
Currently, I just take the "session serialization approach", deserialize at startup, and serialize at shutdown/interrupt, but this may lose some progress. I think that the zs-filecrawler
utility program could benefit from salsa
, but it requires some way to serialize the state (the split into two QueryGroups
would simulate that, but it makes interleaving both parts more difficult, and reduces potential benefits).
from salsa.
A similar, but different feature is to allow to transparently spill rarely used values to disk.
IntelliJ relies on similar feature heavily: when you open a multi-million line project with lots of dependencies, indices become really huge.
Note that this is a significantly different setup from rustc, which operates on a crate at a time, and has a reasonable natural cap on the amount of data it must process simultaneously.
from salsa.
Thank you
from salsa.
Related Issues (20)
- Extend calc example, tutorial with interpreter HOT 1
- book: missing footnote `[^specify]` HOT 2
- tutorial outdated: `parse_statements` is not tagged with `#[salsa::tracked(return_ref)]` anymore HOT 1
- Remove the 'RFCs' section from the book
- Add an API like `new_with_durability` to construct input with specific durability HOT 3
- `debug` method for `DependencyKeyIndex` should leverage `DebugWithDb` HOT 7
- offer a DebugWithDb derive
- Support on-demand inputs
- Limit DebugWithDb impls to avoid generating false dependencies HOT 10
- Visibility of accessors is tied to struct, not field
- tutorial debug chapter: codeblock is empty
- Salsa Book: some chapters exist but are not included in the book
- Panic when reading fields of tracked structs from older revisions HOT 4
- some way to include input fields in DebugWithDb output HOT 4
- Do we want the `debug_all` method? HOT 3
- Singleton inputs: best behavior when a new instance is created? HOT 3
- Support for databases with generic lifetime parameters HOT 3
- Noisy logging, excessive at INFO level HOT 1
- [QUESTION] How can I reuse tracked function results in same revision? HOT 2
- [QUESTION] how to create snapshot from db trait?
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from salsa.