Comments (5)
I believe fn signature would be better like (fn[v]...), then willa could compose it in something like
(fn[[k v]] [k (your-fn v)])
from willa.
I left the key in as an argument because it seems useful to be able to know the key inside the function. In my current use-case the key is also a field in the value, so I don't strictly need it. The reasoning behind including it is that if we do it is easy to ignore, but if we don't include it, there's nothing we can do if we need it.
An alternative could be to have 2 different ones, (map (fn [v] v))
and (map (fn [[k v]] v))
, which would also cover both use-cases.
from willa.
hey @madstap, apologies for taking so long to respond - I only just noticed this issue
I am aware of this issue, but actually I'd like to avoid adding this functionality to Willa if possible. Although it would allow you to avoid unnecessary repartitions, it comes at the cost of making Willa's API larger and requiring the user to have more knowledge of the underlying Kafka Streams library. My aim is to keep Willa's API as simple as possible, so I decided to leave this functionality out. However I'm making the assumption that unnecessary repartitions are a minor annoyance rather than a serious problem, if that's not the case for you please let me know and I'll have a think about how Willa could support this.
For now, if you need to call map-values
there is an (undocumented) "escape hatch" built in to Willa. Instead of calling build-topolgy!
, you can call build-topology-unsafe!
which accepts an extra parameter called overrides
. This is a map of entity id -> builder fn
, where the builder function is responsible for calling methods on the KStreamBuilder
object. It takes in 4 arguments, but I think you'll only need the first 2: entity
(the entity map defined in your topology) and builder
(the KStreamBuilder
object). In your case, you should be able to add an override for the relevant :kstream
entity, which will create the KStream
object and then call map-values
on it.
from willa.
I'll close this issue for now, if you really need this functionality please let me know and I'll reopen the issue
from willa.
@madstap I've added a way to do this in version 0.3.0
. You can now set the willa.overrides/prevent-repartition
option on the kstream
entity to prevent KS creating a repartition.
from willa.
Related Issues (16)
- Undefined var in README.md HOT 1
- Is it possible to name my state store when ::w/entity-type is :ktable? HOT 1
- Custom store name not used for aggregations HOT 2
- Tried to update all dependcies to latest, and lein test failed... HOT 2
- No method in multimethod 'coerce-to-ktable' when building topology HOT 2
- Access state store in xform HOT 2
- Upgrade Willa's Jackdaw Dep to v0.8.0 HOT 1
- Hooking into a Kstream directly from code, rather than from a topic
- Avoid viz dependency in core (HeadlessException in docker) HOT 1
- thoughts on kstream branch? HOT 2
- No method in multimethod 'join-entities*' for dispatch value: [:left :kstream :ktable] HOT 2
- aggregate-subtractor-fn not in willa.experiment HOT 1
- README.md with error HOT 2
- Documentation & error message about graphviz dependency HOT 2
- Typo in readme? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from willa.