Clojure version? about datascript HOT 16 CLOSED

Eventually there will be a Clojure version. Unfortunately it’s not a straightforward port, mostly because performance optimizations are platform-specfic. I’m planning on doing this after all main features of Datomic API will be implemented in DataScript, so I’ll avoid developing 2 implementations at the same time.

from datascript.

Thanks for the quick response. I asked on the Datomic mailinglist [1] and I'm more convinced now that a Clojure DataScript provides new use cases. I'll try with Datomic first, but I'm might go the DataScript route if it turns out to difficult or to hacky.

[1] https://groups.google.com/forum/#!topic/datomic/Cq7ULo0sSBc

from datascript.

+1 for a Clojure version

from datascript.

FYI, I made a start master...jeroenvandijk:feature/clojure-port. I try to get the test suite over to cljx, step by step, until I have both cljsbuild test and test passing via:

lein do clean, cljx once, cljsbuild test, test

Not sure how much time it will take me yet. I've copied the cljx approach from Prismatic Schema. Does this sound ok to you?

from datascript.

Well, test suite can almost transparently being copied to clj—there’s nothing cljs-specific in tests or datascript api at all. Other stuff will need more creative approach: e.g., btset needs a complete rewrite, query partially depends on #js stuff, debug uses a lot of cljs-dependent things. Plus there’re protocol differences in base clojure/clojurescript classes. You can experiment with this to identify possible problems and solutions, but I’m not sure I’ll be able to merge your branch as-is, a lot is changing right now, and merge will probably be a disaster.

from datascript.

That makes sense. I'll regard the first as an experiment. Thanks
Op 28 feb. 2015 09:32 schreef "Nikita Prokopov" [email protected]:

Well, test suite can almost transparently being copied to clj—there’s
nothing cljs-specific in tests or datascript api at all. Other stuff will
need more creative approach: e.g., btset needs a complete rewrite, query
partially depends on #js stuff, debug uses a lot of cljs-dependent things.
Plus there’re protocol differences in base clojure/clojurescript classes.
You can experiment with this to identify possible problems and solutions,
but I’m not sure I’ll be able to merge your branch as-is, a lot is changing
right now, and merge will probably be a disaster.

—
Reply to this email directly or view it on GitHub
#54 (comment).

from datascript.

As github has already added, there is a beginning of a clojure port now: #68

It doesn't merge, and needs more work, but currently seeks feedback.

from datascript.

@benfleis

Wow! That’s a lot of changes :)

Some basic points:

Definetely cljc, not cljx. I admire cljx very much, but everything will only move toward cljc now, no need to do this work twice.

As of 1.6 vs 1.7, I would go with 1.7 because by the time we got this working it’ll probably be widespread.

Definetely performance, not code beauty. E.g. I prefer (.-added datom) to (:added datom) because it translates to direct property access, not a virtual function call. On both CLJ and CLJS. It relates mostly to Datoms, and to datoms inside query.cljs as these are the most hot places. I was even thinking of using positive/negative ints as tx to represent added/retracted instead of separate field (same as Datomic does). So this place is pretty serious :)

I also prefer not to add additional deps until absolutely necessary. I believe we can handle UUID and seqable? without introducing dependency.

I also believe we should feel native for the platform. E.g. you cannot specify FQN functions in queries in CLJS due to lack of ns-resolve (cannot call a fn by name), but in Clojure it’s possible — we should use it.

The idea behind deftype Datom was to make it as small as possible, because there’ll be a lot of them. DB and other stuff can be done as defrecord, so we can avoid reimplementing existing defrecord conveniences.

Is there a reason why you’ve changed (FilteredDB. …) to (->FilteredDB …)? Just curious.

The general idea is not to implement APIs where they make no sense. E.g. assoc for entity, we don’t need it. Etc.

+;; XXX why wouldn’t the RHS of an attr be :v instead of “v”? i’m
+;; assuming it “just works” in CLJS, but in CLJ it fails to access
+;; correctly, since the actual index is :v, not “v”
+;; to deal with this, Datom’s will handle keys as
+;; symbol/keyword/string

This is an low-level cljs implementation detail I made use of. This was the fastest way to store “this is what you need to get out of datom” kind of information. I use it later as (aget datom “e”) for example. Almost as fast as native property access. Have to think what Clojure equivalent will look like.

Also BTSet will probably need separate implementation (maybe even pure-java) as it’s too low-level to be multilingua efficiently.

#+cljs (def Exception js/Error)

Clever! We could probably use ExceptionInfo here though. I believe it is cross-platform.

(reduce #(conj %1 (entity db (.-e %2))) #{} datoms) #+cljs -) ;; XXX what is this trailing dash?

LOL I have no idea :) Probably typo :)

value vs. nuisance

Not sure what do you mean, can you elaborate?

—

All that being said (and repeating again: it’s a huge amount of work! Thanks for putting this together), I think we have to think of some strategy how we can get this merged. It cannot go as one patch: there’s a lot of decisions in there, and there’s no way I can assess them all at once. Also CLJS code is evolving, and I cannot keep track of all changes happening in both master and clj PR.

As I see it, it can be a series of small patches which will smoothly move us towards the end goal. We start introducing portability piece-by-piece, and these changes will co-exist in master. It won’t compile to clj at first, but I want code parts we both agree on to be there anyway: so changes can be made to master and CLJ version at once.

Your current branch can work as a source of ideas and a roadmap. It can be sliced and discussed piece-by-piece as a separate PRs.

Just a heads-up: I don’t recommend touching query right now: it’ll be rewritten soon. Perf is a mess right now: it deserves proper cleaning before any port work can be put into it.

First thing to do I guess is to set up cljc compilation to the project. What do you think?

And we definetely should do it on top of latest master :)

from datascript.

Sorry for late reply. Had a nasty flu since Friday of last week.

Wow! That’s a lot of changes :)

Indeed! More than I hoped for. I tried to minimize superfluous changes (and removed many superfluous to create this version of the diff). My mileage varied...

Okay, there's tons of carrying on, below, so I'll go with the "big steps" thoughts first. Then you can read below so long as your interest allows :)

My idea for discrete steps (with the provision that I haven't yet had a real chance to play with cljc -- that will be tomorrow):
1a. Move all the files that need moving. I hate meaningless diffs. Do you?
1b. Update all existing build stuff to work just like it does now.⁺
2. Start slicing the "inspiration" branch into low hanging fruit which can be more-or-less directly applied, and work them in, on the basis of functional tests.^‡ Low hanging fruit should constitute "minor" changes since fork, with no real impact on performance for the CLJS side.
3. Once the low hanging fruit is done, start playing catch-up with the changes that have come in the last 2 months, and continue to work toward feature parity. Still, all CLJS side tests must run, maintaining good performance characteristics.
4. Once feature parity is achieved, start profiling CLJ side to see what needs tweaking. Tweak.
5. Profit. Or at least smile.

• AFAICT, git stays most sane w/ a commit of just file moves, and a follow up commit with changes. Thus 1a + 1b would be done together, as 2 commits, 1 push. And with some timing awareness and communication to other people working against the tree.
  ‡ (This was my own approach, although the current tests tend to be end-to-end, requiring many other functional things to perform simple tests. Perhaps this could be finagled along the way.)

It feels like these agree with you basic ideas below. If it sounds good, then I'll have a more thorough look @ CLJC, propose a set of file renames / build mods (from head, of course) to make it all work the same, and go from there. I think your biggest first task will be just reviewing that, and understanding whether it has a big impact on other devs -- maybe it's a small enough group that not tooooo many people will be throwing things at my head.

Now, to the carrying on.

Definitely cljc, not cljx.

Easy to agree on, given the 1.7 status. I doubt this will be particularly difficult, given that it's mostly just read restructuring and file renames.

Definitely performance, not code beauty. E.g. ...

Yeah, when I first started, I wondered how much of it was necessary, but as I got deeper, it appeared that you've got some code to support it.

I also prefer not to add additional deps until absolutely necessary. I believe we can handle UUID and seqable? without introducing dependency.

seqable? is trivial to copy; UUID - could use your implementation, I think. I know nothing about that one, but had no such aversion at that moment :)

I also believe we should feel native for the platform.

Agreed. I suspect that these things will also come out quickly as Issues/PRs if people get their hands on a working version.

Is there a reason why you’ve changed (FilteredDB. …) to (->FilteredDB …)? Just curious.

Warranted curiosity -- I had problems with the repl holding a ref to the previous class object, whereas it seems to always get the "right one" when using the -> version. I've experienced this in other projects with heavy use of protocols and the repl, so I did it without looking back. I would love for somebody to tell me "I'm doing it wrong" and that something else was causing my problems, but I've quit and restarted too many repls to fight it.

Also BTSet will probably need separate implementation (maybe even pure-java) as it’s too low-level to be multilingua efficiently.

Yeah, I thought best to start simple, measure performance, then see where we go. This works, in ~20 LOC, so I was pleased afterward :)

#+cljs (def Exception js/Error)
Clever! We could probably use ExceptionInfo here though. I believe it is cross-platform.

Credit due: stolen from prismatic. Probably ExceptionInfo is enough, although I applied Throwable for maximum paranoia in some cases.

value vs. nuisance
Not sure what do you mean, can you elaborate?

Who knows... that was so long ago. I think I was abstractly referring to performance versus code sharing.

All that being said (and repeating again: it’s a huge amount of work! Thanks for putting this together), I think we have to think of some strategy how we can get this merged. It cannot go as one patch: there’s a lot of decisions in there, and there’s no way I can assess them all at once. Also CLJS code is evolving, and I cannot keep track of all changes happening in both master and clj PR.

As I see it, it can be a series of small patches which will smoothly move us towards the end goal. We start introducing portability piece-by-piece, and these changes will co-exist in master. It won’t compile to clj at first, but I want code parts we both agree on to be there anyway: so changes can be made to master and CLJ version at once.

Your current branch can work as a source of ideas and a roadmap. It can be sliced and discussed piece-by-piece as a separate PRs.

This is exactly what I expected. Piece by piece, until a certain unity is found.

from datascript.

@benfleis great! Let’s start with updated deps & cljc renamings. Basically we can rename everything to cljc and it should still work. It won’t compile under clojure, but should work as before under cljs.

(FilteredDB. …) to (->FilteredDB …)

It would be interesting to dig this question. It probably won’t affect anything so I agree on -> form, but we should probably do these renamings in a single separate commit as well.

UUID

I have squuid code working for Clojure, I’ll add it as we have cljc in place.

Let’s start with this and see where to go after that.

I also added you to contributors to tonsky/datascript, please create a branch here (e.g. not in your fork), it’s much simpler to work this way. You can still create pull requests from tonsky/datascript to tonsky/datascript, but I’ll see your branch in my locally checked out git repo.

Thanks! Looking forward to see what will come out of that :)

from datascript.

Any updates on this? I'm personally hoping to use the querying capacity (datalog implementation, basically) against my own implementation of the IDB / ISearch / IIndexAccess protocols under clojure if possible. I'd be willing to do work in that direction if it needs doing?

from datascript.

@seantalts It's in progress; I have a very large diff that is being starting to be incorporated one piece at a time. Check out #78 for the latest bit (from earlier today).

from datascript.

Sweet, thanks.

from datascript.

(FilteredDB. …) to (->FilteredDB …)

I remember having the same problem. I asked the question on
stackoverflow and was told that (FilterDB. …) form is low-level and not
intended for creating records. (Sorry can't find the link to discussion)

from datascript.

All of the tests now pass in Clojure, as well as ClojureScript. There are most certainly still bugs, rough edges, changes still to come, etc., but as of 6d30234 , it's in there. I doubt that it's officially supported yet, but it can be played with. To run the CLJ tests, use lein test-clj; haven't yet tracked down why plain ol' lein test doesn't work, but it doesn't (for me, at least).

from datascript.

DataScript is CLJ/CLJS cross-platform for some time now. Closing this

from datascript.