Plans? about scala-js HOT 25 CLOSED

Btw, I tried running YUI compressor on the scalajs-runtime file but there were many syntax errors due to which it couldn't complete.

from scala-js.

Hello @hrjet! Thanks for your intereset :-)

I know that scalajs-runtime.js is big (huge). There certainly are plans to reduce that size, but some other (functional) features are still missing, and I'm focusing on these first.

I have several ideas to make it smaller, including:

• As you mentioned, take advantage of the modular libraries of Scala 2.11
• Compress using some compressor. It's weird that you got syntax errors with YUI; I will investigate.
• Tweak the encoding scheme to save characters here and there (for starters, $ScalaJSEnvironment could be named $ ^^)
• Use Google Closure or another JS optimizer (not just compressor)
• Translate lambdas to JS lambdas, avoiding the generation of closure classes. This would help a lot in both performance and space, but it is hard to do right.

As for a roadmap, I don't have a precise one yet. I expect to deliver a v0.1 before ScalaDays, though. One big feature that will be included by then is the ability to read TypeScript type definition files.

from scala-js.

Great to hear all this!

Also, nice to hear that you are aware and playing well with TypeScript. I just finished exploring TypeScript this week. One big hole in TypeScript is support for generics (thought that is being addresed, I don't think it will be as rich as Scala's variance model). That is where scala-js could make a difference IMO.

cheers!

from scala-js.

Tweak the encoding scheme to save characters here and there (for starters, $ScalaJSEnvironment could be named $ ^^)

Some update on this one: I managed to reclaim 20 % ( ! ) of the size of the generated code by tweaking the encoding :)

from scala-js.

Btw, I tried running YUI compressor on the scalajs-runtime file but there were many syntax errors due to which it couldn't complete.

@hrjet I don't know about YUI compressor (I couldn't get it to work at all... have to figure that out), but I ran Google's Closure Comiler. I indeed found syntax errors, which I have fixed now.

Using the simple opts of Closure Compiler, I brought down the size of scalajs-runtime.js from 21 MB to 16 MB.

from scala-js.

Great going. Thanks for the updates.

from scala-js.

There's a big issue with the way scala-js encodes methods which subverts the ability of the Closure compiler to eliminate dead code. scala-js defines and accesses all methods via dictionary encoding (i.e. foo["<init>()"] = ... and foo["<init>()"]()) which renders definitions and calls "invisible" to Closure. It can't rename the functions and it can't trace calls to them to determine if they are dead code.

I hacked the scala-js to use a crazy encoding based on UTF8 characters (where, for example, "<init>()" becomes new\u0393\u039e) which substantially improved Closures ability to optimize the results. 23968232 bytes of scalajs-runtime.js and 6442 bytes of helloworld.js becomes 9165914 bytes of helloworld-compiled.js. This is still very far from good, so I want to dig further into why 10MB of Scala code is getting sucked in for something as simple as helloworld.

There are also some issues I haven't resolved with the use of $ScalaJSEnvironment.g and jQuery, but I'm sure that can be fixed.

Before I dug into this further, I wanted to confirm that a) you didn't have other plans for eliminating dead code (it sounds like you don't from the discussion above), and b) you're OK with using a crazy UTF8-based encoding for method names. The method name encoding could be an option, but I highly doubt that anyone is going to consider it acceptable to ship ~20+MB of JavaScript just to be able to use Scala. I think that most people will consider the ability to use Closure to dramatically reduce the size of the final JavaScript code to be an essential requirement.

Anyway, I'll fork the repo and commit my code to a branch on my fork so that you can see exactly what I've done thus far.

from scala-js.

Odd, for some reason when I forked and re-built everything the Closure compiler was able to do substantially better with dead code elimination. Now the helloworld example is more like 40k instead of 9MB. Perhaps I wasn't completely rebuilding all of the old code. In any case, that's well within the realm of reasonable.

from scala-js.

Hi @samskivert! Thanks for your interest in this!

I am well aware that the encoding prevents Closure to do its dead code elimination job. I was actually already considering going for a crazy escaped UTF8 encoding myself ^^ It's great that you actually manage to do something about that!

So, a) no, currently, I don't have better plan. Should this approach be insufficient for some reason, I have a "backup plan" that would involve doing dce myself, using more knowledge.

b) I'm fine with using crazy UTF8-based encoding, as long as it's an option that can be turned on when targeting Closure, and off for easier debugging (or for enabling runtime reflection).

And yes, I know very well that people won't consider 20 MB (or even the 16 MB achieved using only the simple opts of Closure) to be acceptable for a JS runtime. It is my greatest area of interest on the development side, since feature-wise it's next to complete.

Conclusion: by all means, go on!

from scala-js.

Odd, for some reason when I forked and re-built everything the Closure compiler was able to do substantially better with dead code elimination. Now the helloworld example is more like 40k instead of 9MB. Perhaps I wasn't completely rebuilding all of the old code. In any case, that's well within the realm of reasonable.

Oh wow! That is indeed perfectly reasonable! Does the Hello world actually still work?

For the $ScalaJSEnvironment and jQuery thing, you might to want to feed jquery.js to Closure along with the code generated by Scala.js.

from scala-js.

Uh oh, hold the celebrations. Hello World does indeed work, but I think that somehow my scalajs-runtime.js is lacking all of the Scala library. So I'm back to 9MB of reduced size, but hopefully I can still get things closer to 40k once I can get scala-js to generate more Closure-friendly code.

from scala-js.

Ah ah! Well, good luck :-)

from scala-js.

Also, re: sending jQuery.js into Closure, that's certainly possible, but I was aiming to make things work using the Closure extern definitions for jQuery (which allows an existing minified version of jQuery to be used with a Closure-compiled app). I think that will work fine, the issue I'm running into is that helloworld access jQuery via $ScalaJSEnvironment.g.jQuery which I think Closure is renaming in a way that breaks things. Do you know off-hand how that variable is initialized?

from scala-js.

Oh, nevermind. I see that $ScalaJSEnvironment.g is a reference to the global scope and jQuery puts itself there. So clearly when Closure renames that, it will break things. I'll look into accessing jQuery from global as ["jQuery"] which will prevent Closure from renaming it.

from scala-js.

I'll look into accessing jQuery from global as ["jQuery"] which will prevent Closure from renaming it.

There:
https://github.com/lampepfl/scala-js/blob/master/compiler/src/main/scala/scala/tools/nsc/backend/js/GenJSCode.scala#L1855
js.BracketSelect instead of js.Select
:-)

from scala-js.

Great, thanks! I was just poking around in there, but I'm still far from familiar with the (impressively small) codebase.

from scala-js.

OK, I worked around the global issue, though it required more fiddling. genSelectInGlobalScope is used in some cases, but it's also common for env.global to appear as the qualifier to a DynamicSelect. So I just changed DynamicSelect to always use BracketSelect instead of choosing a "normal" select if the identifier is valid. In Hello World, this is clearly the right thing to do, but I'm not familiar enough with the code yet to know if this is a bad idea in general. If so, DynamicSelect.apply could look at the qualifier and only force BracketSelect if the qualifier is env.global.

from scala-js.

Ah, yes. I'm not sure yet whether it would be better to use BracketSelect only if the qualifier is env.global, or always. That will impact selection of any member of any js.Dynamic value. And since static typing is supposed to be equivalent to dynamic typing (wrt the generated code), the same thing must be applied to selections emitted by genPrimitiveJSCall.
I don't know whether it is desirable or not.

from scala-js.

FYI: here's the name mangling changes I made:

samskivert@035eccf

These result in something that Closure can grok, but there is at least one issue that is preventing the Reversi demo from running correctly once run through the Closure compiler. That's the "reflection" going in on RefTypes.js which defines constructors by concatenating strings (which Closure clearly cannot grok):

    var constructorName = "new\u0393"+elemCodeName;

I'm going to write some code to generate the RefTypes instead of magicking them up at runtime. That should allow Closure to see those methods and properly rename them. Then maybe the Reversi demo will work, or maybe there will be other issues like this.

There remains the much bigger issue that scala-js keeps all class definitions in a giant dictionary that is completely invisible to Closure. Once I iron out these other wrinkles, I'm going to start investigating how to define classes in a way that allows Closure to see whether they are used or not and eliminate them if not. Maybe that just means name mangling class names and using property notation to define and access them, instead of dictionary notation. I'll certainly give that a whirl, and if it doesn't work, then I'll actually think about the problem. :)

from scala-js.

Cool!

That's the "reflection" going in on RefTypes.js which defines constructors by concatenating strings (which Closure clearly cannot grok)

Ah yes. Well I suppose I could as well write them by hand, too. It is not that much.

There remains the much bigger issue that scala-js keeps all class definitions in a giant dictionary that is completely invisible to Closure.

Yes, of course. This is trickier to deal with...

Quick thoughts about your encoding:

This omits return types (which will never disambiguate overloaded methods)

Ah ah, you're falling in the same trap as I was two years ago. I thought so, too, but this is wrong. The result type is part of the identity of a method, and it does disambiguate overloaded methods sometimes.

separates packages in non-primitive arguments with $ (instead of .)

That's dangerous, it could conflict with the numerous other $'s emitted by Scala's own name mangling. I suggest using yet another weird code point (or reuse Delta, since it can't be confused in that position).

More generally, I suggest using code points from the Connector punctuation category. They seem very appropriate ^^ And we won't steal away our Greek neighbors' alphabet. WDYT?

Otherwise it looks great! Thank you :-)

from scala-js.

I'm fine with using connectors rather than Greek letters and no dollars. My main concern is keeping the number of different UTF8 characters to a minimum, because I originally started replacing things with UTF8 characters willy nilly and found it very difficult to remember "is this one a \u0394 or a \u039A" and also found it hard to spot the primitive identifiers hiding in something like \u0394O\u3095I\u039A. But as you say, I think we can get by with two special UTF8 characters. Adding the return type is no big deal, since I'm using special characters as separators instead of terminators, I'll just always at the return type last.

from scala-js.

Oh wait, I think I went back to omitting the separators, but anyway, since a method will always have a return type (which may be void), I can still just tack that on last with no chance of ambiguity. There's no need for UTF8 equivalents of ( and ).

from scala-js.

Good.

Yes the identifiers will become cryptic to the human eye. That's why I'd like an option to choose which encoding is used: the human-readable one or the Closure-friendly one.
Now, you could also use _\u0394_ and _\u039A_ (with the _'s) as separators, which makes _\u0394_O_\u3095_I_\u039A_ a little easier to read, without sacrificing unicity nor Closure-friendliness.

from scala-js.

Have you tried without funky utf8 stuff but with gzip?

from scala-js.

Yes, using gzip compression, this gives 1.7 MB if you do not minify, and 1.3 MB if you do minify (simple opts of Closure).

from scala-js.