Cache busting and package versioning about snowpack HOT 31 CLOSED

Back from vacation! @pika/web was built to work with Etag caching support, and doesn't support cache busting across versions.

+1 for adding this via some command-line flag to change the file name to include the version.

OR, better yet, could consumers who care about this add a ?version= param to their imports? We could add this support to the Babel transform that we have as well, so that Babel users would get this cache busting for free.

from snowpack.

@jeremycook Thanks for that perspective and explaining your use-case. I like that idea for a workflow a lot, and I think it lines up strongly with what I need, especially versioning for production. I've got a pretty decent working prototype of my evolution of these brainstormed ideas over at https://github.com/sdegutis/esm-dep-bundler if you'd like to try it out. There are still some rough corners, particularly it has a ton of ugly console.logs, which should be cleaned up in a few hours, but other than that, it works, and I'm pretty excited about it.

from snowpack.

Obviously I'm not the maintainer and naturally it's all up to @FredKSchott, but I'm nervous about the idea of needing to transforming user-code.

My understanding of Pika/web has been that it's meant to be a facilitating tool to promote the idea that, since it's 2019 (almost 2025 in the long scheme of things!), you shouldn't have to transform user-level code, because modern JS can run in pretty much all browsers now, including (and especially?) ES Modules.

But if we abandon that idea (which admittedly might be necessary) of "leave the user's code as-is", how is this different than just using Webpack? Once you open the door to code transformation, what's the reason to avoid all the conveniences like "load json files directly using import syntax", which there can legitimately be arguments for adding into Pika/web, despite these features not being supported natively in browsers, and not even planned to?

So I guess what I'm saying is, if Pika/*'s niche is "you can finally just write modern JavaScript now", then code transformation goes against that, and Pika/* just simply occupies the same role as Rollup does (and Webpack to a lesser extent).

One alternative idea to code-transformation is to hard-code the versions into the import paths. So you'd write:

import React from '/web_modules/react/v16.8.6/index.js'

And then Pika/web would make it exist, one way or another. Between #62 and #68 I think this might be possible via a Rollup configuration.

I think the final "index.js" in that example might also be necessary, because, based on my preliminary testing, I don't think web browsers can infer an extension, and calling it "v16.8.6.js" seems weird.

The obvious downside of this suggestion is that, every time you'd want to change a version, you'd have to find/replace a version in dozens of files. Fortunately most IDEs make it possible to do this in seconds, but it's less than ideal. But it might be better than code-transformation for the above reasons.

from snowpack.

@sdegutis thanks for the input. After spending way to much time trying to formulate a response I'm going to punt on. I'm not sure what the Pika dev(s) think but what I suggest below I believe aligns with how I understand the Pika project's goals.

I think versioning the folders that are output into web_modules is needed (unless HTTP/2 does something amazing to make versioning with folders or cache-busting obsolete). @pika/web must already be rewriting import statements--so having to rewrite import statements of web_modules code to versioned folders doesn't seem like a technical stretch.

Having versioned web_modules, why wouldn't it be possible to opt @pika/web into outputting my code into web_modules via an argument? It would do all the normal things it does to other NPM packages like rewriting imports as needed. If I were to opt into this behavior, it implies that my code references packages in node_modules and not web_modules, and has a valid package.json file. With those pices in place, @pika/web can treat my code just like any other NPM package, CommonJS, ESM or otherwise. (I would have my server side views reference the generated web_modules version of my code instead of what it was sourced from. I would write a little server-side URL helper that looks at my package.json's version number and return a web_modules URL at runtime with the correct path.)

If npx @pika/web can do all that with a couple arguments then I still think @pika/web is holding to its ethos because at the end of my pipe dream are static assets that can be served by any web server to any browser that supports ES modules. I'll have to run npx @pika/web at least once for things to work anyway and will have to do that anytime I update a package, so why not make it possible to output my own code into web_modules? From there I will WebPack or whatever a non-ES distribution from web_modules if I need to support IE, like in enterprise environments.

The asymmetric feel of the NPM packages my code depends on referencing node_modules but my code referencing web_modules bothered me. This may sound odd, but doing local code development the way I described above would feel more balanced to me. It has the added bonus that if I decide to publish my local code as a proper NPM package, the symmetry simplifies that transition and it would already be an ESM-first NPM package.

from snowpack.

@sdegutis I realized that my suggestion to convert "my code" if it is a NPM module project using @pika/web into a web_modules folder would result in bundling. Which really wasn't what I was thinking when I wrote about that. I was thinking my code as a web_module would remain unbundled but just have rewritten import statements that map to the other web_modules it references. However, that is not how @pika/web works out of the box. So if I were to flow with how @pika/web works out of the box my server side views, static HTML pages, or whatever would reference the my code web_modules bundle and not the individual files of my code that were converted to web_modules. I still feel like that would be a better system then the bundle everything in one giant .js file that we have today with WebPack and friends, but not exactly what I had in mind.

I don't want to drown this issue with this other "my code" feature because the primary concern of this issue is versioning NPM libraries that my code references. "My code" versioning would just be a nice bonus, and something that came to mind as I was thinking about how I think I would like to use @pika/web.

from snowpack.

@pika/web must already be rewriting import statements--so having to rewrite import statements of web_modules code to versioned folders doesn't seem like a technical stretch.

Currently @pika/web never rewrites, transforms, compiles or transpiles any of the user's code at all. It leaves your code alone completely. All it does is takes either your package.json's dependencies list, or a webDependencies whitelist if you have one, and bundles those only, putting the results into $dest ("web_modules" by default).

I think versioning the folders that are output into web_modules is needed

We probably will have to version what's in web_modules to bypass the cache. But it can't involve rewriting a user's files, not without changing how @pika/web works right now, and changing its philosophy. And if that happens, how is @pika/web different/better than being a highly specific Webpack configuration? In its current form, it has much more promise than that.

One idea I had is to add the imports into the name, and strip it out when finding the NPM dependency. So if your user code has this:

import ReactDOM from '/web_modules/[email protected]';
import React from '/web_modules/[email protected]';

Then your web_modules will actually look like this:

public/web_modules/
├── [email protected]
├── [email protected]

Thus your imports will work as-is without ever needing to have @pika/web rewrite them.

The above file tree was generated using a prototype I made of this functionality in action. It uses this regular expression to strip out the version in order to find the dependency name:

const r = /@\d+(\.\d+)*/;

console.log('/web_modules/[email protected]'.replace(r, ''));
console.log('/web_modules/[email protected]'  .replace(r, ''));
console.log('/web_modules/[email protected]'    .replace(r, ''));

/*
output:

/web_modules/react.js
/web_modules/react.js
/web_modules/react.js

*/

from snowpack.

The obvious flaws with this plan:

• Your code will be littered with version-specific filenames in all your imports (ugly but feasible).
• You might forget to append a version (easily fixable by adding a version).
• You'll need to keep your import versions in sync with each other (easily fixable with warning logic).
• The version isn't actually tied to the installed version. If it matches, it's just a coincidence.

That last point is the main point that I think we need to address for this to be a clean solution.

My opinion is that @pika/web should go further than just scanning this, and actually install the dependencies for you when it sees one that it doesn't already have installed.

It would then be an all-in-one solution for the front-end, with the following workflow:

1. You run Pika's hypothetical watcher
2. It creates a package.json for you
3. It watches your code (default public/**/*.js)
4. When it sees imports (static or dynamic), it bundles them into web_modules for you.
5. If it isn't already installed and saved via package.json, then it installs them for you.

With this plan, the version number in all your imports, assuming they're all consistent, would then be the canonical version number that your library depends on.

So if you want to upgrade, you'd have to do a find-and-replace of your dependency throughout your whole codebase.

But I think ES Modules might actually have a solution to this too: export them from a "main dependencies" file, and then import them from that file!

So you could have public/deps.js containing this code:

export React from '/web_modules/[email protected]';
// ...

And in any other file in your whole web app, you could do this:

import {React} from '/public/deps.js';

I'm not 100% sure on the semantics of this last idea, but if it's feasible, it actually opens up more possibilities. The one that comes to mind is, @pika/web's watcher no longer has to watch public/**/*.js recursively, but now can just watch public/deps.js by itself.

from snowpack.

I was able to get this syntax working in Chrome but it failed to work in Firefox or Safari:

export * as ReactDOM from '/web_modules/[email protected]';
export * as React from '/web_modules/[email protected]';

Which makes me think it's just Chrome being a little too lenient. But removing * as doesn't work in any of the three browsers.

Checking MDN page on export doesn't really give any other options though.

So I'm not really sure how to get the feature working. But I think if we could, it would be a great simple solution. Your public/deps.js (configurable) would essentially be your dependency list, while also actually importing cache-friendly files when loaded in the browser, without needing to be rewritten.

from snowpack.

Got it working!

export {default as ReactDOM} from '/web_modules/[email protected]';
export {default as React} from '/web_modules/[email protected]';

This gives us some great benefits:

• The watcher only has to watch one file.
• Dependency version numbers can be isolated to a JS single file.
• This JS file would be a good central place to wrap things as needed.
• ... and more!

from snowpack.

This "export" trick would then need a change to my current prototype implementation of #68, so that it scans not only import statements but export statements too. (I found this out because I stopped importing React and ReactDOM directly, and only via public/deps.js, at which point my prototype stopped finding them and bundling them to web_modules.)

from snowpack.

This is sounding good:

export {default as ReactDOM} from '/web_modules/[email protected]';
export {default as React} from '/web_modules/[email protected]';

@sdegutis if you import {React} from '/public/deps.js'; does it also download '/web_modules/[email protected]' even if it isn't imported by my code and it is not used by any library code that executes on the current page?

from snowpack.

@jeremycook according to this half-baked plan of mine described above:

1. Pika/web would see that you are exporting (and thus accessing) "react-dom" and "react" via the public/deps.js file.
2. Pika/web would install that for you via some node-based API to NPM.
3. Pika/web would compile the module and output it to web_modules for you.

This would be true purely because it sees this in public/deps.js, even if no other file in your project ever imports it from this file.

Personally I don't consider this a show-stopper, but an optimization opportunity worth implementing after the fact.

from snowpack.

@sdegutis

@pika/web must already be rewriting import statements--so having to rewrite import statements of web_modules code to versioned folders doesn't seem like a technical stretch.

Currently @pika/web never rewrites, transforms, compiles or transpiles any of the user's code at all. It leaves your code alone completely.

What I meant is that the import statements must be rewritten for those libraries that are output into web_modules folder. For example: if Rollup is converting a Common.js module to ESM the require statements are being rewritten to import statements to make them ES modules and not Common modules. In the cases of ES node_modules modules to ES web_modules modules it still seems like import statements may get rewritten because of the bundling processes that could convert many ES module files into a single shared bundle. Does that make sense, is my assumption correct? I guess my only point is that if this is happening with those NPM packages then it could be done with any NPM package.

from snowpack.

Does that make sense, is my assumption correct?

Ah yes, I misunderstood. Yes, you are correct, that's what Pika/web does.

from snowpack.

Personally I don't consider this a show-stopper, but an optimization opportunity worth implementing after the fact.

Oh I think I see. Would this be an alternative to setting "webDependencies" in package.json?

from snowpack.

Oh yeah, sorry for possibly omitting some context. I've been kind of brainstorming some really big architectural changes in #62, #68 and this issue, and as fate would have it, @FredKSchott just went on vacation for a few weeks, so I'm awaiting feedback.

In the meantime, I do have working prototypes for all the ideas I've come up with. (Except the file-watcher, because that feature is broken in Rollup, see rollup/rollup#2966.)

The basic idea is to let Pika do all the scanning, installing, and bundling your dependencies for you, with nothing but your raw JavaScript code as the sole authority on your dependencies, including the version to install. So you'd just run @pika/web (or maybe another command) and it would just sit there, watching your files, and installing & compiling your dependencies for you as needed. (It might also be good to make it function as a basic static file server since ES Modules don't work in the file:// protocol.)

from snowpack.

Ah I see, having to declare package versions in my code's import statements makes a lot more sense.

One thought I had for managing versions in a central and predictable way might look like this. This way I only need to update version numbers in one place. It seems like it may not work with the scanning system you are suggesting unless your scanner is able to evaluate expressions for dynamic imports.

// web-modules.js
export default {
    "react": "/web_modules/[email protected]",
    "reactDom": "/web_modules/[email protected]"
}

// my-code.js
import {default as webModules} from "/web-modules.js";
var react = await import(webModules.react);
// ...

Here's another version that would perhaps work without your scanner doing any magic. (Not sure if I got the syntax right here.)

// web-modules.js
export default {
    "importReact": () => await import("/web_modules/[email protected]"),
    "importReactDom": () => await import("/web_modules/[email protected]")
}

// my-code.js
import {default as webModules} from "/web-modules.js";
var react = await webModules.importReact();
// ...

from snowpack.

This way I only need to update version numbers in one place.

The plan I outlined above already centralizes package names and version numbers in a single file.

It seems like it may not work with the scanning system you are suggesting unless your scanner is able to evaluate expressions for dynamic imports.

My prototype already can scan dynamic imports, static imports and static exports.

from snowpack.

That's quite impressive. Is it Rollup doing the heavy lifting to make complicated dynamic imports work?

It sounds like all options we have listed will work: your deps.js would work and either variant of my web-modules.js. I guess I proposed mine as a way to not load the universe of dependencies when only one package is needed on a page. It's not clear to me if deps.js will load everything or just what you ask for through an import statement.

from snowpack.

Is it Rollup doing the heavy lifting to make complicated dynamic imports work?

I'm using acorn, acorn-walk and acorn-dynamic-import to find all exports/imports/dynamic-imports which is like 10 lines of code on my part.

I guess I proposed mine as a way to not load the universe of dependencies when only one package is needed on a page. It's not clear to me if deps.js will load everything or just what you ask for through an import statement.

That's a good point I didn't consider. Thinking about it, I think deps.js would load everything up-front.

But deps.js is just a convention, a single file where I export all dependencies, so that this file can use the verbose path that includes the version, and other files can import it from here with the simpler name. The purpose is mainly to "remove" the version from the filename, and export the module contents under a version-less name.

Because it's just a convention, it could be multiple files. You could have public/deps/react.js which contains export {default as React} from '/web_modules/[email protected]'; and in any file you want to load React from, you'd write import {React} from '/public/deps/react.js';. This is more verbose but allows finer-grain control over what's loaded, because you can then have one file per import.

The scanner I wrote in my prototype would still work with this adjustment, because it's still just searching public/**/*.js to scan with acorn et al.

from snowpack.

Using a single file, I believe you'd also be able to use more convenient syntax in the case of packages that have a default export:

// public/deps/react.js
export default from '/web_modules/[email protected]';

// public/main.js
import React from '/public/deps/react.js';

from snowpack.

This issue does bring up an actual limitation that the Pika tooling doesn't really mention: it can't help you with cache-invalidation of your own code, since it never rewrites or touches your code, only dependency code. If you're just serving raw JS files, then you have to manually version them yourself, or use a bundler or something to do it for you. So, thanks to modern caching, it's probably not feasible to use Pika tooling by itself, writing plain old JS and shipping it as-is.

That is, unless HTTP/2 has some nifty cache-invalidating features.

from snowpack.

Yeah it seems like there is one step missing for deploying to production, but Pika (along with the changes you have been talking about) seem like a great development environment and only one build shy of a great deployment.

Also, depending on the website the build step isn't necessarily needed. If it's a personal blog with low traffic for example. I'd Just treat my code the way I always have in those cases.

When uploading files I can imagine usually just adding to web_modules folder, and occasionally cleaning up old versions of web_modules. This is one way to ensure that things "work" if a browser has a cached version of my code that references an older version of a web_module. I guess another way I could do that is keep around (but never actually use) imports to older versions of a library just so Pika adds them to web_modules. This way I could safely refresh my production web_modules and still keep a couple versions back if needed.

// public/obsolete-deps.js
export default from '/web_modules/[email protected]';
export default from '/web_modules/[email protected]';

from snowpack.

I'm not sure what problem that would be solving. Once you push the latest version of your own code, why keep around older dependencies? Assuming the use of a server like nginx that uses last-modified and etags by default (I think), browsers will find the new version of your code as soon as you deploy it, so the old dependencies can be thrown away.

from snowpack.

Perhaps it isn't needed at all and that's overly cautious. It certainly is not ideal. I'm used to browsers sometimes keeping things around and using cached files when I didn't expect them to. Ideally that wouldn't happen if the server is configured correctly. I'm just not sure if that can be guaranteed. Browsers seem to be more aggressive about caching .js files and .css files than .html files or text/html responses, but maybe I'm missing something with the server config. This is only based on anecdotal evidence but too often customer problems are solved by having them hard refresh with CTRL+F5. My sites that have good cache busting/versioning never have those problems. I guess if I really care I'll just figure out a way to build my ES module files and reference them so that they are versioned like the web_modules.

from snowpack.

I feel the same way. I use webpack (via create-react-app) to add hashes to my filenames because I don't really know where I'm ultimately going to deploy my code (nginx, S3, CloudFront, github pages) and how that deployment setup is going to handle caching.

Personally I think the Pika project has the potential to be a great lightweight alternative to webpack and create-react-app. I really like the idea of trying to make web dev as simple as it was almost a decade ago, and I think that dream has merit worth chasing.

That said, I kind of feel like the solutions we came up with so far don't really seem to address all the issues. I think this issue needs more thought, more Hammock Driven Development as Rich Hickey put it.

from snowpack.

Whatever Pika ends up being I think even now it's a good start compared to the alternatives that bundle everything and necessitate HMR to be productive when developing. I've started researching using Riot.js because .riot files very easily convert into individual ES module .js files (npx riot src -o src to build or npx riot -w src -o src to watch and rebuild). I will post my experiment to GitHub if it turns out to be a worthwhile example. VS Code support is OK for Riot (syntax highlighting works with the crisward.riot-tag extension) but ES is great in VS Code. So on balance I accept the tradeoff and would just be encouraged to do complex coding in ES module files and not in .riot files (like I should anyway 🤣).

All of this is quite new to me (ES, Riot, Pika) but developing with minimal ceremony to get started, minimal building in the background while developing, no HMR, and then bundling for production if/when needed sounds great to me. IE11 would also necessitate building for production if it is a concern. Usage is still 4% on caniuse and in many enterprises IE11 is still the default browser on Windows 7 and 10 machines. Who knows what will really happen January 14th, 2020 but I don't think that 4% will change much, they'll just be more IE11 users using Win 10. I don't think IE11 will go away until Microsoft starts hiding Edge inside of IE11, or vice versa and then "removes" IE 11 from new Windows 10 builds and makes it unavailable to download and install. So long as giants like JDE Edwards require IE 11 to be able to use it, IE 11 isn't going anywhere and so long as it is there people will use what they are used to.

from snowpack.

Good point that IE11 is still a concern for a lot of clients. There are probably also a lot like mine who are okay with ditching IE11, and I think tooling like Pika can be considered primarily marketed to them, since Pika is solely meant to support ESM which IE11 doesn't support.

I'm also pretty new to this. Which is why I think brainstorming and hashing things out like this as a community is great and helpful for all of us, not only so we can understand it better but so we can understand it quicker and more correctly, and be more likely to go down productive paths.

That said, I haven't been able to come up with a good solution for the cache-friendly versioning issue yet. For dependencies, yes, for user code without needing to rewrite it, that's tricky. And although this issue you created was only meant to be for the former, I think the latter is relevant here and a real issue.

Ultimately I think I plan to keep TypeScript in my codebase anyway, which at least necessitates one build tool that rewrites user-code. But there is real value to me in getting away from a monolithic "app" framework like create-react-app, and maybe that's where Pika can shine.

More specifically, I want the same benefit Python has with the whole if __name__ == '__main__': runDemo() setup. With my React components, I want to be able to demo them in the browser individually, in isolation from the rest of the codebase.

That's the main practical feature that I think ES Modules are going to help me with, because ES Modules can simply be imported independently of any "parent" modules, and in this case a "demo app" can just require the component it wants to demo, without importing anything else that requires it (e.g. "the whole app").

And that's what's driving my interest in Pika tooling and in getting this to a state where it's as useful as can possibly be.

This is probably a bit off topic for this issue, and I should probably tag @FredKSchott and @backspaces too, but I think it's worth mentioning what my actual use-case is for Pika and how I see it benefiting my client project, which in turn is driving me to contribute back ideas and code.

from snowpack.

@sdegutis Having hung out on your thoughts and my imagination for a few days, here is what I am thinking. For myself I think Pika Web is fine without versioning because versioning is really a production issue, and my thinking is that I would primarily use Pika Web for development and as a early stage in a production build process. When I'm developing, pressing CTRL+F5 to refresh all assets is no biggie. So using Pika Web to develop against ES6 without versioning is fine to me. It matches my current workflow where I primarily develop against modern, standards compliant browsers, and then test and tweak in IE once features stabilize.

What is missing for me is a process for building for testing and production. My two main concerns for those builds is versioning/cache busting and backwards compatibility because IE is still a browser I have to pay attention to.

from snowpack.

@FredKSchott My suggestions and prototypes in this issue and issue #62 evolved into https://github.com/sdegutis/esm-dep-bundler if you want to check it out. It's not meant to be a full-fledged project, only a demo of what I was suggesting Pika/etc could turn into.

from snowpack.

🚚 This feature request has been moved!
Continue the discussion on our project message board:
https://www.pika.dev/packages/@pika/web/discuss/1091 • What is this?

from snowpack.