Links to resource within a Package about webpackage HOT 34 OPEN

from webpackage.

Initially, what if we write the source map's URL, and insist on being online in order to debug?

from webpackage.

@jyasskin I'm not sure I understand. The source maps are generally linked via a URL already ( https://docs.google.com/document/d/1U1RGAehQwRypUTovF1KRlpiOFze0b-_2gc6fAH0KY0k/view#heading=h.lmz475t4mvbx ). Even with an online mandate we don't have a way to refer to contents within different packages so source maps cannot exist within a different package from the generated content.

from webpackage.

@bmeck Say https://example.com/index.package contains https://example.com/index.min.js, which says its sourcemap is https://example.com/index.min.js.map. https://example.com/maps.package happens to contain https://example.com/index.min.js.map, but as you said, there's no way to have index.min.js say its sourcemap is https://example.com/index.min.js.map inside of https://example.com/maps.package. But that's ok as long as you're online: the browser will just fetch https://example.com/index.min.js.map directly. If you want to pre-cache a bunch of sourcemaps, maybe your devtools will have a way to load a package, and then it won't need to re-download the sourcemap, but that doesn't need to be based on fragments.

from webpackage.

@jyasskin ah, I see the disconnect. I am mostly concerned with Node's ability to use webpackage. So we were looking at something more like a app.webpackage and app_debug_info.webpackage they would never refer to a sourcemap outside of a webpackage. Your assertion is that the sourcemap is available is outside a webpackage always, I was under the assumption that there are debug webpackages that are the source of truth and the sourcemaps contained within are not available outside of those webpackages.

from webpackage.

I see. What if app.webpackage included a header:

Link: <app_debug_info.webpackage>; rel=debug; type=application/package

header, and Node interpreted that as a request to load the package when trying to debug. When loading app_debug_info.webpackage, Node would cache index.min.js.map and then use it for requests from app.webpackage. This would also work for offline use in a browser.

from webpackage.

@bmeck but we aren't designing this package format only for Node (or similar). In the context of a web publication, those sourcemaps are a huge security hole since it would allow for reference content across domains - even in signed/trusted content.

While there are some excellent use cases for being able to map relative->absolute URLs, its a major security hole in the general package case.

from webpackage.

@lrosenthol if those urls cannot be accessed (does not exist / lack of perms) how is that a security hole?

I agree that if this allowed cross domain access it would be problematic. I have no desire to allow such, but have a very strong interest in solving packaging debug info. I simply insist it is an important use case. Having a constrained solution is fine to me, but a solution should exist.

from webpackage.

@bmeck How do you restrict the URLs that are allowed? What determines that only "non-existent/non-resolvable" URLs could used?

from webpackage.

@lrosenthol, given that sourcemaps already exist on the web, and can already refer to cross-origin maps, it doesn't seem that we can or need to restrict them further within packages. The UA can process the //# annotation the same as it would if the resource weren't in a package.

from webpackage.

Isn't it where the 'scope' attribute of the Link: header provides necessary mapping? Lets say:

The package contains 2 subpackages, app.webpackage and app_debug_map.webpackage.

The top-level package would contain 2 Link: headers to list the packages:

Link: <https://googleapis.com/packs/jquery.pack>; rel=package; scope=/libs
Link: <https://googleapis.com/packs/jquery_debug_map.pack>; rel=package; scope=/libs/maps ....
Then the URL https://googleapis.com/libs/jquery.min.js will be resolved (scope-matched) to the first webpack, and the map it refers to as https://googleapis.com/libs/maps/jquery.min.js.map will be mapped to the second.

from webpackage.

from webpackage.

@lrosenthol CSS preprocessors and WASM are also affected. I don't know of any HTML/image conventions that use src maps though.

from webpackage.

@lrosenthol Yes, sourcemaps are used for transpilation and little if anything else. When you said "sourcemaps are a huge security hole", what did you mean?

from webpackage.

from webpackage.

@lrosenthol can you clarify the remapping bit. We aren't loading source maps to alter execution, but they can redefine what source text looks like in dev tools.

from webpackage.

from webpackage.

Ah, my presumption is that they would contain paths/urls to something potentially off the machine. Though these urls are contained within the resource bodies generally, not in the headers that would be related to webpackage. This is more a question about what to do when such a url is encountered / how to denote that a resource is within a different file. I have no desire to try and lock down what is allowed content within a .js/.css/.wasm file.

from webpackage.

Ah, my presumption is that they would contain paths/urls to something potentially off the machine. Though these urls are contained within the resource bodies generally, not in the headers that would be related to webpackage. This is more a question about what to do when such a url is encountered / how to denote that a resource is within a different file.

Agreed. To be specific, there are multiple cases here and not sure which you want to support.

The normal case is where an absolute URL is mapped to a relative URL (in the package). For example, you might have a CSS file that refers to an image via absolute URL (eg. background-image: url("http://some.domain.com/image.jpg")). You want to be able to setup a mapping from that URL to something like "/images/image.jp". However, in that case, the resources must be in the unique origin of the package not associated with any other domain!

The second case is relative to relative mapping, where the reference is already to "images/image.jpg", but as part of packaging, it had to be moved to simply "image.jpg", so you need something to map that. Same comment about unique origin here as well.

Finally, we have the really problematic cases, where you are mapping TO an absolute URL (either from a relative or another absolute). You don't want to start having a local reference pull something from a remote source, especially if you then try to map it back into the unique domain.

I have no desire to try and lock down what is allowed content within a .js/.css/.wasm file.

Me either. But we need to do it in a way that is secure.

from webpackage.

Source maps don't alter execution, so even if a malicious sourcemap changes how a string appears while debugging, the original string is still used for all program logic. Sourcemaps can't introduce XSS.

from webpackage.

from webpackage.

They'd be used in exactly the same way when a webpackage is loaded by a UA dynamically.

from webpackage.

from webpackage.

Given a sourcemap, you load and execute resources as if there were no sourcemap. Then, when debugging, you display urls in the devtools as the sourcemap says to remap them. This is the same whether or not there's a package involved.

from webpackage.

from webpackage.

Sourcemaps aren't much use for resources you haven't transpiled, and folks building packages out of just HTML and non-compiled CSS don't need to use sourcemaps.

from webpackage.

from webpackage.

That's a different feature than sourcemaps then, and as you pointed out, it comes with significant XSS risks. If possible, I'd like to avoid that sort of distraction in at least the initial version of the packaging format.

from webpackage.

from webpackage.

Even if we spend the time to design a new mapping system for version 1, sourcemaps already exist and are in wide use, so we'd still have two mapping syntaxes. The only piece that might be redundant is the Link: header I suggested, which @dimich-g pointed out could maybe be subsumed by the scope attribute that we want anyway.

from webpackage.

https://github.com/WICG/webpackage/blob/master/explainers/bundle-urls-and-origins.md proposes a URL scheme to point directly to package/bundle subresources. It's been discussed some at https://www.ietf.org/proceedings/108/minutes/minutes-108-wpack-00 and https://mailarchive.ietf.org/arch/msg/wpack/8fFVJv0AIksODEha8iyJrVDvOek/, and some difficulties with using it as a top-level URL are in w3ctag/design-reviews#509.

from webpackage.

Can someone paint a picture of what is wrong with the original request in the beginning? Why is there a discussion here at all?

If there is /index.min.js in one bundle and /index.min.js.map in another bundle, what is the issue? Imo as a user on the web, posessing these two bundles, the resources of both should be available to me? No?

This question is phrased in terms of linking. That's somewhat nonsensical to me. Bundles are a collection of resources that I the user have. If I have multiple bundles, I would like the content to work across them? There's no "linking" to a bundle needed, is there? My expectation and hope as a user is that packages/bundles are available resources to my offline-capable PWA, for example.

To suppement the development use case outlined at the start, a user-focused use case might be an ebook reader. I go to guten-bundle.example, an example website for hosting Project Gutenberg books, and I download a bundle for Frankenstein and a bundle for Pride and Prejudice. My desire, what I hope would be possible, with a good, user-serving implementation of web packages/web bundles, is to have both /frankenstein.ebook and /pride-and-prejudice.ebook available to my offline service-worker application. This would also permit me to share either of these bundles with others.

The above user needs are heightened the bigger the bundle. If I have a bundle for El Paquet Semanal, this is content that I might want or need to access from a variety of tools & sources. Maybe one day I am in a document editor & want to put a clip art into the document from the EPS bundle. Maybe the next day I am looking for audio samples to mix, and want to pull from the EPS bundle. As a user, my need is a general purpose bundle with assets that I can opt to make use of.

The original post here implies this would not be possible. Why was this issue opened? What impedes /index.min.js.map from being available when requested? How can we empower users to be able to use the resources they have available to them?

from webpackage.

@rektide The original issue exists because the user might have loaded the bundle containing /index.min.js but not yet loaded the bundle containing /index.min.js.map. /index.min.js needs to give the UA instructions for what to do if the user tries to debug it. Just saying "load /index.min.js.map" (as I suggested early in the discussion) isn't really the right thing to do if the user's likely to want a bunch of different .map files: it should instead say which bundle to load to get all of them. But just saying "use /page.wbn" isn't right either: it doesn't say which subresource inside the bundle corresponds to /index.min.js. So two options are to have a global link to the resources that are needed for debugging or to have each link point to a subresource nested inside a bundle.

The debugging case isn't the primary one I care about for the URL scheme, but the issue title matches what that does.

from webpackage.

Rather than invent a new URL scheme, I feel like I'd want the server to throw a Alt-Svc or some kind of designator that the resource is already on it's way via, being sent. That way links can be into plain-old-regular uniform address space, versus needing to be these deep links? I as a rule tend to prefer giving things that can have URLs a URL, but I'm pretty leery of trying to name things inside bundles; that's an expansive realm of complexity.

Thanks for the good overview @jyasskin. I'm also concerned that there may be storage api "shelf" concerns here too that would keep "just load /index.min.js.map" from working? How & when would storage api shelves obstruct this flow? Or is it 100% an issue of the bundle not being loaded yet (which would be fantastic!)?

from webpackage.