It's (somewhat encouraged to be) a global limit and as @tomvangoethem has pointed out, this can lead to cross-origin leaks (XSLeaks).

This relates to #31, but that focuses more on "site storage usage".

cc @whatwg/security

(Firefox: https://bugzilla.mozilla.org/show_bug.cgi?id=1552848.)

If there is an exceptional failure inside estimate(), rejecting the promise makes more sense than resolving a promise with useless StorageEstimate. But now we don't reject the promise in estimate() according to the specification.

draft topics to talk about during the WebApps WG

Storage Buckets

Problem

TODO Maybe something along the lines of:
Developers have consistently wanted browser-supported ways of evictable storage, where it doesn't 'wake up' the website and just evicts things that the website explicitly wants evictable. Boxes allows this
Also - encryption?

Proposal

Spec: TODO
Tracking: TODO

Storage Corruption

Problem

All browsers must deal with the situation where bits on disk change resulting in corruption. Currently there is no specification of what should be done in these cases leading to each browser creating ad hoc solutions.

Proposal

To improve developer confidence in the platform we should:

1. Provide a mechanism to report storage corruption in a consistent manner to sites.
2. Attempt to standardize possible browser mitigations when corruption is encountered.

Spec: Explainer
Tracking: TODO

Low Disk Space Signal API

Problem

Some operating systems exhibit stability problems when free disk space becomes dangerously low. Web applications that want to offer a good UX need to be aware when the disk space is running low, so they can avoid causing more disk space consumption. We’d like to focus on the following use cases.

1. An application with offline functionality wants to hide the ”Enable Offline” UI for users who are low on disk space.
2. An application that syncs content for offline use wants to stop syncing new content and show a warning when disk space is running low.

We are aware of large Web properties that currently rely on Chrome’s quota reporting API to detect when free disk space is running low. In the future, we (the Chrome team) will change this API’s behavior, and applications will not be able to use it to detect free space.

Proposal

Spec: TODO
Tracking: TODO

A previously proposed approach involved waking up origins so they can reduce their disk usage, and was abandoned due to performance concerns. We would like to propose an approach that only sends low disk signals to origins with active execution contexts.

Storage Quota Behaviors

Problem

TODO

Proposal

Spec: https://github.com/whatwg/storage
Tracking: N/A

We would like to explore a cross-browser consensus on standardizing more behavior around quota.

We are starting by kicking off MDN documentation here that covers all browsers.

For estimate() we throw, I think it'd be better for these to return false, unless someone can think of a reason why we'd want to expose the difference.

Let's just make this a method that returns a promise for a boolean, that tells you whether the default box is persistent or not.

@inexorabletash @davidsgrogan I'm guessing we never want to give synchronous access to boxes, correct?

We've been discussing what it would take to allow browsers to sync storage across devices.

Note that this is NOT the scenario where a web app wants to manually sync data to its own servers (or via some CORS-enabled cloud storage), but where the browser would silently replicate offline data across devices, much as Chrome replicates bookmarks and credentials. (Under user consent, of course.)

This would be challenging today as we'd probably need to sync an entire origin's storage as one monolithic entity to avoid breaking web apps that expect consistency. We also can't tell if what's stored in a database is ephemeral data (e.g. cache), data that can and perhaps should be replicated by the app already (e.g. pinned music, videos), or data that should "roam" (e.g. saved game state).

One initial thought is that you could make "syncable" an attribute of boxes. Since initial access to a box is async this also means we can defer access to a box while replication is happening. A user agent could also impose a limit on the size of a syncable box.

Wikipedia claims "Public Suffix" is the current term while "extended top-level domain (eTLD)" is the old term, though they are referring to the same thing.

https://www.wikiwand.com/en/Public_Suffix_List

Based on the experience of F2F discussion earlier, Public Suffix is probably the better term to use in the spec.

The terms here seem confusing and/or restrictive, and even the references from estimate() seem incorrect ("origin's global quota" ???). Things like "on the hard drive" seem prescriptive and don't apply to many devices. Can we start this section from scratch, defining only what's needed?

Specifically, the definitions here just need to support what's in the API substeps for estimate(). I'd suggest renaming this section to e.g. rename 5 to "Usage and Quota" and define "usage of an origin" and "quota of an origin, and have the API steps call out to these definitions.

Session storage is excellent for storing data against the tab. This is useful for storing app state that survives a 'discard' on mobile, or even a crash.

However, session storage only supports strings, and its synchronous API has the same issues as localstorage.

It'd be nice if a bucket, or other unit of storage, could be associated with the session.

Re TAG issue 85, it'd be useful for the persistence permission to be reflected in the Permissions API

/cc @mounirlamouri

It can mostly refer to the user interface guidelines, which already take this into account. Perhaps also mention some concerns around quotas and general interaction with the OS.

3. If origin is an opaque origin, then reject promise with a TypeError.
4. Run these substeps in parallel:

Step 4 needs an "Otherwise, ..."

This helps with the following:

1. It clarifies that it doesn't just request permission, but also changes the box.
2. It's much easier to type.

I think we should just do this, but opening an issue for a day or so to see if I missed anything obvious (e.g., this API already shipping somewhere).

Hi all!

WebIDL recently introduced dedicated syntax for mixins.

This syntax replaces the [NoInterfaceObject] extended attribute and implements statement which have been deprecated (except for a few legacy uses cases explicitely mentioned in the spec).

You can read more about it here.

In most cases, the changes should be relatively straightforward. The now deprecated:

interface Foo { };

[NoInterfaceObject]  // (Mostly) DEPRECATED
interface Bar { };
Foo implementes Bar; // DEPRECATED

should just be rewritten as:

interface Foo { };

interface mixin Bar { };
Foo includes Bar;

Additonaly, a section on when to use partials and mixins has been added to the spec. It's short and might be worth a read.

It seems the following interfaces in this spec are impacted by this change:

• NavigatorStorage

We're sorry for the inconvenience this causes, but our hope is that this ultimately makes things simpler and clearer for everybody.

As always, please feel free to reach out if you have any questions.

Thanks!

Tracked in: whatwg/webidl#472 | Original pull-request: whatwg/webidl#433

For the sake of this discussion, an origin group are all origins that share a domain label + public suffix (and have the same scheme, but not necessarily the same port).

If a user agent wants to group cookies with other kinds of storage in the UI, which is quite reasonable as they can be used to revive each other and generally share a bunch of properties, and cookies use origin group and storage and the "persistentstorage" permission uses origins, how is this reconciled?

And if the UI aligns with origin groups, does it make sense for the API to be origin-based?

I'd be interested in hearing from other implementers. I think at Mozilla we tentatively concluded that's it's probably okay to keep as much origin-bound as possible, while not necessarily surfacing that to the UI.

I could imagine this being done at the box level.

I've heard this requested a few times over the years but the presented use cases escape me at the moment. @inexorabletash might remember.

It's not clear to me what's the expectation if pushState() data is covered by Storage Standard. Can we clarify that a bit more? Perhaps privacy reason?

Re: TAG Issue 85, the estimate() and persistentEstimate() methods are incompatible with the sort of full-origin persistence that Chrome is interested in implementing. They are also disconnected from the existing Quota API drafts. I'd like to see them removed from the spec until there's more clarity.

See these issues:

• w3c/permissions#74
• w3c/permissions#33

The simplest thing we could do for a v2 as @lukewagner suggested is introduce the ability for sites to mint new boxes that are either "atomic" or "persistent" (if they have permission).

There's two ways I see we could integrate with IDB and the Cache API. Either they have accessors on boxes or you pass a box when you create new instances of either IDB or Cache.

This is a very trivial issue, and maybe there's a very good reason not to do this, but given that “box” in CSS has a very specific definition that's used in pretty much every CSS spec and is embedded as a term in some of its APIs, one might consider renaming the Storage concept of “box” to something else?

Limiting persist() and persisted() to secure sites makes sense, but given that some storage apis are available on insecure sites would it make sense to enable estimate() for http?

During discussion in Mozilla London All-hands, it seems clear the term "site" has caused confusing during verbal discussion; we had to put site: eTLD+1, e.g. example.co.uk on the whiteboard and say "the site per spec" to avoid confusion (to the normal concept of site as used in daily life and on UI label).

Would it be easier to use a more obscure term like "origin group" so there can be a distinctive definition? The related terms like "site storage" or "site storage unit" would need to be revised as well.

Sandboxed iframes end up with unique opaque origins. We should define the behavior of the API here.

One option would be to simply reject on any API call (persist(), estimate(), etc). Chrome currently does this, and it's per spec for other storage APIs (localStorage, etc). Indexed DB does this too in Chrome (not sure that's specified anywhere)

Another would be to introduce an analogue to [SecureContext] that removes the API in unique opaque origins.

Or we could let unique opaque origins have some ephemeral storage, with lifetime bound to the browsing context or some such. (But please no...)

Where did the "network storage" divide come from? I think of cookies as more like site storage since they are accessible to script. The http cache isn't AFAIK. Further, considering cookies as site storage gets us closer to the ideal of being able to pitch the "first-run" story to developers: if your origin gets purged due to quota pressure or navigator.storage.purge() you don't have to have any special logic to handle it, it will appear that the user has never visited your site before, so your first-run logic will suffice. Obviously, if we don't purge cookies then this story is a lie.

@slightlyoff said it well in his original durable storage page:

what is an application without its Service Workers, Caches, Databases, Cookies, and all the rest? How is a developer meant to reason about having some but not all of these?

Of course we can't guarantee that the user hasn't modified any data locally, e.g. they could have deleted the IDB directory from the browser's profile directory and left the cookies, but at that point the user is on their own.

We've heard from sites that they don't want to prompt the user for this capability and would only request permission if the browser would autogrant it. We are planning on autogranting if the origin has a high "engagement score" for this user. But we want the origin to have to request the permission, we aren't going to opt the origin in to persistence if it doesn't ask for it.

So we'd like a way for an origin to request permission only if the user wouldn't be prompted. One way would be to add an optional parameter to requestPersistent(): requestPersistent{"noPrompt": true}). Another would be to add a fourth permission, "wouldBeAutoGranted" or something, that would let the site know that requestPersistent() would change the default box's mode to "persistent" without prompting. I'm sure there are other strong possibilities too.

/cc @inexorabletash

In particular lower numbers for the quota, and higher numbers for the usage. That way the difference between them can likely be downloaded and stored without hassle and gives everyone some breathing room.

When can UA resolve promise after calling persist() if other tabs already accessed the resource? It's possible that other tabs hold the storage resources. Can the specification suggest how to deal with this situation?

In 2.3 it should really queue a task to change the default box into a persistent box (if it isn't already) and then resolve the promise within that task. Otherwise this method has no effects.

We have been exploring the behavior around quota while under storage pressure: What value should we return for quota when the user has less available disk space than the quota we'd ordinarily return?

The obvious approach is to return a shrunken value which is less than or equal to the available disk space. The issue with allowing script to identify the remaining disk space is that a bad actor could query the value before and after caching an opaque response and determine it's size. (See #31) Since we don't want to do this, the spec should have a recommendation.

One solution is to return the same quota, regardless of storage pressure. The tradeoff is that apps will lose insight into their own remaining space. This can be addressed by providing a storage pressure API that would let apps know whether there is more/less than X MB/GB.

Thoughts?

This has come up a few times. Use cases include just getting back to a known state for testing, after a major revision across storage types, recovering from XSS, etc.

https://storage.spec.whatwg.org/commit-snapshots/725373fce83643c050f08cfbd8bfca8d531e8352/#example-3a7051a8

  } else if(!persistent && permission.status == "prompt") {

I guess this persistent should be persisted ?

In general, Chrome engineers should only be implementing new APIs that have reasonable Explainer documents attached. Re TAG issues #85, the examples in this spec are pretty thin and there's no adjoining Explainer doc to compensate.

/cc @davidsgrogan

The use case is using a ServiceWorker to fetch and store objects (e.g. as a reaction on a web push) to have them ready when the user visits the site. Having the objects persisted would be an advantage.
I understand the issue of a ServiceWorker prompting the user for permission, but can't that be sorted by having the web page asking for persisted permission to persist, and that is also valid for the SW script (having the same origin)?
The SW script could then check using the permission API if it has permission to persist(), and skip calling persist() if not.

We want to phase out document.domain. We should make usage of document.domain disable this feature and vice versa. See w3c/webappsec-secure-contexts#10 for context.

The disk logo for the spec is not in a green circle, like all other WHATWG logos are. This is unacceptable.

I'm willing to PR this if y'all want. ^_^

@jakearchibald @jungkees hey! I was wondering what kind of hooks you need to make it clear e.g., service worker registrations and the Cache API are stored in a box.

In #4 we are discussing the cleanup steps for when a box gets closed, but maybe we should also have formal language for actually storing something inside?

1. Terminology

• "site" definition here is used only used in UI Guidelines but presence here implies normative parts of the spec may use this concept. Can this be defined inline in UI Guidelines instead?

1. Infrastructure

• Call out that cookies are not considered part of site storage?
• If we include a Privacy section it should note that site storage should be cleared along with cookies to prevent user tracking.
• As written, nothing prevents us from having multiple origin storage units associated with the same origin, but we're starting off with 1:1 mapping. Could we clarify the text given this? I confess I think of things in terms of UA→origin→storage→box not UA→storage→(origin, box) which doesn't match how specs are framed.
• Do we want to clarify that phrases like "clear a origin's site storage" imply clearing the box associated with an origin via the origin storage unit? I suppose that should wait until we need that hook.

1. Persistent permission

• It seems like we should say something informative about what this permission means is, even if it is entirely up to the UA and uses "MAY". Maybe just a "see 6.1 Storage Pressure reference"

1. Quota

• See also #22
• See if we can define usage/quota in an inclusive way to cover Edge's current behavior (which has quota per-origin/per-source rather than just per-origin)

1. User Interface Guidelines

• Define "source" somewhere

We really need to require https://w3c.github.io/webappsec/specs/powerfulfeatures/ for new features, including these.

"default" ? "ephemeral" ?

Not web-exposed, so doesn't really matter, but "atomic" makes me think the other option must therefore be "non-atomic". (IIRC, this is because the spec originally proposed a non-atomic "cache" mode, but since that's absent it's potentially confusing)

Not necessarily bugs, but a couple things I noticed.

If |permission| is "prompt" on entry, there is no way the promise can resolve without setting |permission| to either "grant" or "deny". If a user agent should wish to implement a "Not Now" dismissal of the prompt, the only spec compliant implementation would be to not resolve the promise. I think a resolution to false in that case would be more helpful so logic in the page would not be expecting user input that will never arrive.

Is the ASSERT added between 4 and 5 below expected to always hold true? The steps below may allow the promise to resolve to "true" despite the user having just chosen "deny". That seems wrong.

1. Let origin be settingsObject’s origin.

2. Let permission be permission for origin.

3. If permission is "prompt", then ask the user whether turning origin’s site storageunit’s box
   into a persistent box is acceptable. If it is, then set permission to "granted", and "denied" otherwise.

4. Let persisted be true if origin’s site storage unit’s box is a persistent box, and false otherwise.

  if (permission == "denied") ASSERT(!persisted);

5. Queue a task to run these subsubsteps:

    1., Set permission for origin to permission.

    2. If persisted is false and permission is "granted", set persisted to true and set origin’s
       site storage unit’s box’s mode to "persistent".

    3. Resolve promise with persisted.

Shall we consider 'Application Cache' as network storage? It's unclear for me somehow it looks like site storage. Maybe the specification can provide clear explanation for application cache. We should consider it since application cache has impact on site storage.

As per w3c/webappsec-clear-site-data#24 (comment) by @mikewest we should probably put cookies and the HTTP cache into separate buckets.

It's not entirely clear to me how HTTP auth and channel ID are site-scoped though. It seems channel ID at least could be used for multiple "sites" (origin groups).

Size-exposing attacks

In its current form, the Storage specification makes it very likely for user agents to develop an implementation that allows attackers to leak the size of opaque Responses. There are at least 3 methods that could be used to do this:

Estimate usage and quota

The Fetch standard describes a method (estimate()) that returns a "rough estimate of the amount of bytes used" and "a conservative estimate of the amount of bytes available". As the terms "rough estimate" and "conservative estimate" are not strictly defined, and security considerations are not mentioned in the standard, all bets are off as to what user agents might come up with (Chrome's current implementation, which is currently still behind a flag, seems to return the exact usage instead of an estimate).
An attack that leaks the exact resource size is straightforward, even when a "rough estimate" would be given:

1. Get estimate
2. Fetch resource
3. Store Response in cache
4. Get estimate, and subtract value from (1).

In case the estimate is used to obscure the resource size, repeat steps (3) and/or (4). E.g. if estimate() is implemented to round to the nearest kB value, storing the resource 1k times will give you the exact size.

Per-site quota

Each site has their own fixed quota, and when trying to store something that doesn't fit in storage, this will obviously fail. These features can be abused to leak the response size in the following way:

1. Completely fill up site's storage
2. Free up something like 5MB of storage
3. Fetch resource, and store Response in cache
4. Fill up storage byte per byte until this fails
5. Calculate resource size as 5MB - num_fill_bytes

Global quota

Similar to the per-site quota, there's also a global quota, and user agents will free up space by first clearing non-persistent boxes. This provides the same properties as above to obtain the exact resource size. An attack looks as follows:

1. Fill up storage on multiple sites in order to trigger eviction of all other non-persistent boxes not under your control
2. Force a single box you know the size of to be evicted
3. In a new origin, fetch resource and store Response in cache
4. Fill up storage byte per byte until one of your origins gets evicted
5. Calculate resource size as size_first_evicted_box - num_fill_bytes

Compared to the previous attacks, this one is slightly harder to exploit (especially since the global quota can be substantial), but given the high storage speeds (especially with SSD) the attack is still very practical.

_Note:_ For all browsers that already implement one of the above (i.e. virtually every browser), we managed to devise an attack that exposes the exact size of any resource.

Consequences

Being able to determine the resource size of arbitrary Response objects poses various privacy and security issues. For example, we found that by knowing the exact size of just 5 resources on twitter.com (i.e. https://twitter.com/following, https://twitter.com/followers, …) it is possible to uniquely identify a user from a large set. In an experiment on 500k user profiles, we found that a user could be uniquely identified in 97.62% of the cases. Of course, since virtually every website is sending state-specific responses, the consequences are not just limited to this example, and are applicable to a large number of web services.

Mitigation

Having a usable solution that completely eradicates all size-exposing vectors seems unlikely. Instead, I think it's best to have a solution that limits the practicability that of to existing attacks (i.e. timing attacks). As such, I'd like to suggest an approach where "virtual padding" is applied on Response objects. More concretely: upon creation of a Response object, for instance as the result of a fetch() operation, choose a random value r between 0 and rMax. Next, round up Response.size + r to a multiple of D. The "virtual padding" is then the rounded up value minus Response.size. Note that when a Response object is cloned, it should inherit the same padding value.
The padding is virtual in the sense that it is not actually written do disk. Instead, the user agent just uses it as part of its storage bookkeeping, and will also use these values to provide usage/quota estimates.
This method works quite well as it prevents an attacker to quickly obtain many measurements (for each measurement, a fetch() operation is required).
For values of rMax and D, I'd suggest 100kb and 20kb respectively. Even after 50 measurements, these values seem to obfuscate the actual size somewhere in the range of 4.5kb (for 10 measurements, this is approximately 10kb). I made a little script that allows you to play around with the values a bit, but note that most likely a better method can be used to improve the accuracy of attacks, so these values should only be seen as an upper bound.

The introduction starts with

Over the years the web has grown various APIs that can be used for storage, e.g., IndexedDB, localStorage, and showNotification(). The Storage Standard consolidates these APIs by defining:

Under 3. I found this note:

This specification primarily concerns itself with site storage.

Which helps explain why there's nothing about actually storing and retrieving data. But as of now, I can't tell, for example, how I'd actually store something in combination with navigator.storage. That StoreManager interface doesn't expose anything that could be used to actually store something; the Promises made only contain booleans.

The spec is re-introducing its own permission enum instead of re-using the one from the Permissions API. It would be great to re-use the Permissions API's PermissionState instead of PersistentStoragePermission.

Currently the spec requires that a Cache object continue to function even if Caches.delete() has deleted it from CacheStorage. Once the Cache js wrapper is GC'd the backing data can be removed. With Storage.estimate() it is now possible to observe when that GC takes place by polling disk usage.

This was originally noted here in a chromium issue:

https://bugs.chromium.org/p/chromium/issues/detail?id=782869

Since its a polling issue I'm not sure how critical it is. I just wanted to file this so @annevk and others are aware.

whatwg/html#3038 would enable us to ensure that questions are asked only once per origin and do not race with each other.

We've had a frequent request from users of estimate() to provide an per storage type breakdown estimation. For example, something like:

{
   quota: 440922000000,
   usage: 27300000,
   details: {
     indexeddb: 676000,
     cacheapi: 26500000,
     serviceworker: 52800
   }
}

This stems from web apps that have supported for years and/or where multiple apps are present within an origin makes reasoning about what is using up storage difficult. Also exacerbating the problem:

• Storage APIs don't simply store bytes. There's encoding, metadata, compression, padding, and so on.
• Not all APIs allow enumeration (e.g. w3c/IndexedDB#31)

Lots of details to figure out, but:

• Numbers would be estimates.
• No need for everything to add up to the "usage" value. There should be an understanding that there will overhead/metadata in the system.
• Would add hooks to this spec to allow other storage specs to define the key.
  • You could imagine sharing that with a more granular Clear-Site-Data header in the future.
• In Chrome we'd want to surface non-standard/deprecated storage type usage (specifically, file system, websql, appcache), so the spec shouldn't specifically preclude that.

In Chrome we have this plumbed through internally for our DevTools implementation:

Thoughts?