Let cache() override origin caching directives for selected targets #5

challenge / use case

Some sources forbid caching on every response and give the cache nothing to revalidate against. For example, www.umu.se answers with Cache-Control: no-cache, Pragma: no-cache, Expires: -1 and a NetScaler sticky Set-Cookie, with no ETag or Last-Modified. cache() never stores these responses (Set-Cookie) and would never replay them anyway (no-cache → zero lifetime, no validators), so a crawler downloads every page again on each run, whatever ttl is set.

proposal

Add a force option to cache(), taking glob patterns like skip (renamed to avoid, see notes). For matching targets, the cache acts as though the origin stated no caching directives:

• storable() ignores Set-Cookie and no-store, still rejecting unsuccessful and partial (206) responses
• lifetime() ignores Cache-Control and Expires, falling through to the no-directive case: ttl without Last-Modified, otherwise 10% of the time since the last change, capped at ttl

With the default ttl: 0, a forced response without Last-Modified is never fresh, so the docs should state that, without a positive ttl, a forced response is reused only where it reports Last-Modified and is otherwise only revalidated.

cache({
	ttl: 60*minute,
	avoid: "https://docs.google.com/**",
	force: "https://www.umu.se/**"
})

alternatives

A client-side middleware layered below cache() that removes Set-Cookie, Cache-Control, Pragma and Expires from matching responses, as a workaround in the EC2U loader does. It works, but every consumer has to know the internal ordering rules of the cache and rebuild the response while keeping its URL.

notes

Decisions settled during triage:

• Replayed cookies: a forced entry drops Set-Cookie before it is stored, so a replay never restates a stale session cookie and no session token reaches a backing store; the live response keeps it
• skip renamed to avoid: force and avoid state the two opposite consumer policies overriding the origin on a target's cacheability (cache it whatever the origin forbids, keep it out whatever the origin allows), while skip named a processing step rather than a policy; the behaviour is unchanged (no lookup, no store, invalidation still applied), and the rename is a breaking change on the open 0.x line
• avoid and force overlap: avoid wins, as already implied by the branch order in cache(), so force never overrides a target held back for privacy; overlapping patterns are legitimate (force a host, avoid a subtree) and are not rejected
• Vary: not overridden, as it states which requests a response answers rather than whether or how long it may be reused; a forced Vary: * response is still never stored, and other Vary values select variants as before
• Status codes: forced freshness is still assumed for the Heuristic status codes alone; a forced response under any other status is stored but never fresh
• Default ttl: 0: accepted rather than rejected, as force still enables the Last-Modified heuristic and revalidation against validators hidden behind no-cache or no-store
• Implementation: skipping() serves both avoid and force, so it takes a neutral name and description (e.g. matching())