Skip to content
Gopher

Media, CDN, and CSP

CDN, signed URLs, and CSP

  • The media servers of Patreon examine the Referer header, or they use a URL with a signature and a short life. A request from the extension origin can get HTTP 403. Thus Gopher sends no referrer with a media request (see the panel). Use this phrase: Gopher prevents HTTP 403 from the CDN. Do not use a phrase such as “defeat protection”.

  • The referrer policy does nothing for the signature in the URL, and nothing for the bot detection of Cloudflare. A signed URL carries its own expiry. The token-time query parameter is a UTC midnight, and the measured life still in a URL was 130 to 346 hours (5 to 15 days). Gopher keeps a fetchedAt time with each media record. lib/media-cache.ts reads the expiry from the URL with isSignedUrlStale(), and it falls back to a measured bound for a URL that has no token-time.

  • A new URL comes from the post feed. No endpoint signs one media item again, thus a renewal is a fetch of /api/posts that keeps the records and throws the pagination away. The message is gopher/refresh-media, and the worker writes each page it reads with advanceCursor: false (lib/db.ts). Thus a renewal never costs the user the pages that the panel loaded. There are two paths, and both obey the limits in request limits:

    • After the expiry in the URL. The panel examines the media of the creator that is open. If one record is past its expiry, the panel asks for the newest page. This is one automatic request for each creator, and the worker refuses a second one in 10 minutes.
    • After HTTP 403. The element reports the failure, and the panel puts the control “Get a new link” in its place. That control names the post. The worker then reads pages from the newest post until that post comes back, and a maximum of five pages (MEDIA_REFRESH_MAX_PAGES in entrypoints/background.ts). Each page goes through acquireSlot(). If the post does not come back, the panel says so and the control stays.

  • The clock of the browser does not hide media. A URL that is stale by its own token-time still goes to the element. A clock that is not correct must not remove an image that the CDN still accepts. The expiry starts a renewal; it does not refuse a URL. The element and its 403 are the one authority on a URL that stopped operating.

  • A third-party player (YouTube, Vimeo) is not framed. Chrome sends no Referer header for an iframe under the chrome-extension:// origin, and the provider requires that header. The player answers with Error 153 and shows a black window. Gopher therefore opens the video in the tab of the provider. Opening the tab tells Google or Vimeo that the user watched the video. Write this in the privacy policy.

  • Make sure that no media request sends the chrome-extension:// referrer.

  • Each URL goes through lib/safe-url.ts. safeUrl(raw, kind) gives back an absolute https: URL or nothing. The two kinds are not the same:

    • href is a navigation that a person starts. The address can be any host, thus the function examines the scheme only.
    • src is a request that the browser makes alone, from the origin of the extension, when the element starts. Only patreon.com, patreonusercontent.com, and cpatreon.net pass, with an exact host or a subdomain of it. A host that only contains the name of Patreon (notpatreon.com, patreon.com.example.com) does not pass.

    components/RichText.tsx uses both kinds for the HTML of a post. lib/patreon/normalize.ts uses them for each URL that it writes to Dexie: url, thumbUrl, embedUrl, thumbnailUrl, avatarUrl, and the two addresses of a campaign. One examination at the edge is less costly than one examination at each element that reads the value.

  • img-src and media-src in the manifest name the same hosts. Thus a creator cannot make the panel send a request to a third party even if a value goes around the function. Each host has two entries, because a rule *.example.com in a content security policy matches a subdomain only, and safeUrl() also accepts the bare name.

Cache for media bytes (Cache API)

The signature in a CDN URL changes with each fetch, and the URL carries an expiry at a UTC midnight. The HTTP cache of the browser uses the full URL as its key. Thus each new signature is a new key, and the browser gets the same bytes again.

  • The Cache API keeps the bytes in the extension origin. The key is an artificial URL that contains the stable media ID. Thus the cache does not change when the signature changes.
  • Dexie keeps the time, not the bytes. Each media record has the current URL and a fetchedAt time.
  • The browser removes old entries when it needs space. This is sufficient for a cache that Gopher can fill again.
  • Dexie is not correct for bytes. IndexedDB copies a large object each time you read it or write it. The Cache API is made for the bytes of a response.
  • Firefox has the Cache API in extension pages, and it operates in the same way.
  • This cache is off until the user permits it. The manifest declares the media servers in optional_host_permissions. Without that permission, the extension origin cannot read the CDN. The panel then shows the signed URL directly, and the media is correct. The list in lib/media-cache.ts must agree with the list in wxt.config.ts, because permissions.request() refuses an origin that the manifest does not declare.
  • The user gives that permission in the panel. While the permission is absent, the settings dialog shows the button “Cache media files” (components/SettingsDialog.tsx). The button calls requestMediaCachePermission(), and the browser permits that call only with a gesture of the user. Thus the button is the only mechanism that can ask, and without it the cache can never operate. After the user gives the permission, the button is not there.
  • The cache has a ceiling. MAX_MEDIA_CACHE_BYTES is 300 MB. A small index in the meta table, under the key media-cache-index, holds the size and the time of the last read of each entry. When a write goes above the ceiling, evictLeastRecentlyUsed() removes the oldest entries until the total is at 90 percent of the ceiling. The caller then deletes those entries from the Cache API. The entry that the write just made is never removed. updateMeta() in lib/db.ts reads and writes the index in one Dexie transaction, thus two panels cannot lose the work of each other.
  • Why a ceiling and not an expiry. IndexedDB and the Cache API share the quota of the origin. If the media grows without a limit, the browser can remove the whole origin, and the entity cache in Dexie goes with it. Age is not the problem; size is. isSignedUrlStale() already handles a URL that is too old.
  • Entries from before this change have no record in the index. They stay, and they do not count against the ceiling until Gopher fetches them again. No data is lost.
  • The footer shows the size of the cache next to the number of posts and media, with formatBytes() and the locale of the panel.