dat-thang/docs/beaker-networking.md

# Beaker networking notes for `dat-gpt`

These notes use the archived `beakerbrowser/beaker` 1.1.0 source and the
included old `dillo-dat` plugin as references. Beaker was archived in 2022.
Its later Hyperdrive network stack is useful for subsystem boundaries, but it
is not proof of compatibility with the old Dat protocol or a replacement for
the Dillo plugin's `hyperdrive` + `hyperdiscovery` path.

## What Beaker separates

1. **Name resolution.** URLs are parsed first. A 64-character key is used
   directly; a domain requires a resolver. Beaker has a `dat-dns` based
   resolver and local name/key maps. Its v1.1 HyperDNS implementation leaves
   the remote DNS logic disabled, and its `dat://` handler calls a separate
   `dat-dns` resolver. Name resolution therefore belongs above transport and
   should return the archive key before opening feeds.
2. **Drive sessions and persistence.** Beaker's later stack talks to a
   long-lived Hyperspace process through `hyperdrive-daemon-client`. Sessions
   are keyed by archive key plus version and read/write mode. The daemon owns
   the storage and network handles; Beaker expires idle sessions and reconnects
   them if the daemon restarts.
3. **Announce versus lookup.** Beaker explicitly configures a drive with
   `announce: true, lookup: true` when it should be hosted and discover peers.
   Merely opening a drive and resolving its key are separate operations.
4. **Archive reads.** The newer `hyper://` handler resolves the host to a key,
   opens/loads the drive (which starts network lookup), checks out an optional
   version, reads its manifest, resolves the path, and streams the selected
   file from the drive. It supports `GET`/`HEAD`, byte ranges, redirects for
   directory URLs without a trailing slash, manifest MIME overrides, and
   content sniffing. A drive with version zero becomes a clear “no peers found
   yet” response, so lookup-in-progress is distinct from a missing path.
   The separate legacy `dat://` handler resolves through `dat-dns`, runs the
   legacy conversion/download tool into temporary storage, and then serves
   that filesystem through the same path resolver. That conversion route is
   materially different from the native Hyperdrive read path.
5. **Operational visibility.** Beaker has separate debug categories for the
   application, Dat, DHT discovery, DNS discovery, and the Hypercore protocol.
   That separation makes it possible to tell whether a failure is lookup,
   connection, replication, or file resolution.

## What applies to this C implementation

- Preserve the existing `NETWORK_PROTOCOL` switch and `cesspool` spelling.
  The default Cesspool build uses the V7 message format. `hypercore` selects
  the old Dillo dependency's V6 channels, nonce-bearing feed handshake, and
  XSalsa20 stream encryption. The `CSP0` UDP discovery packet format is still
  separate from the old JS `hyperdiscovery` format.
- Keep hostname resolution, discovery, authenticated peer sessions, feed
  replication, and browser path handling as separate layers. The Dillo DPI
  helper already consumes the URL/path and local archive layers; its cache
  lookup is not yet a remote archive loader.
- The archive session joins a key's topic, consumes topic-tagged Cesspool
  announcements, carries peer feed channels, exchanges HAVE/WANT state,
  requests blocks, and verifies/forwards signed proofs. The UDP discovery
  packet format still needs old JS `hyperdiscovery` interoperability work;
  Noise authentication and socket-to-session transport integration are
  separate networking layers.
- Beaker's daemon restart/session handling suggests a lifecycle boundary for
  the C version: one archive session owns the discovery membership, peer set,
  replication state, pending block requests, and close/retry behavior. Keep
  this below both the CLI and Dillo DPI callers so they can share it.
- `dat-dns` names should be resolved as a mapping to a key, not treated as a
  second archive identity. Cache the mapping separately from archive content;
  keep the resolved key visible for debugging and canonical `dat://<key>`
  links.
- `dat_dns_resolve` follows the old `dat-dns` URL-to-key contract. The bundled
  Dillo plugin disables DNS-over-HTTPS and resolves through HTTPS
  `/.well-known/dat`; the C resolver follows that same path after checking its
  local alias map. It does not call the operating system's DNS TXT resolver.
  The record contains a `dat://<64-hex-key>` line and optional `TTL=<seconds>`.
  This lookup returns a key only; archive discovery and transfer remain
  separate Cesspool/session operations.

## Suggested implementation order

1. Define an archive network-session API with explicit `open`, `announce`,
   `lookup`, `peer-up`, `peer-down`, `request-block`, `block-arrived`, and
   `close` transitions. Keep transport details behind the existing `cesspool`
   boundary.
2. Make one peer connection carry a real authenticated protocol handshake and
   identify the metadata and content feeds. Do not start with browser DNS or
   directory listing behavior.
3. Connect remote metadata block requests to feed verification and archive
   reads. A cold `dat://<key>/index.html` should be able to wait for and serve
   the requested bytes, while earlier verified blocks remain cached.
4. Add retry, peer removal, and session cleanup. Expose peer and missing-block
   state through a diagnostic command before adding a dashboard.
5. Add name resolution and any Beaker-style convenience behaviors only after
   key-addressed remote reads work.

## Source references

- Beaker 1.1.0 `app/package.json`: `dat-dns`, `discovery-swarm`, `hyperspace`,
  `hyperdrive-daemon-client`, and `pauls-dat-api2` dependencies.
- `app/bg/hyper/daemon.js`: daemon setup/reconnect, per-drive sessions, peer
  addresses, and session cleanup.
- `app/bg/hyper/drives.js`: key/domain resolution, network configuration,
  drive loading, manifest reads, and version checkouts.
- `app/bg/hyper/dns.js`: local name/key map; remote DNS implementation is
  commented out in this archived revision.
- `app/bg/protocols/dat.js`: legacy `dat://` URL host resolution, local
  legacy download/conversion to temporary storage, path resolution, MIME
  selection, and response handling.
- `app/bg/protocols/hyper.js`: native Hyperdrive network open, historic
  checkout, path resolution, no-peer state, range reads, and browser response
  behavior.
- Beaker's repository README identifies version 1.1.0 as the last release and
  the repository as archived; its debug namespaces include `dat`,
  `bittorrent-dht`, `dns-discovery`, and `hypercore-protocol`.