SIP-01: The Search Index Protocol — One Index, No Owner

An open Nostr standard for decentralized web search indexing. Kind 39697 web index observations, deterministic URL identity, indexer agreement signals, and a full production stack — crawlers, relays, engines. Built by 0xPrivacy.

The Problem with Search Today#

The web's discovery layer is controlled by a handful of corporations who decide:

What gets crawled and when
What gets indexed and what gets buried
What ranking signals matter
What you see when you search

This is not a technical limitation. It is an architectural choice — the choice to centralize. SIP-01 makes the opposite choice.

The SIP-01 Answer#

SIP-01 (Search Index Protocol) is an open Nostr standard for decentralized web search indexing. It defines a single, shared, permissionless index of web documents that anyone can crawl, anyone can publish to, anyone can relay, and anyone can build a search engine on top of.

The core primitive is elegant in its simplicity:

"Indexer pubkey observed this web document at this time, and here is its lightweight metadata."

That's it. One signed statement per observation. Everything else — ranking, filtering, presentation — is left to the search engines that consume the index. The protocol defines what an indexed document looks like on Nostr. It deliberately does not define how to rank it.

The Event#

Every SIP-01 observation is a kind 39697 Nostr event. Here is a real, self-consistent example:

{
  "kind": 39697,
  "content": "{\"title\":\"Example Page\",\"description\":\"A page about examples.\"}",
  "tags": [
    ["d", "widx:3641c5f2274c5471278ab5bf1df6d185"],
    ["u", "https://example.com/page"],
    ["t", "nostr"],
    ["l", "en"],
    ["x", "2a5cbdf44513f552fb571d6c6de2ddf16c5452b235cc887980b52898fb38e7c1"],
    ["v", "1"],
    ["source", "crawlstr/1"],
    ["alt", "Web index observation: Example Page"]
  ]
}

Three identities, kept separate on purpose:

Identity	Field	Rule
URL identity	`d` tag	`widx:` + sha256(normalized_url)[0:32] — identical across ALL indexers
Canonical URL	`u` tag	Normalized per spec (www-strip, tracker-params removed, query sorted)
Content identity	`x` tag	sha256(title + "\n" + description) — the agreement signal

The d tag is the genius. Because every indexer normalizes URLs the same way, every crawler that observes the same page produces the same d tag with a different signing pubkey. Group by d, count distinct authors, and you get: "7 independent indexers saw this page." That is indexer agreement — a trust signal no single crawler can fake.

The x hash is the freshness signal. Same d, same x = indexers agree on content. Same d, different x = the page changed or indexers disagree. Both are useful.

The wire schema is frozen at v: "1". The spec is at v1.2 (document revision), audited line-by-line against the upstream NIPs repository. 45 conformance tests pin the test vectors byte-for-byte. If you implement SIP-01, you are byte-compatible with every other implementation or the tests tell you exactly where you drifted.

The Ecosystem#

SIP-01 is not a paper protocol. It is running in production across a full stack:

Crawlers — Who Finds Pages#

Crawlstr is the lightweight browser-based crawler. Turn your browser into a voluntary crawl node — no backend, no tracking, no account. It respects robots.txt, rate-limits per domain, guards against crawl traps, and publishes observations signed by a per-device anonymous keypair. Crawlstr v2 added adaptive recrawls (changed pages revisit in 24h, unchanged pages back off to 30 days), an IndexedDB outbox for offline operation, and relay-ACKed accounting — it only counts a page as published when a relay confirms it.

Indexstr is the heavyweight counterpart. Where Crawlstr is a scout ("I found something"), Indexstr is a systematic indexer ("I operate indexing capacity"). Sharded crawling across 256 slots, bundled curated collections, node heartbeats (kind 16919). Desktop/VPS-class contribution.

Both speak the same SIP-01 wire format to the same relays. Zero coupling.

Relays — Who Stores and Serves#

UNCAGED-Index-Relay is a high-performance Nostr relay (forked from Ditto Relay) with a SIP-01-native indexing layer on top of OpenSearch. It strictly validates observations at ingestion, indexes them into dedicated structured fields, and exposes web-search operators via NIP-50 extensions: site:, domain:, url:, inurl:, title:, topic:, type:, platform:, network:, country:, mime:, filetype:, source:, lang:, before:, after:, distinct:domain.

It supports NIP-77 Negentropy federation — two index relays can reconcile their SIP-01 indexes efficiently. No relay is the permanent global index. If one disappears, the index lives on elsewhere.

SIP-Booster-Relay is the serverless Cloudflare Workers variant. Crawlstr-SIP-Relay brings index-relay capability to Android.

Engines — Who Ranks and Presents#

Dsearch (dsearch.com) is the flagship community search engine. It consolidates the lineage of 0xSearchstr → UNCAGED → Presearchstr → Dsearch into one coherent project. 18 parallel providers (SIP-01 community index, NIP-50 Nostr search, SearXNG, DuckDuckGo, Brave BYOK, Hacker News, wiki, git, Tor…), structured query parsing (site:, lang:, after:, boolean operators, grouping), query classification for privacy (npub inputs never hit clearnet engines), and keyword staking with your Nostr key.

0xSearchstr was the original aggregator. 0xPresearchstr added keyword stakes. UNCAGED-ENGINE is the minimal forkable template. Savedd rounds out the family. All of them read the same shared index.

sip-01-core is the reusable reference implementation — protocol, federation contracts, provider system, query/rank stack, AI layer. Any search engine can build on it. 45 protocol tests pin the spec vectors byte-for-byte.

SIP-dashboard provides live network telemetry — observations, documents, indexers, crawler heartbeats, relay provenance.

Why This Matters#

No Single Point of Failure#

Google's index is a single point of failure. If Google disappears, the index disappears. SIP-01's index lives on Nostr relays, replicated via NIP-77 federation. If one relay dies, the index converges elsewhere.

No Ranking Monopoly#

Google decides what you see. SIP-01 relays store signals, not rankings. Every search engine ranks independently. The community index results get priority in Dsearch, but any engine can weight signals however it chooses. Ranking is a competitive layer, not a protocol layer.

No Query Surveillance#

Every Google search is logged, profiled, and monetized. SIP-01 observations contain page metadata — never search queries, never user identity. The auto-indexing identity is a per-device pseudonymous keypair, cryptographically separate from your personal Nostr identity. Searching needs no login, no key, no profile.

Verifiable Agreement#

When 7 independent indexers observe the same page with the same content hash, that is a trust signal no single crawler can fabricate. The deterministic d tag makes indexer agreement a query, not a guess.

The Philosophy#

SIP-01 is built with the same principles that drive everything we make:

Open source. The spec is public domain (CC0). The code is MIT.
No tracking. No accounts. No query logging. Per-device pseudonymous keys.
Decentralized. No mandatory identity. No single owner. No mandatory relay.
Modular. The core schema is frozen. Extensions ship through a registry without forking the protocol.
Honest. The spec documents what it does NOT define (ranking, moderation, reputation) as carefully as what it does.

One shared decentralized index. Many independent indexers. Many independent search nodes. Many independent search engines. No mandatory identity. No single owner.

Try It#

Crawl the web: crawlstr.shakespeare.wtf
Search the index: dsearch.com
Read the spec: sip.shakespeare.wtf / github.com/NostrDanish/SIP-01
Run an index relay: UNCAGED-Index-Relay
Build an engine: sip-01-core

Built by 0xPrivacy. Privacy is a human right. Knowledge should remain free. Communities build stronger than corporations.

#nostr #sip01 #decentralized #search #privacy #opensource