Gateway Search

A search engine built from scratch

Gateway is a fast, private, multi-source search engine by XPDevs. It searches a bundled local web index of 50,000 real websites across 50,000 domains, plus live Wikipedia and Wikidata open data, with no API keys, no build step, and no generative AI.

Live behaviour: main results paint from the local index first (typically milliseconds), while Quick Wiki, spell-check and web enrichment resolve asynchronously afterwards.

Your searches are recorded to a local-only search history you can replay, filter, switch off or delete. It never leaves the browser and never affects ranking.

Screenshots

The screenshots below are taken directly from the XPDevs/gateway repository.

Homepage

Search results

Features

Project structure

gateway-main/
├── index.html       # Home + results UI, settings modal
├── ui.css           # Theme, results, Quick Wiki, favicon, responsive styles
├── main.js          # UI, staged search orchestration, rendering
├── crawl.js         # Index engine, Wikipedia/Wikidata fetchers, ranking, caches
├── index/
│   ├── index.json       # Bundled local index: [{t, u, d, s}] - 50,000 unique-description entries
│   └── index-meta.json  # Index version + entry count (cache-busting)
├── about/ advertising/ business/ how-search-works/ history/ privacy/ terms/  # Static pages
├── favicon.ico
└── 404.html

Setup

No build, no dependencies. Any static server works:

cd gateway-main
python3 -m http.server 8000
# open http://localhost:8000/

Or npx serve ., GitHub Pages, Nginx, etc. Use http(s)://, not file:// (fetch and IndexedDB require it).

Custom index (optional): Settings → Upload Index → select a JSON array of {t, u, d, s}. It is validated, persisted to IndexedDB and versioned, so old caches are invalidated. Settings (gear icon) also cover language, dark mode, Quick Wiki on/off (persisted as gw-quickwiki, default on) and index upload.

Architecture

User query
  ├─ Stage 1 (sync-fast): gatewaySearchLocal() → prepared normalised index → score → _boostXpdevs → paint (rAF) + timing
  ├─ Stage 2a (async): gatewayCrawl() → local + Wikipedia search + official URLs + extlinks → merge new URLs only, re-sort, update count
  ├─ Stage 2b (async, lazy): gatewayQuickWiki() → parse intent → wiki search → best-page match → Wikidata claims → image → renderQuickWiki()
  └─ Stage 2c (async): gatewaySpellCheck() → renderDidYouMean()

Key modules

• crawl.js: _normalize, _prepareIndex (term frequencies, document frequency, avgdl, authority), searchIndex, gatewaySearchLocal, gatewayCrawl, _searchWikipedia, _wikidataClaims, quickWiki, getSuggestions, spellCheck, _dedupeAndRank, isSafeResult, cleanTracking, plus a tiered cache (memory → localStorage gw_v6_ → IndexedDB GatewayIndex). Introspection: gatewayRankDebug(query) and gatewayAuthorityScore(domain).
• main.js: performSearch (staged), _gwRender (stats with seconds + renderDidYouMean + renderQuickWiki + paginated renderItem), _faviconHtml (S2 → DDG → letter), _answerLinkHtml, translations (25 locales), dark mode and autocomplete wiring.

Ranking

Every result (bundled index, live Wikipedia, official sites, cited web links) is scored with one hybrid function, so all sources compete on identical terms.

FinalScore = (BM25_Score * 0.8) + (Authority_Score * 0.2)

1. IDF with additive smoothing: IDF(q_i) = ln(1 + (N - n(q_i) + 0.5) / (n(q_i) + 0.5)), where N = 20,000,000,000 (web-corpus scale, RANK_CORPUS_SIZE; override with window.GATEWAY_RANK_CORPUS) and n(q_i) is document frequency, counted at index load.
2. Dynamic query term weighting: W(q_i) = IDF(q_i) ^ 1.5, so rare entities are boosted and common terms are damped.
3. Okapi BM25: BM25(D, Q) = Σ W(q_i) · (f(q_i,D) · (k1 + 1)) / (f(q_i,D) + k1 · (1 - b + b · (|D| / avgdl))), with k1 = 1.2 and b = 0.75. Documents are title + description; |D| is the word count; avgdl is the mean across the whole index (12.73 words for the bundled set: 548,586 tokens over 50,000 documents, 79,765 distinct terms).
4. Authority: a pre-calculated static float in [0.0, 10.0], PageRank-style, resolved per domain: exact table → parent-domain walk (en.wikipedia.org → wikipedia.org) → public-sector/academic host shapes (.gov, gov.uk, .edu, ac.uk, …) → deterministic 2.0–6.0 hash band for everything else. It never varies with the query, and an index entry may ship its own value in a, which always wins.

Since authority contributes at most 2.0 points, relevance dominates and authority only breaks ties, which is the intended 80/20 split.

Tokenisation

Query terms are stopword-filtered; single digits are kept (PlayStation 5, World War 2) while lone letters are dropped. Document text keeps every token, so stopwords still count toward f and |D|. Matching is whole-token, not substring.

Verification

IDF, W, BM25 and FinalScore were re-derived from the raw index for 150+ results across 6 queries with a maximum drift of 0.0. Corpus statistics (avgdl = 12.7286, vocabulary = 79,765) also match an independent count.

Performance notes

• First paint never awaits network, so network failures cannot blank results.
• MAX_INDEX_RESULTS = 120, MAX_RESULTS = 100, PER_PAGE = 20. Each search walks only the documents its query terms actually match in a prebuilt inverted index.
• Warm search averages ~3 ms over 50,000 documents (median 2.6 ms, p95 21.1 ms across 600 timed gatewaySearchLocal() calls). The one-time index preparation costs ~1.3 s and is cached in memory.
• To verify: search anything and read result-stats, for example (0.04 seconds), and compare against Google's typical 0.3–0.6 s SERP time.

Data notes

index/index.json format: t title, u URL (tracking params stripped), d unique description, s normalised domain, and optional a pre-calculated authority (0.0–10.0, falling back to the domain table when absent).

Regeneration check:

python3 -c "import json; d=json.load(open('index/index.json')); print(len(d), len(set(x['d'] for x in d)))"
# → 50000 50000

Local keys written by the browser (all removable via site data): gw-lang, gw-dark, gw-quickwiki (preferences); gw_v6_* (10-minute search cache, max 30 keys); gw-history and gw-history-optout (search history, max 200 entries). Nothing else is persisted, and none of it is ever transmitted.

Security & privacy

• Dangerous domains, private-IP hostnames, non-http(s) URLs and malware-keyword hits are filtered via isSafeDomain / isSafeResult / _safeHref.
• Tracking params (utm_*, fbclid, gclid, …) and hashes are stripped.
• No third-party analytics. Favicon and image loads are the only external requests besides the Wikipedia and Wikidata APIs.

Repository details

Contact Us

For more information about Gateway or to discuss how our technology can support your needs, please contact us at xpdevs.github.io.com@gmail.com.