⚡

Dataset RAG Gateway

RAG over the entire HuggingFace dataset collection — ask a question, the gateway discovers the most relevant datasets via a relevance hierarchy, probes their real contents, and returns a cited answer. 100% client-side · no keys · free.

Your question

budget = # of candidate datasets whose rows get probed

Top-k chunks

Answer model (optional)

HF token (gated models, optional)

Discover Probe content Retrieve Synthesize

Answer

Relevance hierarchy — why these datasets?

Retrieved chunks

Dataset id

Question

Top-k

Answer model (optional)

HF token (optional)

Search all datasets

Limit

ranked by downloads — from here you can send any dataset straight into the one-dataset RAG tab.

🧩 JSON API — callable from anywhere (any model/agent)

Hash routes render raw JSON that any LLM can parse. No auth. CORS-friendly.

Endpoint	What it returns
`#/api/rag?query=…&top_k=5&budget=4&model=…`	The full pipeline: auto-discovered datasets (relevance hierarchy), probed rows, ranked chunks, and the synthesized answer.
`#/api/rag_manual?dataset=…&query=…&top_k=5`	RAG inside a single dataset.
`#/api/datasets?q=…&limit=10`	Hub discovery, ranked by downloads.
`#/api/info?dataset=…`	Configs & splits of any dataset.
`#/api/rows?dataset=…&split=…&length=…`	Raw rows of any dataset.
`#/api/health`	Service info + supported endpoints.

💡 Tip for agents: the response is plain, parseable JSON inside a <pre> block. Point any LLM at …hf.space/#/api/rag?query=… and it can read the answer + citations directly.

How “RAG over the whole collection” works

This is a two-stage retrieval hierarchy — it never embeds the whole hub (petabytes, impossible client-side), it instead retrieves the right datasets first, then retrieves inside them:

Stage	What happens
1 · Term extraction	The question is broken into weighted keywords (stopwords removed).
2 · Hub discovery	Those terms hit the HF Hub search API — candidates ranked by downloads/likes.
3 · Relevance hierarchy	Each candidate dataset is tiered: tier 1 id/tag match → tier 2 description match → tier 3 broad/popular — plus a meta-score.
4 · Content probe	The top candidates’ real rows are fetched and lexically scored against the question (TF-style).
5 · Prune & re-rank	Datasets with no relevant rows are dropped; chunks from all survivors are merged and ranked.
6 · Synthesize	Extractive answer by default (no key needed) or model-generated (`&model=…`) with row citations.

⚡ Everything runs in your browser — the Space is pure static (HTML/JS). The only network calls go to HF’s public, free APIs. No server, no billing, no API key required.