Fan-Out Coverage Study
16,877-query observational research pipeline measuring whether ChatGPT cites pages that cover more of its internal fan-out sub-query space, against a deterministic seen-but-not-cited control group. Queue-orchestrated multi-worker pipeline with novel headings-as-index coverage scoring and on-demand section-text embedding.
The Fan-Out Coverage Study is a 16,877-query observational research pipeline measuring whether GPT-5.3 cites pages that meaningfully cover more of its internal fan-out sub-query space. It compares cited pages against a deterministic seen-but-not-cited control group to isolate the citation decision from the retrieval decision, and uses a novel headings-as-index coverage scoring method to avoid conflating coverage with page length or topical adjacency.
The gap it closes
ChatGPT does not cite every page it retrieves. It expands the user’s query into multiple fan-out sub-queries during browsing, retrieves pages for each, then chooses a subset to cite. Most public claims about why a page got cited collapse those decisions into one and rely on simple length or topical-adjacency heuristics. The sharper question is whether a page that meaningfully covers more of the fan-out sub-query space has a measurable association with being cited, after controlling for whether ChatGPT retrieved the page at all. Without a structured coverage measurement, against a controlled seen-but-not-cited comparison group, the question cannot be answered defensibly. This data pipeline was built setup in Postgres and Docker to capture data needed to conduct the study.
Experimental design
Conditioned-on-being-seen control group
For each primary query: all cited URLs plus the top 10 seen-but-not-cited URLs, ordered by the sequence ChatGPT saw them in the browsing payload. Seen order is deterministic. URLs are normalized and deduplicated within each query before the top 10 selection. Conditioning on “being seen by ChatGPT” strips out the retrieval decision and isolates page-level differences associated with being chosen for citation.
Headings-as-index coverage scoring
Each scraped page is indexed by its H1 through H4 headings in order of appearance, with the section text under each heading preserved. For each fan-out sub-query ChatGPT generated for a given primary query, the pipeline checks whether any heading on the page matches the fan-out query above a similarity threshold. If a heading matches, the section text under that heading is embedded on demand and validated against the fan-out query. Coverage is marked only when both heading and section text answer the fan-out query.
Output per page: a fan-out coverage ratio (share of fan-out queries the page covers), average similarity across matched fan-out queries, a binary covers-primary flag, and a fan-out-only flag for pages that cover fan-out queries without addressing the primary query.
Pipeline architecture
Eight idempotent and resumable steps coordinated through a relational analytical store and a job queue, run inside a containerized multi-worker stack.
- Load the query set.
- Send each query through ChatGPT’s consumer interface, capture cited URLs, seen-but-not-cited URLs, fan-out queries, and enqueue URLs for scraping.
- Scrape all cited URLs and the top 10 seen-but-not-cited URLs per query.
- Extract page structure (headings, sections, page controls including word count, page age, page type, readability, internal link counts, structured-data type).
- Pull domain authority, backlinks, and spam-score signals from third-party authority providers.
- Coverage scoring per the methodology above.
- Join everything into a per-(query, URL) row, export CSV and Parquet.
An optional SERP collection step adds Google SERP coverage for primary and fan-out queries when needed as a diagnostic signal. A run orchestrator loads queries, starts worker containers, monitors progress with a live dashboard, and triggers the final join once all per-step queues are drained.
Schema for joinability
Tables separate the three relevant decisions (ChatGPT retrieval, ChatGPT citation, Google SERP rank) on query_id, normalized_url, response_id, and coverage_id join keys. A coverage_summary table is the one-row-per-(query, URL) primary analysis surface, with citation flag and coverage scores. A coverage_fanout_detail table holds the per-fan-out match diagnostics. Controls join through page_controls and domain_metrics.
What shipped
Full 16,877-query observational dataset processed end-to-end. Roughly 2-hour total runtime on full pipeline. One row per (query, URL) pair in the export with all coverage scores, content signals, and domain metrics joined. Analysis-ready Parquet and CSV outputs, schema diagram, analyst guide, verification queries doc, and the headline research report all shipped.