Sitemap Topic Cluster
Python CLI for sitemap-driven topic gap analysis. Lemmatization with POS filtering and a generic-SEO-term blacklist tokenizes URL slugs, groups URLs into topic clusters by token frequency, and surfaces under-covered topics as content gap candidates with example slugs.
Sitemap Topic Cluster is a CLI that turns a sitemap into a topical distribution and surfaces under-covered topics as content gap candidates. Six single-responsibility modules behind one entry point.
The gap it closes
Content teams plan topic coverage from gut feel. A site’s existing sitemap already encodes which topics it covers and how deeply, but the raw URL list is unreadable at scale. The obvious approach (count keywords in the URL path) is contaminated by generic SEO terms like guide, best, how, and year markers that inflate counts without representing real topical depth. Stripped of that noise, the topical distribution becomes readable.
Pipeline
Six modules behind a single entry point.
- A sitemap loader pulls and parses the sitemap.
- A URL parser extracts slugs.
- A token analyzer tokenizes each slug with POS-filtered lemmatization. Keeps only noun and proper-noun lemmas. Drops stopwords and tokens under 3 characters. Blacklists generic SEO terms (
guide,best,how,what,why, year markers). - A cluster builder counts token frequencies. Treats any token at or above a minimum frequency threshold (default 5) as a core topic. Assigns each URL the subset of core topics it contains.
- A gap detector groups by topic cluster and flags clusters with three or fewer URLs as gap candidates, returning example slugs per cluster for context.
- A CLI orchestrator wires the pipeline and exposes the command-line interface.
The CLI accepts a sitemap URL as a positional argument or via interactive prompt. A configurable minimum-frequency flag tunes the topic threshold per site (a 500-page site needs a different threshold than a 50,000-page site).
What shipped
Reusable CLI for sitemap-driven topic gap analysis. Tokenization that strips generic-SEO contamination so the resulting topical distribution reflects real coverage depth rather than templated URL conventions. Per-cluster example slugs in the gap output, so the gap candidates are interpretable without going back to the source sitemap.