How it works¶

This page describes what happens when you crawl a site and search it, in enough detail to find your way around the source.

The big picture¶

┌─────────────┐     ┌──────────────┐     ┌───────────┐     ┌──────────┐
│  Flask API  │────▶│   Crawler    │────▶│  Indexer  │────▶│  MySQL   │
│  (app.py)   │     │ (crawler.py) │     │(indexer.py│     │ Database │
└─────────────┘     └──────────────┘     └───────────┘     └──────────┘
       │                                                        ▲
       │              ┌──────────────┐                          │
       └─────────────▶│    Search    │──────────────────────────┘
                      │  (search.py) │
                      └──────────────┘

The API receives requests and coordinates. The crawler fetches pages. The indexer extracts content and stores it. The search module queries the database and ranks what it finds.

The crawl process¶

1. robots.txt¶

The crawler fetches robots.txt and honours it, including the * and $ wildcards that RFC 9309 defines. That matters more than it sounds: Python's standard-library robots parser does not implement wildcards, so a perfectly ordinary rule like Disallow: /search/?* silently matches nothing, and the crawler walks straight into the URL space the site asked it to avoid. Dead Simple Search uses protego instead. A missing or unreadable robots.txt means everything is allowed.

2. Sitemap discovery¶

Sitemaps are found through Sitemap: directives in robots.txt, falling back to common locations such as /sitemap.xml and /sitemap_index.xml. <sitemapindex> files are followed recursively to a depth of three. Their URLs seed the crawl queue, which is faster and more complete than discovering pages by following links alone.

Each <lastmod> is read too, but only kept if the sitemap looks like it tracks real per-page dates. Many generators stamp every entry with the moment the file was written; when a single value covers more than half the dated URLs in a sitemap, that document's dates are discarded. Sitemaps are judged one at a time, so a site with one well-maintained sitemap and one build-stamped sitemap keeps the good one.

3. Fetching¶

The crawler works through a queue, fetching with an async HTTP client and pausing CRAWL_DELAY_SECONDS between requests. Only text/html responses are processed. Redirects are followed manually, one hop at a time, so each hop can be re-checked. Links are collected from each page and the crawler stays on the original domain.

Every DNS resolution passes through an SSRF filter that drops private, loopback, link-local and reserved addresses. Because it sits at the resolver, it covers robots.txt, sitemap probes and page fetches alike, and re-checks on each resolution rather than trusting an earlier answer.

4. Deciding what not to index¶

A page is fetched before its instructions can be read, so several checks happen after the response arrives. Each one removes any existing entry for that URL, so a page that gains a noindex on Tuesday is gone from the index after Tuesday's crawl.

noindex, from <meta name="robots">, the googlebot variant, content="none", or an X-Robots-Tag response header.
Redirects. The entry under the original URL is deleted. If the target is off-domain it is not indexed at all.
HTTP errors. 4xx and 5xx responses drop the page from the index.
Meta refresh with a target, which is a client-side redirect to a stub page. Delays over ten seconds are self-refreshing pages like dashboards, and index normally.
Non-canonical URLs, where rel=canonical names a different page. The canonical is queued instead. Links on the page are still followed, since a filtered listing is often the only route to what it links to — except links that are themselves query-string variants of the same canonical, which would otherwise walk pagination forever.

5. Indexing¶

For each surviving page the indexer extracts the title, meta description, all H1 and H2 headings, the body text with scripts and navigation stripped out, the language from <html lang> or by detection, and the publication dates.

Rows are upserted on a hash of the URL: an existing page is updated in place, a new one inserted.

After a complete and healthy crawl, pages that were not seen are deleted — this is how pages removed from your site leave the index. Pruning is skipped if the crawl hit its page cap or looked unhealthy, so an outage cannot empty a working index.

Page dates¶

Two dates are stored per page, resolved independently.

page_published comes only from the page's own markup, checked in order: Schema.org JSON-LD, Schema.org Microdata, Open Graph, Dublin Core. Nothing else can supply it, because neither a sitemap nor an HTTP header can express when a page was created.

page_modified takes the first of three sources that speaks:

Source	Claim
The page's own `dateModified`	the content changed
The sitemap's `<lastmod>`	the URL changed
The HTTP `Last-Modified` header	the file changed

They are consulted in that order and never merged, and the most recent value deliberately does not win. The second and third sources both fail toward the present — build timestamps, pages regenerated on every request — so preferring the newest date would hand the least reliable source control of nearly every page. A date that is wrong but recent is worse than a missing one, because it outranks genuinely fresh content whenever results are sorted by date.

Everything doubtful is discarded rather than stored: dates before 1990 or more than a day ahead, a Last-Modified matching the response's own Date header, and any weaker date that precedes the page's own datePublished. page_modified_source records which source won, so a client that trusts only editorial dates can filter on it.

The two dates are never reconciled against each other. A page claiming it was published after it was modified is reported exactly that way; the markup is the site's to fix.

The search process¶

Mode detection. A query containing boolean operators is passed to MySQL's boolean mode as written.

Stemming. Otherwise, with stemming on, each term expands to a stemmed prefix and all terms are required. If nothing matches, the same terms are retried as optional, so a query where one word is absent still returns partial matches. Failing that, the query runs in plain natural-language mode. Stemming covers Swedish, Danish, Norwegian, Finnish, Icelandic and English.

Ranking. MySQL's full-text index scores each match. Pages matching in the title, meta description or H1 have that score multiplied by SEARCH_TITLE_BOOST, using a second index covering only those fields — MySQL cannot weight fields within a single full-text index, so without this a title match counts no more than a passing mention.

Pagination. Twenty results at a time by default, up to a hundred.

The database¶

Three tables:

sites — one row per registered website: domain, start URL, discovered sitemaps, whether crawling is enabled.

pages — one row per indexed page, holding the extracted content and dates. Two full-text indexes: one across title, description, headings and body, another across title, description and H1 for the ranking boost.

crawl_log — one row per crawl run, recording when it started and finished, how many pages succeeded and failed, and the outcome.

There is no migration framework. database.py is the source of truth, and init_schema() creates anything missing on startup. Because CREATE TABLE IF NOT EXISTS never alters an existing table, columns added later are applied by small idempotent helpers that check information_schema first.

File structure¶

Around 3,200 lines of Python:

File	Lines	Purpose
`indexer.py`	860	HTML parsing, date extraction, database upserts
`crawler.py`	586	Async crawler, robots and canonical handling
`app.py`	429	Flask application, API endpoints, auth, rate limits
`sitemap.py`	251	Sitemap discovery and XML parsing
`search.py`	243	Full-text search and ranking
`database.py`	194	Connection pool, schema, in-place migrations
`stemming.py`	160	Query-time stemming for Nordic languages and English
`config.py`	110	Configuration from environment variables
`ssrf.py`	78	IP blocklist and DNS resolver filter
`recrawl_all.py`	77	CLI helper to re-crawl every site in sequence
`robots.py`	74	robots.txt matching with wildcard support
`scheduler.py`	69	Periodic re-crawling
`wsgi.py`	35	Production entrypoint for gunicorn
`passenger_wsgi.py`	12	Entrypoint for Passenger and cPanel hosting

Design principles¶

Ordinary technology. Python, Flask and MySQL are mature and widely documented. Whatever goes wrong, someone has written about it.

No magic. Every SQL query is hand-written. There is no ORM between you and the database, so reading the code tells you exactly which queries run.

Small surface area. Each module does one thing and fits in your head. There are no deep abstraction layers to trace through.

Honest trade-offs. The snippet in search results is the first 300 characters of body text, not a window around the matched terms. That is a real limitation, accepted because the alternative costs more complexity than it returns here.

Discard rather than guess. Where a signal is unreliable — a date, a sitemap's timestamps, a header — the code prefers null to a plausible-looking value. Missing data is visible; wrong data quietly spreads.