Crawler
openbookschools-bot fetches public records for OpenBookSchools. This page says how it identifies itself, what it requests, how fast, and how to ask us to slow down or stop.
How it identifies itself
Every request carries these headers:
User-Agent: openbookschools-bot/1.0 (+https://openbookschools.com/bot)From: bot@openbookschools.comX-OpenBookSchools-Bot: openbookschools-bot/1.0 (+https://openbookschools.com/bot)
One exception to the User-Agent is described under Hosts that refuse the name. The other two headers always go with the request.
What it requests
A short, fixed list of public-records sources and the documents they list. It reads the index pages of these sources and fetches the documents they point to. It does not search the web for pages, and it does not follow links beyond the documents the sources list.
- DPSCD Board of Education agendas, minutes and attachments on BoardDocs (go.boarddocs.com): checked daily.
- DPSCD budget and salary transparency reports on detroitk12.org: checked monthly.
- Detroit Financial Review Commission resolutions and meetings on michigan.gov: checked monthly.
A document it already holds is re-checked with a conditional request (If-None-Match and If-Modified-Since), so an unchanged document is not downloaded again. Requests are GET requests, except on BoardDocs, where an agenda item can only be read with the same form request (a POST) that BoardDocs' own public pages send. Nothing else is sent.
How fast
- No more than two requests at a time to any one host.
- At least half a second between requests to the same host.
- These limits are kept through a shared cache and are best effort: two jobs can briefly overlap.
- A request that takes longer than two minutes is abandoned. A response over 64 MiB is refused.
- When a host answers 429 or a server error, or does not answer, the request is retried later with a growing delay, not at once. If a source answers 403 twice in a row, that run stops.
robots.txt
The crawler does not read robots.txt. The sources above are fixed, and each is a public body's records page that it reads by name. If you run one of them and want a different rate, a narrower path or no requests at all, write to bot@openbookschools.com and we will change it or stop.
Hosts that refuse the name
Two hosts, michigan.gov and boarddocs.com, answer 403 to any User-Agent that contains the word bot (we saw this on October 4, 2026). For those hosts only, a request that gets a 403 is retried once with an ordinary browser User-Agent. The From and X-OpenBookSchools-Bot headers are still sent with it, so the request can still be recognised as ours.
How to contact us
Write to
Say which host and, if you can, when, and we will answer.
How to block it
Any of these works, and none needs our agreement:
- Block the User-Agent
openbookschools-bot. - Block requests that carry an
X-OpenBookSchools-Botheader, or whoseFromheader isbot@openbookschools.com. This also catches the browser User-Agent retries above. - Write to us, and we will take the source off our schedule.
Documents already retrieved stay on the site unless a removal is requested under the content policy.
Last updated