How SchemaCraft reads websites.

All website requests made by SchemaCraft send this user agent. The last part names the feature that made the request:

SchemaCraftBot/2.0 (+https://www.schemacraft.org/bot; <feature>)

SchemaCraftBot sends only GET requests to public http/https addresses on ports 80 and 443. It does not log in, submit forms, add items to carts, run website JavaScript or follow links on its own beyond the limits described here.

Automated, scheduled, multi-page and third-party reading checks robots.txt (RFC 9309) before every connection, redirects included. The group for SchemaCraftBot applies first, then the earlier SchemaCraft group, then *. If robots.txt answers with 4xx it states no rules; if it cannot be reached (5xx, 429 or a network error) automated reading pauses.

• monitoringScheduled monitoring of a page a user added
• bulk-auditBulk audit of up to 25 listed URLs
• competitor-radarRadar comparison of 2–4 websites
• competitor-fixPage context for a Radar repair draft (sent to AI)
• site-mapAgency website map of up to 50 project URLs
• customer-questionsCustomer-question source check (passages sent to AI)

A single page that a signed-in or visiting person explicitly asks to check is read once on their behalf, like a browser visit, without consulting robots.txt. This mirrors how search engines treat user-triggered fetchers:

• page-checkFree page check of one URL
• homepage-checkHomepage inspector
• first-scanFirst scan of a client page
• change-verification / deployment-verificationVerification of a change the user deployed
• site-knowledgeOne page an agency asks to read for its client's site knowledge
• identity / content-check / ai-file-checkChecks of up to three selected project pages or files

To stop all automated reading by SchemaCraft:

User-agent: SchemaCraftBot
Disallow: /

Or keep it out of selected paths:

User-agent: SchemaCraftBot
Disallow: /private/
Allow: /

The website map also honours Crawl-delay up to five seconds. For anything robots.txt cannot express, write to us (section 6).

Only two features send website text to an AI provider (Google Gemini): the customer-question source check and the Radar repair context. Before that, SchemaCraft checks robots.txt and a machine-readable reservation. Radar also leaves out any third-party website that declares a reservation. A page is not analysed this way when it declares any of these:

tdm-reservation: 1                                   (HTTP header, W3C TDMRep)
<meta name="tdm-reservation" content="1">            (HTML)
<meta name="robots" content="noai">                   (HTML)
X-Robots-Tag: noai                                   (HTTP header)

SchemaCraft itself does not train AI models on website content; the provider processes the sent text under its API terms.

Each request has a 5–10 second deadline that includes DNS and redirects, a 0.2–3 MB response limit and at most three redirects. Private network addresses are refused. Endpoints are rate limited per account and IP; the website map allows at most 30 connections per website per minute across all accounts. Monitoring reads a page at most about once per hour.

SchemaCraft stores only the observations a feature needs, such as the title, meta description, canonical, indexing rules, JSON-LD, page facts (language, viewport, main heading, image and link counts, hreflang, redirects) and capture time, plus short visible passages for the customer-question check. It does not deliberately collect personal data from websites; see the privacy notice.

Questions, problems or a request to stop reading your website: fesojcube@gmail.com. Include the domain and, if possible, the time and user agent of the request.