The HeadlessHorse Grader

If you see HeadlessHorse-Grader in your logs, this page explains what it was doing.

The Grader checks how a software vendor's public website responds to automated visitors on the way to a demo or pricing request. HeadlessHorse uses the results to show vendors where an AI agent acting for a buyer might get stuck.

What it checks

• Whether the site is served through Cloudflare, and what an automated request without a signature gets on the home, pricing and demo pages: the page, a security challenge, a block, a payment request or a rate limit
• Whether the demo form uses Cloudflare Turnstile
• Which form and scheduling tools the page loads, such as HubSpot, Chili Piper, RevenueHero or Calendly
• Which fields the demo form requires, and whether it asks for a phone number, as the page shows them in a browser
• What the site's robots.txt says to AI crawlers and agents

How it behaves

• It reads public pages only. It never submits a form, signs in, books a meeting or buys anything.
• It respects robots.txt. A rule for HeadlessHorse-Grader stops it, including the browser's requests to your site.
• Most demo forms are drawn by scripts, so to read one it opens the demo page once in a headless Chrome browser, which runs the page's scripts. The browser does not type, click or scroll. It allows only requests that fetch something (GET and HEAD) and refuses every other request, including form submissions and beacons, while the page loads, while it is read and as it closes. So a form that sends by POST, as demo forms do, cannot be submitted. Like any visit, the page's own GET requests still go out: the page, its scripts and frames, and whatever those scripts fetch, which can include analytics services that are not on our skip list. It does not load images, fonts, video, known CAPTCHA or challenge scripts, or a fixed list of common analytics and advertising services, and it stops if it is shown a challenge.
• It runs slowly. It checks one site at a time and waits a few seconds between pages. Its plain checks request each page once per run, usually eight requests or fewer per site, including robots.txt for each host they visit; opening the demo page in the browser also loads the scripts and frames that page loads, up to 150 requests, and it closes any page that attempts more than 2,000.
• It finds pages only by following the site's own links on the site's domain. It does not guess addresses.
• It identifies itself as HeadlessHorse-Grader/0.1 (+https://headlesshorse.ai/grader), in the browser too. It starts each page with no cookies and discards them afterwards, and it does not try to get past challenges or CAPTCHAs.
• It keeps what it found, such as response codes, headers, the tools a page loads and the demo form's fields. It keeps copies of your pages only until the run's report is written, the same day at the latest, and then deletes them.

Opting out

Add this to your robots.txt:

User-agent: HeadlessHorse-Grader
Disallow: /

Or email contact@headlesshorse.ai with your domain, and we will stop checking it.

Questions and corrections

Email contact@headlesshorse.ai. If a finding about your site is wrong, we will correct it. Corrections never depend on buying anything from us.