TheUsedCarIndexBot

TheUsedCarIndexBot reads public vehicle listing pages for The Used Car Index. The feed contains listing facts only: VIN, price, mileage, status, dates and the link back. It never includes photos or dealer-written text.

Our identity

The default user agent is:

TheUsedCarIndexBot/1.0 (+https://theusedcarindex.com/bot)

How requests work

• Before inventory requests, it fetches robots.txt and checks the applicable policy. The policy is reused during a run. Disallow rules and Crawl-delay apply to listing requests and redirect targets.
• It waits several seconds between requests to a host. The default is at least 6 seconds after a request finishes, or longer if Crawl-delay requires it.
• It stops on a detected block or challenge and does not try to get around one. It records a cooldown before another run can request that host. It also bounds retries after temporary failures.
• We designed the bot to visit each site about once a day. We skip a run when a site is unavailable, blocked or asks us to wait.

Pages are read to extract facts. For incomplete pagination, the collector can retain a public page's HTML for local diagnostics. That HTML can contain dealer-written text and image references. Referenced photos are not downloaded. Captures are capped at 50 files per date. Captures older than 7 days by folder date or file modification time are purged at run start and end, on status checks, or by an explicit purge command. Old captures are deleted each time the bot starts and finishes and when its status is checked; a capture that cannot be deleted is retried on the next run. Diagnostic HTML does not enter the feed.

Opt out or ask a question

To stop collection, add this to your site's robots.txt:

User-agent: TheUsedCarIndexBot
Disallow: /

Or email jorge@theusedcarindex.com with your dealer website. Your dealer's listings will be removed from the feed within 7 days of the request. This is handled by Jorge. Robots.txt stops future collection; email to request removal of existing listings. Questions about a request, a listing or the bot are welcome at the same address.