RateDiffBot

RateDiffBot is the crawler for Rate Diff (ratediff.com), a weekly newsletter on hospital prices. It fetches one thing: the machine-readable standard charges file that each US hospital is required to publish under the federal Hospital Price Transparency rule (45 CFR Part 180), found through the hospital's own cms-hpt.txt pointer file or the link on its price transparency page. It checks each file about once a week with a conditional request, so a file that has not changed is not downloaded again. It identifies itself as RateDiffBot/1.0 and never uses a browser disguise or a proxy. On a hospital's own website it honors robots.txt, both to find the file and to fetch it, and fetches nothing else. When a hospital publishes its file on a third-party host (Box, Dropbox, Google Drive, a CDN), it fetches the exact published address and nothing else on that host, because that host's robots.txt is its policy about crawling the host, not the hospital's about its own file. A link wrapped by an email security service is unwrapped to the address underneath first. Questions, or a request to stop: hello@ratediff.com.

Who runs it

Rate Diff is published by Upstream Logic. Write to hello@ratediff.com with any question about the crawler.

For firewall and security teams

User agent. Every request carries exactly this header:

User-Agent: RateDiffBot/1.0 (+https://www.ratediff.com/bot; hello@ratediff.com)

Its robots.txt product token is RateDiffBot, matched without regard to case.

Source IP addresses. From Monday, October 12, 2026, RateDiffBot runs from Trigger.dev's cloud, in its us-east-1 region, and its requests leave from 5.60.65.64 to 5.60.65.127 (5.60.65.64/26). Those addresses are shared with other customers of the same cloud provider, so match RateDiffBot's user agent as well as the address: a rule that allows the whole range on its own also lets in other customers' traffic. To confirm that a request in your logs came from it, send the time and the address requested to hello@ratediff.com: every request it makes is logged with its time and the address requested.

What it requests on a hospital's own website.

• /robots.txt, before anything else on the site.
• /cms-hpt.txt and /.well-known/cms-hpt.txt, on the domain and on its www host.
• The standard charges files your cms-hpt.txt names. Before downloading a file it may send a HEAD request, or read its first 64 KB, to learn its size and format.
• Only when a site has no cms-hpt.txt that names a usable file: up to 8 pages of the site, starting at the home page and following only links that look like price transparency, standard charges, or billing pages, at most two links deep, until it finds the link to the file. It may also read a price transparency page of the site that a web search returned.

How often, and how hard. It checks each cms-hpt.txt and each standard charges file about once a week. The check is conditional: a HEAD request compared with the ETag, Last-Modified, and Content-Length your server last sent, or a GET with If-None-Match and If-Modified-Since, so a file that has not changed is not downloaded again. An unchanged file is downloaded in full at most once every 90 days. It sends at most two requests a second to any one host and downloads one file at a time from a host. A small request (robots.txt, cms-hpt.txt, a page) that gets a 429, a 5xx, or no answer is tried at most three times in all, a few seconds apart. A file download that fails waits for the next weekly check. A 403 from a hospital's website is never retried. The one exception to one file at a time is two large cloud storage services that host many hospitals' files, where it downloads up to four at once.

How to ask it to stop. Add this to the robots.txt of your site:

User-agent: RateDiffBot
Disallow: /

It reads your robots.txt at the start of each check and requests nothing your robots.txt disallows for it. A robots.txt that cannot be read (a server error or a timeout) is treated as disallowing everything until it can be read. A file your hospital publishes on a third-party host is fetched at its exact address, so to stop those fetches, or for anything else, write to hello@ratediff.com.