indexing API

An indexing API inverts discovery. The default model is crawl-pull: a search engine decides when to revisit your site, and a page you changed this morning may not be re-fetched for days. Push means you send an HTTP request naming the URLs that changed, and the index queues a fetch immediately. Nothing about ranking or eligibility changes — you are only removing the waiting, which matters most when the whole point of a deploy was to change what a machine reads.

The two are confused constantly and they have almost nothing in common besides the category. IndexNow is an open protocol operated by Bing and Yandex, accepts any content type, requires only a key file hosted at your domain root, and is implemented natively by several CDNs and CMSes. Google's Indexing API is a Google-specific service with OAuth service-account authentication, and its documentation scopes it to two structured types: JobPosting and BroadcastEvent. Using it for general content is not a clever hack, it is outside the documented surface.

The third thing in this space is the sitemap ping, and it belongs in a different tier. A sitemap tells an engine which URLs exist and roughly when they changed; pinging it says the file has been updated. That is a hint about a list, not a request to fetch a specific page, and the engine schedules whatever it wants afterwards. Sitemaps remain the correct mechanism for breadth and discovery. Indexing APIs are the mechanism for latency on pages you just changed.

The metric is latency, measured end to end: the wall-clock gap between a successful submission and the engine's crawler actually re-fetching the URL. You read it from your own server logs, which is the only place the truth lives — an accepted submission is a receipt, not a fetch. Measure it per engine and per URL type, because a page the engine considers low-value gets queued behind everything else regardless of how you submitted it.

Two secondary numbers are worth watching. Acceptance rate is the share of submissions returning success rather than an authentication or validation failure, and a rate below total almost always means a key file has drifted rather than that the URLs were rejected on merit. Change ratio is the share of submitted URLs whose content actually differed from the previously crawled version; submitting unchanged pages trains an engine to deprioritise your endpoint, so this number wants to stay near total.

IndexNow needs two things: a key you invent, published as a plain text file at your domain root whose filename is the key itself and whose body is the key repeated. That file is the proof of ownership — no OAuth, no console verification, no service account. Once it resolves, a single POST submits a batch of changed URLs, all of which must belong to the host named in the payload.

GET https://www.example.com/6a1f3c9d2b4e47f8a0c5d7e9b1234567.txt
6a1f3c9d2b4e47f8a0c5d7e9b1234567

POST https://api.indexnow.org/indexnow
Content-Type: application/json; charset=utf-8

{
  "host": "www.example.com",
  "key": "6a1f3c9d2b4e47f8a0c5d7e9b1234567",
  "keyLocation": "https://www.example.com/6a1f3c9d2b4e47f8a0c5d7e9b1234567.txt",
  "urlList": [
    "https://www.example.com/pricing",
    "https://www.example.com/docs/getting-started",
    "https://www.example.com/compare/alternatives"
  ]
}

The keyLocation field is the part most implementations get wrong. It exists so the key file can live somewhere other than the exact root path, and if it points at a URL that 404s or redirects, the whole submission is rejected even though the URLs themselves were fine. The host field must match the URLs in urlList — a batch mixing apex and www hostnames fails as a unit. Fetch the key file yourself with a plain HTTP client before you debug anything else.

• The firehose submit. Someone wires the endpoint to a nightly job that resubmits the entire sitemap, so ten thousand unchanged URLs arrive every day. The protocol accepts them, because acceptance is cheap, and the engine quietly learns that your signal carries no information. The fix is to submit only on actual content change — hook the publish and update events in your CMS rather than a cron schedule, and let the URL count per day look as small as your real change rate.
• The unverified key. The key file was created once during setup, then a hosting migration, a trailing-slash redirect rule or a new CDN cache rule made it 404 or return HTML. Every submission after that fails silently in the background, and nobody notices for months because the receipts were never checked. The fix is to log the response status of every submission and alert on the first failure, plus a scheduled fetch of the key file itself as a canary.
• The push without a change. A team submits URLs after a deploy that only touched CSS or a shared component, so the engine re-fetches and finds byte-identical content. Do this often enough and the crawl budget you were trying to spend efficiently gets allocated elsewhere. The fix is to diff the rendered output before submitting and skip URLs whose extractable content is unchanged — a deploy is not the same event as a content change, even when they arrive together.

The ordering the table implies is real: these mechanisms are sequential, not alternative. robots.txt decides whether any of them get read, the sitemap handles breadth, llms.txt handles priority among what was discovered, and an indexing API handles speed on the subset that just changed. Pushing a URL that your robots.txt blocks accomplishes nothing, and pushing a page whose content is unchanged accomplishes less than nothing. Push last, on the pages where the delta is the whole point.