Guest writer Barry Adams shares his considerable expertise on crawl budget, the Crawls Stats report, common errors and ways publisher can optimize their crawl budget.
I find it very interesting you mention the 'duplicate' URLs for stories - that's kind of baked in when we need to produce a normal web rendition plus the AMP rendition for the same article :(
Would crawl budgets basically be halved if we dropped AMP - a near and dear subject to your heart Barry?
Theoretically AMP should not consume crawl budget, but in practice it does. Decommissioning AMP can result in more crawl effort being spent on regular pages, but it can take a while (3+ months) for the crawl budget allocated to AMP URLs to be 'released' and re-assigned to other URLs.
I find it very interesting you mention the 'duplicate' URLs for stories - that's kind of baked in when we need to produce a normal web rendition plus the AMP rendition for the same article :(
Would crawl budgets basically be halved if we dropped AMP - a near and dear subject to your heart Barry?
Theoretically AMP should not consume crawl budget, but in practice it does. Decommissioning AMP can result in more crawl effort being spent on regular pages, but it can take a while (3+ months) for the crawl budget allocated to AMP URLs to be 'released' and re-assigned to other URLs.
,