User-agent: * Crawl-delay: 1 Disallow: /maintenance.html Disallow: *=undefined* Disallow: /item/upsell_counter Disallow: /compare-add Disallow: /articles/search/ Sitemap: https://www.avbuyer.com/sitemap.xml User-agent: MJ12bot Disallow: / User-agent: psbot Disallow: / User-agent: Yandex Disallow: / User-agent: PetalBot Disallow: /