User-agent: * Disallow: Crawl-Delay: 6 User-agent: Amazonbot Disallow: / User-agent: Bytespider Disallow: / User-agent: Yahoo Pipes 1.0 Disallow: / User-agent: Yandex Disallow: / User-agent: Mail.Ru Disallow: / User-agent: SemrushBot Disallow: / User-agent: SemrushBot-SA Disallow: / User-agent: BLEXBot Disallow: / User-agent: MJ12bot Disallow: / user-agent: AhrefsBot disallow: / User-agent: PetalBot Disallow: /