Crawler settings allow you to choose the type of bot that crawls your website.
Posted: Mon Jan 20, 2025 6:05 am
Note that the “ Website ” option will cause bots to crawl your site like Google does, while a crawl using “ On-site Sitemaps ” uses the URLs in your robots.txt sitemap.
Alternatively, click " URL from file " to select the pages you want the list of djibouti consumer email bots to crawl.
Adjust Crawler settings
Site Audit Crawler Settings Page
You can choose between Googlebot and SemrushBot and between mobile and desktop versions.
Then select the crawl delay settings. The bots crawl at a normal speed with "minimal delay" and prioritize user experience in the "1 URL every 2 seconds" settings.
Finally, select “Respect robots.txt” if you have the file in question and need a specific delay for crawling.
Allow/Reject URLs
Customize your site audit with allow/deny URL settings by entering URLs into the corresponding boxes.
Alternatively, click " URL from file " to select the pages you want the list of djibouti consumer email bots to crawl.
Adjust Crawler settings
Site Audit Crawler Settings Page
You can choose between Googlebot and SemrushBot and between mobile and desktop versions.
Then select the crawl delay settings. The bots crawl at a normal speed with "minimal delay" and prioritize user experience in the "1 URL every 2 seconds" settings.
Finally, select “Respect robots.txt” if you have the file in question and need a specific delay for crawling.
Allow/Reject URLs
Customize your site audit with allow/deny URL settings by entering URLs into the corresponding boxes.