On top of monitoring your website's pages and XML Sitemaps, Conductor Website Monitoring also monitors your robots.txt file. Your robots.txt file is checked every 5 minutes. Learn more about why this is important in our Robots.txt guide for SEO.
Working with robots.txt in Conductor Website Monitoring
Conductor Website Monitoring monitors your robots.txt file 24/7 and you're alerted in case it changes or becomes inaccessible.
Conductor Website Monitoring also stores the robots.txt versions historically, so you can compare the changes between the versions, see the response headers that Conductor Website Monitoring received, or download the robots.txt data!
Comparing different versions of robots.txt
Conductor Website Monitoring not only stores robots.txt versions historically - you can also compare the changes between them!
You noticed that the search engines haven't been crawling certain pages due to the robots.txt blocking them?
Easily compare different robots.txt versions that Conductor Website Monitoring indexed and see when was the undesired rule implemented:
Robots.txt and Alerts
Alerts in Conductor Website Monitoring notify you when any important changes happen on your website and allow you to take action before they hurt your bottom line.
Conductor Website Monitoring out of the box alerts you whenever any changes are made to the robots.txt file, or when it becomes inaccessible.
You can find more details about Alerts in Conductor Website Monitoring here: Alerts
Robots.txt and Issues
Conductor Website Monitoring audits the robots.txt file for the following SEO issues:
- Robots.txt is accessible
- Robots.txt is present on canonical variant only
- All directives are valid
- Contains no crawl-delay directives
- Syntax is valid
- All assets accessible to search engines
- XML sitemap reference is present
You can read more about issues that Conductor Website Monitoring audits here: Issues
See HTTP headers for any robots.txt version
Ever wondered what the exact response was for your robots.txt? Or are you seeing two alternating versions of your robots.txt and need to investigate?
Check the HTTP headers for any robots.txt version to see:
- the headers of the HTTP request that we sent.
- the headers of the HTTP response that we received.
Download your robots.txt data with one click
Need to show the response and HTTP headers to your colleagues or the technical team? Easily download your robots.txt data and forward it directly to them.
See when your robots.txt is visited by search engines in real-time
You implemented an important change to the robots.txt file and are wondering if search engines picked it up?
With our new Log File Analysis feature, you can know whether or not Google and Bing have seen each version directly in the robots.txt viewer.