Conductor Website Monitoring allows you to adjust the monitoring settings for each website in your account to ensure your websites have the optimal monitoring setup that is tailored to your specific needs.
To access the monitoring settings in Conductor Website Monitoring, go to the Website settings and click on the Monitoring tab on the left side of the screen.
This is a highlight box. The borders that appear in the editor and this explainer will not appear in the published article. Use the Title and Body below for examples, best practices, or other information you'd like to call out.
End of Highlight box
Monitoring Settings
Device Type
You can choose the device type for which you want Conductor to use as it crawls your website. You can choose either:
- Mobile, which uses a 360px width by 640px height viewport.
- Desktop, which uses 1,350px width by 940px height viewport
See instruction below for how to configure this setting.
User-Agent String
What is a user-agent string?
Browsers and web crawlers send a special identification, called the user-agent string, to the website they're accessing. This user-agent can be used to tailor the reply, for example: when the user-agent string makes it clear that the website is being accessed on a phone the mobile variant of the website may be returned. If the web server software keeps logs of all visitors to the website it usually also stores the user-agent string to allow identification of the different traffic.
See instruction below for how to configure this setting.
Default user-agent string
By default, Conductor sends this user-agent string when making requests:
Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (+https://whatis.contentkingapp.com)
This may change in the future, so check this page regularly for updates.
Monitoring speed and frequency
This is a highlight box. The borders that appear in the editor and this explainer will not appear in the published article. Use the Title and Body below for examples, best practices, or other information you'd like to call out.
As an auditing product, Conductor Website Monitoring differs from other SEO auditing tools in that it monitors websites 24/7. Search engines check your website 24/7, so it’s essential that your auditing solution does the same. Our monitoring is driven by smart algorithms that take a vast array of signals into account to determine when to crawl a specific page.
As a product that reports technical SEO data, Conductor Intelligence provides regularly collected data from crawls of your websites, no less often than a week.
End of Highlight box
The speed with which Conductor monitors and the overhead (or impact) on your website depends on your website’s size, your webserver’s capacity and the monitoring speed setting.
See instruction below for how to configure this setting.
The relation between web server capacity and monitoring speed
In order to avoid performance issues on your website and to align with your business needs, before using Conductor it is essential that you verify what crawl speed your infrastructure can support and adjust your settings accordingly.
Although it is important to assess webserver capacity before using the application, Conductor helps predict server overload by continuously assessing your website’s performance. It does this through its throttling algorithm, which looks at the time it takes to get a response from the webserver and at the response itself. If the algorithm detects an issue (for example, slower speeds, increased errors, etc.), it may slow down crawling automatically until the impact on the webserver has been remedied.
The relation between website size and monitoring speed
Once you have verified webserver capacity, for larger websites you may decide to crawl at a faster speed in order to keep up to date on your website's changes as well as possible. For example, if your website has 200,000 pages, you may wish to configure Conductor to request a page every second.
For smaller websites, you may decide to crawl at a slower speed. So, if your website is only a couple of thousand pages, the optimal crawl frequency may be very low. For example, if your website has 2,000 pages, you may only need Conductor to request a page once every 10 seconds.
Changing the Monitoring Settings in Conductor Website Monitoring
Changing the Device Type
You can configure which Device Type Conductor Website Monitoring should use when monitoring your website in the Monitoring section of the Website settings.
You can pick from Desktop or Mobile.
Changing the User-Agent
You can configure which user-agent string Conductor Website Monitoring should use when monitoring your website in the Monitoring section of the Website settings.
You can pick a predefined user-agent string from the list or set a fully custom user-agent string if you have validated ownership of the website. You can validate ownership of the website by connecting Google Search Console or installing the Cloudflare Worker. More about website verification here
Predefined user-agents:
- Google Chrome (desktop)
- Google Chrome (mobile)
- Googlebot (desktop)
- Googlebot (mobile)
- BingBot (desktop)
- BingBot (mobile)
- Slurp
- YandexBot
- Baiduspider
- Screaming Frog
- DuckDuckGo
- SeznamBot
- Conductor Website Monitoring
Setting a different monitoring speed in Conductor Website Monitoring
You can speed up or slow down Conductor Website Monitoring's crawling by setting a different monitoring speed in the website settings in Conductor Website Monitoring.
To crawl faster than the default monitoring speed you also need to verify domain ownership by connecting with Google Search Console or installing the Cloudflare Worker. You can read more about this here: Website verification.
Setting different monitoring speeds in Conductor Website Monitoring for different times of the day
If your website sees or may see significantly higher loads due to traffic at peak times, you should configure Conductor Website Monitoring to slow down, or even pause, monitoring during these times by using the Peak / Off-peak monitoring speed feature:
Javascript Rendering & Lighthouse
You can enable and disable wheter Conductor renders javascript during its crawl of your website and whether Conductor tracks the Web Vitals metrics for every page on your website and work with them in the same way as with all the other properties and metrics Conductor monitors.
You can enable Javascript Rendering and Lighthouse monitoring here in Monitoring settings.
To learn more, refer to the Javascript Rendering and Core Web Vitals Lighthouse Monitoring articles.
Advanced Settings
Monitoring location
Out of the box, Conductor monitors websites from the EU, the UK, or the US:
- If you create your account from the US or from Canada, your websites will be monitored from the US by default.
- If you create your account from the UK, your websites will be monitored from the UK by default.
- In all other cases, your websites will be monitored from the EU by default.
However, you can also change the monitoring location (see below).
When does the monitoring location matter?
Geo redirects
When you operate globally or in multiple countries, you might have multiple domains with different language versions of your website (example.com, example.co.uk, example.de, etc.) and a geo redirect setup in place that redirects visitors to the relevant domain version based on their location.
So when you want to monitor example.com but the default monitoring location is set to the UK, Conductor will get redirected to example.co.uk and it will not be able to monitor example.com.
If this is the case, you can change the default monitoring location to the US to ensure that Conductor doesn’t get redirected to a different domain and will be able to monitor example.com.
GeoIP blocking
Some websites are only available for visitors located in a certain region, and visitors outside of this region are not able to access the website.
So for example, when your website is only available in the US, but your default monitoring location is set to the EU, Conductor will not be able to access and monitor the website.
If this is the case, you can change the monitoring location for the website to the US, which will allow Conductor to access the website and monitor it properly.
HTTP Auth
Staging and testing environments
Conductor provides great solutions for monitoring your staging and testing environments, allowing you to do comparisons with the production environment before green lighting a release or migration.
Staging and testing environments are usually protected from all outside visitors as well as search engines.
If you’re wondering why you should be protecting your staging environment or you want to learn about different protection methods, be sure to read our Academy article on this topic.
Allowing Conductor access to the staging environment
There are several ways in which you can give Conductor Website Monitoring access to your staging environment based on what protection method you are using.
HTTP Auth is the best way to protect your staging environment and you can read more about it here.
Conductor allows you to set up HTTP Auth directly in its apps, which means that you can easily enable us to monitor your staging environment.
See instruction below for how to configure this setting.
Other ways to whitelist Conductor's monitoring
You can also give Conductor access to your staging environment based on a unique identifier of our crawler.
You can whitelist Conductor based on:
Cookies
By default Conductor doesn’t accept or transmit cookies when requesting URLs on websites.
However, some websites can require cookies to load certain elements which Conductor needs to monitor the website properly and report accurate data.
For example, some websites have a cookie consent which is used to load analytics tracking. In such cases Conductor wouldn’t be able to discover the analytics tracking, as it doesn’t transmit cookies.
This is why you can set custom cookies in Conductor.
See instruction below for how to configure this setting.
HTTP Headers
HTTP headers serve for transferring additional (not displayed) information between Conductor and your website’s server.
Based on HTTP headers you can identify our monitoring as well as whitelist Conductor in case our crawler is being blocked by your web server.
See instruction below for how to configure this setting.
Default HTTP headers
By default, Conductor sends the following HTTP headers with the monitoring requests:
Accept: /Accept-Encoding: gzip,deflate
Setting custom HTTP headers
On top of the default HTTP headers, you can also set custom HTTP headers that will be part of Conductor’s monitoring requests.
Out of the box, Conductor supports the following custom HTTP headers:
AcceptAccept-LanguageAuthorizationCache-controlForwardedProxy-AuthorizationRefererX-*
If you want to set a different custom header, don’t hesitate to contact us and we will be happy to set it up for you.
Changing Advanced Settings in Conductor Website Monitoring
How to change the monitoring location in Conductor Website Monitoring
In Conductor Website Monitoring, you can change the monitoring location on the website level as well as on the account level.
-
Changing the monitoring location for a website
To change the monitoring location for a specific website, go to the Monitoring tab of the Website settings and adjust the Monitor from setting. -
Changing the default monitoring location for an account
You can also set the default monitoring location for the whole account. To do this, go to the Account settings and adjust the Monitor from setting. Note that changing this setting doesn’t have a retroactive effect and the newly selected default monitoring location will only apply to websites that are added to your Conductor Website Monitoring account after the change is made.
Custom monitoring location
On the Basic, Standard, and Pro plans you can monitor your websites from one of the three monitoring locations as described above.
If you want to monitor your websites from a custom location, you can go with the Enterprise plan where it is possible for us to set up a monitoring location anywhere in the world. If you are interested in the Enterprise plan, don’t hesitate to contact us.
Setting up HTTP Auth for Conductor Website Monitoring
To set up HTTP Auth for a website in Conductor Website Monitoring, follow these steps:
- Go to the Monitoring tab of the Website settings
- In the top-right corner of the Advanced monitoring section click Edit and then click configure next to the Auth setting.
- In the window that appears, set the Username and Password that Conductor Website Monitoring will use to access the website.
- Click Save.
Setting up HTTP headers for Conductor Website Monitoring
To set custom HTTP headers for a website, follow these steps:
- Go to the Monitoring tab of the Website settings
- In the top-right corner of the Advanced monitoring section click Edit and then click configure next to the HTTP headers setting.
- In the window that appears, set the custom HTTP headers and their values.
- Click Apply changes.
Setting up custom cookies for Conductor Website Monitoring
To set custom cookies for a website, follow these steps:
- Go to the Monitoring tab of the Website settings
- In the top-right corner of the Advanced monitoring section click Edit and then click configure next to the Cookies setting.
- In the window that appears, set the custom cookies and their values.
- Click Apply changes.
More Miscellaneous Settings
Conductor also allows you to configure the following aspects of its crawl:
Relevant Search Engine
This setting affects how Conductor reviews the robots.txt and meta robots rules on your website. Because search engines have different rules and regulations for how they crawl websites, you can specify whether Conductor should look at directives for certain search engines.
Also track H2-H6 changes
Conductor can include or exclude tracked changes to H2–H6 tags on your website. Keep in mind that setting this toggle to Yes may greatly increase the total number of tracked changes for each URL—and Conductor does limit the total number of tracked changes to 70,000.
Also track number-only changes
Conductor can include or exclude tracked changes to tags where only a number value has changed. This might be helpful for your site if you want ignore changes to numbers that frequently change, such as quantities, prices, or dates. Turning this setting off can decrease the overall number of tracked changes on your website.
FAQs
Does Conductor Website Monitoring affect my Web Analytics data?
In general, Conductor Website Monitoring’s monitoring doesn’t have any effect on your analytics data. However, there are certain edge-case scenarios in which Conductor Website Monitoring can impact the data.
These scenarios also depend on the monitoring configuration you are using in Conductor Website Monitoring, namely on the JavaScript Rendering feature.
Effect on web analytics when JavaScript rendering is disabled.
If you're using a JavaScript-based web analytics solution such as Google Analytics, Conductor Website Monitoring won't affect your analytics data.
Conductor Website Monitoring may inflate your web analytics data in one situation only: when you're using a server-side web analytics solution.
In this case you should prevent our crawlers from being tracked by excluding our crawlers' IP addresses.
Effect on web analytics when JavaScript rendering is enabled.
When monitoring a website with JavaScript Rendering enabled, Conductor Website Monitoring blocks requests to the most common web analytics and ad services to ensure that your web analytics data don’t get inflated by Conductor Website Monitoring’s monitoring.
This means that if you’re using a common JavaScript-based web analytics solution such as Google Analytics or Adobe Analytics, Conductor Website Monitoring’s monitoring does not have any effect on your web analytics data and you can safely enable the feature without any concerns.
If you’re using a non-standard web analytics solution and you want to ensure that Conductor Website Monitoring’s monitoring doesn’t affect your analytics data, make sure to configure custom on-page request blocking in Conductor Website Monitoring.
More about how to set up the blocking you can find here: Custom on-page request blocking
Can I monitor only specific sections of my website?
Although we strongly recommend against doing this it is possible to limit monitoring to specific sections of your website. The reason why we don't recommend doing this is that you create a blind spot for yourself, and tests that rely on multiple data points (such as auditing for broken links or duplicate-content issues) will have limited reliability.
We do understand that not every page may be equally important to the success of your business, but we've learned from experience that serious issues do occur in pages initially deemed to be "uninteresting," and these issues may in turn hurt the entire website's SEO performance.
To support the concept of having your entire website monitored, we have made our pricing model as fair and flexible as possible: you only pay for pages (not for errors, redirects, etc.), and the more pages you monitor, the lower the price per page.
Having said all of this, if you still want to continue with monitoring only parts of your website, you can do this through the "URL Exclusion List". See this Support article on how to set this up: How can I exclude certain parts of my website from monitoring?
Can I pause monitoring on my website?
You can pause (and resume) monitoring through the website's monitoring speed settings. See our article on monitoring speed to learn how to do that.
Keep in mind that when monitoring is paused, your subscription remains active (you are still charged), but no changes will be detected and no alerts will be sent out.
What IP addresses does Conductor user when monitoring websites?
We monitor websites from the following IP address subnets:
- 89.149.192.96/27
- 81.17.55.192/27
- 23.105.12.64/27
- 173.234.16.0/28