Website Crawl Settings in Conductor Intelligence

The Crawling activity (Settings > Crawling) in Conductor Intelligence lets you fine-tune Conductor's behavior as it crawls your website for technical SEO information in Pages and Page Details.

For websites you manage in Conductor Intelligence (that you do not manage with our Conductor Website Monitoring product—see the note below), you can configure a number of different settings that determine how Conductor crawls your sites.

---

This is a highlight box. The borders that appear in the editor and this explainer will not appear in the published article. Use the Title and Body below for examples, best practices, or other information you'd like to call out.

---

Note!

If you also use the Conductor Website Monitoring (CWM) product to manage any of your tracked web properties, you will configure those settings in CWM's Website settings screen.

Any web properties you track with Conductor Intelligence but do not mange in CWM can be configured here in Conductor Intelligence's Crawling activity. 

---

End of Highlight box

---

Monitoring Settings

Device Type

You can choose the device type for which you want Conductor to use as it crawls your website. You can choose either:

• Mobile, which uses a 360px width by 640px height viewport.
• Desktop, which uses 1,350px width by 940px height viewport

See instruction below for how to configure this setting.

User-Agent String

What is a user-agent string?

Browsers and web crawlers send a special identification, called the user-agent string, to the website they're accessing. This user-agent can be used to tailor the reply, for example: when the user-agent string makes it clear that the website is being accessed on a phone the mobile variant of the website may be returned. If the web server software keeps logs of all visitors to the website it usually also stores the user-agent string to allow identification of the different traffic.

See instruction below for how to configure this setting.

Default user-agent string

By default, Conductor sends this user-agent string when making requests:

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/60.0.3112.113 Safari/537.36 (+https://whatis.contentkingapp.com)

This may change in the future, so check this page regularly for updates.

Monitoring speed and frequency

The speed with which Conductor monitors and the overhead (or impact) on your website depends on your website’s size, your webserver’s capacity and the monitoring speed setting.

See instruction below for how to configure this setting.

The relation between web server capacity and monitoring speed

In order to avoid performance issues on your website and to align with your business needs, before using Conductor it is essential that you verify what crawl speed your infrastructure can support and adjust your settings accordingly.

Although it is important to assess webserver capacity before using the application, Conductor helps predict server overload by continuously assessing your website’s performance. It does this through its throttling algorithm, which looks at the time it takes to get a response from the webserver and at the response itself. If the algorithm detects an issue (for example, slower speeds, increased errors, etc.), it may slow down crawling automatically until the impact on the webserver has been remedied.

The relation between website size and monitoring speed

Once you have verified webserver capacity, for larger websites you may decide to crawl at a faster speed in order to keep up to date on your website's changes as well as possible. For example, if your website has 200,000 pages, you may wish to configure Conductor to request a page every second.

For smaller websites, you may decide to crawl at a slower speed. So, if your website is only a couple of thousand pages, the optimal crawl frequency may be very low. For example, if your website has 2,000 pages, you may only need Conductor to request a page once every 10 seconds.

Changing Crawl Settings in Conductor Intelligence

Changing the Device Type

You can configure which Device Type Conductor Intelligence should use when monitoring your website in the Monitoring section of the Website settings.

You can pick from Desktop or Mobile.

Changing the User-Agent

You can pick a predefined user-agent string from the list. Note that if you want to use a fully custom user-agent string, you'll need to manage the website in the Conductor Website Monitoring product.

Predefined user-agents:

• Google Chrome (desktop)
• Google Chrome (mobile)
• Googlebot (desktop)
• Googlebot (mobile)
• BingBot (desktop)
• BingBot (mobile)
• Slurp
• YandexBot
• Baiduspider
• Screaming Frog
• DuckDuckGo
• SeznamBot
• Conductor Website Monitoring

Setting a different monitoring speed in Conductor Intelligence

You can speed up or slow down Conductor's crawling by setting a different monitoring speed in the website settings in Conductor Website Monitoring.

You can also choose to set different speeds depending on the time of day to account for peak and off-peak periods. To do so, click Set peak / off-peak speeds.

From there, you can configure the specific times you consider Peak and the speeds associated with Peak and Off-Peak.

Advanced Settings

Monitoring location

Out of the box, Conductor monitors websites from the EU, the UK, or the US:

• If you create your account from the US or from Canada, your websites will be monitored from the US by default.
• If you create your account from the UK, your websites will be monitored from the UK by default.
• In all other cases, your websites will be monitored from the EU by default.

However, you can also change the monitoring location (see below).

When does the monitoring location matter?

Geo redirects

When you operate globally or in multiple countries, you might have multiple domains with different language versions of your website (example.com, example.co.uk, example.de, etc.) and a geo redirect setup in place that redirects visitors to the relevant domain version based on their location.

So when you want to monitor example.com but the default monitoring location is set to the UK, Conductor will get redirected to example.co.uk and it will not be able to monitor example.com.

If this is the case, you can change the default monitoring location to the US to ensure that Conductor doesn’t get redirected to a different domain and will be able to monitor example.com.

GeoIP blocking

Some websites are only available for visitors located in a certain region, and visitors outside of this region are not able to access the website.

So for example, when your website is only available in the US, but your default monitoring location is set to the EU, Conductor will not be able to access and monitor the website.

If this is the case, you can change the monitoring location for the website to the US, which will allow Conductor to access the website and monitor it properly.

HTTP Auth

Staging and testing environments

Conductor provides great solutions for monitoring your staging and testing environments, allowing you to do comparisons with the production environment before green lighting a release or migration.

Staging and testing environments are usually protected from all outside visitors as well as search engines.

If you’re wondering why you should be protecting your staging environment or you want to learn about different protection methods, be sure to read our Academy article on this topic.

Allowing Conductor access to the staging environment

There are several ways in which you can give Conductor Website Monitoring access to your staging environment based on what protection method you are using.

HTTP Auth is the best way to protect your staging environment and you can read more about it here.

Conductor allows you to set up HTTP Auth directly in its apps, which means that you can easily enable us to monitor your staging environment.

See instruction below for how to configure this setting.

Other ways to whitelist Conductor's monitoring

You can also give Conductor access to your staging environment based on a unique identifier of our crawler.

You can whitelist Conductor based on:

• Monitoring IP addresses
• User-agent string
• Custom HTTP header

Cookies

By default Conductor doesn’t accept or transmit cookies when requesting URLs on websites.

However, some websites can require cookies to load certain elements which Conductor needs to monitor the website properly and report accurate data.

For example, some websites have a cookie consent which is used to load analytics tracking. In such cases Conductor wouldn’t be able to discover the analytics tracking, as it doesn’t transmit cookies.

This is why you can set custom cookies in Conductor.

See instruction below for how to configure this setting.

HTTP Headers

HTTP headers serve for transferring additional (not displayed) information between Conductor and your website’s server.

Based on HTTP headers you can identify our monitoring as well as whitelist Conductor in case our crawler is being blocked by your web server.

See instruction below for how to configure this setting.

Default HTTP headers

By default, Conductor sends the following HTTP headers with the monitoring requests:

• Accept: /
• Accept-Encoding: gzip,deflate

Setting custom HTTP headers

On top of the default HTTP headers, you can also set custom HTTP headers that will be part of Conductor’s monitoring requests.

Out of the box, Conductor supports the following custom HTTP headers:

• Accept
• Accept-Language
• Authorization
• Cache-control
• Forwarded
• Proxy-Authorization
• Referer
• X-*

If you want to set a different custom header, don’t hesitate to contact us and we will be happy to set it up for you.

Changing Advanced Settings in Conductor Intelligence

1. Follow the path Settings > Crawling, and go to the Advanced tab.
2. Use the following sections to configure your advanced settings:
   • Monitoring From determines the monitoring location.
   • HTTP Auth lets you enter a Username and Password for the site.
   • Cookies allows you to add a Cookie Name and Value for one or more cookies.
   • HTTP Headers allows you to enter one or more HTTP Headers.

More Miscellaneous Settings

Conductor also allows you to configure the following aspects of its crawl:

Relevant Search Engine

This setting affects how Conductor reviews the robots.txt and meta robots rules on your website. Because search engines have different rules and regulations for how they crawl websites, you can specify whether Conductor should look at directives for certain search engines.

Also track H2-H6 changes

Conductor can include or exclude tracked changes to H2–H6 tags on your website. Keep in mind that setting this toggle to Yes may greatly increase the total number of tracked changes for each URL—and Conductor does limit the total number of tracked changes to 70,000.

Also track number-only changes

Conductor can include or exclude tracked changes to tags where only a number value has changed. This might be helpful for your site if you want ignore changes to numbers that frequently change, such as quantities, prices, or dates. Turning this setting off can decrease the overall number of tracked changes on your website.

Changing "More" Settings in Conductor Intelligence

1. Follow the path Settings > Crawling, and go to the More tab.
2. Use the following sections to configure these settings:
   • Relevant Search Engines
   • Track H2-H6 Changes
   • Track Number-Only Changes

FAQ