Data Collection and Crawl FAQs
When does Conductor crawl and publish data?
Conductor crawls and collects data continually, but publishes data on a weekly basis. The data collection time period is Sunday to Saturday, though most collection occurs during Sunday through Tuesday. If Caliperbot is unable to crawl a page, it will attempt to crawl again during the current week's crawl period. If, after those two attempts, Caliperbot still cannot crawl the page, it will wait until the next week's crawl period to make its next attempt.
Conductor publishes collected data on Monday after the time period closes.
Keywords added to Conductor before 5:00 P.M. Eastern on Thursdays appear in Conductor on Monday. Keywords added after that time will appear in Conductor after the following time period closes and data is published on the following Monday.
The same schedule applies to details associated with keywords, such as preferred URLs.
Why does Conductor show that there is no title for a page for which I know there is a title?
There are a few possible answers to this question:
Allow Lists and Block Lists
Your organization may be blocking Conductor's web crawlers, which need to access your website so that they can record information from your pages' HTML, such as the title tag. For more information, refer to the What Is Caliperbot and how can I ensure it can crawl my site? FAQ.
No sessions or visits
If the page in question did not receive a session or visit during the most recent reporting period, Conductor's crawlers would not have crawled the page and therefore could not determine whether the page had a title or not.
404s
If the page in question redirects to a 404 page with no title, Conductor's will not be able to crawl the title from the original page and instead attempt to read the title from the 404 page.
Page encryption
One answer relates to the encryption associated with the page. When your organization configured an SSL certificate for your https protocol, it likely chose either a 1024 bit or 2048 bit encryption. Conductor supports up to a 1024 bit encryption. If your page uses a 2048 bit encryption, Conductor's web crawlers cannot read the page's HTML to determine a title, and cannot then report title in the platform.
Why does Conductor show a title for a page that is not the actual title tag on my page?
There are a few reasons you might see different titles. In some cases, the actual circumstance might be a combination of some of these reasons.
Google rewrote your title tag for your page on their results page
Google rewrites many title tags. As a result, what appears for your page on a search engine results page may differ from what your organization has designated as a title tag. Consequently, you might see these different title tags appear in Conductor across different reports, such as Keywords, Page Details, and Content Guidance.
You've updated a title tag, Conductor has crawled the page, but Google has not
When you make a change to your page (like updating a title tag) Google has to re-crawl the page to reflect that change in its results. Conductor crawls your pages weekly, so we might show your most recent updates before Google does—assuming they have not reindexed the page since you made those updates.
You researched a page in Content Guidance, and the page was updated since Conductor last crawled it
Content Guidance investigates pages in realtime. If you've made changes to your pages title tag since the last time Conductor crawled your page, Content Guidance might show a different title tag for your page from the title tag that appears in Page Insights.
The result does not have a title tag
Some results reported in the Keywords report are universal results, which don't include a page title on the SERP. Consider the following example: A standard result appears with the actual page title ("How StreamCo Streaming TV Works | StreamCo") and a People Also Ask result ("What channels are available on StreamCo?"), where we indicate the question related with that result.
The SERP does not show the actual page title in the People Also Ask result, only the question.
As a result, Conductor shows that question to provide context for the result.
Note this also means that you may see the same page appear in your ranking content for a single keyword.
Why can't Conductor crawl my page?
What Is Caliperbot?
To collect web data, Conductor uses a single web crawler that identifies itself as Caliperbot. Caliperbot crawls your site's HTML code to provide information and SEO recommendations in Conductor. As it crawls, Caliperbot takes note of on-page elements like title tags, header tags, and other metadata—just like Google's Googlebot does when it crawls your site to add it to their search index.
What prevents Caliperbot from crawling my page?
Frequently, we'll see customers that have sign-ins, pop-ups, or other interstitial user experiences that prevent Caliperbot from being able to crawl pages. Examples include:
- Sign-in forms.
- Cookie preference forms.
- Paywalls.
Identifying Caliperbot in Your Server Logs
To assess how Caliperbot interacts with your site, your IT team can use the following user agent string when evaluating your web server logs: Caliperbot/1.0 (+https://www.conductor.com/caliperbot)
Allowing Caliperbot
For Caliperbot to successfully collect your site data, your IT department should allow Conductor's IP address range: 162.246.176.1–162.246.183.254
Rate Limit Requests
By default, Caliperbot crawls at a 3–5 request per second crawl rate. If your IT team notices a high collection volume from caliperbot, Conductor can limit Caliperbot's collection rate for your website. To do so, send a request to Conductor Support with the request rate—in RPM (requests per minute)—that your IT team finds acceptable.
Questions?
If you have any questions about Conductor’s Caliperbot, contact Conductor Support.