Setting up Log File Analysis with the Akamai DataStream in Conductor Website Monitoring is very straightforward! Follow the steps outlined below to enable Log File Analysis in Conductor Website Monitoring, and set up the Akamai integration.
Setting up the Log File Analysis
The process of configuring the Log File Analysis consists of two phases:
- Enabling the Log File Analysis feature and creating an AWS S3 bucket in Conductor Website Monitoring
- Configuring the Akamai DataStream in the Akamai UI and associating the DataStream to the corresponding property
1. Enabling the Log File Analysis feature and creating an AWS S3 bucket in Conductor Website Monitoring
If you navigate to Account and then Websites section in Conductor Website Monitoring, you can easily filter and see which of the websites you are monitoring in Conductor Website Monitoring are on Akamai:
From there, just follow the steps outlined below to enable Log File Analysis on the wanted website:
- Click any website that is running on Akamai in the Websites section ↗️ of Conductor Website Monitoring.
- Click the Log File Analysis tab in Settings.
- Enable the Log File Analysis toggle.
After this is done, follow the next steps to create an AWS S3 bucket in Conductor Website Monitoring:
- In the same Log File Analysis section, click the How to install link under Log Sources.
- Specify in which region should the AWS S3 bucket be created (EU or US) and click on Create bucket.
- After this, Conductor Website Monitoring will automatically generate the AWS credentials and the AWS S3 bucket.
- The credentials will be used to associate the AWS S3 bucket with the Akamai DataStream in the next section.
2. Configuring the Akamai DataStream
The Akamai DataStream needs to be manually configured in the Akamai UI for every website. The configuration of the Akamai DataStream consists of two steps:
- Creating a new Akamai DataStream
- Associating the DataStream to the corresponding property (domain)
Creating a new Akamai DataStream
Follow these steps to create a new Akamai DataStream:
- Log in to your Akamai Control Center and click on DataStream in the navigation bar:
- On the DataStream screen, click Create stream:
-
Configure the DataStream: Specify the stream's name (e.g. "Conductor Website Monitoring LFA example.com") and pick the website for which you want to configure the Akamai DataStream in the Properties list:
- Configure the Data Sets:
Important:* The log format needs to be *JSON.
Choose the following required data sets to be sent to the AWS S3 bucket. All the fields listed below need to be selected for the Log File Analysis feature to work properly. You can also add other data sets, but the ones listed above are required.- Log information
- Request ID
- Request time
- Message exchange data
- Client IP
- HTTP status code
- Protocol type
- Request host
- Request method
- Request path
- Request port
- Response-Content-Type
- User-Agent
- SSL version
- Query string
- Log information
This is a highlight box. The borders that appear in the editor and this explainer will not appear in the published article. Use the Title and Body below for examples, best practices, or other information you'd like to call out.
We require streaming only non-PII (non-personally identifiable information) data to our AWS S3.
To do that, you can filter away non-search engine traffic based on the User Agent string when setting up the DataStream directly in Akamai UI.
This is covered in the section Filter away non-search engine traffic below.
End of Highlight box
- Choose Amazon S3 as the destination where the data will be streamed:
- Fill in the Display name (e.g. contentking-lfa-s3-example).
- Enter the Bucket name generated by Conductor Website Monitoring to the Bucket field.
- Path (default suggested by Akamai): logs/{%Y}/{%m}/{%d}/{%H}
- Enter the Region that you selected in Conductor Website Monitoring:
- If you've selected US: enter us-east-2 (located in Ohio)
- If you've selected EU: enter eu-central-1 (located in Frankfurt)
- Enter the Access key ID and Secret access key from Conductor Website Monitoring.
- Once done, click Validate & Save.
- Finish the setup and click Save stream:
This is a highlight box. The borders that appear in the editor and this explainer will not appear in the published article. Use the Title and Body below for examples, best practices, or other information you'd like to call out.
End of Highlight box
Associate the created DataStream to the corresponding property
After creating the new DataStream, you need to associate it with the corresponding property in your Akamai account.
- Log in to your Akamai Control Center and click on Properties in the navigation bar:
- Click the property which you want to associate with the created DataStream:
- Click the Property Version being monitored by Conductor Website Monitoring (this can be either the Staging Version or the Production Version):
-
Scroll to the DataStream section in Property Configuration Settings. If the DataStream section is missing, follow the steps below to add it. If not, skip to the step 5.
- Scroll all the way down, and click Edit new version.
- Scroll down to Property Configuration Settings, and click Add Behavior.
- Search for DataStream in the modal that appears and select it.
- Click Insert Behavior.
- Scroll all the way down, and click Edit new version.
- In the DataStream box in the Behaviors list, choose DataStream2 as the Stream version.
- In the DataStream2 section, enable the toggle to On, choose the Stream name of the new DataStream you created, and then click the Save button:
This is a highlight box. The borders that appear in the editor and this explainer will not appear in the published article. Use the Title and Body below for examples, best practices, or other information you'd like to call out.
Once this is done, ALL the traffic on your website will be streamed to Conductor Website Monitoring's AWS S3 bucket. However, Conductor Website Monitoring will process only search engine visits, and filter out all other traffic, including all PII data.
Follow the next steps to learn how to filter away non-search engine traffic directly in the Akamai DataStream that you've created in the previous steps.
These steps are required!
End of Highlight box
Ensure Logging of the User Agent header
- Go to Property Configuration Settings following steps 1 through 3 in the section Associate the created DataStream to the corresponding property.
- Scroll down to the Log Request Details section in Property Configuration Settings:
In case the Log Request Details section is not there, follow the same steps as within step 4. in the section Associate the created DataStream to the corresponding property to add it. - Enable the "Log User-Agent header" and click the Save button:
Filter away non-search engine traffic
-
Go to Property Configuration Settings following steps 1 through 3 in the section Associate the created DataStream to the corresponding property, and click Add Rule:
- Insert the rule for specific User Agents: choose a Blank rule template, then give it a name (e.g. "Enable DataStream for specific user agents", and click Insert rule:
-
Click Add Behavior and add DataStream behavior following steps 4.3 and 4.4. in Associate the created DataStream to the corresponding property:
-
Click Add Match and set up the criteria for the User Agent to match either the Googlebot or the bingbot user agent:Example:
User agent is one of:- Googlebot
- bingbot
- Finally, click Save:
This is a highlight box. The borders that appear in the editor and this explainer will not appear in the published article. Use the Title and Body below for examples, best practices, or other information you'd like to call out.
End of Highlight box
Reinstalling the Akamai DataStream
If you want to change the AWS S3 bucket region, or reinstall the Akamai DataStream, you can do it in the following way:
- Click on the website on which you want to reinstall the Akamai DataStream in the Websites section ↗️ of Conductor Website Monitoring.
- Click Log File Analysis in Settings, and then on the reinstall link under Log Sources.
- If needed change the region, and click Create bucket.
- Create a new DataStream by following the steps above.
Disabling Log File Analysis
Same as with enabling Log File Analysis, you need to disable the feature in Conductor Website Monitoring and then remove the Akamai DataStream in the Akamai UI.
Disabling Log File Analysis
- Click on the website on which you want to disable Log File Analysis in the Websites section ↗️ of Conductor Website Monitoring.
- Click the Log File Analysis in the Settings section.
- Disable the Log File Analysis toggle.
Once this is done, Conductor Website Monitoring will automatically disable Akamai DataStream access to the AWS S3 bucket where the data was streamed.
Removing the Akamai DataStream
If you have disabled the Log File Analysis feature, you still need to remove the Akamai DataStream in your Akamai account.
This needs to be done manually, as Conductor Website Monitoring doesn't have access to your Akamai account.
Security FAQs
For the most common security-related questions about Conductor Website Monitoring's Log File Analysis, refer to the FAQ section in the Log File Analysis support article