Setting Up the Fastly Integration

Setting up Log File Analysis with the Fastly Real-Time Log Streaming in Conductor Monitoring is very straightforward! Follow the steps outlined below to enable Log File Analysis in Conductor Monitoring, and set up the Fastly integration.

Setting up the Log File Analysis

The process of configuring the Log File Analysis consists of two phases:

• Enabling the Log File Analysis feature and creating an AWS S3 bucket in Conductor Monitoring.
• Configuring the Fastly Real-Time Log Streaming in the Fastly UI.

1. Enabling the Log File Analysis feature and creating an AWS S3 bucket in Conductor Monitoring

If you navigate to Account and then Websites section in Conductor Monitoring, you can easily filter and see which of the websites you are monitoring in Conductor Monitoring are on Fastly:

From there, just follow the steps outlined below to enable Log File Analysis on the wanted website:

1. Click any website that is running on Fastly in the Websites section of Conductor Monitoring.
2. Click the Log File Analysis tab in Settings.
3. Enable the Log File Analysis toggle.

After this is done, follow the next steps to create an AWS S3 bucket in Conductor Monitoring:

1. In the same Log File Analysis section, click the How to install link under the Log Sources.
2. Specify the region in which the AWS S3 bucket should be created (EU or US) and click on Create bucket.
3. After this, Conductor Monitoring will automatically generate the AWS credentials and the AWS S3 bucket.
4. Save the credentials as they will be used to associate the AWS S3 bucket with the Fastly Real-Time Log Streaming in the next section.

2. Configuring the Fastly Real-Time Log Streaming

The Fastly Real-Time Log Streaming needs to be manually configured in the Fastly UI for every website.

The configuration of the Fastly Real-Time Log Streaming consists of these steps:

1. Log in to your Fastly account and click on the website (service) for which you want to enable the Log File Analysis feature.
   
2. In the service summary screen, click Edit configuration.
   
3. In the left-hand navigation bar, click Logging, and then click Create endpoint.
   
4. In the Amazon S3 section click Create endpoint.
   
5. In the S3 endpoint details screen add the following settings:
   
   • Name: enter any name that would help easily identify the endpoint, e.g, "Conductor Monitoring LFA S3"
   • Placement: Format Version Default (this is the default, no change needed)

Log format: Paste the following format

• {
  "timestampMs": %{time.start.msec}V,
  "ip": "%{req.http.Fastly-Client-IP}V",
  "url": "%{if(req.is_ssl, "https", "http")}V://%v%U%q",
  "method": "%{json.escape(req.method)}V",
  "userAgent": "%{json.escape(req.http.User-Agent)}V",
  "host": "%{if(req.http.Fastly-Orig-Host, req.http.Fastly-Orig-Host, req.http.Host)}V"
  }

• Timestamp format: `%Y-%m-%dT%H:%M:%S.000` (this is the default one, no change needed)
• Bucket name: copy-paste here the bucket name from the modal in Conductor Monitoring that was generated in the steps above.
• Access Method: User Credentials
• Access key: copy-paste here the Access key from the modal in Conductor Monitoring that was generated in the steps above.
• Secret key: copy-paste here the Access key from the modal in Conductor Monitoring that was generated in the steps above.
• Period: 60 seconds.

1. Click Advanced options and edit more properties:
   
   1. Path: / (this is default, no need to change this)
   2. Domain: copy-paste here the Domain from the modal in Conductor Monitoring that was generated in the steps above.
   3. PGP public key: keep this empty (this is default, no need to change this)
   4. Select a log line format: Blank (this is default, no need to change this)
   5. Compression: Gzip
   6. The rest of the settings: keep the default ones
2. Click `Update`

Filter away non-search engine traffic

Setting up the response condition

To configure a response condition, so only search engine traffic is sent to the S3 bucket, follow the steps below.

1. In the left-hand navigation bar, select Conditions:
   
2. In the Condition screen, click Create condition:
   
3. In the Condition details screen, fill in the following condition:
   • Type: Response
   • Name: any name, e.g. "Is Crawler"

Apply if... Paste the exact condition below:

• req.http.user-agent ~ "bingbot" || req.http.user-agent ~ "Googlebot" || req.http.user-agent ~ "OAI-SearchBot" || req.http.user-agent ~ "ChatGPT-User" || req.http.user-agent ~ "GPTBot" || req.http.user-agent ~ "PerplexityBot" || req.http.user-agent ~ "Perplexity-User" 

(Note: These strings are case sensitive. Confirm you have the precise upper- and lower-case letters.)

1. Click Save.
   

Attaching the condition to the needed endpoint

To attach the condition to the created endpoint follow these steps:

1. Open the Logging endpoints screen by clicking on Logging in the left-hand navigation bar, and clicking Attach condition for the endpoint you've just created.
   
2. In the Add a condition screen, select the condition you've just set up:
   
3. To save all changes, just click Activate.
   

If everything has been configured correctly, you should see a success message.

Reinstalling the Fastly Real-Time Log Streaming

If you want to change the AWS S3 bucket region, or reinstall the Fastly Real-Time Log Steaming, you can do it in the following way:

1. Click on the website on which you want to reinstall the Fastly Real-Time Log Steaming in the Websites section of Conductor Monitoring.
2. Click Log File Analysis in Settings, and then on the reinstall link under Log Sources.
3. If needed change the region, and click Create bucket.
4. Create a new Real-Time Log stream by following the steps above

Disabling Log File Analysis

Same as with enabling Log File Analysis, you need to disable the feature in Conductor Monitoring and then remove the Fastly Real-Time Log Streaming in the Fastly UI.

Disabling Log File Analysis

1. Click on the website on which you want to disable Log File Analysis in the Websites section of Conductor Monitoring.
2. Click Log File Analysis in the Settings section.
3. Disable the Log File Analysis toggle.

Once this is done, Conductor Monitoring will automatically disable Fastly Real-Time Log Streaming access to the AWS S3 bucket where the data was streamed.

Removing the Fastly Real-Time Log Streaming

If you have disabled the Log File Analysis feature, you still need to remove the Fastly Real-Time Log Streaming in your Fastly account.

This needs to be done manually, as Conductor Monitoring doesn't have access to your Fastly account.

Security FAQs

For the most common security-related questions about Conductor Monitoring's Log File Analysis, refer to the Log File Analysis FAQ article.