GCS Logshare Setup Script

Description

The main.sh script from your local machine creates a GCC VM that runs Cloudflare's Logshare tool on a cron schedule to push Enterprise Logs to BigQuery. Logs are written to GCS and then automatically pushed into BigQuery using the GCS-To-Big-Query code.

Defaults

1. The cron job pushed runs every 1 minute and pulls logs from 10 minutes ago to 11 minutes ago.
2. The ELS endpoint's fields are subject to change. At the time the VM is built, we cache a local version of the available fields. This is subject to change, but can be updated manually by modifying the fields.txt file.

Requirements

• jq
• Python 2.7.9
• curl

Setup Instructions

1. Select or create a Cloud Platform project:
   • https://console.cloud.google.com/
2. Clone the GCS Automation Script on your local machine:
   • git clone https://github.com/cloudflare/GCS-Logshare-Setup-Script.git
3. Enable the Service Management API for your project:
   • https://console.developers.google.com/apis/api/servicemanagement.googleapis.com/overview
4. Configure and enable your Google Billing profile:
   • https://support.google.com/cloud/answer/6293499#enable-billing
5. Enable the following Google APIs here:
   • Google Cloud Storage,
   • Google BigQuery,
   • Cloud Function
6. Create a copy of default.config.json and rename to config.json
   • mv config.default.json config.json
7. Modify config.json with your cloudflare account details
   • Cloudflare_api_key - Cloudflare API Key
   • Cloudflare_api_email - Cloudflare user account email address
   • Zone_name - domain name; example: mydomain.com
   • Gcs_project_id - Google Cloud Project ID
8. Run the main orchestration script:
   • bash main.sh