This code will parse cloudfront logs from s3 bucket and will send it to open search
APACHE-2.0 License
This project is a Serverless application that parses CloudFront logs stored in an S3 bucket and ingests them into an OpenSearch cluster. The application is built using Python and deployed using the Serverless Framework. CI/CD is managed with GitHub Actions.
Update the serverless.yml with below details
Add the following secrets to your GitHub repository settings:
MY_AWS_ACCESS_KEY
MY_AWS_SECRET_KEY
OPENSEARCH_HOST
OPENSEARCH_USERNAME
OPENSEARCH_PASSWORD
Deploy the Serverless application to AWS using deploy action
The Serverless configuration is set up to handle S3 events for the specified bucket. It uses the serverless-python-requirements plugin to manage Python dependencies.
Source Code src/cloudfront_parser.py: Contains the logic to parse CloudFront logs. src/lambda_handler.py: AWS Lambda handler to process S3 events, parse logs, and ingest into OpenSearch. src/version.py: Contains version information.
To run tests, use the following command:
pytest
git clone [email protected]:malikparvez/cloudfront-logs-parser-lambda-opensearch.git
cd cloudfront-logs-parser-lambda-opensearch
Install Python Dependencies: Ensure you have Python 3.11 installed. Install the necessary Python packages using pip:
pip3 install --target ./package -r requirements-lambda.txt
cd package
zip -r ../my_deployment_package.zip .
cd ..
cd src
zip -r ../my_deployment_package.zip cloudfront_parser.py lambda_handler.py version.py
Upload the generated zip file (my_deployment_package.zip) to AWS Lambda. Set the following environment variables in Lambda:
Set the following AWS and OpenSearch credentials as environment variables:
MY_AWS_ACCESS_KEY = ''
MY_AWS_SECRET_KEY = ''
OPENSEARCH_HOST = ''
OPENSEARCH_USERNAME = ''
OPENSEARCH_PASSWORD = ''
To automatically trigger the Lambda function when new CloudFront logs are added to your S3 bucket, follow these steps:
Now, whenever new CloudFront logs are added to the specified S3 bucket, the Lambda function will be automatically triggered to parse and send the data to OpenSearch.
Contributions are welcome! Please create a pull request or open an issue to discuss any changes.
This project is licensed under the MIT License. See the LICENSE file for details.