BigQuery Ecosystem
Google BigQuery enables companies to handle large amounts of data without having to manage infrastructure. Google’s documentation describes it as a « serverless architecture (that) lets you use SQL queries to answer your organization’s biggest questions with zero infrastructure management. BigQuery’s scalable, distributed analysis engine lets you query terabytes in seconds and petabytes in minutes. » Its client libraries allow the use of widely known languages such as Python, Java, JavaScript, and Go. Federated queries are also supported, making it flexible to read data from external sources.
📖 A highly rated canonical book on it is « Google BigQuery: The Definitive Guide », a comprehensive reference. Another enriching read on the subject is the inside story told in the article by the founding product manager of BigQuery celebrating its 10th anniversary.
Core Projects
Moremicroservices-demo
terraformer
training-data-analyst
Popular Projects
More
sqlfluff
A modular SQL linter and auto-formatter with support for multiple dialects and templated code
airbyte
The leading data integration platform for ETL / ELT data pipelines from APIs, databases & files to data warehouses, data lakes & data lakehouses
graphql-engine
Blazing fast, instant realtime GraphQL APIs on your DB with fine grained access control, also trigger webhooks on database events
rudder-server
Privacy and Security focused Segment-alternative, in Golang and React
ingestr
ingestr is a CLI tool to copy data between any databases with a single command seamlessly
elementary
The dbt-native data observability solution for data & analytics engineers
Up and Coming Projects
Morepromptweaver
PromptWeaver streamlines prompt development and management in Generative AI workflows
vertex-ai-creative-studio
Creative Studio is a Vertex AI generative media example user experience to highlight the use of Imagen and other generative media APIs on Google Cloud
cluster-health-scanner
Google-Cloud-Professional-Data-Engineer-ACompleteGuide
This Repo contains all study, lab and supportive materials for Udemy course on "Google Cloud Professional Data Engineer - A Complete Guide"
cloudrun-gpus-opencv-cuda-demo
energy-data-analysis
This is the cloud model analyzing real world dataset with BigQuery and other big-data analyzing tools
terraform-gcp-datadog-integration
Terraform code to make the Google Cloud Platform to Datadog log collection integration easier
reCAPTCHA-PLD
ra_dbt_to_dataform
An open-source tool that partially automates the migration of dbt packages to Dataform
pypi-package-stats
Project for ingest pypi packages data from BigQuery and send to DataDog for analysis and insights with dashboards, monitors and more
tag-automator
Simple tool to easily visualize and manage tag bindings across organization, folder or project
enterprise-knowledge-solution
purdue
The internship was broadly to understand if the topics/events are being covered differently in the different countries and how they affect stock market returns
accelerated-platforms
fluentbit-bigquery-writeapi-sink