BigQuery agent for Triglav, data-driven workflow tool
MIT License
Triglav Agent for BigQuery
Add this line to your application's Gemfile:
gem 'triglav-agent-bigquery'
And then execute:
$ bundle
Or install it yourself as:
$ gem install triglav-agent-bigquery
Usage: triglav-agent-bigquery [options]
-c, --config VALUE Config file (default: config.yml)
-s, --status VALUE Status stroage file (default: status.yml)
-t, --token VALUE Triglav access token storage file (default: token.yml)
--dotenv Load environment variables from .env file (default: false)
-h, --help help
--log VALUE Log path (default: STDOUT)
--log-level VALUE Log level (default: info)
Run as:
TRIGLAV_ENV=development bundle exec triglav-agent-bigquery --dotenv -c config.yml
Prepare config.yml as example/config.yml.
You can use erb template. You may load environment variables from .env file with --dotenv
option as an example/example.env file shows.
You can specify any serverengine options at this section
Specify triglav api url, and a credential to authenticate.
The access token obtained is stored into a token storage file (--token option).
This section is the special section for triglav-agent-bigquery.
service_account
, authorized_user
(for oauth2), compute_engine
, and application_default
. Default obtains from credentials.credentials_file
, you may pass json contents as a stringResource URI must be a form of:
https://bigquery.cloud.google.com/table/#{project}:#{dataset}.#{table}
#{table}
also accepts strftime formatted suffix such as
#{table}_%Y%m%d
and strftime formatted partition decorator for a partitioned table such as
#{table}$%Y%m%d
monitor_interval
seconds:./prepare.sh
Edit .env
or config.yml
file directly.
Start up triglav api on localhost.
Run triglav-anget-bigquery as:
TRIGLAV_ENV=development bundle exec triglav-agent-bigquery --dotenv --debug -c example/config.yml
The debug mode with --debug option ignores the last_modified_time
value in status file.
Bug reports and pull requests are welcome on GitHub at https://github.com/triglav-dataflow/triglav-agent-bigquery. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the Contributor Covenant code of conduct.
The gem is available as open source under the terms of the MIT License.