Fluentd plugin to add or replace fields of a event record
MIT License
Fluentd plugin to add or replace fields of a event record
See .travis.yml
Note that fluent-plugin-record-reformer
supports both v0.14 API and v0.12 API in one gem.
Use RubyGems:
gem install fluent-plugin-record-reformer
Example:
<match foo.**>
type record_reformer
remove_keys remove_me
renew_record false
enable_ruby false
tag reformed.${tag_prefix[-2]}
<record>
hostname ${hostname}
input_tag ${tag}
last_tag ${tag_parts[-1]}
message ${record['message']}, yay!
</record>
</match>
Assume following input is coming (indented):
foo.bar {
"remove_me":"bar",
"not_remove_me":"bar",
"message":"Hello world!"
}
then output becomes as below (indented):
reformed.foo {
"not_remove_me":"bar",
"hostname":"YOUR_HOSTNAME",
"input_tag":"foo.bar",
"last_tag":"bar",
"message":"Hello world!, yay!",
}
Example:
<match foo.**>
type record_reformer
remove_keys remove_me
renew_record false
enable_ruby false
tag reformed.${tag_prefix[-2]}
hostname ${hostname}
input_tag ${tag}
last_tag ${tag_parts[-1]}
message ${record['message']}, yay!
</match>
This results in same, but please note that following option parameters are reserved, so can not be used as a record key.
output_tag (obsolete)
The output tag name. This option is deprecated. Use tag
option instead
tag
The output tag name.
remove_keys
Specify record keys to be removed by a string separated by , (comma) like
remove_keys message,foo
renew_record bool
renew_record true
creates an output record newly without extending (merging) the input record fields. Default is false
.
renew_time_key string
renew_time_key foo
overwrites the time of events with a value of the record field foo
if exists. The value of foo
must be a unix time.
keep_keys
You may want to remain some record fields although you specify renew_record true
. Then, specify record keys to be kept by a string separated by , (comma) like
keep_keys message,foo
enable_ruby bool
Enable to use ruby codes in placeholders. See Placeholders
section.
Default is true
(just for lower version compatibility).
auto_typecast bool
Automatically cast the field types. Default is false. NOTE: This option is effective only for field values comprised of a single placeholder.
Effective Examples:
foo ${foo}
Non-Effective Examples:
foo ${foo}${bar}
foo ${foo}bar
foo 1
Internally, this keeps the type of value if the value text is comprised of a single placeholder, otherwise, values are treated as strings.
When you need to cast field types manually, out_typecast and filter_typecast are available.
Following placeholders are available:
${record["key"]} Record value of key
such as ${record["message"]}
in the above example (available from v0.8.0).
${key}
such as ${message}
. This is still kept for the backward compatibility, but would be removed in the future.${hostname} Hostname of the running machine
${tag} Input tag
${time} Time of the event
${tags[N]} (Obsolete. Use tag_parts) Input tag splitted by '.'
${tag_parts[N]} Input tag splitted by '.' indexed with N such as ${tag_parts[0]}
, ${tag_parts[-1]}
.
${tag_prefix[N]} Tag parts before and on the index N. For example,
Input tag: prefix.test.tag.suffix
${tag_prefix[0]} => prefix
${tag_prefix[1]} => prefix.test
${tag_prefix[-2]} => prefix.test.tag
${tag_prefix[-1]} => prefix.test.tag.suffix
${tag_suffix[N]} Tag parts after and on the index N. For example,
Input tag: prefix.test.tag.suffix
${tag_suffix[0]} => prefix.test.tag.suffix
${tag_suffix[1]} => test.tag.suffix
${tag_suffix[-2]} => tag.suffix
${tag_suffix[-1]} => suffix
It is also possible to write a ruby code in placeholders if you set enable_ruby true
option, so you may write some codes as
but, please note that enabling ruby codes is not encouraged by security reasons and also in terms of the performance.
Following plugins look similar:
See CHANGELOG.md for details.
git checkout -b my-new-feature
)git commit -am 'Add some feature'
)git push origin my-new-feature
)Copyright (c) 2013 - 2015 Naotoshi Seo. See LICENSE for details.