detect anomal sequential input casually
To detect anomaly for log stream, use this plugin. Then you can find changes in logs casually.
Add this line to your application's Gemfile:
gem 'fluent-plugin-anomalydetect'
And then execute:
$ bundle
Or install it yourself as:
$ gem install fluent-plugin-anomalydetect
<source>
type file
...
tag access.log
</source>
<match access.**>
type anomalydetect
tag anomaly.access
tick 86400
</match>
<match anomaly.access>
type file
...
</match>
Then the plugin output anomaly log counts in each day.
This plugin watches a value of input record number in the interval set with tick
.
If you want to watch a value for a target field in data, write below:
<match access.**>
type anomalydetect
tag anomaly.access
tick 86400
target fieldname
</match>
<match access.**>
type anomalydetect
tag anomaly.access
tick 86400
target fieldname
outlier_term 7
outlier_discount 0.5
smooth_term 7
score_term 28
score_discount 0.01
</match>
If you want to know detail of these parameters, see "Theory".
<match access.**>
type anomalydetect
...
store_file /path/to/anomalydetect.dat
</match>
If "store_file" option was specified, a historical stat will be stored to the file at shutdown, and it will be restored on started.
<match access.**>
type anomalydetect
...
threshold 3
</match>
If "threshold" option was specified, plugin only ouput when the anomalyscore is more than threshold.
<match access.**>
type anomalydetect
...
trend up
</match>
If "trend" option was specified, plugin only ouput when the input data tends to up (or down).
outlier_term
outlier_discount
smooth_term
score_term
score_discount
tick
The time interval to watch in seconds.
tag
The output tag name. Required for aggregate all
. Default is anomaly
.
add_tag_prefix
Add tag prefix for output message. Required for aggregate tag
.
remove_tag_prefix
Remove tag prefix for output message.
aggragate
Process data for each tag
or all
. The default is all
.
target
Watch a value of a target field in data. If not specified, the number of records is watched (default). The output would become like:
{"outlier":1.783,"score":4.092,"target":10}
threshold
Emit message only if the score is greater than the threshold. Default is -1.0
.
trend
Emit message only if the input data trend is up
(or down
). Default is nil.
store_file
Store the learning results into a file, and reload it on restarting.
targets
Watch target fields in data. Specify by comma separated value like x,y
. The output messsages would be like:
{"x_outlier":1.783,"x_score":4.092,"x":10,"y_outlier":2.310,"y_score":3.982,"y":3}
thresholds
Threahold values for each target. Specify by comma separated value like 1.0,2.0
. Use with targets
option.
outlier_suffix
Change the suffix of emitted messages of targets
option. Default is _outlier
.
score_suffix
Change the suffix of emitted messages of targets
option. Default is _score
.
target_suffix
Change the suffix of emitted messages of targets
option. Default is `` (empty).
suppress_tick
Suppress to emit output messsages during specified seconds after starting up.
Copyright
License