Django app for storing time-series metrics in Elasticsearch.
MIT License
Django app for storing time-series metrics in Elasticsearch.
pip install django-elasticsearch-metrics
Add "elasticseach_metrics"
to INSTALLED_APPS
.
INSTALLED_APPS += ["elasticsearch_metrics"]
Define the ELASTICSEARCH_DSL
setting.
ELASTICSEARCH_DSL = {"default": {"hosts": "localhost:9200"}}
This setting is passed to elasticsearch_dsl.connections.configure
so
it takes the same parameters.
In one of your apps, define a new metric in metrics.py
.
A Metric
is a subclass of elasticsearch_dsl.Document
.
# myapp/metrics.py
from elasticsearch_metrics import metrics
class PageView(metrics.Metric):
user_id = metrics.Integer(index=True, doc_values=True)
Use the sync_metrics
management command to ensure that the index template
for your metric is created in Elasticsearch.
# This will create an index template called myapp_pageview
python manage.py sync_metrics
Now add some data:
from myapp.metrics import PageView
user = User.objects.latest()
# By default we create an index for each day.
# Therefore, this will persist the document
# to an index called, e.g. "myapp_pageview_2020.02.04"
PageView.record(user_id=user.id)
Go forth and search!
# perform a search across all page views
PageView.search()
By default, an index is created for every day that a metric is saved.
You can change this to create an index per month or per year by changing
the ELASTICSEARCH_METRICS_DATE_FORMAT
setting.
# settings.py
# Monthly:
ELASTICSEARCH_METRICS_DATE_FORMAT = "%Y.%m"
# Yearly:
ELASTICSEARCH_METRICS_DATE_FORMAT = "%Y"
You can configure the index template settings by setting
Metric.Index.settings
.
class PageView(metrics.Metric):
user_id = metrics.Integer()
class Index:
settings = {"number_of_shards": 2, "refresh_interval": "5s"}
Each Metric
will have its own index template.
The index template name and glob pattern are computed from the app label
for the containing app and the class's name. For example, a PageView
class defined in myapp/metrics.py
will have an index template with the
name myapp_pageview
and a template glob pattern of myapp_pageview_*
.
If you declare a Metric
outside of an app, you will need to set
app_label
.
class PageView(metrics.Metric):
class Meta:
app_label = "myapp"
Alternatively, you can set template_name
and/or template
explicitly.
class PageView(metrics.Metric):
user_id = metrics.Integer()
class Meta:
template_name = "myapp_pviews"
template = "myapp_pviews_*"
from elasticsearch_metrics import metrics
class MyBaseMetric(metrics.Metric):
user_id = metrics.Integer()
class Meta:
abstract = True
class PageView(MyBaseMetric):
class Meta:
app_label = "myapp"
import factory
from elasticsearch_metrics.factory import MetricFactory
from ..myapp.metrics import MyMetric
class MyMetricFactory(MetricFactory):
my_int = factory.Faker("pyint")
class Meta:
model = MyMetric
def test_something():
metric = MyMetricFactory() # index metric in ES
assert isinstance(metric.my_int, int)
ELASTICSEARCH_DSL
: Required. Connection settings passed toelasticsearch_dsl.connections.configure
.ELASTICSEARCH_METRICS_DATE_FORMAT
: Date format to use when creating%Y.%m.%d
(same date format Elasticsearch uses forsync_metrics
: Ensure that index templates have been created forshow_metrics
: Pretty-print a listing of all registered metrics.check_metrics
: Check if index templates are in sync. ExitsSignals are located in the elasticsearch_metrics.signals
module.
pre_index_template_create(Metric, index_template, using)
: Sent before PUT
ting a new indexpost_index_template_create(Metric, index_template, using)
: Sent after PUT
ting a new indexpre_save(Metric, instance, using, index)
: Sent at the beginning of asave()
method.post_save(Metric, instance, using, index)
: Sent at the end of asave()
method._source
is disabled by default on metric indices in order to save_source
, you can override it in class Meta
.class MyMetric(metrics.Metric):
class Meta:
source = metrics.MetaField(enabled=True)
MIT Licensed.