Export events to Elasticsearch
Before you start
If you already have an up and running otoroshi instance, you can skip the following instructions
Let’s start by downloading the latest Otoroshi.
curl -L -o otoroshi.jar 'https://github.com/MAIF/otoroshi/releases/download/v16.20.0-dev/otoroshi.jar'
then you can run start Otoroshi :
java -Dotoroshi.adminPassword=password -jar otoroshi.jar
Now you can log into Otoroshi at http://otoroshi.oto.tools:8080 with admin@otoroshi.io/password
Create a new route, exposed on http://myservice.oto.tools:8080
, which will forward all requests to the mirror https://request.otoroshi.io
. Each call to this service will returned the body and the headers received by the mirror.
curl -X POST 'http://otoroshi-api.oto.tools:8080/api/routes' \
-H "Content-type: application/json" \
-u admin-api-apikey-id:admin-api-apikey-secret \
-d @- <<'EOF'
{
"name": "my-service",
"frontend": {
"domains": ["myservice.oto.tools"]
},
"backend": {
"targets": [
{
"hostname": "request.otoroshi.io",
"port": 443,
"tls": true
}
]
}
}
EOF
Deploy a Elasticsearch and kibana stack on Docker
Let’s start by creating an Elasticsearch and Kibana stack on our machine (if it’s already done for you, you can skip this section).
To start an Elasticsearch container for development or testing, run:
docker network create elastic
docker pull docker.elastic.co/elasticsearch/elasticsearch:7.15.1
docker run --name es01-test --net elastic -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.15.1
docker pull docker.elastic.co/kibana/kibana:7.15.1
docker run --name kib01-test --net elastic -p 5601:5601 -e "ELASTICSEARCH_HOSTS=http://es01-test:9200" docker.elastic.co/kibana/kibana:7.15.1
To access Kibana, go to http://localhost:5601.
Create an Elasticsearch exporter
Let’s create an exporter. The exporter will export by default all events generated by Otoroshi.
- Go ahead, and navigate to http://otoroshi.oto.tools:8080
- Click on the cog icon on the top right
- Then
Exporters
button - And add a new configuration when clicking on the
Add item
button - Select the
elastic
in thetype
selector field - Jump to
Exporter config
- Set the following values:
Cluster URI
->http://localhost:9200
Then test your configuration by clicking on the Check connection
button. This should output a modal with the Elasticsearch version and the number of loaded docs.
Save at the bottom of the page and enable the exporter (on the top of the page or in list of exporters).
Testing your configuration
One simple way to test is to setup the reading of our Elasticsearch instance by Otoroshi.
Navigate to the danger zone (click on the cog on the top right and scroll to danger zone
). Jump to the Analytics: Elastic dashboard datasource (read)
section.
Set the following values : Cluster URI
-> http://localhost:9200
Then click on the Check connection
. This should ouput the same result as the previous part. Save the global configuration and navigate to http://otoroshi.oto.tools:8080/bo/dashboard/stats.
This should output a list of graphs.
Advanced usage
By default, an exporter handle all events from Otoroshi. In some case, you need to filter the events to send to elasticsearch.
To filter the events, jump to the Filtering and projection
field in exporter view. Otoroshi supports to include a kind of events or to exclude a list of events (if you want to deep learn about this section, read this part).
An example which keep only events with a field @type
of value AlertEvent
:
{
"include": [
{ "@type": "AlertEvent" }
],
"exclude": []
}
An example which exclude only events with a field @type
of value GatewayEvent
:
{
"exclude": [
{ "@type": "GatewayEvent" }
],
"include": []
}
The next field is the Projection. This field is a json when you can list the fields to keep for each event.
{
"@type": true,
"@timestamp": true,
"@id": true
}
With this example, only @type
, @timestamp
and @id
will be send to ES.
Debug your configuration
Missing user rights on Elasticsearch
When creating an exporter, Otoroshi try to join the index route of the elasticsearch instance. If you have a specific management access rights on Elasticsearch, you have two possiblities :
- set a full access to the user used in Otoroshi for write in Elasticsearch
- set the version of Elasticsearch inside the
Version
field of your exporter.
None event appear in your Elasticsearch
When creating an exporter, Otoroshi try to push the index template on Elasticsearch. If the post failed, Otoroshi will fail for each push of events and your database will keep empty.
To fix this problem, you can try to send the index template with the Manually apply index template
button in your exporter.