Data exporters

The data exporters are the way to export alerts and events from Otoroshi to an external storage.

To try them, you can folllow this tutorial.

Common fields

  • Type: the type of event exporter
  • Enabled: enabled or not the exporter
  • Name: given name to the exporter
  • Description: the data exporter description
  • Tags: list of tags associated to the module
  • Metadata: list of metadata associated to the module

All exporters are split in three parts. The first and second parts are common and the last are specific by exporter.

  • Filtering and projection : section to filter the list of sent events and alerts. The projection field allows you to export only certain event fields and reduce the size of exported data. It’s composed of Filtering and Projection fields. To get a full usage of this elements, read this section
  • Queue details: set of fields to adjust the workers of the exporter.
  • Buffer size: if elements are pushed onto the queue faster than the source is consumed the overflow will be handled with a strategy specified by the user. Keep in memory the number of events.
  • JSON conversion workers: number of workers used to transform events to JSON format in paralell
  • Send workers: number of workers used to send transformed events
  • Group size: chunk up this stream into groups of elements received within a time window (the time window is the next field)
  • Group duration: waiting time before sending the group of events. If the group size is reached before the group duration, the events will be instantly sent

For the last part, the Exporter configuration will be detail individually.

Matching and projections

Filtering is used to include or exclude some kind of events and alerts. For each include and exclude field, you can add a list of key-value.

Let’s say we only want to keep Otoroshi alerts

{ "include": [{ "@type": "AlertEvent" }] }

Otoroshi provides a list of rules to keep only events with specific values. We will use the following event to illustrate.

{
 "foo": "bar",
 "type": "AlertEvent",
 "alert": "big-alert",
 "status": 200,
 "codes": ["a", "b"],
 "inner": {
   "foo": "bar",
   "bar": "foo"
 }
}

The rules apply with the previous example as event.

{
 "<key>": "<value>"
}
a field key with value
{
 "foo": "bar"
}
Keep events with foo as key and bar as value
{
 "<key>": {
  "$wildcard": "<value>*"
 }
}
a field starting with value
{
 "type": {
  "$wildcard": "Alert*"
 }
}
Keep events with type field starting with Alert
{
 "<key>": {
  "<key2>": "<value>"
 }
}
a sub-field with value
{
 "inner": {
  "foo": "bar"
 }
}
Keep events with sub field foo at value bar
{
 "<key>": "<number>"
}
a field with the specific value as number
{
 "status": 200
}
Keep events with status code at 200 (as number check)
{
 "<key>": {
  "$gt": "<number>"
 }
}
a field with number value greater than number
{
 "status": {
  "$gt": 100
 }
}
Keep events with status code greater than 100
{
 "<key>": {
  "$gte": "<number>"
 }
}
a field with number value greater or equal to number
{
 "status": {
  "$gte": 100
 }
}
Keep events with status code greater or equal to 100
{
 "<key>": {
  "$lt": "<number>"
 }
}
a field with number value lower than number
{
 "status": {
  "$lt": 100
 }
}
Keep events with status code lower than 100
{
 "<key>": {
  "$lte": "<number>"
 }
}
a field with number value lower or equal to number
{
 "status": {
  "$lte": 100
 }
}
Keep events with status code lower or equal to 100
{
 "<key>": {
  "$between": {
   "min": "<number>",
   "max": "<number>"
  }
 }
}
a field with value between two values (exclusive)
{
 "status": {
  "$between": {
   "min": 100,
   "max": 200
  }
 }
}
Keep events with status code between 100 and 200 (100 and 200 won't match)
{
 "<key>": {
  "$between": {
   "min": "<number>",
   "max": "<number>"
  }
 }
}
a field with value between two values (inclusive)
{
 "status": {
  "$between": {
   "min": 100,
   "max": 200
  }
 }
}
Keep events with status code between 100 and 200 (100 and 200 will match)
{
 "<key>": {
  "$and": [
   {
    "<key2>": "<value>"
   },
   {
    "<key3>": "<value>"
   }
  ]
 }
}
an object with two fields with values
{
 "inner": {
  "$and": [
   {
    "foo": "bar"
   },
   {
    "bar": "foo"
   }
  ]
 }
}
Keep events matching the list of key-value
{
 "$or": [
  {
   "<key2>": "<value>"
  },
  {
   "<key3>": "<value>"
  }
 ]
}
an object matching at least one condition of the list
{
 "$or": [
  {
   "method": "DELETE"
  },
  {
   "protocol": "http"
  }
 ]
}
Keep event whose method is http OR method is DELETE OR both
{
 "$nor": [
  {
   "<key2>": "<value>"
  },
  {
   "<key3>": "<value>"
  }
 ]
}
an object that matches no conditions of the list
{
 "$nor": [
  {
   "method": "DELETE"
  },
  {
   "protocol": "http"
  }
 ]
}
Keep events whose method is not DELETE AND protocol is not http
{
 "<key>": [
  "<value>",
  "<value2>"
 ]
}
an array field with values
{
 "codes": [
  "a",
  "b"
 ]
}
Keep events with an array codes which strictly containing values a and b
{
 "<key>": {
  "$contains": "<value>"
 }
}
an array field containing a specific value
{
 "codes": {
  "$contains": "a"
 }
}
Keep events with an array codes containing an a value
{
 "<key>": {
  "$contains": {
   "key": "<subkey>",
   "value": "<value>"
  }
 }
}
an object containing a key subkey with given value
{
 "target": {
  "$contains": {
   "key": "scheme",
   "value": "https"
  }
 }
}
Keep events whose target contains a field 'scheme' valued to 'https'
{
 "<key>": {
  "$all": [
   "<value>",
   "<value>"
  ]
 }
}
and array field containing all specific values
{
 "codes": {
  "$all": [
   "a",
   "b"
  ]
 }
}
Keep events with an array codes containing at minima a and b values
{
 "<key>": {
  "$regex": "<value>"
 }
}
a string field whose value match given regular expression
{
 "url": {
  "$regex": ".*api.*"
 }
}
Keep events with url containing 'api'
{
 "<key>": {
  "$in": [
   "<value>",
   "<value2>"
  ]
 }
}
a field containing one of the given value
{
 "method": {
  "$in": [
   "PUT",
   "POST"
  ]
 }
}
Keep events whose method is PUT or POST
{
 "<key>": {
  "$nin": [
   "<value>",
   "<value2>"
  ]
 }
}
a field containing none of the given value
{
 "method": {
  "$nin": [
   "PUT",
   "POST"
  ]
 }
}
Keep events whose method is neither PUT nor POST
{
 "<key>": {
  "$size": "<number>"
 }
}
an array field whose size is given value
{
 "headers": {
  "$size": 12
 }
}
Keep events with exactly 12 headers
{
 "<key>": {
  "$not": "<condition>"
 }
}
an object that does not satisfy condition
{
 "url": {
  "$not": {
   "$regex": ".*api.*"
  }
 }
}
Keep events whose url does not contain 'api'
{
 "<key>": {
  "$eq": "<value>"
 }
}
a field key with value
{
 "foo": {
  "$eq": "bar"
 }
}
Keep events with foo as key and bar as value
{
 "<key>": {
  "$ne": "<value>"
 }
}
a field key whose value is not provided value
{
 "foo": {
  "$ne": "bar"
 }
}
Keep events with foo field not equal to bar
{
 "<key>": {
  "$exists": "<entry>"
 }
}
an object field containing given entry as key
{
 "target": {
  "$exists": "scheme"
 }
}
Keep events whose target object contains a schema field

Projection is a list of fields to export. In the case of an empty list, all the fields of an event will be exported. In other case, only the listed fields will be exported.

"<field>": true

Description

{
 "<field>": true
}

Include given field in result.

Expression

{
 "foo": true
}

Event

{
 "headers": [
  {
   "key": 1,
   "value": "1"
  },
  {
   "key": 2,
   "value": "2"
  }
 ],
 "foo": 1
}

Result

{
 "foo": 1
}

$at

Description

{
 "<target>": {
  "$at": "<location>"
 }
}

Values <target> with value <at> location

Expression

{
 "h1": {
  "$at": "headers.0.value"
 }
}

Event

{
 "headers": [
  {
   "key": 1,
   "value": "1"
  },
  {
   "key": 2,
   "value": "2"
  }
 ]
}

Result

{
 "h1": "1"
}

$atIf

Description

{
 "<target>": {
  "$atIf": {
   "path": "<path>",
   "predicate": {
    "at": "<at>",
    "value": "<value>"
   }
  }
 }
}

Put <path> value in <target> if value at <at> match <value>

Expression

{
 "r": {
  "$atIf": {
   "path": "hs.0.value",
   "predicate": {
    "at": "scheme",
    "value": "HTTPS"
   }
  }
 }
}

Event

{
 "hs": [
  {
   "key": 1,
   "value": "1"
  },
  {
   "key": 2,
   "value": "2"
  }
 ],
 "scheme": "HTTPS"
}

Result

{
 "r": "1"
}

$pointer

Description

{
 "<target>": {
  "$pointer": "<jsonPointer>"
 }
}

Allow to get a json value using JSON pointer spec

Expression

{
 "h1": {
  "$pointer": "/headers/0/value"
 }
}

Event

{
 "headers": [
  {
   "key": 1,
   "value": "1"
  },
  {
   "key": 2,
   "value": "2"
  }
 ]
}

Result

{
 "h1": "1"
}

$pointerIf

Description

{
 "<target>": {
  "$pointerIf": {
   "path": "<path>",
   "predicate": {
    "pointer": "<pointer>",
    "value": "<value>"
   }
  }
 }
}

Put value at <path> in <target> field if and only if value at <pointer> equals <value>. <path> and <pointer> fields are resolved using json pointer spec.

Expression

{
 "h1": {
  "$pointerIf": {
   "path": "/headers/0/value",
   "predicate": {
    "pointer": "/scheme",
    "value": "HTTP"
   }
  }
 }
}

Event

{
 "headers": [
  {
   "key": 1,
   "value": "1"
  },
  {
   "key": 2,
   "value": "2"
  }
 ],
 "scheme": "HTTP"
}

Result

{
 "h1": "1"
}

$path

Description

{
 "<target>": {
  "$path": "<jsonPath>"
 }
}

Put value located at <jsonPath> in <target>. <jsonPath> is resolved using json path specification.

Expression

{
 "hs": {
  "$path": "$.headers[1:]"
 }
}

Event

{
 "headers": [
  {
   "key": "k1",
   "value": "v1"
  },
  {
   "key": "k2",
   "value": "v2"
  },
  {
   "key": "k3",
   "value": "v3"
  }
 ]
}

Result

{
 "hs": [
  {
   "key": "k2",
   "value": "v2"
  },
  {
   "key": "k3",
   "value": "v3"
  }
 ]
}

$pathIf

Description

{
 "<target>": {
  "$pathIf": {
   "path": "<jsonPath>",
   "predicate": {
    "path": "<predicateJsonPath>",
    "value": "<value>"
   }
  }
 }
}

Put value located at <jsonPath> in <target>, if and only if value at <predicateJsonPath> equals <value>.

Expression

{
 "test": {
  "$pathIf": {
   "path": "$.headers[1:]",
   "predicate": {
    "path": "$.scheme",
    "value": "HTTPS"
   }
  }
 }
}

Event

{
 "headers": [
  {
   "key": "k1",
   "value": "v1"
  },
  {
   "key": "k2",
   "value": "v2"
  },
  {
   "key": "k3",
   "value": "v3"
  }
 ],
 "scheme": "HTTPS"
}

Result

{
 "test": [
  {
   "key": "k2",
   "value": "v2"
  },
  {
   "key": "k3",
   "value": "v3"
  }
 ]
}

$compose (array result)

Description

{
 "<target>": {
  "$compose": [
   {
    "<subtarget>": {
     "$path": "<path>"
    }
   }
  ]
 }
}

Build an array in <target> field, by composing severals operators : $path, $at, $pointer.

Expression

{
 "result": {
  "$compose": [
   {
    "values": {
     "$path": "$.headers[0:].value"
    }
   },
   {
    "keys": {
     "$path": "$.headers[0:].key"
    }
   }
  ]
 }
}

Event

{
 "headers": [
  {
   "key": "k1",
   "value": "v1"
  },
  {
   "key": "k2",
   "value": "v2"
  },
  {
   "key": "k3",
   "value": "v3"
  }
 ],
 "scheme": "HTTPS"
}

Result

{
 "result": [
  {
   "values": [
    "v1",
    "v2",
    "v3"
   ]
  },
  {
   "keys": [
    "k1",
    "k2",
    "k3"
   ]
  }
 ]
}

$compose (object result)

Description

{
 "<target>": {
  "$compose": {
   "<subtarget>": {
    "$path": "<path>"
   }
  }
 }
}

Build an object in <target> field, by composing severals operators : $path, $at, $pointer.

Expression

{
 "result": {
  "$compose": {
   "values": {
    "$path": "$.headers[0:].value"
   },
   "keys": {
    "$path": "$.headers[0:].key"
   }
  }
 }
}

Event

{
 "headers": [
  {
   "key": "k1",
   "value": "v1"
  },
  {
   "key": "k2",
   "value": "v2"
  },
  {
   "key": "k3",
   "value": "v3"
  }
 ]
}

Result

{
 "result": {
  "values": [
   "v1",
   "v2",
   "v3"
  ],
  "keys": [
   "k1",
   "k2",
   "k3"
  ]
 }
}

$value

Description

{
 "<target>": {
  "$value": "<value>"
 }
}

Put <value> in <target> field. <value> can be either a primtive type, an object or an array.

Expression

{
 "headers": {
  "$value": []
 }
}

Event

{
 "headers": [
  {
   "key": "k1",
   "value": "v1"
  },
  {
   "key": "k2",
   "value": "v2"
  },
  {
   "key": "k3",
   "value": "v3"
  }
 ]
}

Result

{
 "headers": []
}

$spread

Description

{
 "$spread": true
}

Include all properties of source object, this can be used as base for futher transformation with below operators/

Expression

{
 "$spread": true
}

Event

{
 "headers": [
  {
   "key": "k1",
   "value": "v1"
  }
 ],
 "scheme": "HTTPS",
 "foo": 1
}

Result

{
 "headers": [
  {
   "key": "k1",
   "value": "v1"
  }
 ],
 "scheme": "HTTPS",
 "foo": 1
}

{"<field>": false}

Description

{
 "$spread": true,
 "<field>": false
}

Exclude <field> field from resulting object

Expression

{
 "$spread": true,
 "foo": false
}

Event

{
 "headers": [
  {
   "key": "k1",
   "value": "v1"
  }
 ],
 "scheme": "HTTPS",
 "foo": 1
}

Result

{
 "headers": [
  {
   "key": "k1",
   "value": "v1"
  }
 ],
 "scheme": "HTTPS"
}

$remove

Description

{
 "$spread": true,
 "<field>": {
  "$remove": true
 }
}

Exclude <field> field from resulting object, like for {"<field>": false}

Expression

{
 "$spread": true,
 "foo": {
  "$remove": true
 }
}

Event

{
 "headers": [
  {
   "key": "k1",
   "value": "v1"
  }
 ],
 "scheme": "HTTPS",
 "foo": 1
}

Result

{
 "headers": [
  {
   "key": "k1",
   "value": "v1"
  }
 ],
 "scheme": "HTTPS"
}

$header

Description

{
 "<target>": {
  "$header": {
   "path": "<path>",
   "name": "<name>"
  }
 }
}

Put specified header value in <target>. <path> indicate an array of key/value headers, <name> indicate the name of the header.

Expression

{
 "hostValue": {
  "$header": {
   "path": "headers",
   "name": "Host"
  }
 }
}

Event

{
 "headers": [
  {
   "key": "Host",
   "value": "otoroshi.oto.tools:9999"
  },
  {
   "key": "Accept",
   "value": "application/json"
  }
 ]
}

Result

{
 "hostValue": "otoroshi.oto.tools:9999"
}

$includeAllKeysMatching

Description

{
 "$spread": true,
 "<subname>": {
  "$includeAllKeysMatching": [
   "<expression>"
  ]
 }
}

Filter an object entries based on StartsWith, Wildcard or Regex expression. This filter must be used in a sub-object whose name <subname> doesn't matter, since it won't appear in resulting projection. This operator is an array, which mean you can have several filters. A logical OR will be applied between filters, therefore an entry just need to match one filter to be kept.

Expression

{
 "$spread": true,
 "_": {
  "$includeAllKeysMatching": [
   "Wildcard(f*)",
   "StartsWith(fi)",
   "bar",
   "Regex(.*lo)"
  ]
 }
}

Event

{
 "foo": 1,
 "foobar": 2,
 "bar": 3,
 "baz": 4,
 "hello": "world",
 "fifou": "test"
}

Result

{
 "foo": 1,
 "foobar": 2,
 "bar": 3,
 "hello": "world",
 "fifou": "test"
}

$excludeAllKeysMatching

Description

{
 "$spread": true,
 "<subname>": {
  "$excludeAllKeysMatching": [
   "<expression>"
  ]
 }
}

Filter an object entries, excluding them based on StartsWith, Wildcard or Regex expression. This filter must be used in a sub-object whose name <subname> doesn't matter, since it won't appear in resulting projection. This operator is an array, which mean you can have several filters. A logical OR will be applied between filters, therefore an entry just need to match one filter to be deleted.

Expression

{
 "$spread": true,
 "_": {
  "$excludeAllKeysMatching": [
   "Wildcard(fo*)",
   "StartsWith(fi)",
   "bar",
   "Regex(.*lo)"
  ]
 }
}

Event

{
 "foo": 1,
 "foobar": 2,
 "bar": 3,
 "baz": 4,
 "hello": "world",
 "fifou": "test"
}

Result

{
 "baz": 4
}

$jq

Description

{
 "<target>": {
  "$jq": "<jqExpression>"
 }
}

Fill target field with provided JQ selector

Expression

{
 "headerKeys": {
  "$jq": "[.headers[].key]"
 }
}

Event

{
 "headers": [
  {
   "key": "k1",
   "value": "v1"
  },
  {
   "key": "k2",
   "value": "v2"
  }
 ]
}

Result

{
 "headerKeys": [
  "k1",
  "k2"
 ]
}

$jqIf

Description

{
 "<target>": {
  "$jqIf": {
   "filter": "<jqExpression>",
   "predicate": {
    "path": "<path>",
    "value": "<value>"
   }
  }
 }
}

Fill target field with provided JQ selector if and only if value located at <path> is equal to <value>

Expression

{
 "headerKeys": {
  "$jqIf": {
   "filter": "[.headers[].key]",
   "predicate": {
    "path": "target.scheme",
    "value": "https"
   }
  }
 }
}

Event

{
 "headers": [
  {
   "key": "k1",
   "value": "v1"
  },
  {
   "key": "k2",
   "value": "v2"
  }
 ],
 "target": {
  "scheme": "https"
 }
}

Result

{
 "headerKeys": [
  "k1",
  "k2"
 ]
}

Elastic

With this kind of exporter, every matching event will be sent to an elastic cluster (in batch). It is quite useful and can be used in combination with elastic read in global config

  • Cluster URI: Elastic cluster URI
  • Index: Elastic index
  • Type: Event type (not needed for elasticsearch above 6.x)
  • User: Elastic User (optional)
  • Password: Elastic password (optional)
  • Version: Elastic version (optional, if none provided it will be fetched from cluster)
  • Apply template: Automatically apply index template
  • Check Connection: Button to test the configuration. It will displayed a modal with checked point, and if the case of it’s successfull, it will displayed the found version of the Elasticsearch and the index used
  • Manually apply index template: try to put the elasticsearch template by calling the api of elasticsearch
  • Show index template: try to retrieve the current index template presents in elasticsearch
  • Client side temporal indexes handling: When enabled, Otoroshi will manage the creation of indexes. When it’s disabled, Otoroshi will push in the same index
  • One index per: When the previous field is enabled, you can choose the interval of time between the creation of a new index in elasticsearch
  • Custom TLS Settings: Enable the TLS configuration for the communication with Elasticsearch
  • TLS loose: if enabled, will block all untrustful ssl configs
  • TrustAll: allows any server certificates even the self-signed ones
  • Client certificates: list of client certificates used to communicate with elasticsearch
  • Trusted certificates: list of trusted certificates received from elasticsearch

Webhook

With this kind of exporter, every matching event will be sent to a URL (in batch) using a POST method and an JSON array body.

  • Alerts hook URL: url used to post events
  • Hook Headers: headers add to the post request
  • Custom TLS Settings: Enable the TLS configuration for the communication with Elasticsearch
  • TLS loose: if enabled, will block all untrustful ssl configs
  • TrustAll: allows any server certificates even the self-signed ones
  • Client certificates: list of client certificates used to communicate with elasticsearch
  • Trusted certificates: list of trusted certificates received from elasticsearch

Pulsar

With this kind of exporter, every matching event will be sent to an Apache Pulsar topic

  • Pulsar URI: URI of the pulsar server
  • Custom TLS Settings: Enable the TLS configuration for the communication with Elasticsearch
  • TLS loose: if enabled, will block all untrustful ssl configs
  • TrustAll: allows any server certificates even the self-signed ones
  • Client certificates: list of client certificates used to communicate with elasticsearch
  • Trusted certificates: list of trusted certificates received from elasticsearch
  • Pulsar tenant: tenant on the pulsar server
  • Pulsar namespace: namespace on the pulsar server
  • Pulsar topic: topic on the pulsar server

Kafka

With this kind of exporter, every matching event will be sent to an Apache Kafka topic. You can find few tutorials about the connection between Otoroshi and Kafka based on docker images.

  • Kafka Servers: the list of servers to contact to connect the Kafka client with the Kafka cluster
  • Kafka topic: the topic on which Otoroshi alerts will be sent

By default, Kafka is installed with no authentication. Otoroshi supports the following authentication mechanisms and protocols for Kafka brokers.

SASL

The Simple Authentication and Security Layer (SASL) [RFC4422] is a method for adding authentication support to connection-based protocols.

  • SASL username: the client username
  • SASL password: the client username
  • SASL Mechanism:
    • PLAIN: SASL/PLAIN uses a simple username and password for authentication.
    • SCRAM-SHA-256 and SCRAM-SHA-512: SASL/SCRAM uses usernames and passwords stored in ZooKeeper. Credentials are created during installation.

SSL

  • Kafka keypass: the keystore password if you use a keystore/truststore to connect to Kafka cluster
  • Kafka keystore path: the keystore path on the server if you use a keystore/truststore to connect to Kafka cluster
  • Kafka truststore path: the truststore path on the server if you use a keystore/truststore to connect to Kafka cluster
  • Custom TLS Settings: enable the TLS configuration for the communication with Elasticsearch
    • TLS loose: if enabled, will block all untrustful ssl configs
    • TrustAll: allows any server certificates even the self-signed ones
    • Client certificates: list of client certificates used to communicate with elasticsearch
    • Trusted certificates: list of trusted certificates received from elasticsearch

SASL + SSL

This mechanism uses the SSL configuration and the SASL configuration.

Mailer

With this kind of exporter, every matching event will be sent in batch as an email (using one of the following email provider)

Otoroshi supports 5 exporters of email type.

Console

Nothing to add. The events will be write on the standard output.

Generic

  • Mailer url: URL used to push events
  • Headers: headers add to the push requests
  • Email addresses: recipients of the emails

Mailgun

  • EU: is EU server ? if enabled, *https://api.eu.mailgun.net/* will be used, otherwise, the US URL will be used : *https://api.mailgun.net/*
  • Mailgun api key: API key of the mailgun account
  • Mailgun domain: domain name of the mailgun account
  • Email addresses: recipients of the emails

Mailjet

  • Public api key: public key of the mailjet account
  • Private api key: private key of the mailjet account
  • Email addresses: recipients of the emails

Sendgrid

  • Sendgrid api key: api key of the sendgrid account
  • Email addresses: recipients of the emails

File

  • File path: path where the logs will be write
  • Max file size: when size is reached, Otoroshi will create a new file postfixed by the current timestamp

GoReplay file

With this kind of exporter, every matching event will be sent to a .gor file compatible with GoReplay.

Warning

this exporter will only be able to catch TrafficCaptureEvent. Those events are created when a route (or the global config) of the new proxy engine is setup to capture traffic using the capture flag.

  • File path: path where the logs will be write
  • Max file size: when size is reached, Otoroshi will create a new file postfixed by the current timestamp
  • Capture requests: capture http requests in the .gor file
  • Capture responses: capture http responses in the .gor file

Console

Nothing to add. The events will be write on the standard output.

Custom

This type of exporter let you the possibility to write your own exporter with your own rules. To create an exporter, we need to navigate to the plugins page, and to create a new item of type exporter.

When it’s done, the exporter will be visible in this list.

  • Exporter config.: the configuration of the custom exporter.

Metrics

This plugin is useful to rewrite the metric labels exposed on the /metrics endpoint.

  • Labels: list of metric labels. Each pair contains an existing field name and the new name.