Kubernetes
Starting at version 1.5.0, Otoroshi provides a native Kubernetes support. Multiple otoroshi jobs (that are actually kubernetes controllers) are provided in order to
- sync kubernetes secrets of type
kubernetes.io/tls
to otoroshi certificates - act as a standard ingress controller (supporting
Ingress
objects) - provide Custom Resource Definitions (CRDs) to manage Otoroshi entities from Kubernetes and act as an ingress controller with its own resources
Installing otoroshi on your kubernetes cluster
You need to have cluster admin privileges to install otoroshi and its service account, role mapping and CRDs on a kubernetes cluster. We also advise you to create a dedicated namespace (you can name it otoroshi
for example) to install otoroshi
If you want to deploy otoroshi into your kubernetes cluster, you can download the deployment descriptors from https://github.com/MAIF/otoroshi/tree/master/kubernetes and use kustomize to create your own overlay.
You can also create a kustomization.yaml
file with a remote base
bases:
- github.com/MAIF/otoroshi/kubernetes/kustomize/overlays/simple/?ref=v16.20.0-dev
Then deploy it with kubectl apply -k ./overlays/myoverlay
.
You can also use Helm to deploy a simple otoroshi cluster on your kubernetes cluster
helm repo add otoroshi https://maif.github.io/otoroshi/helm
helm install my-otoroshi otoroshi/otoroshi
Below, you will find example of deployment. Do not hesitate to adapt them to your needs. Those descriptors have value placeholders that you will need to replace with actual values like
env:
- name: APP_STORAGE_ROOT
value: otoroshi
- name: APP_DOMAIN
value: ${domain}
you will have to edit it to make it look like
env:
- name: APP_STORAGE_ROOT
value: otoroshi
- name: APP_DOMAIN
value: 'apis.my.domain'
if you don’t want to use placeholders and environment variables, you can create a secret containing the configuration file of otoroshi
apiVersion: v1
kind: Secret
metadata:
name: otoroshi-config
type: Opaque
stringData:
oto.conf: >
include "application.conf"
otoroshi {
storage = "redis"
domain = "apis.my.domain"
}
and mount it in the otoroshi container
apiVersion: apps/v1
kind: Deployment
metadata:
name: otoroshi-deployment
spec:
selector:
matchLabels:
run: otoroshi-deployment
template:
metadata:
labels:
run: otoroshi-deployment
spec:
serviceAccountName: otoroshi-admin-user
terminationGracePeriodSeconds: 60
hostNetwork: false
containers:
- image: maif/otoroshi:16.20.0-dev
imagePullPolicy: IfNotPresent
name: otoroshi
args: ['-Dconfig.file=/usr/app/otoroshi/conf/oto.conf']
ports:
- containerPort: 8080
name: "http"
protocol: TCP
- containerPort: 8443
name: "https"
protocol: TCP
volumeMounts:
- name: otoroshi-config
mountPath: "/usr/app/otoroshi/conf"
readOnly: true
volumes:
- name: otoroshi-config
secret:
secretName: otoroshi-config
...
You can also create several secrets for each placeholder, mount them to the otoroshi container then use their file path as value
env:
- name: APP_STORAGE_ROOT
value: otoroshi
- name: APP_DOMAIN
value: 'file:///the/path/of/the/secret/file'
you can use the same trick in the config. file itself
Note on bare metal kubernetes cluster installation
Bare metal kubernetes clusters don’t come with support for external loadbalancers (service of type LoadBalancer
). So you will have to provide this feature in order to route external TCP traffic to Otoroshi containers running inside the kubernetes cluster. You can use projects like MetalLB that provide software LoadBalancer
services to bare metal clusters or you can use and customize examples below.
We don’t recommand running Otoroshi behind an existing ingress controller (or something like that) as you will not be able to use features like TCP proxying, TLS, mTLS, etc. Also, this additional layer of reverse proxy will increase call latencies.
Common manifests
the following manifests are always needed. They create otoroshi CRDs, tokens, role, etc. Redis deployment is not mandatory, it’s just an example. You can use your own existing setup.
When updating an existing otoroshi cluster, new kubernetes entities can be expected to be available by the kubernetes plugins. So it could be helpful to check if that’s the case before deploying the new version. If you want to automate this process, you can check the following section
- rbac.yaml
-
--- kind: ServiceAccount apiVersion: v1 metadata: name: otoroshi-admin-user --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: name: otoroshi-admin-user roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: otoroshi-admin-user subjects: - kind: ServiceAccount name: otoroshi-admin-user namespace: $namespace --- kind: ClusterRole apiVersion: rbac.authorization.k8s.io/v1 metadata: name: otoroshi-admin-user rules: - apiGroups: - "" resources: - services - endpoints - secrets - configmaps - deployments - namespaces - pods verbs: - get - list - watch - apiGroups: - "apps" resources: - deployments verbs: - get - list - watch - apiGroups: - "" resources: - secrets - configmaps verbs: - update - create - delete - apiGroups: - extensions resources: - ingresses - ingressclasses verbs: - get - list - watch - apiGroups: - extensions resources: - ingresses/status verbs: - update - apiGroups: - admissionregistration.k8s.io resources: - validatingwebhookconfigurations - mutatingwebhookconfigurations verbs: - get - update - patch - apiGroups: - proxy.otoroshi.io resources: - routes - backends - route-compositions - service-descriptors - tcp-services - error-templates - apikeys - certificates - jwt-verifiers - auth-modules - admin-sessions - admins - auth-module-users - service-groups - organizations - tenants - teams - data-exporters - scripts - wasm-plugins - global-configs - green-scores - coraza-configs - http-listeners verbs: - get - list - watch
- crds.yaml
- redis.yaml
Updating rbac and crds when upgrading Otoroshi using otoroshictl
Updating rbac and crds definition for an Otoroshi upgrade can be tedious and is quite a manual process. But it can be largely simplified by using otoroshictl from our friends at Cloud APIM.
otoroshictl
has some commands to automate the creation for rbac.yaml
and crds.yaml
for your otoroshi deployments. Using it is quite handy as it will also generate the descriptor for any custom extension your using on your Otoroshi instance.
First add your otoroshi instance to otoroshictl
(it may be already done, in that case, just use it with otoroshictl config use new-cluster
).
$ otoroshictl config add new-cluster \
--current \
--hostname otoroshi.foo.bar \
--port 8443 \
--tls \
--client-id xxx \
--client-secret xxxxx
then you can generate the crds.yaml
file using the command
$ otoroshictl resources crds > crds.yaml
and you can generate the rbac.yaml
file using the command
$ otoroshictl resources rbac --namespace mynamespace > rbac.yaml
you can also directly apply it on your kubernetes cluster
$ otoroshictl resources rbac --namespace mynamespace | kubectl apply -f -
$ otoroshictl resources crds | kubectl apply -f -
Deploy a simple otoroshi instanciation on a cloud provider managed kubernetes cluster
Here we have 2 replicas connected to the same redis instance. Nothing fancy. We use a service of type LoadBalancer
to expose otoroshi to the rest of the world. You have to setup your DNS to bind otoroshi domain names to the LoadBalancer
external CNAME
(see the example below)
- deployment.yaml
-
apiVersion: apps/v1 kind: Deployment metadata: name: otoroshi-deployment spec: selector: matchLabels: run: otoroshi-deployment replicas: 2 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 template: metadata: labels: run: otoroshi-deployment spec: serviceAccountName: otoroshi-admin-user terminationGracePeriodSeconds: 60 hostNetwork: false containers: - image: maif/otoroshi:16.20.0-dev-jdk11 imagePullPolicy: IfNotPresent name: otoroshi ports: - containerPort: 8080 name: "http" protocol: TCP - containerPort: 8443 name: "https" protocol: TCP env: - name: APP_STORAGE_ROOT value: otoroshi - name: OTOROSHI_INITIAL_ADMIN_PASSWORD value: ${password} - name: APP_DOMAIN value: ${domain} - name: APP_STORAGE value: lettuce - name: REDIS_URL value: ${redisUrl} # value: redis://redis-leader-service:6379/0 - name: ADMIN_API_CLIENT_ID value: ${clientId} - name: ADMIN_API_CLIENT_SECRET value: ${clientSecret} - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN value: otoroshi-api-service.${namespace}.svc.cluster.local - name: OTOROSHI_SECRET value: ${otoroshiSecret} - name: HEALTH_LIMIT value: "5000" - name: SSL_OUTSIDE_CLIENT_AUTH value: Want - name: HTTPS_WANT_CLIENT_AUTH value: "true" - name: OTOROSHI_EXPOSED_PORTS_HTTP value: "80" - name: OTOROSHI_EXPOSED_PORTS_HTTPS value: "443" - name: OTOROSHI_INITIAL_CUSTOMIZATION value: > { "config":{ "tlsSettings": { "defaultDomain": "www.${domain}", "randomIfNotFound": false }, "scripts":{ "enabled":true, "sinkRefs":[ "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator", "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector" ], "sinkConfig": {}, "jobRefs":[ "cp:otoroshi.plugins.jobs.kubernetes.KubernetesOtoroshiCRDsControllerJob" ], "jobConfig":{ "KubernetesConfig": { "trust": false, "namespaces": [ "*" ], "labels": {}, "namespacesLabels": {}, "ingressClasses": [ "otoroshi" ], "defaultGroup": "default", "ingresses": false, "crds": true, "coreDnsIntegration": false, "coreDnsIntegrationDryRun": false, "kubeLeader": false, "restartDependantDeployments": false, "watch": false, "syncDaikokuApikeysOnly": false, "kubeSystemNamespace": "kube-system", "coreDnsConfigMapName": "coredns", "coreDnsDeploymentName": "coredns", "corednsPort": 53, "otoroshiServiceName": "otoroshi-service", "otoroshiNamespace": "${namespace}", "clusterDomain": "cluster.local", "syncIntervalSeconds": 60, "coreDnsEnv": null, "watchTimeoutSeconds": 60, "watchGracePeriodSeconds": 5, "mutatingWebhookName": "otoroshi-admission-webhook-injector", "validatingWebhookName": "otoroshi-admission-webhook-validation", "templates": { "service-group": {}, "service-descriptor": {}, "apikeys": {}, "global-config": {}, "jwt-verifier": {}, "tcp-service": {}, "certificate": {}, "auth-module": {}, "script": {}, "organizations": {}, "teams": {}, "webhooks": { "flags": { "requestCert": true, "originCheck": true, "tokensCheck": true, "displayEnv": false, "tlsTrace": false } } } } } } } } - name: JAVA_OPTS value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0' readinessProbe: httpGet: path: /ready port: 8080 failureThreshold: 1 initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 livenessProbe: httpGet: path: /live port: 8080 failureThreshold: 3 initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 resources: # requests: # cpu: "100m" # memory: "50Mi" # limits: # cpu: "4G" # memory: "4Gi" --- apiVersion: v1 kind: Service metadata: name: otoroshi-service spec: selector: run: otoroshi-deployment ports: - port: 8080 name: "http" targetPort: "http" - port: 8443 name: "https" targetPort: "https" --- apiVersion: v1 kind: Service metadata: name: otoroshi-external-service spec: type: LoadBalancer selector: run: otoroshi-deployment ports: - port: 80 name: "http" targetPort: "http" - port: 443 name: "https" targetPort: "https" --- apiVersion: proxy.otoroshi.io/v1 kind: Certificate metadata: name: otoroshi-service-certificate spec: description: certificate for otoroshi-service autoRenew: true csr: issuer: CN=Otoroshi Root hosts: - otoroshi-service - otoroshi-service.${namespace}.svc.cluster.local - otoroshi-api-service.${namespace}.svc.cluster.local - otoroshi.${domain} - otoroshi-api.${domain} - privateapps.${domain} key: algo: rsa size: 2048 subject: uid=otoroshi-service-cert, O=Otoroshi client: false ca: false duration: 31536000000 signatureAlg: SHA256WithRSAEncryption digestAlg: SHA-256
- dns.example
Deploy a simple otoroshi instanciation on a bare metal kubernetes cluster
Here we have 2 replicas connected to the same redis instance. Nothing fancy. The otoroshi instance are exposed as nodePort
so you’ll have to add a loadbalancer in front of your kubernetes nodes to route external traffic (TCP) to your otoroshi instances. You have to setup your DNS to bind otoroshi domain names to your loadbalancer (see the example below).
- deployment.yaml
-
apiVersion: apps/v1 kind: Deployment metadata: name: otoroshi-deployment spec: selector: matchLabels: run: otoroshi-deployment replicas: 2 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 template: metadata: labels: run: otoroshi-deployment spec: serviceAccountName: otoroshi-admin-user terminationGracePeriodSeconds: 60 hostNetwork: false containers: - image: maif/otoroshi:16.20.0-dev-jdk11 imagePullPolicy: IfNotPresent name: otoroshi ports: - containerPort: 8080 name: "http" protocol: TCP - containerPort: 8443 name: "https" protocol: TCP env: - name: APP_STORAGE_ROOT value: otoroshi - name: OTOROSHI_INITIAL_ADMIN_PASSWORD value: ${password} - name: APP_DOMAIN value: ${domain} - name: APP_STORAGE value: lettuce - name: REDIS_URL value: ${redisUrl} # value: redis://redis-leader-service:6379/0 - name: ADMIN_API_CLIENT_ID value: ${clientId} - name: ADMIN_API_CLIENT_SECRET value: ${clientSecret} - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN value: otoroshi-api-service.${namespace}.svc.cluster.local - name: OTOROSHI_SECRET value: ${otoroshiSecret} - name: HEALTH_LIMIT value: "5000" - name: SSL_OUTSIDE_CLIENT_AUTH value: Want - name: HTTPS_WANT_CLIENT_AUTH value: "true" - name: OTOROSHI_EXPOSED_PORTS_HTTP value: "80" - name: OTOROSHI_EXPOSED_PORTS_HTTPS value: "443" - name: OTOROSHI_INITIAL_CUSTOMIZATION value: > { "config":{ "tlsSettings": { "defaultDomain": "www.${domain}", "randomIfNotFound": false }, "scripts":{ "enabled":true, "sinkRefs":[ "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator", "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector" ], "sinkConfig": {}, "jobRefs":[ "cp:otoroshi.plugins.jobs.kubernetes.KubernetesOtoroshiCRDsControllerJob" ], "jobConfig":{ "KubernetesConfig": { "trust": false, "namespaces": [ "*" ], "labels": {}, "namespacesLabels": {}, "ingressClasses": [ "otoroshi" ], "defaultGroup": "default", "ingresses": false, "crds": true, "coreDnsIntegration": false, "coreDnsIntegrationDryRun": false, "kubeLeader": false, "restartDependantDeployments": false, "watch": false, "syncDaikokuApikeysOnly": false, "kubeSystemNamespace": "kube-system", "coreDnsConfigMapName": "coredns", "coreDnsDeploymentName": "coredns", "corednsPort": 53, "otoroshiServiceName": "otoroshi-service", "otoroshiNamespace": "${namespace}", "clusterDomain": "cluster.local", "syncIntervalSeconds": 60, "coreDnsEnv": null, "watchTimeoutSeconds": 60, "watchGracePeriodSeconds": 5, "mutatingWebhookName": "otoroshi-admission-webhook-injector", "validatingWebhookName": "otoroshi-admission-webhook-validation", "templates": { "service-group": {}, "service-descriptor": {}, "apikeys": {}, "global-config": {}, "jwt-verifier": {}, "tcp-service": {}, "certificate": {}, "auth-module": {}, "script": {}, "organizations": {}, "teams": {}, "webhooks": { "flags": { "requestCert": true, "originCheck": true, "tokensCheck": true, "displayEnv": false, "tlsTrace": false } } } } } } } } - name: JAVA_OPTS value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0' readinessProbe: httpGet: path: /ready port: 8080 failureThreshold: 1 initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 livenessProbe: httpGet: path: /live port: 8080 failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 resources: # requests: # cpu: "100m" # memory: "50Mi" # limits: # cpu: "4G" # memory: "4Gi" --- apiVersion: v1 kind: Service metadata: name: otoroshi-service spec: selector: run: otoroshi-deployment ports: - port: 8080 name: "http" targetPort: "http" - port: 8443 name: "https" targetPort: "https" --- apiVersion: v1 kind: Service metadata: name: otoroshi-external-service spec: selector: run: otoroshi-deployment ports: - port: 80 name: "http" targetPort: "http" nodePort: 31080 - port: 443 name: "https" targetPort: "https" nodePort: 31443 --- apiVersion: proxy.otoroshi.io/v1 kind: Certificate metadata: name: otoroshi-service-certificate spec: description: certificate for otoroshi-service autoRenew: true csr: issuer: CN=Otoroshi Root hosts: - otoroshi-service - otoroshi-service.${namespace}.svc.cluster.local - otoroshi-api-service.${namespace}.svc.cluster.local - otoroshi.${domain} - otoroshi-api.${domain} - privateapps.${domain} key: algo: rsa size: 2048 subject: uid=otoroshi-service-cert, O=Otoroshi client: false ca: false duration: 31536000000 signatureAlg: SHA256WithRSAEncryption digestAlg: SHA-256
- haproxy.example
- nginx.example
- dns.example
Deploy a simple otoroshi instanciation on a bare metal kubernetes cluster using a DaemonSet
Here we have one otoroshi instance on each kubernetes node (with the otoroshi-kind: instance
label) with redis persistance. The otoroshi instances are exposed as hostPort
so you’ll have to add a loadbalancer in front of your kubernetes nodes to route external traffic (TCP) to your otoroshi instances. You have to setup your DNS to bind otoroshi domain names to your loadbalancer (see the example below).
- deployment.yaml
-
apiVersion: apps/v1 kind: DaemonSet metadata: name: otoroshi-deployment spec: selector: matchLabels: run: otoroshi-deployment template: metadata: labels: run: otoroshi-deployment spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: otoroshi-kind operator: In values: - instance tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule serviceAccountName: otoroshi-admin-user terminationGracePeriodSeconds: 60 restartPolicy: Always hostNetwork: false containers: - image: maif/otoroshi:16.20.0-dev-jdk11 imagePullPolicy: IfNotPresent name: otoroshi ports: - containerPort: 8080 hostPort: 41080 name: "http" protocol: TCP - containerPort: 8443 hostPort: 41443 name: "https" protocol: TCP env: - name: APP_STORAGE_ROOT value: otoroshi - name: OTOROSHI_INITIAL_ADMIN_PASSWORD value: ${password} - name: APP_DOMAIN value: ${domain} - name: APP_STORAGE value: lettuce - name: REDIS_URL value: ${redisUrl} # value: redis://redis-leader-service:6379/0 - name: ADMIN_API_CLIENT_ID value: ${clientId} - name: ADMIN_API_CLIENT_SECRET value: ${clientSecret} - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN value: otoroshi-api-service.${namespace}.svc.cluster.local - name: OTOROSHI_SECRET value: ${otoroshiSecret} - name: HEALTH_LIMIT value: "5000" - name: SSL_OUTSIDE_CLIENT_AUTH value: Want - name: HTTPS_WANT_CLIENT_AUTH value: "true" - name: OTOROSHI_EXPOSED_PORTS_HTTP value: "80" - name: OTOROSHI_EXPOSED_PORTS_HTTPS value: "443" - name: OTOROSHI_INITIAL_CUSTOMIZATION value: > { "config":{ "tlsSettings": { "defaultDomain": "www.${domain}", "randomIfNotFound": false }, "scripts":{ "enabled":true, "sinkRefs":[ "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator", "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector" ], "sinkConfig": {}, "jobRefs":[ "cp:otoroshi.plugins.jobs.kubernetes.KubernetesOtoroshiCRDsControllerJob" ], "jobConfig":{ "KubernetesConfig": { "trust": false, "namespaces": [ "*" ], "labels": {}, "namespacesLabels": {}, "ingressClasses": [ "otoroshi" ], "defaultGroup": "default", "ingresses": false, "crds": true, "coreDnsIntegration": false, "coreDnsIntegrationDryRun": false, "kubeLeader": false, "restartDependantDeployments": false, "watch": false, "syncDaikokuApikeysOnly": false, "kubeSystemNamespace": "kube-system", "coreDnsConfigMapName": "coredns", "coreDnsDeploymentName": "coredns", "corednsPort": 53, "otoroshiServiceName": "otoroshi-service", "otoroshiNamespace": "${namespace}", "clusterDomain": "cluster.local", "syncIntervalSeconds": 60, "coreDnsEnv": null, "watchTimeoutSeconds": 60, "watchGracePeriodSeconds": 5, "mutatingWebhookName": "otoroshi-admission-webhook-injector", "validatingWebhookName": "otoroshi-admission-webhook-validation", "templates": { "service-group": {}, "service-descriptor": {}, "apikeys": {}, "global-config": {}, "jwt-verifier": {}, "tcp-service": {}, "certificate": {}, "auth-module": {}, "script": {}, "organizations": {}, "teams": {}, "webhooks": { "flags": { "requestCert": true, "originCheck": true, "tokensCheck": true, "displayEnv": false, "tlsTrace": false } } } } } } } } - name: JAVA_OPTS value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0' readinessProbe: httpGet: path: /ready port: 8080 failureThreshold: 1 initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 livenessProbe: httpGet: path: /live port: 8080 failureThreshold: 3 initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 resources: # requests: # cpu: "100m" # memory: "50Mi" # limits: # cpu: "4G" # memory: "4Gi" --- apiVersion: v1 kind: Service metadata: name: otoroshi-service spec: selector: run: otoroshi-deployment ports: - port: 8080 name: "http" targetPort: "http" - port: 8443 name: "https" targetPort: "https" --- apiVersion: proxy.otoroshi.io/v1 kind: Certificate metadata: name: otoroshi-service-certificate spec: description: certificate for otoroshi-service autoRenew: true csr: issuer: CN=Otoroshi Root hosts: - otoroshi-service - otoroshi-service.${namespace}.svc.cluster.local - otoroshi-api-service.${namespace}.svc.cluster.local - otoroshi.${domain} - otoroshi-api.${domain} - privateapps.${domain} key: algo: rsa size: 2048 subject: uid=otoroshi-service-cert, O=Otoroshi client: false ca: false duration: 31536000000 signatureAlg: SHA256WithRSAEncryption digestAlg: SHA-256
- haproxy.example
- nginx.example
- dns.example
Deploy an otoroshi cluster on a cloud provider managed kubernetes cluster
Here we have 2 replicas of an otoroshi leader connected to a redis instance and 2 replicas of an otoroshi worker connected to the leader. We use a service of type LoadBalancer
to expose otoroshi leader/worker to the rest of the world. You have to setup your DNS to bind otoroshi domain names to the LoadBalancer
external CNAME
(see the example below)
- deployment.yaml
-
--- apiVersion: apps/v1 kind: Deployment metadata: name: otoroshi-leader-deployment spec: selector: matchLabels: run: otoroshi-leader-deployment template: metadata: labels: run: otoroshi-leader-deployment replicas: 2 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 spec: serviceAccountName: otoroshi-admin-user terminationGracePeriodSeconds: 60 hostNetwork: false restartPolicy: Always containers: - image: maif/otoroshi:16.20.0-dev-jdk11 imagePullPolicy: IfNotPresent name: otoroshi-leader ports: - containerPort: 8080 name: "http" protocol: TCP - containerPort: 8443 name: "https" protocol: TCP env: - name: APP_STORAGE_ROOT value: otoroshi - name: OTOROSHI_INITIAL_ADMIN_PASSWORD value: ${password} - name: APP_DOMAIN value: ${domain} - name: APP_STORAGE value: lettuce - name: REDIS_URL value: ${redisUrl} # value: redis://redis-leader-service:6379/0 - name: CLUSTER_MODE value: Leader - name: CLUSTER_AUTO_UPDATE_STATE value: 'true' - name: CLUSTER_MTLS_ENABLED value: 'true' - name: CLUSTER_MTLS_LOOSE value: 'true' - name: CLUSTER_MTLS_TRUST_ALL value: 'true' - name: CLUSTER_LEADER_URL value: https://otoroshi-leader-api-service.${namespace}.svc.cluster.local:8443 - name: CLUSTER_LEADER_HOST value: otoroshi-leader-api-service.${namespace}.svc.cluster.local - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN value: otoroshi-leader-api-service.${namespace}.svc.cluster.local - name: ADMIN_API_CLIENT_ID value: ${clientId} - name: CLUSTER_LEADER_CLIENT_ID value: ${clientId} - name: ADMIN_API_CLIENT_SECRET value: ${clientSecret} - name: CLUSTER_LEADER_CLIENT_SECRET value: ${clientSecret} - name: OTOROSHI_SECRET value: ${otoroshiSecret} - name: HEALTH_LIMIT value: "5000" - name: SSL_OUTSIDE_CLIENT_AUTH value: Want - name: HTTPS_WANT_CLIENT_AUTH value: "true" - name: OTOROSHI_INITIAL_CUSTOMIZATION value: > { "config":{ "tlsSettings": { "defaultDomain": "www.${domain}", "randomIfNotFound": false }, "scripts":{ "enabled":true, "sinkRefs":[ "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator", "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector" ], "sinkConfig": {}, "jobRefs":[ "cp:otoroshi.plugins.jobs.kubernetes.KubernetesOtoroshiCRDsControllerJob" ], "jobConfig":{ "KubernetesConfig": { "trust": false, "namespaces": [ "*" ], "labels": {}, "namespacesLabels": {}, "ingressClasses": [ "otoroshi" ], "defaultGroup": "default", "ingresses": false, "crds": true, "coreDnsIntegration": false, "coreDnsIntegrationDryRun": false, "kubeLeader": false, "restartDependantDeployments": false, "watch": false, "syncDaikokuApikeysOnly": false, "kubeSystemNamespace": "kube-system", "coreDnsConfigMapName": "coredns", "coreDnsDeploymentName": "coredns", "corednsPort": 53, "otoroshiServiceName": "otoroshi-worker-service", "otoroshiNamespace": "${namespace}", "clusterDomain": "cluster.local", "syncIntervalSeconds": 60, "coreDnsEnv": null, "watchTimeoutSeconds": 60, "watchGracePeriodSeconds": 5, "mutatingWebhookName": "otoroshi-admission-webhook-injector", "validatingWebhookName": "otoroshi-admission-webhook-validation", "templates": { "service-group": {}, "service-descriptor": {}, "apikeys": {}, "global-config": {}, "jwt-verifier": {}, "tcp-service": {}, "certificate": {}, "auth-module": {}, "script": {}, "organizations": {}, "teams": {}, "webhooks": { "flags": { "requestCert": true, "originCheck": true, "tokensCheck": true, "displayEnv": false, "tlsTrace": false } } } } } } } } - name: JAVA_OPTS value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0' readinessProbe: httpGet: path: /ready port: 8080 failureThreshold: 1 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 livenessProbe: httpGet: path: /live port: 8080 failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 --- apiVersion: apps/v1 kind: Deployment metadata: name: otoroshi-worker-deployment spec: selector: matchLabels: run: otoroshi-worker-deployment template: metadata: labels: run: otoroshi-worker-deployment replicas: 2 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 spec: serviceAccountName: otoroshi-admin-user terminationGracePeriodSeconds: 60 hostNetwork: false restartPolicy: Always containers: - image: maif/otoroshi:16.8.0-rc.1-dev imagePullPolicy: IfNotPresent name: otoroshi-worker ports: - containerPort: 8080 name: "http" protocol: TCP - containerPort: 8443 name: "https" protocol: TCP env: - name: APP_STORAGE_ROOT value: otoroshi - name: OTOROSHI_INITIAL_ADMIN_PASSWORD value: ${password} - name: APP_DOMAIN value: ${domain} - name: CLUSTER_MODE value: Worker - name: CLUSTER_AUTO_UPDATE_STATE value: 'true' - name: CLUSTER_MTLS_ENABLED value: 'true' - name: CLUSTER_MTLS_LOOSE value: 'true' - name: CLUSTER_MTLS_TRUST_ALL value: 'true' - name: CLUSTER_LEADER_URL value: https://otoroshi-leader-api-service.${namespace}.svc.cluster.local:8443 - name: CLUSTER_LEADER_HOST value: otoroshi-leader-api-service.${namespace}.svc.cluster.local - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN value: otoroshi-leader-api-service.${namespace}.svc.cluster.local - name: ADMIN_API_CLIENT_ID value: ${clientId} - name: CLUSTER_LEADER_CLIENT_ID value: ${clientId} - name: ADMIN_API_CLIENT_SECRET value: ${clientSecret} - name: CLUSTER_LEADER_CLIENT_SECRET value: ${clientSecret} - name: HEALTH_LIMIT value: "5000" - name: SSL_OUTSIDE_CLIENT_AUTH value: Want - name: HTTPS_WANT_CLIENT_AUTH value: "true" - name: JAVA_OPTS value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0' readinessProbe: httpGet: path: /ready port: 8080 failureThreshold: 1 initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 livenessProbe: httpGet: path: /live port: 8080 failureThreshold: 3 initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 --- apiVersion: v1 kind: Service metadata: name: otoroshi-leader-api-service spec: selector: run: otoroshi-leader-deployment ports: - port: 8080 name: "http" targetPort: "http" - port: 8443 name: "https" targetPort: "https" --- apiVersion: v1 kind: Service metadata: name: otoroshi-leader-service spec: selector: run: otoroshi-leader-deployment ports: - port: 8080 name: "http" targetPort: "http" - port: 8443 name: "https" targetPort: "https" --- apiVersion: v1 kind: Service metadata: name: otoroshi-worker-service spec: selector: run: otoroshi-worker-deployment ports: - port: 8080 name: "http" targetPort: "http" - port: 8443 name: "https" targetPort: "https" --- apiVersion: v1 kind: Service metadata: name: otoroshi-leader-external-service spec: type: LoadBalancer selector: run: otoroshi-leader-deployment ports: - port: 80 name: "http" targetPort: "http" - port: 443 name: "https" targetPort: "https" --- apiVersion: v1 kind: Service metadata: name: otoroshi-worker-external-service spec: type: LoadBalancer selector: run: otoroshi-worker-deployment ports: - port: 80 name: "http" targetPort: "http" - port: 443 name: "https" targetPort: "https" --- apiVersion: proxy.otoroshi.io/v1 kind: Certificate metadata: name: otoroshi-service-certificate spec: description: certificate for otoroshi-service autoRenew: true csr: issuer: CN=Otoroshi Root hosts: - otoroshi-service - otoroshi-service.${namespace}.svc.cluster.local - otoroshi-api-service.${namespace}.svc.cluster.local - otoroshi.${domain} - otoroshi-api.${domain} - privateapps.${domain} key: algo: rsa size: 2048 subject: uid=otoroshi-service-cert, O=Otoroshi client: false ca: false duration: 31536000000 signatureAlg: SHA256WithRSAEncryption digestAlg: SHA-256
- dns.example
Deploy an otoroshi cluster on a bare metal kubernetes cluster
Here we have 2 replicas of otoroshi leader connected to the same redis instance and 2 replicas for otoroshi worker. The otoroshi instances are exposed as nodePort
so you’ll have to add a loadbalancer in front of your kubernetes nodes to route external traffic (TCP) to your otoroshi instances. You have to setup your DNS to bind otoroshi domain names to your loadbalancer (see the example below).
- deployment.yaml
-
--- apiVersion: apps/v1 kind: Deployment metadata: name: otoroshi-leader-deployment spec: selector: matchLabels: run: otoroshi-leader-deployment template: metadata: labels: run: otoroshi-leader-deployment replicas: 2 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 spec: serviceAccountName: otoroshi-admin-user terminationGracePeriodSeconds: 60 hostNetwork: false restartPolicy: Always containers: - image: maif/otoroshi:16.20.0-dev-jdk11 imagePullPolicy: IfNotPresent name: otoroshi-leader ports: - containerPort: 8080 name: "http" protocol: TCP - containerPort: 8443 name: "https" protocol: TCP env: - name: APP_STORAGE_ROOT value: otoroshi - name: OTOROSHI_INITIAL_ADMIN_PASSWORD value: ${password} - name: APP_DOMAIN value: ${domain} - name: APP_STORAGE value: lettuce - name: REDIS_URL value: ${redisUrl} # value: redis://redis-leader-service:6379/0 - name: CLUSTER_MODE value: Leader - name: CLUSTER_AUTO_UPDATE_STATE value: 'true' - name: CLUSTER_MTLS_ENABLED value: 'true' - name: CLUSTER_MTLS_LOOSE value: 'true' - name: CLUSTER_MTLS_TRUST_ALL value: 'true' - name: CLUSTER_LEADER_URL value: https://otoroshi-leader-api-service.${namespace}.svc.cluster.local:8443 - name: CLUSTER_LEADER_HOST value: otoroshi-leader-api-service.${namespace}.svc.cluster.local - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN value: otoroshi-leader-api-service.${namespace}.svc.cluster.local - name: ADMIN_API_CLIENT_ID value: ${clientId} - name: CLUSTER_LEADER_CLIENT_ID value: ${clientId} - name: ADMIN_API_CLIENT_SECRET value: ${clientSecret} - name: CLUSTER_LEADER_CLIENT_SECRET value: ${clientSecret} - name: OTOROSHI_SECRET value: ${otoroshiSecret} - name: HEALTH_LIMIT value: "5000" - name: SSL_OUTSIDE_CLIENT_AUTH value: Want - name: HTTPS_WANT_CLIENT_AUTH value: "true" - name: OTOROSHI_INITIAL_CUSTOMIZATION value: > { "config":{ "tlsSettings": { "defaultDomain": "www.${domain}", "randomIfNotFound": false }, "scripts":{ "enabled":true, "sinkRefs":[ "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator", "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector" ], "sinkConfig": {}, "jobRefs":[ "cp:otoroshi.plugins.jobs.kubernetes.KubernetesOtoroshiCRDsControllerJob" ], "jobConfig":{ "KubernetesConfig": { "trust": false, "namespaces": [ "*" ], "labels": {}, "namespacesLabels": {}, "ingressClasses": [ "otoroshi" ], "defaultGroup": "default", "ingresses": false, "crds": true, "coreDnsIntegration": false, "coreDnsIntegrationDryRun": false, "kubeLeader": false, "restartDependantDeployments": false, "watch": false, "syncDaikokuApikeysOnly": false, "kubeSystemNamespace": "kube-system", "coreDnsConfigMapName": "coredns", "coreDnsDeploymentName": "coredns", "corednsPort": 53, "otoroshiServiceName": "otoroshi-worker-service", "otoroshiNamespace": "${namespace}", "clusterDomain": "cluster.local", "syncIntervalSeconds": 60, "coreDnsEnv": null, "watchTimeoutSeconds": 60, "watchGracePeriodSeconds": 5, "mutatingWebhookName": "otoroshi-admission-webhook-injector", "validatingWebhookName": "otoroshi-admission-webhook-validation", "templates": { "service-group": {}, "service-descriptor": {}, "apikeys": {}, "global-config": {}, "jwt-verifier": {}, "tcp-service": {}, "certificate": {}, "auth-module": {}, "script": {}, "organizations": {}, "teams": {}, "webhooks": { "flags": { "requestCert": true, "originCheck": true, "tokensCheck": true, "displayEnv": false, "tlsTrace": false } } } } } } } } - name: JAVA_OPTS value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0' readinessProbe: httpGet: path: /ready port: 8080 failureThreshold: 1 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 livenessProbe: httpGet: path: /live port: 8080 failureThreshold: 3 initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 --- apiVersion: apps/v1 kind: Deployment metadata: name: otoroshi-worker-deployment spec: selector: matchLabels: run: otoroshi-worker-deployment template: metadata: labels: run: otoroshi-worker-deployment replicas: 2 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 spec: serviceAccountName: otoroshi-admin-user terminationGracePeriodSeconds: 60 hostNetwork: false restartPolicy: Always containers: - image: maif/otoroshi:16.19.0-dev-dev imagePullPolicy: IfNotPresent name: otoroshi-worker ports: - containerPort: 8080 name: "http" protocol: TCP - containerPort: 8443 name: "https" protocol: TCP env: - name: APP_STORAGE_ROOT value: otoroshi - name: OTOROSHI_INITIAL_ADMIN_PASSWORD value: ${password} - name: APP_DOMAIN value: ${domain} - name: CLUSTER_MODE value: Worker - name: CLUSTER_AUTO_UPDATE_STATE value: 'true' - name: CLUSTER_MTLS_ENABLED value: 'true' - name: CLUSTER_MTLS_LOOSE value: 'true' - name: CLUSTER_MTLS_TRUST_ALL value: 'true' - name: CLUSTER_LEADER_URL value: https://otoroshi-leader-api-service.${namespace}.svc.cluster.local:8443 - name: CLUSTER_LEADER_HOST value: otoroshi-leader-api-service.${namespace}.svc.cluster.local - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN value: otoroshi-leader-api-service.${namespace}.svc.cluster.local - name: ADMIN_API_CLIENT_ID value: ${clientId} - name: CLUSTER_LEADER_CLIENT_ID value: ${clientId} - name: ADMIN_API_CLIENT_SECRET value: ${clientSecret} - name: CLUSTER_LEADER_CLIENT_SECRET value: ${clientSecret} - name: HEALTH_LIMIT value: "5000" - name: SSL_OUTSIDE_CLIENT_AUTH value: Want - name: HTTPS_WANT_CLIENT_AUTH value: "true" - name: JAVA_OPTS value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0' readinessProbe: httpGet: path: /ready port: 8080 failureThreshold: 1 initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 livenessProbe: httpGet: path: /live port: 8080 failureThreshold: 3 initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 --- apiVersion: v1 kind: Service metadata: name: otoroshi-leader-api-service spec: selector: run: otoroshi-leader-deployment ports: - port: 8080 name: "http" targetPort: "http" - port: 8443 name: "https" targetPort: "https" --- apiVersion: v1 kind: Service metadata: name: otoroshi-leader-service spec: selector: run: otoroshi-leader-deployment ports: - port: 8080 nodePort: 31080 name: "http" targetPort: "http" - port: 8443 nodePort: 31443 name: "https" targetPort: "https" --- apiVersion: v1 kind: Service metadata: name: otoroshi-worker-service spec: selector: run: otoroshi-worker-deployment ports: - port: 8080 nodePort: 32080 name: "http" targetPort: "http" - port: 8443 nodePort: 32443 name: "https" targetPort: "https" --- apiVersion: proxy.otoroshi.io/v1 kind: Certificate metadata: name: otoroshi-service-certificate spec: description: certificate for otoroshi-service autoRenew: true csr: issuer: CN=Otoroshi Root hosts: - otoroshi-service - otoroshi-service.${namespace}.svc.cluster.local - otoroshi-api-service.${namespace}.svc.cluster.local - otoroshi.${domain} - otoroshi-api.${domain} - privateapps.${domain} key: algo: rsa size: 2048 subject: uid=otoroshi-service-cert, O=Otoroshi client: false ca: false duration: 31536000000 signatureAlg: SHA256WithRSAEncryption digestAlg: SHA-256
- nginx.example
- dns.example
- dns.example
Deploy an otoroshi cluster on a bare metal kubernetes cluster using DaemonSet
Here we have 1 otoroshi leader instance on each kubernetes node (with the otoroshi-kind: leader
label) connected to the same redis instance and 1 otoroshi worker instance on each kubernetes node (with the otoroshi-kind: worker
label). The otoroshi instances are exposed as nodePort
so you’ll have to add a loadbalancer in front of your kubernetes nodes to route external traffic (TCP) to your otoroshi instances. You have to setup your DNS to bind otoroshi domain names to your loadbalancer (see the example below).
- deployment.yaml
-
--- apiVersion: apps/v1 kind: DaemonSet metadata: name: otoroshi-leader-deployment spec: selector: matchLabels: run: otoroshi-leader-deployment template: metadata: labels: run: otoroshi-leader-deployment strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: otoroshi-kind operator: In values: - leader tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule serviceAccountName: otoroshi-admin-user terminationGracePeriodSeconds: 60 hostNetwork: false restartPolicy: Always containers: - image: maif/otoroshi:16.20.0-dev-jdk11 imagePullPolicy: IfNotPresent name: otoroshi-leader ports: - containerPort: 8080 hostPort: 41080 name: "http" protocol: TCP - containerPort: 8443 hostPort: 41443 name: "https" protocol: TCP env: - name: APP_STORAGE_ROOT value: otoroshi - name: OTOROSHI_INITIAL_ADMIN_PASSWORD value: ${password} - name: APP_DOMAIN value: ${domain} - name: APP_STORAGE value: lettuce - name: REDIS_URL value: ${redisUrl} # value: redis://redis-leader-service:6379/0 - name: CLUSTER_MODE value: Leader - name: CLUSTER_AUTO_UPDATE_STATE value: 'true' - name: CLUSTER_MTLS_ENABLED value: 'true' - name: CLUSTER_MTLS_LOOSE value: 'true' - name: CLUSTER_MTLS_TRUST_ALL value: 'true' - name: CLUSTER_LEADER_URL value: https://otoroshi-leader-api-service.${namespace}.svc.cluster.local:8443 - name: CLUSTER_LEADER_HOST value: otoroshi-leader-api-service.${namespace}.svc.cluster.local - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN value: otoroshi-leader-api-service.${namespace}.svc.cluster.local - name: ADMIN_API_CLIENT_ID value: ${clientId} - name: CLUSTER_LEADER_CLIENT_ID value: ${clientId} - name: ADMIN_API_CLIENT_SECRET value: ${clientSecret} - name: CLUSTER_LEADER_CLIENT_SECRET value: ${clientSecret} - name: OTOROSHI_SECRET value: ${otoroshiSecret} - name: HEALTH_LIMIT value: "5000" - name: SSL_OUTSIDE_CLIENT_AUTH value: Want - name: HTTPS_WANT_CLIENT_AUTH value: "true" - name: OTOROSHI_INITIAL_CUSTOMIZATION value: > { "config":{ "tlsSettings": { "defaultDomain": "www.${domain}", "randomIfNotFound": false }, "scripts":{ "enabled":true, "sinkRefs":[ "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator", "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector" ], "sinkConfig": {}, "jobRefs":[ "cp:otoroshi.plugins.jobs.kubernetes.KubernetesOtoroshiCRDsControllerJob" ], "jobConfig":{ "KubernetesConfig": { "trust": false, "namespaces": [ "*" ], "labels": {}, "namespacesLabels": {}, "ingressClasses": [ "otoroshi" ], "defaultGroup": "default", "ingresses": false, "crds": true, "coreDnsIntegration": false, "coreDnsIntegrationDryRun": false, "kubeLeader": false, "restartDependantDeployments": false, "watch": false, "syncDaikokuApikeysOnly": false, "kubeSystemNamespace": "kube-system", "coreDnsConfigMapName": "coredns", "coreDnsDeploymentName": "coredns", "corednsPort": 53, "otoroshiServiceName": "otoroshi-worker-service", "otoroshiNamespace": "${namespace}", "clusterDomain": "cluster.local", "syncIntervalSeconds": 60, "coreDnsEnv": null, "watchTimeoutSeconds": 60, "watchGracePeriodSeconds": 5, "mutatingWebhookName": "otoroshi-admission-webhook-injector", "validatingWebhookName": "otoroshi-admission-webhook-validation", "templates": { "service-group": {}, "service-descriptor": {}, "apikeys": {}, "global-config": {}, "jwt-verifier": {}, "tcp-service": {}, "certificate": {}, "auth-module": {}, "script": {}, "organizations": {}, "teams": {}, "webhooks": { "flags": { "requestCert": true, "originCheck": true, "tokensCheck": true, "displayEnv": false, "tlsTrace": false } } } } } } } } - name: JAVA_OPTS value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0' readinessProbe: httpGet: path: /ready port: 8080 failureThreshold: 1 initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 livenessProbe: httpGet: path: /live port: 8080 failureThreshold: 3 initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 --- apiVersion: apps/v1 kind: DaemonSet metadata: name: otoroshi-worker-deployment spec: selector: matchLabels: run: otoroshi-worker-deployment template: metadata: labels: run: otoroshi-worker-deployment replicas: 2 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 maxSurge: 1 spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: otoroshi-kind operator: In values: - worker tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule serviceAccountName: otoroshi-admin-user terminationGracePeriodSeconds: 60 hostNetwork: false restartPolicy: Always containers: - image: maif/otoroshi:16.19.0-dev-dev imagePullPolicy: IfNotPresent name: otoroshi-worker ports: - containerPort: 8080 hostPort: 42080 name: "http" protocol: TCP - containerPort: 8443 hostPort: 42443 name: "https" protocol: TCP env: - name: APP_STORAGE_ROOT value: otoroshi - name: OTOROSHI_INITIAL_ADMIN_PASSWORD value: ${password} - name: APP_DOMAIN value: ${domain} - name: CLUSTER_MODE value: Worker - name: CLUSTER_AUTO_UPDATE_STATE value: 'true' - name: CLUSTER_MTLS_ENABLED value: 'true' - name: CLUSTER_MTLS_LOOSE value: 'true' - name: CLUSTER_MTLS_TRUST_ALL value: 'true' - name: CLUSTER_LEADER_URL value: https://otoroshi-leader-api-service.${namespace}.svc.cluster.local:8443 - name: CLUSTER_LEADER_HOST value: otoroshi-leader-api-service.${namespace}.svc.cluster.local - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN value: otoroshi-leader-api-service.${namespace}.svc.cluster.local - name: ADMIN_API_CLIENT_ID value: ${clientId} - name: CLUSTER_LEADER_CLIENT_ID value: ${clientId} - name: ADMIN_API_CLIENT_SECRET value: ${clientSecret} - name: CLUSTER_LEADER_CLIENT_SECRET value: ${clientSecret} - name: HEALTH_LIMIT value: "5000" - name: SSL_OUTSIDE_CLIENT_AUTH value: Want - name: HTTPS_WANT_CLIENT_AUTH value: "true" - name: JAVA_OPTS value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0' readinessProbe: httpGet: path: /ready port: 8080 failureThreshold: 1 initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 livenessProbe: httpGet: path: /live port: 8080 failureThreshold: 3 initialDelaySeconds: 60 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 2 --- apiVersion: v1 kind: Service metadata: name: otoroshi-leader-api-service spec: selector: run: otoroshi-leader-deployment ports: - port: 8080 name: "http" targetPort: "http" - port: 8443 name: "https" targetPort: "https" --- apiVersion: v1 kind: Service metadata: name: otoroshi-leader-service spec: selector: run: otoroshi-leader-deployment ports: - port: 8080 name: "http" targetPort: "http" - port: 8443 name: "https" targetPort: "https" --- apiVersion: v1 kind: Service metadata: name: otoroshi-worker-service spec: selector: run: otoroshi-worker-deployment ports: - port: 8080 name: "http" targetPort: "http" - port: 8443 name: "https" targetPort: "https" --- apiVersion: proxy.otoroshi.io/v1 kind: Certificate metadata: name: otoroshi-service-certificate spec: description: certificate for otoroshi-service autoRenew: true csr: issuer: CN=Otoroshi Root hosts: - otoroshi-service - otoroshi-service.${namespace}.svc.cluster.local - otoroshi-api-service.${namespace}.svc.cluster.local - otoroshi.${domain} - otoroshi-api.${domain} - privateapps.${domain} key: algo: rsa size: 2048 subject: uid=otoroshi-service-cert, O=Otoroshi client: false ca: false duration: 31536000000 signatureAlg: SHA256WithRSAEncryption digestAlg: SHA-256
- nginx.example
- dns.example
- dns.example
Using Otoroshi as an Ingress Controller
If you want to use Otoroshi as an Ingress Controller, just go to the danger zone, and in Global scripts
add the job named Kubernetes Ingress Controller
.
Then add the following configuration for the job (with your own tweaks of course)
{
"KubernetesConfig": {
"enabled": true,
"endpoint": "https://127.0.0.1:6443",
"token": "eyJhbGciOiJSUzI....F463SrpOehQRaQ",
"namespaces": [
"*"
]
}
}
the configuration can have the following values
{
"KubernetesConfig": {
"endpoint": "https://127.0.0.1:6443", // the endpoint to talk to the kubernetes api, optional
"token": "xxxx", // the bearer token to talk to the kubernetes api, optional
"userPassword": "user:password", // the user password tuple to talk to the kubernetes api, optional
"caCert": "/etc/ca.cert", // the ca cert file path to talk to the kubernetes api, optional
"trust": false, // trust any cert to talk to the kubernetes api, optional
"namespaces": ["*"], // the watched namespaces
"labels": ["label"], // the watched namespaces
"ingressClasses": ["otoroshi"], // the watched kubernetes.io/ingress.class annotations, can be *
"defaultGroup": "default", // the group to put services in otoroshi
"ingresses": true, // sync ingresses
"crds": false, // sync crds
"kubeLeader": false, // delegate leader election to kubernetes, to know where the sync job should run
"restartDependantDeployments": true, // when a secret/cert changes from otoroshi sync, restart dependant deployments
"templates": { // template for entities that will be merged with kubernetes entities. can be "default" to use otoroshi default templates
"service-group": {},
"service-descriptor": {},
"apikeys": {},
"global-config": {},
"jwt-verifier": {},
"tcp-service": {},
"certificate": {},
"auth-module": {},
"data-exporter": {},
"script": {},
"organization": {},
"team": {},
"data-exporter": {},
"routes": {},
"route-compositions": {},
"backends": {}
}
}
}
If endpoint
is not defined, Otoroshi will try to get it from $KUBERNETES_SERVICE_HOST
and $KUBERNETES_SERVICE_PORT
. If token
is not defined, Otoroshi will try to get it from the file at /var/run/secrets/kubernetes.io/serviceaccount/token
. If caCert
is not defined, Otoroshi will try to get it from the file at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
. If $KUBECONFIG
is defined, endpoint
, token
and caCert
will be read from the current context of the file referenced by it.
Now you can deploy your first service ;)
Deploy an ingress route
now let’s say you want to deploy an http service and route to the outside world through otoroshi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: http-app-deployment
spec:
selector:
matchLabels:
run: http-app-deployment
replicas: 1
template:
metadata:
labels:
run: http-app-deployment
spec:
containers:
- image: kennethreitz/httpbin
imagePullPolicy: IfNotPresent
name: otoroshi
ports:
- containerPort: 80
name: "http"
---
apiVersion: v1
kind: Service
metadata:
name: http-app-service
spec:
ports:
- port: 8080
targetPort: http
name: http
selector:
run: http-app-deployment
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: http-app-ingress
annotations:
kubernetes.io/ingress.class: otoroshi
spec:
tls:
- hosts:
- httpapp.foo.bar
secretName: http-app-cert
rules:
- host: httpapp.foo.bar
http:
paths:
- path: /
backend:
serviceName: http-app-service
servicePort: 8080
once deployed, otoroshi will sync with kubernetes and create the corresponding service to route your app. You will be able to access your app with
curl -X GET https://httpapp.foo.bar/get
Support for Ingress Classes
Since Kubernetes 1.18, you can use IngressClass
type of manifest to specify which ingress controller you want to use for a deployment (https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#extended-configuration-with-ingress-classes). Otoroshi is fully compatible with this new manifest kind
. To use it, configure the Ingress job to match your controller
{
"KubernetesConfig": {
...
"ingressClasses": ["otoroshi.io/ingress-controller"],
...
}
}
then you have to deploy an IngressClass
to declare Otoroshi as an ingress controller
apiVersion: "networking.k8s.io/v1beta1"
kind: "IngressClass"
metadata:
name: "otoroshi-ingress-controller"
spec:
controller: "otoroshi.io/ingress-controller"
parameters:
apiGroup: "proxy.otoroshi.io/v1alpha"
kind: "IngressParameters"
name: "otoroshi-ingress-controller"
and use it in your Ingress
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: http-app-ingress
spec:
ingressClassName: otoroshi-ingress-controller
tls:
- hosts:
- httpapp.foo.bar
secretName: http-app-cert
rules:
- host: httpapp.foo.bar
http:
paths:
- path: /
backend:
serviceName: http-app-service
servicePort: 8080
Use multiple ingress controllers
It is of course possible to use multiple ingress controller at the same time (https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/#using-multiple-ingress-controllers) using the annotation kubernetes.io/ingress.class
. By default, otoroshi reacts to the class otoroshi
, but you can make it the default ingress controller with the following config
{
"KubernetesConfig": {
...
"ingressClass": "*",
...
}
}
Supported annotations
if you need to customize the service descriptor behind an ingress rule, you can use some annotations. If you need better customisation, just go to the CRDs part. The following annotations are supported :
ingress.otoroshi.io/groups
ingress.otoroshi.io/group
ingress.otoroshi.io/groupId
ingress.otoroshi.io/name
ingress.otoroshi.io/targetsLoadBalancing
ingress.otoroshi.io/stripPath
ingress.otoroshi.io/enabled
ingress.otoroshi.io/userFacing
ingress.otoroshi.io/privateApp
ingress.otoroshi.io/forceHttps
ingress.otoroshi.io/maintenanceMode
ingress.otoroshi.io/buildMode
ingress.otoroshi.io/strictlyPrivate
ingress.otoroshi.io/sendOtoroshiHeadersBack
ingress.otoroshi.io/readOnly
ingress.otoroshi.io/xForwardedHeaders
ingress.otoroshi.io/overrideHost
ingress.otoroshi.io/allowHttp10
ingress.otoroshi.io/logAnalyticsOnServer
ingress.otoroshi.io/useAkkaHttpClient
ingress.otoroshi.io/useNewWSClient
ingress.otoroshi.io/tcpUdpTunneling
ingress.otoroshi.io/detectApiKeySooner
ingress.otoroshi.io/letsEncrypt
ingress.otoroshi.io/publicPatterns
ingress.otoroshi.io/privatePatterns
ingress.otoroshi.io/additionalHeaders
ingress.otoroshi.io/additionalHeadersOut
ingress.otoroshi.io/missingOnlyHeadersIn
ingress.otoroshi.io/missingOnlyHeadersOut
ingress.otoroshi.io/removeHeadersIn
ingress.otoroshi.io/removeHeadersOut
ingress.otoroshi.io/headersVerification
ingress.otoroshi.io/matchingHeaders
ingress.otoroshi.io/ipFiltering.whitelist
ingress.otoroshi.io/ipFiltering.blacklist
ingress.otoroshi.io/api.exposeApi
ingress.otoroshi.io/api.openApiDescriptorUrl
ingress.otoroshi.io/healthCheck.enabled
ingress.otoroshi.io/healthCheck.url
ingress.otoroshi.io/jwtVerifier.ids
ingress.otoroshi.io/jwtVerifier.enabled
ingress.otoroshi.io/jwtVerifier.excludedPatterns
ingress.otoroshi.io/authConfigRef
ingress.otoroshi.io/redirection.enabled
ingress.otoroshi.io/redirection.code
ingress.otoroshi.io/redirection.to
ingress.otoroshi.io/clientValidatorRef
ingress.otoroshi.io/transformerRefs
ingress.otoroshi.io/transformerConfig
ingress.otoroshi.io/accessValidator.enabled
ingress.otoroshi.io/accessValidator.excludedPatterns
ingress.otoroshi.io/accessValidator.refs
ingress.otoroshi.io/accessValidator.config
ingress.otoroshi.io/preRouting.enabled
ingress.otoroshi.io/preRouting.excludedPatterns
ingress.otoroshi.io/preRouting.refs
ingress.otoroshi.io/preRouting.config
ingress.otoroshi.io/issueCert
ingress.otoroshi.io/issueCertCA
ingress.otoroshi.io/gzip.enabled
ingress.otoroshi.io/gzip.excludedPatterns
ingress.otoroshi.io/gzip.whiteList
ingress.otoroshi.io/gzip.blackList
ingress.otoroshi.io/gzip.bufferSize
ingress.otoroshi.io/gzip.chunkedThreshold
ingress.otoroshi.io/gzip.compressionLevel
ingress.otoroshi.io/cors.enabled
ingress.otoroshi.io/cors.allowOrigin
ingress.otoroshi.io/cors.exposeHeaders
ingress.otoroshi.io/cors.allowHeaders
ingress.otoroshi.io/cors.allowMethods
ingress.otoroshi.io/cors.excludedPatterns
ingress.otoroshi.io/cors.maxAge
ingress.otoroshi.io/cors.allowCredentials
ingress.otoroshi.io/clientConfig.useCircuitBreaker
ingress.otoroshi.io/clientConfig.retries
ingress.otoroshi.io/clientConfig.maxErrors
ingress.otoroshi.io/clientConfig.retryInitialDelay
ingress.otoroshi.io/clientConfig.backoffFactor
ingress.otoroshi.io/clientConfig.connectionTimeout
ingress.otoroshi.io/clientConfig.idleTimeout
ingress.otoroshi.io/clientConfig.callAndStreamTimeout
ingress.otoroshi.io/clientConfig.callTimeout
ingress.otoroshi.io/clientConfig.globalTimeout
ingress.otoroshi.io/clientConfig.sampleInterval
ingress.otoroshi.io/enforceSecureCommunication
ingress.otoroshi.io/sendInfoToken
ingress.otoroshi.io/sendStateChallenge
ingress.otoroshi.io/secComHeaders.claimRequestName
ingress.otoroshi.io/secComHeaders.stateRequestName
ingress.otoroshi.io/secComHeaders.stateResponseName
ingress.otoroshi.io/secComTtl
ingress.otoroshi.io/secComVersion
ingress.otoroshi.io/secComInfoTokenVersion
ingress.otoroshi.io/secComExcludedPatterns
ingress.otoroshi.io/secComSettings.size
ingress.otoroshi.io/secComSettings.secret
ingress.otoroshi.io/secComSettings.base64
ingress.otoroshi.io/secComUseSameAlgo
ingress.otoroshi.io/secComAlgoChallengeOtoToBack.size
ingress.otoroshi.io/secComAlgoChallengeOtoToBack.secret
ingress.otoroshi.io/secComAlgoChallengeOtoToBack.base64
ingress.otoroshi.io/secComAlgoChallengeBackToOto.size
ingress.otoroshi.io/secComAlgoChallengeBackToOto.secret
ingress.otoroshi.io/secComAlgoChallengeBackToOto.base64
ingress.otoroshi.io/secComAlgoInfoToken.size
ingress.otoroshi.io/secComAlgoInfoToken.secret
ingress.otoroshi.io/secComAlgoInfoToken.base64
ingress.otoroshi.io/securityExcludedPatterns
for more informations about it, just go to https://maif.github.io/otoroshi/swagger-ui/index.html
with the previous example, the ingress does not define any apikey, so the route is public. If you want to enable apikeys on it, you can deploy the following descriptor
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
name: http-app-ingress
annotations:
kubernetes.io/ingress.class: otoroshi
ingress.otoroshi.io/group: http-app-group
ingress.otoroshi.io/forceHttps: 'true'
ingress.otoroshi.io/sendOtoroshiHeadersBack: 'true'
ingress.otoroshi.io/overrideHost: 'true'
ingress.otoroshi.io/allowHttp10: 'false'
ingress.otoroshi.io/publicPatterns: ''
spec:
tls:
- hosts:
- httpapp.foo.bar
secretName: http-app-cert
rules:
- host: httpapp.foo.bar
http:
paths:
- path: /
backend:
serviceName: http-app-service
servicePort: 8080
now you can use an existing apikey in the http-app-group
to access your app
curl -X GET https://httpapp.foo.bar/get -u existing-apikey-1:secret-1
Use Otoroshi CRDs for a better/full integration
Otoroshi provides some Custom Resource Definitions for kubernetes in order to manage Otoroshi related entities in kubernetes
routes
backends
route-compositions
service-descriptors
tcp-services
error-templates
apikeys
certificates
jwt-verifiers
auth-modules
admin-sessions
admins
auth-module-users
service-groups
organizations
tenants
teams
data-exporters
scripts
wasm-plugins
global-configs
green-scores
coraza-configs
using CRDs, you will be able to deploy and manager those entities from kubectl or the kubernetes api like
sudo kubectl get apikeys --all-namespaces
sudo kubectl get service-descriptors --all-namespaces
curl -X GET \
-H 'Authorization: Bearer eyJhbGciOiJSUzI....F463SrpOehQRaQ' \
-H 'Accept: application/json' -k \
https://127.0.0.1:6443/apis/proxy.otoroshi.io/v1/apikeys | jq
You can see this as better Ingress
resources. Like any Ingress
resource can define which controller it uses (using the kubernetes.io/ingress.class
annotation), you can chose another kind of resource instead of Ingress
. With Otoroshi CRDs you can even define resources like Certificate
, Apikey
, AuthModules
, JwtVerifier
, etc. It will help you to use all the power of Otoroshi while using the deployment model of kubernetes.
when using Otoroshi CRDs, Kubernetes becomes the single source of truth for the synced entities. It means that any value in the descriptors deployed will overrides the one in Otoroshi datastore each time it’s synced. So be careful if you use the Otoroshi UI or the API, some changes in configuration may be overriden by CRDs sync job.
Resources examples
- group.yaml
-
apiVersion: proxy.otoroshi.io/v1 kind: ServiceGroup metadata: name: http-app-group annotations: io.otoroshi/id: http-app-group spec: description: a group to hold services about the http-app
- apikey.yaml
- service-descriptor.yaml
- certificate.yaml
- jwt.yaml
- auth.yaml
- organization.yaml
- team.yaml
Configuration
To configure it, just go to the danger zone, and in Global scripts
add the job named Kubernetes Otoroshi CRDs Controller
. Then add the following configuration for the job (with your own tweak of course)
{
"KubernetesConfig": {
"enabled": true,
"crds": true,
"endpoint": "https://127.0.0.1:6443",
"token": "eyJhbGciOiJSUzI....F463SrpOehQRaQ",
"namespaces": [
"*"
]
}
}
the configuration can have the following values
{
"KubernetesConfig": {
"endpoint": "https://127.0.0.1:6443", // the endpoint to talk to the kubernetes api, optional
"token": "xxxx", // the bearer token to talk to the kubernetes api, optional
"userPassword": "user:password", // the user password tuple to talk to the kubernetes api, optional
"caCert": "/etc/ca.cert", // the ca cert file path to talk to the kubernetes api, optional
"trust": false, // trust any cert to talk to the kubernetes api, optional
"namespaces": ["*"], // the watched namespaces
"labels": ["label"], // the watched namespaces
"ingressClasses": ["otoroshi"], // the watched kubernetes.io/ingress.class annotations, can be *
"defaultGroup": "default", // the group to put services in otoroshi
"ingresses": false, // sync ingresses
"crds": true, // sync crds
"kubeLeader": false, // delegate leader election to kubernetes, to know where the sync job should run
"restartDependantDeployments": true, // when a secret/cert changes from otoroshi sync, restart dependant deployments
"templates": { // template for entities that will be merged with kubernetes entities. can be "default" to use otoroshi default templates
"service-group": {},
"service-descriptor": {},
"apikeys": {},
"global-config": {},
"jwt-verifier": {},
"tcp-service": {},
"certificate": {},
"auth-module": {},
"data-exporter": {},
"script": {},
"organization": {},
"team": {},
"data-exporter": {}
}
}
}
If endpoint
is not defined, Otoroshi will try to get it from $KUBERNETES_SERVICE_HOST
and $KUBERNETES_SERVICE_PORT
. If token
is not defined, Otoroshi will try to get it from the file at /var/run/secrets/kubernetes.io/serviceaccount/token
. If caCert
is not defined, Otoroshi will try to get it from the file at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
. If $KUBECONFIG
is defined, endpoint
, token
and caCert
will be read from the current context of the file referenced by it.
you can find a more complete example of the configuration object here
Note about apikeys
and certificates
resources
Apikeys and Certificates are a little bit different than the other resources. They have ability to be defined without their secret part, but with an export setting so otoroshi will generate the secret parts and export the apikey or the certificate to kubernetes secret. Then any app will be able to mount them as volumes (see the full example below)
In those resources you can define
exportSecret: true
secretName: the-secret-name
and omit clientSecret
for apikey or publicKey
, privateKey
for certificates. For certificate you will have to provide a csr
for the certificate in order to generate it
csr:
issuer: CN=Otoroshi Root
hosts:
- httpapp.foo.bar
- httpapps.foo.bar
key:
algo: rsa
size: 2048
subject: UID=httpapp-front, O=OtoroshiApps
client: false
ca: false
duration: 31536000000
signatureAlg: SHA256WithRSAEncryption
digestAlg: SHA-256
when apikeys are exported as kubernetes secrets, they will have the type otoroshi.io/apikey-secret
with values clientId
and clientSecret
apiVersion: v1
kind: Secret
metadata:
name: apikey-1
type: otoroshi.io/apikey-secret
data:
clientId: TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdA==
clientSecret: TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdA==
when certificates are exported as kubernetes secrets, they will have the type kubernetes.io/tls
with the standard values tls.crt
(the full cert chain) and tls.key
(the private key). For more convenience, they will also have a cert.crt
value containing the actual certificate without the ca chain and ca-chain.crt
containing the ca chain without the certificate.
apiVersion: v1
kind: Secret
metadata:
name: certificate-1
type: kubernetes.io/tls
data:
tls.crt: TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdA==
tls.key: TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdA==
cert.crt: TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdA==
ca-chain.crt: TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdA==
Full CRD example
then you can deploy the previous example with better configuration level, and using mtls, apikeys, etc
Let say the app looks like :
const fs = require('fs');
const https = require('https');
// here we read the apikey to access http-app-2 from files mounted from secrets
const clientId = fs.readFileSync('/var/run/secrets/kubernetes.io/apikeys/clientId').toString('utf8')
const clientSecret = fs.readFileSync('/var/run/secrets/kubernetes.io/apikeys/clientSecret').toString('utf8')
const backendKey = fs.readFileSync('/var/run/secrets/kubernetes.io/certs/backend/tls.key').toString('utf8')
const backendCert = fs.readFileSync('/var/run/secrets/kubernetes.io/certs/backend/cert.crt').toString('utf8')
const backendCa = fs.readFileSync('/var/run/secrets/kubernetes.io/certs/backend/ca-chain.crt').toString('utf8')
const clientKey = fs.readFileSync('/var/run/secrets/kubernetes.io/certs/client/tls.key').toString('utf8')
const clientCert = fs.readFileSync('/var/run/secrets/kubernetes.io/certs/client/cert.crt').toString('utf8')
const clientCa = fs.readFileSync('/var/run/secrets/kubernetes.io/certs/client/ca-chain.crt').toString('utf8')
function callApi2() {
return new Promise((success, failure) => {
const options = {
// using the implicit internal name (*.global.otoroshi.mesh) of the other service descriptor passing through otoroshi
hostname: 'http-app-service-descriptor-2.global.otoroshi.mesh',
port: 433,
path: '/',
method: 'GET',
headers: {
'Accept': 'application/json',
'Otoroshi-Client-Id': clientId,
'Otoroshi-Client-Secret': clientSecret,
},
cert: clientCert,
key: clientKey,
ca: clientCa
};
let data = '';
const req = https.request(options, (res) => {
res.on('data', (d) => {
data = data + d.toString('utf8');
});
res.on('end', () => {
success({ body: JSON.parse(data), res });
});
res.on('error', (e) => {
failure(e);
});
});
req.end();
})
}
const options = {
key: backendKey,
cert: backendCert,
ca: backendCa,
// we want mtls behavior
requestCert: true,
rejectUnauthorized: true
};
https.createServer(options, (req, res) => {
res.writeHead(200, {'Content-Type': 'application/json'});
callApi2().then(resp => {
res.write(JSON.stringify{ ("message": `Hello to ${req.socket.getPeerCertificate().subject.CN}`, api2: resp.body }));
});
}).listen(433);
then, the descriptors will be :
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: http-app-deployment
spec:
selector:
matchLabels:
run: http-app-deployment
replicas: 1
template:
metadata:
labels:
run: http-app-deployment
spec:
containers:
- image: foo/http-app
imagePullPolicy: IfNotPresent
name: otoroshi
ports:
- containerPort: 443
name: "https"
volumeMounts:
- name: apikey-volume
# here you will be able to read apikey from files
# - /var/run/secrets/kubernetes.io/apikeys/clientId
# - /var/run/secrets/kubernetes.io/apikeys/clientSecret
mountPath: "/var/run/secrets/kubernetes.io/apikeys"
readOnly: true
volumeMounts:
- name: backend-cert-volume
# here you will be able to read app cert from files
# - /var/run/secrets/kubernetes.io/certs/backend/tls.crt
# - /var/run/secrets/kubernetes.io/certs/backend/tls.key
mountPath: "/var/run/secrets/kubernetes.io/certs/backend"
readOnly: true
- name: client-cert-volume
# here you will be able to read app cert from files
# - /var/run/secrets/kubernetes.io/certs/client/tls.crt
# - /var/run/secrets/kubernetes.io/certs/client/tls.key
mountPath: "/var/run/secrets/kubernetes.io/certs/client"
readOnly: true
volumes:
- name: apikey-volume
secret:
# here we reference the secret name from apikey http-app-2-apikey-1
secretName: secret-2
- name: backend-cert-volume
secret:
# here we reference the secret name from cert http-app-certificate-backend
secretName: http-app-certificate-backend-secret
- name: client-cert-volume
secret:
# here we reference the secret name from cert http-app-certificate-client
secretName: http-app-certificate-client-secret
---
apiVersion: v1
kind: Service
metadata:
name: http-app-service
spec:
ports:
- port: 8443
targetPort: https
name: https
selector:
run: http-app-deployment
---
apiVersion: proxy.otoroshi.io/v1
kind: ServiceGroup
metadata:
name: http-app-group
annotations:
otoroshi.io/id: http-app-group
spec:
description: a group to hold services about the http-app
---
apiVersion: proxy.otoroshi.io/v1
kind: ApiKey
metadata:
name: http-app-apikey-1
# this apikey can be used to access the app
spec:
# a secret name secret-1 will be created by otoroshi and can be used by containers
exportSecret: true
secretName: secret-1
authorizedEntities:
- group_http-app-group
---
apiVersion: proxy.otoroshi.io/v1
kind: ApiKey
metadata:
name: http-app-2-apikey-1
# this apikey can be used to access another app in a different group
spec:
# a secret name secret-1 will be created by otoroshi and can be used by containers
exportSecret: true
secretName: secret-2
authorizedEntities:
- group_http-app-2-group
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
name: http-app-certificate-frontend
spec:
description: certificate for the http-app on otorshi frontend
autoRenew: true
csr:
issuer: CN=Otoroshi Root
hosts:
- httpapp.foo.bar
key:
algo: rsa
size: 2048
subject: UID=httpapp-front, O=OtoroshiApps
client: false
ca: false
duration: 31536000000
signatureAlg: SHA256WithRSAEncryption
digestAlg: SHA-256
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
name: http-app-certificate-backend
spec:
description: certificate for the http-app deployed on pods
autoRenew: true
# a secret name http-app-certificate-backend-secret will be created by otoroshi and can be used by containers
exportSecret: true
secretName: http-app-certificate-backend-secret
csr:
issuer: CN=Otoroshi Root
hosts:
- http-app-service
key:
algo: rsa
size: 2048
subject: UID=httpapp-back, O=OtoroshiApps
client: false
ca: false
duration: 31536000000
signatureAlg: SHA256WithRSAEncryption
digestAlg: SHA-256
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
name: http-app-certificate-client
spec:
description: certificate for the http-app
autoRenew: true
secretName: http-app-certificate-client-secret
csr:
issuer: CN=Otoroshi Root
key:
algo: rsa
size: 2048
subject: UID=httpapp-client, O=OtoroshiApps
client: false
ca: false
duration: 31536000000
signatureAlg: SHA256WithRSAEncryption
digestAlg: SHA-256
---
apiVersion: proxy.otoroshi.io/v1
kind: ServiceDescriptor
metadata:
name: http-app-service-descriptor
spec:
description: the service descriptor for the http app
groups:
- http-app-group
forceHttps: true
hosts:
- httpapp.foo.bar # hostname exposed oustide of the kubernetes cluster
# - http-app-service-descriptor.global.otoroshi.mesh # implicit internal name inside the kubernetes cluster
matchingRoot: /
targets:
- url: https://http-app-service:8443
# alternatively, you can use serviceName and servicePort to use pods ip addresses
# serviceName: http-app-service
# servicePort: https
mtlsConfig:
# use mtls to contact the backend
mtls: true
certs:
# reference the DN for the client cert
- UID=httpapp-client, O=OtoroshiApps
trustedCerts:
# reference the DN for the CA cert
- CN=Otoroshi Root
sendOtoroshiHeadersBack: true
xForwardedHeaders: true
overrideHost: true
allowHttp10: false
publicPatterns:
- /health
additionalHeaders:
x-foo: bar
# here you can specify everything supported by otoroshi like jwt-verifiers, auth config, etc ... for more informations about it, just go to https://maif.github.io/otoroshi/swagger-ui/index.html
now with this descriptor deployed, you can access your app with a command like
CLIENT_ID=`kubectl get secret secret-1 -o jsonpath="{.data.clientId}" | base64 --decode`
CLIENT_SECRET=`kubectl get secret secret-1 -o jsonpath="{.data.clientSecret}" | base64 --decode`
curl -X GET https://httpapp.foo.bar/get -u "$CLIENT_ID:$CLIENT_SECRET"
Expose Otoroshi to outside world
If you deploy Otoroshi on a kubernetes cluster, the Otoroshi service is deployed as a loadbalancer (service type: LoadBalancer
). You’ll need to declare in your DNS settings any name that can be routed by otoroshi going to the loadbalancer endpoint (CNAME or ip addresses) of your kubernetes distribution. If you use a managed kubernetes cluster from a cloud provider, it will work seamlessly as they will provide external loadbalancers out of the box. However, if you use a bare metal kubernetes cluster, id doesn’t come with support for external loadbalancers (service of type LoadBalancer
). So you will have to provide this feature in order to route external TCP traffic to Otoroshi containers running inside the kubernetes cluster. You can use projects like MetalLB that provide software LoadBalancer
services to bare metal clusters or you can use and customize examples in the installation section.
We don’t recommand running Otoroshi behind an existing ingress controller (or something like that) as you will not be able to use features like TCP proxying, TLS, mTLS, etc. Also, this additional layer of reverse proxy will increase call latencies.
Access a service from inside the k8s cluster
Using host header overriding
You can access any service referenced in otoroshi, through otoroshi from inside the kubernetes cluster by using the otoroshi service name (if you use a template based on https://github.com/MAIF/otoroshi/tree/master/kubernetes/base deployed in the otoroshi namespace) and the host header with the service domain like :
CLIENT_ID="xxx"
CLIENT_SECRET="xxx"
curl -X GET -H 'Host: httpapp.foo.bar' https://otoroshi-service.otoroshi.svc.cluster.local:8443/get -u "$CLIENT_ID:$CLIENT_SECRET"
Using dedicated services
it’s also possible to define services that targets otoroshi deployment (or otoroshi workers deployment) and use then as valid hosts in otoroshi services
apiVersion: v1
kind: Service
metadata:
name: my-awesome-service
spec:
selector:
# run: otoroshi-deployment
# or in cluster mode
run: otoroshi-worker-deployment
ports:
- port: 8080
name: "http"
targetPort: "http"
- port: 8443
name: "https"
targetPort: "https"
and access it like
CLIENT_ID="xxx"
CLIENT_SECRET="xxx"
curl -X GET https://my-awesome-service.my-namspace.svc.cluster.local:8443/get -u "$CLIENT_ID:$CLIENT_SECRET"
Using coredns integration
You can also enable the coredns integration to simplify the flow. You can use the the following keys in the plugin config :
{
"KubernetesConfig": {
...
"coreDnsIntegration": true, // enable coredns integration for intra cluster calls
"kubeSystemNamespace": "kube-system", // the namespace where coredns is deployed
"corednsConfigMap": "coredns", // the name of the coredns configmap
"otoroshiServiceName": "otoroshi-service", // the name of the otoroshi service, could be otoroshi-workers-service
"otoroshiNamespace": "otoroshi", // the namespace where otoroshi is deployed
"clusterDomain": "cluster.local", // the domain for cluster services
...
}
}
otoroshi will patch coredns config at startup then you can call your services like
CLIENT_ID="xxx"
CLIENT_SECRET="xxx"
curl -X GET https://my-awesome-service.my-awesome-service-namespace.otoroshi.mesh:8443/get -u "$CLIENT_ID:$CLIENT_SECRET"
By default, all services created from CRDs service descriptors are exposed as ${service-name}.${service-namespace}.otoroshi.mesh
or ${service-name}.${service-namespace}.svc.otoroshi.local
Using coredns with manual patching
you can also patch the coredns config manually
kubectl edit configmaps coredns -n kube-system # or your own custom config map
and change the Corefile
data to add the following snippet in at the end of the file
otoroshi.mesh:53 {
errors
health
ready
kubernetes cluster.local in-addr.arpa ip6.arpa {
pods insecure
upstream
fallthrough in-addr.arpa ip6.arpa
}
rewrite name regex (.*)\.otoroshi\.mesh otoroshi-worker-service.otoroshi.svc.cluster.local
forward . /etc/resolv.conf
cache 30
loop
reload
loadbalance
}
you can also define simpler rewrite if it suits you use case better
rewrite name my-service.otoroshi.mesh otoroshi-worker-service.otoroshi.svc.cluster.local
do not hesitate to change otoroshi-worker-service.otoroshi
according to your own setup. If otoroshi is not in cluster mode, change it to otoroshi-service.otoroshi
. If otoroshi is not deployed in the otoroshi
namespace, change it to otoroshi-service.the-namespace
, etc.
By default, all services created from CRDs service descriptors are exposed as ${service-name}.${service-namespace}.otoroshi.mesh
then you can call your service like
CLIENT_ID="xxx"
CLIENT_SECRET="xxx"
curl -X GET https://my-awesome-service.my-awesome-service-namespace.otoroshi.mesh:8443/get -u "$CLIENT_ID:$CLIENT_SECRET"
Using old kube-dns system
if your stuck with an old version of kubernetes, it uses kube-dns that is not supported by otoroshi, so you will have to provide your own coredns deployment and declare it as a stubDomain in the old kube-dns system.
Here is an example of coredns deployment with otoroshi domain config
- coredns.yaml
-
--- apiVersion: v1 kind: ConfigMap metadata: name: otoroshi-dns labels: app: otoroshi component: coredns data: Corefile: | otoroshi.mesh:5353 { errors health ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } rewrite name regex (.*)\.otoroshi\.mesh otoroshi-service.otoroshi.svc.cluster.local forward . /etc/resolv.conf cache 30 loop reload loadbalance } .:5353 { errors health kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } forward . /etc/resolv.conf cache 30 loop reload loadbalance } --- apiVersion: v1 kind: Service metadata: name: otoroshi-dns labels: app: otoroshi component: coredns spec: # clusterIP: 1.1.1.1 selector: app: otoroshi component: coredns type: ClusterIP ports: - name: dns port: 5353 protocol: UDP - name: dns-tcp port: 5353 protocol: TCP --- apiVersion: apps/v1 kind: Deployment metadata: name: otoroshi-dns labels: app: otoroshi component: coredns spec: replicas: 2 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 selector: matchLabels: app: otoroshi component: coredns template: metadata: labels: app: otoroshi component: coredns spec: serviceAccountName: otoroshi-admin-user affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - otoroshi - key: component operator: In values: - coredns topologyKey: "kubernetes.io/hostname" tolerations: - key: "CriticalAddonsOnly" operator: "Exists" containers: - name: coredns image: coredns/coredns:1.8.0 imagePullPolicy: IfNotPresent resources: limits: memory: 170Mi requests: cpu: 100m memory: 70Mi args: [ "-conf", "/etc/coredns/Corefile" ] volumeMounts: - name: config-volume mountPath: /etc/coredns readOnly: true ports: - containerPort: 5353 name: dns protocol: UDP - containerPort: 5353 name: dns-tcp protocol: TCP securityContext: allowPrivilegeEscalation: false capabilities: add: - NET_BIND_SERVICE drop: - all readOnlyRootFilesystem: true livenessProbe: httpGet: path: /health port: 8080 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 dnsPolicy: Default volumes: - name: config-volume configMap: name: otoroshi-dns items: - key: Corefile path: Corefile
then you can enable the kube-dns integration in the otoroshi kubernetes job
{
"KubernetesConfig": {
...
"kubeDnsOperatorIntegration": true, // enable kube-dns integration for intra cluster calls
"kubeDnsOperatorCoreDnsNamespace": "otoroshi", // namespace where coredns is installed
"kubeDnsOperatorCoreDnsName": "otoroshi-dns", // name of the coredns service
"kubeDnsOperatorCoreDnsPort": 5353, // port of the coredns service
...
}
}
Using Openshift DNS operator
Openshift DNS operator does not allow to customize DNS configuration a lot, so you will have to provide your own coredns deployment and declare it as a stub in the Openshift DNS operator.
Here is an example of coredns deployment with otoroshi domain config
- coredns.yaml
-
--- apiVersion: v1 kind: ConfigMap metadata: name: otoroshi-dns labels: app: otoroshi component: coredns data: Corefile: | otoroshi.mesh:5353 { errors health ready kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } rewrite name regex (.*)\.otoroshi\.mesh otoroshi-service.otoroshi.svc.cluster.local forward . /etc/resolv.conf cache 30 loop reload loadbalance } .:5353 { errors health kubernetes cluster.local in-addr.arpa ip6.arpa { pods insecure fallthrough in-addr.arpa ip6.arpa } forward . /etc/resolv.conf cache 30 loop reload loadbalance } --- apiVersion: v1 kind: Service metadata: name: otoroshi-dns labels: app: otoroshi component: coredns spec: # clusterIP: 1.1.1.1 selector: app: otoroshi component: coredns type: ClusterIP ports: - name: dns port: 5353 protocol: UDP - name: dns-tcp port: 5353 protocol: TCP --- apiVersion: apps/v1 kind: Deployment metadata: name: otoroshi-dns labels: app: otoroshi component: coredns spec: replicas: 2 strategy: type: RollingUpdate rollingUpdate: maxUnavailable: 1 selector: matchLabels: app: otoroshi component: coredns template: metadata: labels: app: otoroshi component: coredns spec: serviceAccountName: otoroshi-admin-user affinity: podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - weight: 100 podAffinityTerm: labelSelector: matchExpressions: - key: app operator: In values: - otoroshi - key: component operator: In values: - coredns topologyKey: "kubernetes.io/hostname" tolerations: - key: "CriticalAddonsOnly" operator: "Exists" containers: - name: coredns image: coredns/coredns:1.8.0 imagePullPolicy: IfNotPresent resources: limits: memory: 170Mi requests: cpu: 100m memory: 70Mi args: [ "-conf", "/etc/coredns/Corefile" ] volumeMounts: - name: config-volume mountPath: /etc/coredns readOnly: true ports: - containerPort: 5353 name: dns protocol: UDP - containerPort: 5353 name: dns-tcp protocol: TCP securityContext: allowPrivilegeEscalation: false capabilities: add: - NET_BIND_SERVICE drop: - all readOnlyRootFilesystem: true livenessProbe: httpGet: path: /health port: 8080 scheme: HTTP initialDelaySeconds: 30 timeoutSeconds: 5 successThreshold: 1 failureThreshold: 5 dnsPolicy: Default volumes: - name: config-volume configMap: name: otoroshi-dns items: - key: Corefile path: Corefile
then you can enable the Openshift DNS operator integration in the otoroshi kubernetes job
{
"KubernetesConfig": {
...
"openshiftDnsOperatorIntegration": true, // enable openshift dns operator integration for intra cluster calls
"openshiftDnsOperatorCoreDnsNamespace": "otoroshi", // namespace where coredns is installed
"openshiftDnsOperatorCoreDnsName": "otoroshi-dns", // name of the coredns service
"openshiftDnsOperatorCoreDnsPort": 5353, // port of the coredns service
...
}
}
don’t forget to update the otoroshi ClusterRole
- apiGroups:
- operator.openshift.io
resources:
- dnses
verbs:
- get
- list
- watch
- update
CRD validation in kubectl
In order to get CRD validation before manifest deployments right inside kubectl, you can deploy a validation webhook that will do the trick. Also check that you have otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator
request sink enabled.
- validation-webhook.yaml
-
apiVersion: admissionregistration.k8s.io/v1 kind: ValidatingWebhookConfiguration metadata: name: otoroshi-admission-webhook-validation labels: app: otoroshi component: otoroshi-validation-webhook webhooks: - name: otoroshi-admission-webhook.otoroshi.io rules: - operations: - "CREATE" - "UPDATE" apiGroups: - "proxy.otoroshi.io" apiVersions: - "*" resources: - "*" scope: "Namespaced" clientConfig: # url: "https://otoroshi-kubernetes-admission-webhook.otoroshi.svc.cluster.local:8443/apis/webhooks/validation" service: name: otoroshi-service namespace: otoroshi path: "/apis/webhooks/validation" port: 8443 caBundle: "" # injected at runtime failurePolicy: Ignore # inject at runtime sideEffects: None admissionReviewVersions: - "v1"
Easier integration with otoroshi-sidecar
Otoroshi can help you to easily use existing services without modifications while gettings all the perks of otoroshi like apikeys, mTLS, exchange protocol, etc. To do so, otoroshi will inject a sidecar container in the pod of your deployment that will handle call coming from otoroshi and going to otoroshi. To enable otoroshi-sidecar, you need to deploy the following admission webhook. Also check that you have otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector
request sink enabled.
- sidecar-webhook.yaml
-
apiVersion: admissionregistration.k8s.io/v1 kind: MutatingWebhookConfiguration metadata: name: otoroshi-admission-webhook-injector labels: app: otoroshi component: otoroshi-validation-webhook webhooks: - name: otoroshi-admission-webhook-injector.otoroshi.io rules: - operations: - "CREATE" apiGroups: - "" apiVersions: - "v1" resources: - "pods" scope: "Namespaced" # namespaceSelector: # matchLabels: # otoroshi.io/sidecar: inject objectSelector: matchLabels: otoroshi.io/sidecar: inject clientConfig: # url: "https://otoroshi-kubernetes-admission-webhook.otoroshi.svc.cluster.local:8443/apis/webhooks/inject" service: name: otoroshi-service namespace: otoroshi path: "/apis/webhooks/inject" port: 8443 caBundle: "" # inject at runtime failurePolicy: Ignore # inject at runtime sideEffects: None admissionReviewVersions: - "v1"
then it’s quite easy to add the sidecar, just add the following label to your pod otoroshi.io/sidecar: inject
and some annotations to tell otoroshi what certificates and apikeys to use.
annotations:
otoroshi.io/sidecar-apikey: backend-apikey
otoroshi.io/sidecar-backend-cert: backend-cert
otoroshi.io/sidecar-client-cert: oto-client-cert
otoroshi.io/token-secret: secret
otoroshi.io/expected-dn: UID=oto-client-cert, O=OtoroshiApps
now you can just call you otoroshi handled apis from inside your pod like curl http://my-service.namespace.otoroshi.mesh/api
without passing any apikey or client certificate and the sidecar will handle everything for you. Same thing for call from otoroshi to your pod, everything will be done in mTLS fashion with apikeys and otoroshi exchange protocol
here is a full example
- sidecar.yaml
-
--- apiVersion: apps/v1 kind: Deployment metadata: name: app-deployment spec: selector: matchLabels: run: app-deployment app: node replicas: 1 template: metadata: labels: run: app-deployment app: node foo: bar otoroshi.io/sidecar: inject annotations: otoroshi.io/sidecar-apikey: backend-apikey otoroshi.io/sidecar-backend-cert: backend-cert otoroshi.io/sidecar-client-cert: oto-client-cert otoroshi.io/token-secret: secret otoroshi.io/expected-dn: UID=oto-client-cert, O=OtoroshiApps spec: containers: - image: containous/whoami:latest name: whoami args: ["--port", "8081"] ports: - name: main-port containerPort: 8081 --- apiVersion: v1 kind: Service metadata: name: app-service spec: selector: run: app-deployment ports: - port: 8443 name: "https" targetPort: "https" --- apiVersion: proxy.otoroshi.io/v1 kind: Certificate metadata: name: backend-cert spec: description: backend-cert autoRenew: true exportSecret: true secretName: backend-cert csr: hosts: - app-service.default.svc.cluster.local issuer: otoroshi-intermediate-ca key: algo: rsa size: 2048 subject: UID=backend-cert, O=OtoroshiApps duration: 31536000000 signatureAlg: SHA256WithRSAEncryption digestAlg: SHA-256 --- apiVersion: proxy.otoroshi.io/v1 kind: Certificate metadata: name: client-cert annotations: otoroshi.io/id: client-cert spec: description: client-cert autoRenew: true exportSecret: true client: true secretName: client-cert csr: client: true issuer: otoroshi-intermediate-ca key: algo: rsa size: 2048 subject: UID=client-cert, O=OtoroshiApps duration: 31536000000 signatureAlg: SHA256WithRSAEncryption digestAlg: SHA-256 --- apiVersion: proxy.otoroshi.io/v1 kind: Certificate metadata: name: oto-client-cert annotations: otoroshi.io/id: oto-client-cert spec: description: oto-client-cert autoRenew: true exportSecret: true client: true secretName: oto-client-cert csr: client: true issuer: otoroshi-intermediate-ca key: algo: rsa size: 2048 subject: UID=oto-client-cert, O=OtoroshiApps duration: 31536000000 signatureAlg: SHA256WithRSAEncryption digestAlg: SHA-256 --- apiVersion: proxy.otoroshi.io/v1 kind: Certificate metadata: name: frontend-cert spec: description: frontend-cert autoRenew: true csr: issuer: otoroshi-intermediate-ca hosts: - backend.oto.tools key: algo: rsa size: 2048 subject: UID=frontend-cert, O=OtoroshiApps duration: 31536000000 signatureAlg: SHA256WithRSAEncryption digestAlg: SHA-256 --- apiVersion: proxy.otoroshi.io/v1 kind: Certificate metadata: name: mesh-cert spec: description: mesh-cert autoRenew: true csr: issuer: otoroshi-intermediate-ca hosts: - '*.default.otoroshi.mesh' key: algo: rsa size: 2048 subject: O=Otoroshi, OU=Otoroshi Certificates, CN=kubernetes-mesh duration: 31536000000 signatureAlg: SHA256WithRSAEncryption digestAlg: SHA-256 --- apiVersion: proxy.otoroshi.io/v1 kind: ApiKey metadata: name: backend-apikey spec: exportSecret: true secretName: backend-apikey authorizedEntities: - group_default --- apiVersion: proxy.otoroshi.io/v1 kind: ServiceDescriptor metadata: name: backend spec: description: backend groups: - default forceHttps: false hosts: - backend.oto.tools matchingRoot: / publicPatterns: - /.* secComUseSameAlgo: true secComVersion: 2 secComInfoTokenVersion: Latest secComSettings: type: HSAlgoSettings size: 512 secret: secret base64: false secComAlgoChallengeOtoToBack: type: HSAlgoSettings size: 512 secret: secret base64: false secComAlgoChallengeBackToOto: type: HSAlgoSettings size: 512 secret: secret base64: false secComAlgoInfoToken: type: HSAlgoSettings size: 512 secret: secret base64: false targets: - url: https://app-service.default.svc.cluster.local:8443 mtlsConfig: mtls: true certs: - UID=oto-client-cert, O=OtoroshiApps trustedCerts: - otoroshi-intermediate-ca
Please avoid to use port 80
for your pod as it’s the default port to access otoroshi from your pod and the call will be redirect to the sidecar via an iptables rule
Daikoku integration
It is possible to easily integrate daikoku generated apikeys without any human interaction with the actual apikey secret. To do that, create a plan in Daikoku and setup the integration mode to Automatic
then when a user subscribe for an apikey, he will only see an integration token
then just create an ApiKey manifest with this token and your good to go
apiVersion: proxy.otoroshi.io/v1
kind: ApiKey
metadata:
name: http-app-2-apikey-3
spec:
exportSecret: true
secretName: secret-3
daikokuToken: RShQrvINByiuieiaCBwIZfGFgdPu7tIJEN5gdV8N8YeH4RI9ErPYJzkuFyAkZ2xy