Kubernetes

Starting at version 1.5.0, Otoroshi provides a native Kubernetes support. Multiple otoroshi jobs (that are actually kubernetes controllers) are provided in order to

• sync kubernetes secrets of type kubernetes.io/tls to otoroshi certificates
• act as a standard ingress controller (supporting Ingress objects)
• provide Custom Resource Definitions (CRDs) to manage Otoroshi entities from Kubernetes and act as an ingress controller with its own resources

Installing otoroshi on your kubernetes cluster

If you want to deploy otoroshi into your kubernetes cluster, you can download the deployment descriptors from https://github.com/MAIF/otoroshi/tree/master/kubernetes and use kustomize to create your own overlay.

You can also create a kustomization.yaml file with a remote base

bases:
- github.com/MAIF/otoroshi/kubernetes/kustomize/overlays/simple/?ref=v16.16.1

Then deploy it with kubectl apply -k ./overlays/myoverlay.

You can also use Helm to deploy a simple otoroshi cluster on your kubernetes cluster

helm repo add otoroshi https://maif.github.io/otoroshi/helm
helm install my-otoroshi otoroshi/otoroshi

Below, you will find example of deployment. Do not hesitate to adapt them to your needs. Those descriptors have value placeholders that you will need to replace with actual values like

 env:
  - name: APP_STORAGE_ROOT
    value: otoroshi
  - name: APP_DOMAIN
    value: ${domain}

you will have to edit it to make it look like

 env:
  - name: APP_STORAGE_ROOT
    value: otoroshi
  - name: APP_DOMAIN
    value: 'apis.my.domain'

if you don’t want to use placeholders and environment variables, you can create a secret containing the configuration file of otoroshi

apiVersion: v1
kind: Secret
metadata:
  name: otoroshi-config
type: Opaque
stringData:
  oto.conf: >
    include "application.conf"
    app {
      storage = "redis"
      domain = "apis.my.domain"
    }

and mount it in the otoroshi container

apiVersion: apps/v1
kind: Deployment
metadata:
  name: otoroshi-deployment
spec:
  selector:
    matchLabels:
      run: otoroshi-deployment
  template:
    metadata:
      labels:
        run: otoroshi-deployment
    spec:
      serviceAccountName: otoroshi-admin-user
      terminationGracePeriodSeconds: 60
      hostNetwork: false
      containers:
      - image: maif/otoroshi:16.16.1
        imagePullPolicy: IfNotPresent
        name: otoroshi
        args: ['-Dconfig.file=/usr/app/otoroshi/conf/oto.conf']
        ports:
          - containerPort: 8080
            name: "http"
            protocol: TCP
          - containerPort: 8443
            name: "https"
            protocol: TCP
        volumeMounts:
        - name: otoroshi-config
          mountPath: "/usr/app/otoroshi/conf"
          readOnly: true
      volumes:
      - name: otoroshi-config
        secret:
          secretName: otoroshi-config
        ...

You can also create several secrets for each placeholder, mount them to the otoroshi container then use their file path as value

 env:
  - name: APP_STORAGE_ROOT
    value: otoroshi
  - name: APP_DOMAIN
    value: 'file:///the/path/of/the/secret/file'

you can use the same trick in the config. file itself

Note on bare metal kubernetes cluster installation

Common manifests

the following manifests are always needed. They create otoroshi CRDs, tokens, role, etc. Redis deployment is not mandatory, it’s just an example. You can use your own existing setup.

rbac.yaml

copy---
kind: ServiceAccount
apiVersion: v1
metadata:
  name: otoroshi-admin-user
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: otoroshi-admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: otoroshi-admin-user
subjects:
- kind: ServiceAccount
  name: otoroshi-admin-user
  namespace: $namespace
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: otoroshi-admin-user
rules:
  - apiGroups:
      - ""
    resources:
      - services
      - endpoints
      - secrets
      - configmaps
      - deployments
      - namespaces
      - pods
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "apps"
    resources:
      - deployments
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - secrets
      - configmaps
    verbs:
      - update
      - create
      - delete
  - apiGroups:
      - extensions
    resources:
      - ingresses
      - ingressclasses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - extensions
    resources:
      - ingresses/status
    verbs:
      - update
  - apiGroups:
      - admissionregistration.k8s.io
    resources:
      - validatingwebhookconfigurations
      - mutatingwebhookconfigurations
    verbs:
      - get
      - update
      - patch
  - apiGroups:
      - proxy.otoroshi.io
    resources:
      - routes
      - backends
      - route-compositions
      - service-descriptors
      - tcp-services
      - error-templates
      - apikeys
      - certificates
      - jwt-verifiers
      - auth-modules
      - admin-sessions
      - admins
      - auth-module-users
      - service-groups
      - organizations
      - tenants
      - teams
      - data-exporters
      - scripts
      - wasm-plugins
      - global-configs
      - green-scores
      - coraza-configs
    verbs:
      - get
      - list
      - watch

crds.yaml

copy---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "routes.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "Route"
    plural: "routes"
    singular: "route"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "backends.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "Backend"
    plural: "backends"
    singular: "backend"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "route-compositions.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "RouteComposition"
    plural: "route-compositions"
    singular: "route-composition"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "service-descriptors.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "ServiceDescriptor"
    plural: "service-descriptors"
    singular: "service-descriptor"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "tcp-services.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "TcpService"
    plural: "tcp-services"
    singular: "tcp-service"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "error-templates.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "ErrorTemplate"
    plural: "error-templates"
    singular: "error-templates"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "apikeys.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "Apikey"
    plural: "apikeys"
    singular: "apikey"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "certificates.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "Certificate"
    plural: "certificates"
    singular: "certificate"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "jwt-verifiers.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "JwtVerifier"
    plural: "jwt-verifiers"
    singular: "jwt-verifier"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "auth-modules.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "AuthModule"
    plural: "auth-modules"
    singular: "auth-module"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "admin-sessions.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "AdminSession"
    plural: "admin-sessions"
    singular: "admin-session"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "admins.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "SimpleAdminUser"
    plural: "admins"
    singular: "admins"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "auth-module-users.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "AuthModuleUser"
    plural: "auth-module-users"
    singular: "auth-module-user"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "service-groups.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "ServiceGroup"
    plural: "service-groups"
    singular: "service-group"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "organizations.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "Organization"
    plural: "organizations"
    singular: "organization"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "tenants.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "Tenant"
    plural: "tenants"
    singular: "tenant"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "teams.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "Team"
    plural: "teams"
    singular: "team"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "data-exporters.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "DateExporter"
    plural: "data-exporters"
    singular: "data-exporter"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "scripts.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "Script"
    plural: "scripts"
    singular: "script"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "wasm-plugins.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "WasmPlugin"
    plural: "wasm-plugins"
    singular: "wasm-plugin"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "global-configs.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "GlobalConfig"
    plural: "global-configs"
    singular: "global-config"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "green-scores.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "GreenScore"
    plural: "green-scores"
    singular: "green-score"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
---
apiVersion: "apiextensions.k8s.io/v1"
kind: "CustomResourceDefinition"
metadata:
  name: "coraza-configs.proxy.otoroshi.io"
spec:
  group: "proxy.otoroshi.io"
  names:
    kind: "CorazaConfig"
    plural: "coraza-configs"
    singular: "coraza-config"
  scope: "Namespaced"
  versions:
  - name: "v1alpha1"
    served: false
    storage: false
    deprecated: true
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"
  - name: "v1"
    served: true
    storage: true
    deprecated: false
    schema:
      openAPIV3Schema:
        x-kubernetes-preserve-unknown-fields: true
        type: "object"

redis.yaml

copyapiVersion: v1
kind: Service
metadata:
  name: redis-leader-service
spec:
  ports:
    - port: 6379
      name: redis
  selector:
    run: redis-leader-deployment
---
apiVersion: v1
kind: Service
metadata:
  name: redis-follower-service
spec:
  ports:
    - port: 6379
      name: redis
  selector:
    run: redis-follower-deployment
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-leader-deployment
spec:
  selector:
    matchLabels:
      run: redis-leader-deployment
  serviceName: redis-leader-service
  replicas: 1
  template:
    metadata:
      labels:
        run: redis-leader-deployment
    spec:
      containers:
        - name: redis-leader-container
          image: redis
          imagePullPolicy: Always
          command: ["redis-server", "--appendonly", "yes"]
          ports:
            - containerPort: 6379
              name: redis
          volumeMounts:
          - name: redis-leader-storage
            mountPath: /data
            readOnly: false
          readinessProbe:
            exec:
              command:
              - sh
              - -c
              - "redis-cli -h $(hostname) ping"
            initialDelaySeconds: 15
            timeoutSeconds: 5
          livenessProbe:
            exec:
              command:
              - sh
              - -c
              - "redis-cli -h $(hostname) ping"
            initialDelaySeconds: 20
            periodSeconds: 3
  volumeClaimTemplates:
  - metadata:
      name: redis-leader-storage
      labels:
        name: redis-leader-storage
      annotations:
        volume.alpha.kubernetes.io/storage-class: anything
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 100Mi
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-follower-deployment
spec:
  selector:
    matchLabels:
      run: redis-follower-deployment
  serviceName: redis-follower-service
  replicas: 1
  template:
    metadata:
      labels:
        run: redis-follower-deployment
    spec:
      containers:
        - name: redis-follower-container
          image: redis
          imagePullPolicy: Always
          command: ["redis-server", "--appendonly", "yes", "--slaveof", "redis-leader-service", "6379"]
          ports:
            - containerPort: 6379
              name: redis
          volumeMounts:
          - name: redis-follower-storage
            mountPath: /data
            readOnly: false
          readinessProbe:
            exec:
              command:
              - sh
              - -c
              - "redis-cli -h $(hostname) ping"
            initialDelaySeconds: 15
            timeoutSeconds: 5
          livenessProbe:
            exec:
              command:
              - sh
              - -c
              - "redis-cli -h $(hostname) ping"
            initialDelaySeconds: 20
            periodSeconds: 3
  volumeClaimTemplates:
  - metadata:
      name: redis-follower-storage
      labels:
        name: redis-follower-storage
      annotations:
        volume.alpha.kubernetes.io/storage-class: anything
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 100Mi

Deploy a simple otoroshi instanciation on a cloud provider managed kubernetes cluster

Here we have 2 replicas connected to the same redis instance. Nothing fancy. We use a service of type LoadBalancer to expose otoroshi to the rest of the world. You have to setup your DNS to bind otoroshi domain names to the LoadBalancer external CNAME (see the example below)

deployment.yaml

copyapiVersion: apps/v1
kind: Deployment
metadata:
  name: otoroshi-deployment
spec:
  selector:
    matchLabels:
      run: otoroshi-deployment
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        run: otoroshi-deployment
    spec:
      serviceAccountName: otoroshi-admin-user
      terminationGracePeriodSeconds: 60
      hostNetwork: false
      containers:
      - image: maif/otoroshi:16.16.1-jdk11
        imagePullPolicy: IfNotPresent
        name: otoroshi
        ports:
          - containerPort: 8080
            name: "http"
            protocol: TCP
          - containerPort: 8443
            name: "https"
            protocol: TCP
        env:
          - name: APP_STORAGE_ROOT
            value: otoroshi
          - name: OTOROSHI_INITIAL_ADMIN_PASSWORD
            value: ${password}
          - name: APP_DOMAIN
            value: ${domain}
          - name: APP_STORAGE
            value: lettuce
          - name: REDIS_URL
            value: ${redisUrl}
            # value: redis://redis-leader-service:6379/0
          - name: ADMIN_API_CLIENT_ID
            value: ${clientId}
          - name: ADMIN_API_CLIENT_SECRET
            value: ${clientSecret}
          - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN
            value: otoroshi-api-service.${namespace}.svc.cluster.local
          - name: OTOROSHI_SECRET
            value: ${otoroshiSecret}
          - name: HEALTH_LIMIT
            value: "5000"
          - name: SSL_OUTSIDE_CLIENT_AUTH
            value: Want
          - name: HTTPS_WANT_CLIENT_AUTH
            value: "true"
          - name: OTOROSHI_EXPOSED_PORTS_HTTP
            value: "80"
          - name: OTOROSHI_EXPOSED_PORTS_HTTPS
            value: "443"
          - name: OTOROSHI_INITIAL_CUSTOMIZATION
            value: >
              {
                "config":{
                  "tlsSettings": {
                    "defaultDomain": "www.${domain}",
                    "randomIfNotFound": false
                  },
                  "scripts":{
                    "enabled":true,
                    "sinkRefs":[
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator",
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector"
                    ],
                    "sinkConfig": {},
                    "jobRefs":[
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesOtoroshiCRDsControllerJob"
                    ],
                    "jobConfig":{
                      "KubernetesConfig": {
                        "trust": false,
                        "namespaces": [
                          "*"
                        ],
                        "labels": {},
                        "namespacesLabels": {},
                        "ingressClasses": [
                          "otoroshi"
                        ],
                        "defaultGroup": "default",
                        "ingresses": false,
                        "crds": true,
                        "coreDnsIntegration": false,
                        "coreDnsIntegrationDryRun": false,
                        "kubeLeader": false,
                        "restartDependantDeployments": false,
                        "watch": false,
                        "syncDaikokuApikeysOnly": false,
                        "kubeSystemNamespace": "kube-system",
                        "coreDnsConfigMapName": "coredns",
                        "coreDnsDeploymentName": "coredns",
                        "corednsPort": 53,
                        "otoroshiServiceName": "otoroshi-service",
                        "otoroshiNamespace": "${namespace}",
                        "clusterDomain": "cluster.local",
                        "syncIntervalSeconds": 60,
                        "coreDnsEnv": null,
                        "watchTimeoutSeconds": 60,
                        "watchGracePeriodSeconds": 5,
                        "mutatingWebhookName": "otoroshi-admission-webhook-injector",
                        "validatingWebhookName": "otoroshi-admission-webhook-validation",
                        "templates": {
                          "service-group": {},
                          "service-descriptor": {},
                          "apikeys": {},
                          "global-config": {},
                          "jwt-verifier": {},
                          "tcp-service": {},
                          "certificate": {},
                          "auth-module": {},
                          "script": {},
                          "organizations": {},
                          "teams": {},
                          "webhooks": {
                            "flags": {
                              "requestCert": true,
                              "originCheck": true,
                              "tokensCheck": true,
                              "displayEnv": false,
                              "tlsTrace": false
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }  
          - name: JAVA_OPTS
            value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0'
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          failureThreshold: 1
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /live
            port: 8080
          failureThreshold: 3
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        resources:
          # requests:
          #   cpu: "100m"
          #   memory: "50Mi"
          # limits:
          #   cpu: "4G"
          #   memory: "4Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-service
spec:
  selector:
    run: otoroshi-deployment
  ports:
  - port: 8080
    name: "http"
    targetPort: "http"
  - port: 8443
    name: "https"
    targetPort: "https"
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-external-service
spec:
  type: LoadBalancer
  selector:
    run: otoroshi-deployment
  ports:
  - port: 80
    name: "http"
    targetPort: "http"
  - port: 443
    name: "https"
    targetPort: "https"
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
  name: otoroshi-service-certificate
spec:
  description: certificate for otoroshi-service
  autoRenew: true
  csr:
    issuer: CN=Otoroshi Root
    hosts: 
    - otoroshi-service
    - otoroshi-service.${namespace}.svc.cluster.local
    - otoroshi-api-service.${namespace}.svc.cluster.local
    - otoroshi.${domain}
    - otoroshi-api.${domain}
    - privateapps.${domain}
    key:
      algo: rsa
      size: 2048
    subject: uid=otoroshi-service-cert, O=Otoroshi
    client: false
    ca: false
    duration: 31536000000
    signatureAlg: SHA256WithRSAEncryption
    digestAlg: SHA-256

dns.example

copyotoroshi.your.otoroshi.domain      IN CNAME generated.cname.of.your.cluster.loadbalancer
otoroshi-api.your.otoroshi.domain  IN CNAME generated.cname.of.your.cluster.loadbalancer
privateapps.your.otoroshi.domain   IN CNAME generated.cname.of.your.cluster.loadbalancer
api1.another.domain                IN CNAME generated.cname.of.your.cluster.loadbalancer
api2.another.domain                IN CNAME generated.cname.of.your.cluster.loadbalancer
*.api.the.api.domain               IN CNAME generated.cname.of.your.cluster.loadbalancer

Deploy a simple otoroshi instanciation on a bare metal kubernetes cluster

Here we have 2 replicas connected to the same redis instance. Nothing fancy. The otoroshi instance are exposed as nodePort so you’ll have to add a loadbalancer in front of your kubernetes nodes to route external traffic (TCP) to your otoroshi instances. You have to setup your DNS to bind otoroshi domain names to your loadbalancer (see the example below).

deployment.yaml

copyapiVersion: apps/v1
kind: Deployment
metadata:
  name: otoroshi-deployment
spec:
  selector:
    matchLabels:
      run: otoroshi-deployment
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
      maxSurge: 1
  template:
    metadata:
      labels:
        run: otoroshi-deployment
    spec:
      serviceAccountName: otoroshi-admin-user
      terminationGracePeriodSeconds: 60
      hostNetwork: false
      containers:
      - image: maif/otoroshi:16.16.1-jdk11
        imagePullPolicy: IfNotPresent
        name: otoroshi
        ports:
          - containerPort: 8080
            name: "http"
            protocol: TCP
          - containerPort: 8443
            name: "https"
            protocol: TCP
        env:
          - name: APP_STORAGE_ROOT
            value: otoroshi
          - name: OTOROSHI_INITIAL_ADMIN_PASSWORD
            value: ${password}
          - name: APP_DOMAIN
            value: ${domain}
          - name: APP_STORAGE
            value: lettuce
          - name: REDIS_URL
            value: ${redisUrl}
            # value: redis://redis-leader-service:6379/0
          - name: ADMIN_API_CLIENT_ID
            value: ${clientId}
          - name: ADMIN_API_CLIENT_SECRET
            value: ${clientSecret}
          - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN
            value: otoroshi-api-service.${namespace}.svc.cluster.local
          - name: OTOROSHI_SECRET
            value: ${otoroshiSecret}
          - name: HEALTH_LIMIT
            value: "5000"
          - name: SSL_OUTSIDE_CLIENT_AUTH
            value: Want
          - name: HTTPS_WANT_CLIENT_AUTH
            value: "true"
          - name: OTOROSHI_EXPOSED_PORTS_HTTP
            value: "80"
          - name: OTOROSHI_EXPOSED_PORTS_HTTPS
            value: "443"
          - name: OTOROSHI_INITIAL_CUSTOMIZATION
            value: >
              {
                "config":{
                  "tlsSettings": {
                    "defaultDomain": "www.${domain}",
                    "randomIfNotFound": false
                  },
                  "scripts":{
                    "enabled":true,
                    "sinkRefs":[
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator",
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector"
                    ],
                    "sinkConfig": {},
                    "jobRefs":[
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesOtoroshiCRDsControllerJob"
                    ],
                    "jobConfig":{
                      "KubernetesConfig": {
                        "trust": false,
                        "namespaces": [
                          "*"
                        ],
                        "labels": {},
                        "namespacesLabels": {},
                        "ingressClasses": [
                          "otoroshi"
                        ],
                        "defaultGroup": "default",
                        "ingresses": false,
                        "crds": true,
                        "coreDnsIntegration": false,
                        "coreDnsIntegrationDryRun": false,
                        "kubeLeader": false,
                        "restartDependantDeployments": false,
                        "watch": false,
                        "syncDaikokuApikeysOnly": false,
                        "kubeSystemNamespace": "kube-system",
                        "coreDnsConfigMapName": "coredns",
                        "coreDnsDeploymentName": "coredns",
                        "corednsPort": 53,
                        "otoroshiServiceName": "otoroshi-service",
                        "otoroshiNamespace": "${namespace}",
                        "clusterDomain": "cluster.local",
                        "syncIntervalSeconds": 60,
                        "coreDnsEnv": null,
                        "watchTimeoutSeconds": 60,
                        "watchGracePeriodSeconds": 5,
                        "mutatingWebhookName": "otoroshi-admission-webhook-injector",
                        "validatingWebhookName": "otoroshi-admission-webhook-validation",
                        "templates": {
                          "service-group": {},
                          "service-descriptor": {},
                          "apikeys": {},
                          "global-config": {},
                          "jwt-verifier": {},
                          "tcp-service": {},
                          "certificate": {},
                          "auth-module": {},
                          "script": {},
                          "organizations": {},
                          "teams": {},
                          "webhooks": {
                            "flags": {
                              "requestCert": true,
                              "originCheck": true,
                              "tokensCheck": true,
                              "displayEnv": false,
                              "tlsTrace": false
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }  
          - name: JAVA_OPTS
            value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0'
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          failureThreshold: 1
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /live
            port: 8080
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        resources:
          # requests:
          #   cpu: "100m"
          #   memory: "50Mi"
          # limits:
          #   cpu: "4G"
          #   memory: "4Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-service
spec:
  selector:
    run: otoroshi-deployment
  ports:
  - port: 8080
    name: "http"
    targetPort: "http"
  - port: 8443
    name: "https"
    targetPort: "https"
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-external-service
spec:
  selector:
    run: otoroshi-deployment
  ports:
  - port: 80
    name: "http"
    targetPort: "http"
    nodePort: 31080
  - port: 443
    name: "https"
    targetPort: "https"
    nodePort: 31443
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
  name: otoroshi-service-certificate
spec:
  description: certificate for otoroshi-service
  autoRenew: true
  csr:
    issuer: CN=Otoroshi Root
    hosts: 
    - otoroshi-service
    - otoroshi-service.${namespace}.svc.cluster.local
    - otoroshi-api-service.${namespace}.svc.cluster.local
    - otoroshi.${domain}
    - otoroshi-api.${domain}
    - privateapps.${domain}
    key:
      algo: rsa
      size: 2048
    subject: uid=otoroshi-service-cert, O=Otoroshi
    client: false
    ca: false
    duration: 31536000000
    signatureAlg: SHA256WithRSAEncryption
    digestAlg: SHA-256

haproxy.example

copyfrontend front_nodes_http
    bind *:80
    mode tcp
    default_backend back_http_nodes
    timeout client          1m

frontend front_nodes_https
    bind *:443
    mode tcp
    default_backend back_https_nodes
    timeout client          1m

backend back_http_nodes
    mode tcp
    balance roundrobin
    server kubernetes-node1 10.2.2.40:31080
    server kubernetes-node2 10.2.2.41:31080
    server kubernetes-node3 10.2.2.42:31080
    timeout connect        10s
    timeout server          1m

backend back_https_nodes
    mode tcp
    balance roundrobin
    server kubernetes-node1 10.2.2.40:31443
    server kubernetes-node2 10.2.2.41:31443
    server kubernetes-node3 10.2.2.42:31443
    timeout connect        10s
    timeout server          1m

nginx.example

copystream {

  upstream back_http_nodes {
    zone back_http_nodes 64k;
    server 10.2.2.40:31080 max_fails=1;
    server 10.2.2.41:31080 max_fails=1;
    server 10.2.2.42:31080 max_fails=1;
  }

  upstream back_https_nodes {
    zone back_https_nodes 64k;
    server 10.2.2.40:31443 max_fails=1;
    server 10.2.2.41:31443 max_fails=1;
    server 10.2.2.42:31443 max_fails=1;
  }

  server {
    listen     80;
    proxy_pass back_http_nodes;
    health_check;
  }

  server {
    listen     443;
    proxy_pass back_https_nodes;
    health_check;
  }
  
}

dns.example

copy# if your loadbalancer is at ip address 10.2.2.50

otoroshi.your.otoroshi.domain      IN A 10.2.2.50
otoroshi-api.your.otoroshi.domain  IN A 10.2.2.50
privateapps.your.otoroshi.domain   IN A 10.2.2.50
api1.another.domain                IN A 10.2.2.50
api2.another.domain                IN A 10.2.2.50
*.api.the.api.domain               IN A 10.2.2.50

Deploy a simple otoroshi instanciation on a bare metal kubernetes cluster using a DaemonSet

Here we have one otoroshi instance on each kubernetes node (with the otoroshi-kind: instance label) with redis persistance. The otoroshi instances are exposed as hostPort so you’ll have to add a loadbalancer in front of your kubernetes nodes to route external traffic (TCP) to your otoroshi instances. You have to setup your DNS to bind otoroshi domain names to your loadbalancer (see the example below).

deployment.yaml

copyapiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otoroshi-deployment
spec:
  selector:
    matchLabels:
      run: otoroshi-deployment
  template:
    metadata:
      labels:
        run: otoroshi-deployment
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: otoroshi-kind
                operator: In
                values:
                - instance
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      serviceAccountName: otoroshi-admin-user
      terminationGracePeriodSeconds: 60
      restartPolicy: Always
      hostNetwork: false
      containers:
      - image: maif/otoroshi:16.16.1-jdk11
        imagePullPolicy: IfNotPresent
        name: otoroshi
        ports:
          - containerPort: 8080
            hostPort: 41080
            name: "http"
            protocol: TCP
          - containerPort: 8443
            hostPort: 41443
            name: "https"
            protocol: TCP
        env:
          - name: APP_STORAGE_ROOT
            value: otoroshi
          - name: OTOROSHI_INITIAL_ADMIN_PASSWORD
            value: ${password}
          - name: APP_DOMAIN
            value: ${domain}
          - name: APP_STORAGE
            value: lettuce
          - name: REDIS_URL
            value: ${redisUrl}
            # value: redis://redis-leader-service:6379/0
          - name: ADMIN_API_CLIENT_ID
            value: ${clientId}
          - name: ADMIN_API_CLIENT_SECRET
            value: ${clientSecret}
          - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN
            value: otoroshi-api-service.${namespace}.svc.cluster.local
          - name: OTOROSHI_SECRET
            value: ${otoroshiSecret}
          - name: HEALTH_LIMIT
            value: "5000"
          - name: SSL_OUTSIDE_CLIENT_AUTH
            value: Want
          - name: HTTPS_WANT_CLIENT_AUTH
            value: "true"
          - name: OTOROSHI_EXPOSED_PORTS_HTTP
            value: "80"
          - name: OTOROSHI_EXPOSED_PORTS_HTTPS
            value: "443"
          - name: OTOROSHI_INITIAL_CUSTOMIZATION
            value: >
              {
                "config":{
                  "tlsSettings": {
                    "defaultDomain": "www.${domain}",
                    "randomIfNotFound": false
                  },
                  "scripts":{
                    "enabled":true,
                    "sinkRefs":[
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator",
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector"
                    ],
                    "sinkConfig": {},
                    "jobRefs":[
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesOtoroshiCRDsControllerJob"
                    ],
                    "jobConfig":{
                      "KubernetesConfig": {
                        "trust": false,
                        "namespaces": [
                          "*"
                        ],
                        "labels": {},
                        "namespacesLabels": {},
                        "ingressClasses": [
                          "otoroshi"
                        ],
                        "defaultGroup": "default",
                        "ingresses": false,
                        "crds": true,
                        "coreDnsIntegration": false,
                        "coreDnsIntegrationDryRun": false,
                        "kubeLeader": false,
                        "restartDependantDeployments": false,
                        "watch": false,
                        "syncDaikokuApikeysOnly": false,
                        "kubeSystemNamespace": "kube-system",
                        "coreDnsConfigMapName": "coredns",
                        "coreDnsDeploymentName": "coredns",
                        "corednsPort": 53,
                        "otoroshiServiceName": "otoroshi-service",
                        "otoroshiNamespace": "${namespace}",
                        "clusterDomain": "cluster.local",
                        "syncIntervalSeconds": 60,
                        "coreDnsEnv": null,
                        "watchTimeoutSeconds": 60,
                        "watchGracePeriodSeconds": 5,
                        "mutatingWebhookName": "otoroshi-admission-webhook-injector",
                        "validatingWebhookName": "otoroshi-admission-webhook-validation",
                        "templates": {
                          "service-group": {},
                          "service-descriptor": {},
                          "apikeys": {},
                          "global-config": {},
                          "jwt-verifier": {},
                          "tcp-service": {},
                          "certificate": {},
                          "auth-module": {},
                          "script": {},
                          "organizations": {},
                          "teams": {},
                          "webhooks": {
                            "flags": {
                              "requestCert": true,
                              "originCheck": true,
                              "tokensCheck": true,
                              "displayEnv": false,
                              "tlsTrace": false
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }  
          - name: JAVA_OPTS
            value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0'
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          failureThreshold: 1
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /live
            port: 8080
          failureThreshold: 3
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        resources:
          # requests:
          #   cpu: "100m"
          #   memory: "50Mi"
          # limits:
          #   cpu: "4G"
          #   memory: "4Gi"
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-service
spec:
  selector:
    run: otoroshi-deployment
  ports:
  - port: 8080
    name: "http"
    targetPort: "http"
  - port: 8443
    name: "https"
    targetPort: "https"
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
  name: otoroshi-service-certificate
spec:
  description: certificate for otoroshi-service
  autoRenew: true
  csr:
    issuer: CN=Otoroshi Root
    hosts: 
    - otoroshi-service
    - otoroshi-service.${namespace}.svc.cluster.local
    - otoroshi-api-service.${namespace}.svc.cluster.local
    - otoroshi.${domain}
    - otoroshi-api.${domain}
    - privateapps.${domain}
    key:
      algo: rsa
      size: 2048
    subject: uid=otoroshi-service-cert, O=Otoroshi
    client: false
    ca: false
    duration: 31536000000
    signatureAlg: SHA256WithRSAEncryption
    digestAlg: SHA-256

haproxy.example

copyfrontend front_nodes_http
    bind *:80
    mode tcp
    default_backend back_http_nodes
    timeout client          1m

frontend front_nodes_https
    bind *:443
    mode tcp
    default_backend back_https_nodes
    timeout client          1m

backend back_http_nodes
    mode tcp
    balance roundrobin
    server kubernetes-node1 10.2.2.40:41080
    server kubernetes-node2 10.2.2.41:41080
    server kubernetes-node3 10.2.2.42:41080
    timeout connect        10s
    timeout server          1m

backend back_https_nodes
    mode tcp
    balance roundrobin
    server kubernetes-node1 10.2.2.40:41443
    server kubernetes-node2 10.2.2.41:41443
    server kubernetes-node3 10.2.2.42:41443
    timeout connect        10s
    timeout server          1m

nginx.example

copystream {

  upstream back_http_nodes {
    zone back_http_nodes 64k;
    server 10.2.2.40:41080 max_fails=1;
    server 10.2.2.41:41080 max_fails=1;
    server 10.2.2.42:41080 max_fails=1;
  }

  upstream back_https_nodes {
    zone back_https_nodes 64k;
    server 10.2.2.40:41443 max_fails=1;
    server 10.2.2.41:41443 max_fails=1;
    server 10.2.2.42:41443 max_fails=1;
  }

  server {
    listen     80;
    proxy_pass back_http_nodes;
    health_check;
  }

  server {
    listen     443;
    proxy_pass back_https_nodes;
    health_check;
  }
  
}

dns.example

copy# if your loadbalancer is at ip address 10.2.2.50

otoroshi.your.otoroshi.domain      IN A 10.2.2.50
otoroshi-api.your.otoroshi.domain  IN A 10.2.2.50
privateapps.your.otoroshi.domain   IN A 10.2.2.50
api1.another.domain                IN A 10.2.2.50
api2.another.domain                IN A 10.2.2.50
*.api.the.api.domain               IN A 10.2.2.50

Deploy an otoroshi cluster on a cloud provider managed kubernetes cluster

Here we have 2 replicas of an otoroshi leader connected to a redis instance and 2 replicas of an otoroshi worker connected to the leader. We use a service of type LoadBalancer to expose otoroshi leader/worker to the rest of the world. You have to setup your DNS to bind otoroshi domain names to the LoadBalancer external CNAME (see the example below)

deployment.yaml

copy---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otoroshi-leader-deployment
spec:
  selector:
    matchLabels:
      run: otoroshi-leader-deployment
  template:
    metadata:
      labels:
        run: otoroshi-leader-deployment
    replicas: 2
    strategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: 1
        maxSurge: 1
    spec:
      serviceAccountName: otoroshi-admin-user
      terminationGracePeriodSeconds: 60
      hostNetwork: false
      restartPolicy: Always
      containers:
      - image: maif/otoroshi:16.16.1-jdk11
        imagePullPolicy: IfNotPresent
        name: otoroshi-leader
        ports:
          - containerPort: 8080
            name: "http"
            protocol: TCP
          - containerPort: 8443
            name: "https"
            protocol: TCP
        env:
          - name: APP_STORAGE_ROOT
            value: otoroshi
          - name: OTOROSHI_INITIAL_ADMIN_PASSWORD
            value: ${password}
          - name: APP_DOMAIN
            value: ${domain}
          - name: APP_STORAGE
            value: lettuce
          - name: REDIS_URL
            value: ${redisUrl}
            # value: redis://redis-leader-service:6379/0
          - name: CLUSTER_MODE
            value: Leader
          - name: CLUSTER_AUTO_UPDATE_STATE
            value: 'true'
          - name: CLUSTER_MTLS_ENABLED
            value: 'true'
          - name: CLUSTER_MTLS_LOOSE
            value: 'true'
          - name: CLUSTER_MTLS_TRUST_ALL
            value: 'true'
          - name: CLUSTER_LEADER_URL
            value: https://otoroshi-leader-api-service.${namespace}.svc.cluster.local:8443
          - name: CLUSTER_LEADER_HOST
            value: otoroshi-leader-api-service.${namespace}.svc.cluster.local
          - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN
            value: otoroshi-leader-api-service.${namespace}.svc.cluster.local
          - name: ADMIN_API_CLIENT_ID
            value: ${clientId}
          - name: CLUSTER_LEADER_CLIENT_ID
            value: ${clientId}
          - name: ADMIN_API_CLIENT_SECRET
            value: ${clientSecret}
          - name: CLUSTER_LEADER_CLIENT_SECRET
            value: ${clientSecret}
          - name: OTOROSHI_SECRET
            value: ${otoroshiSecret}
          - name: HEALTH_LIMIT
            value: "5000"
          - name: SSL_OUTSIDE_CLIENT_AUTH
            value: Want
          - name: HTTPS_WANT_CLIENT_AUTH
            value: "true"
          - name: OTOROSHI_INITIAL_CUSTOMIZATION
            value: >
              {
                "config":{
                  "tlsSettings": {
                    "defaultDomain": "www.${domain}",
                    "randomIfNotFound": false
                  },
                  "scripts":{
                    "enabled":true,
                    "sinkRefs":[
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator",
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector"
                    ],
                    "sinkConfig": {},
                    "jobRefs":[
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesOtoroshiCRDsControllerJob"
                    ],
                    "jobConfig":{
                      "KubernetesConfig": {
                        "trust": false,
                        "namespaces": [
                          "*"
                        ],
                        "labels": {},
                        "namespacesLabels": {},
                        "ingressClasses": [
                          "otoroshi"
                        ],
                        "defaultGroup": "default",
                        "ingresses": false,
                        "crds": true,
                        "coreDnsIntegration": false,
                        "coreDnsIntegrationDryRun": false,
                        "kubeLeader": false,
                        "restartDependantDeployments": false,
                        "watch": false,
                        "syncDaikokuApikeysOnly": false,
                        "kubeSystemNamespace": "kube-system",
                        "coreDnsConfigMapName": "coredns",
                        "coreDnsDeploymentName": "coredns",
                        "corednsPort": 53,
                        "otoroshiServiceName": "otoroshi-worker-service",
                        "otoroshiNamespace": "${namespace}",
                        "clusterDomain": "cluster.local",
                        "syncIntervalSeconds": 60,
                        "coreDnsEnv": null,
                        "watchTimeoutSeconds": 60,
                        "watchGracePeriodSeconds": 5,
                        "mutatingWebhookName": "otoroshi-admission-webhook-injector",
                        "validatingWebhookName": "otoroshi-admission-webhook-validation",
                        "templates": {
                          "service-group": {},
                          "service-descriptor": {},
                          "apikeys": {},
                          "global-config": {},
                          "jwt-verifier": {},
                          "tcp-service": {},
                          "certificate": {},
                          "auth-module": {},
                          "script": {},
                          "organizations": {},
                          "teams": {},
                          "webhooks": {
                            "flags": {
                              "requestCert": true,
                              "originCheck": true,
                              "tokensCheck": true,
                              "displayEnv": false,
                              "tlsTrace": false
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }        
          - name: JAVA_OPTS
            value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0'
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          failureThreshold: 1
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /live
            port: 8080
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otoroshi-worker-deployment
spec:
  selector:
    matchLabels:
      run: otoroshi-worker-deployment
  template:
    metadata:
      labels:
        run: otoroshi-worker-deployment
    replicas: 2
    strategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: 1
        maxSurge: 1
    spec:
      serviceAccountName: otoroshi-admin-user
      terminationGracePeriodSeconds: 60
      hostNetwork: false
      restartPolicy: Always   
      containers:
      - image: maif/otoroshi:16.8.0-rc.1-dev
        imagePullPolicy: IfNotPresent
        name: otoroshi-worker
        ports:
          - containerPort: 8080
            name: "http"
            protocol: TCP
          - containerPort: 8443
            name: "https"
            protocol: TCP
        env:
          - name: APP_STORAGE_ROOT
            value: otoroshi
          - name: OTOROSHI_INITIAL_ADMIN_PASSWORD
            value: ${password}
          - name: APP_DOMAIN
            value: ${domain}
          - name: CLUSTER_MODE
            value: Worker
          - name: CLUSTER_AUTO_UPDATE_STATE
            value: 'true'
          - name: CLUSTER_MTLS_ENABLED
            value: 'true'
          - name: CLUSTER_MTLS_LOOSE
            value: 'true'
          - name: CLUSTER_MTLS_TRUST_ALL
            value: 'true'
          - name: CLUSTER_LEADER_URL
            value: https://otoroshi-leader-api-service.${namespace}.svc.cluster.local:8443
          - name: CLUSTER_LEADER_HOST
            value: otoroshi-leader-api-service.${namespace}.svc.cluster.local
          - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN
            value: otoroshi-leader-api-service.${namespace}.svc.cluster.local
          - name: ADMIN_API_CLIENT_ID
            value: ${clientId}
          - name: CLUSTER_LEADER_CLIENT_ID
            value: ${clientId}
          - name: ADMIN_API_CLIENT_SECRET
            value: ${clientSecret}
          - name: CLUSTER_LEADER_CLIENT_SECRET
            value: ${clientSecret}
          - name: HEALTH_LIMIT
            value: "5000"
          - name: SSL_OUTSIDE_CLIENT_AUTH
            value: Want
          - name: HTTPS_WANT_CLIENT_AUTH
            value: "true"
          - name: JAVA_OPTS
            value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0'
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          failureThreshold: 1
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /live
            port: 8080
          failureThreshold: 3
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-leader-api-service
spec:
  selector:
    run: otoroshi-leader-deployment
  ports:
  - port: 8080
    name: "http"
    targetPort: "http"
  - port: 8443
    name: "https"
    targetPort: "https"
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-leader-service
spec:
  selector:
    run: otoroshi-leader-deployment
  ports:
  - port: 8080
    name: "http"
    targetPort: "http"
  - port: 8443
    name: "https"
    targetPort: "https"
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-worker-service
spec:
  selector:
    run: otoroshi-worker-deployment
  ports:
  - port: 8080
    name: "http"
    targetPort: "http"
  - port: 8443
    name: "https"
    targetPort: "https"
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-leader-external-service
spec:
  type: LoadBalancer
  selector:
    run: otoroshi-leader-deployment
  ports:
  - port: 80
    name: "http"
    targetPort: "http"
  - port: 443
    name: "https"
    targetPort: "https"
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-worker-external-service
spec:
  type: LoadBalancer
  selector:
    run: otoroshi-worker-deployment
  ports:
  - port: 80
    name: "http"
    targetPort: "http"
  - port: 443
    name: "https"
    targetPort: "https"
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
  name: otoroshi-service-certificate
spec:
  description: certificate for otoroshi-service
  autoRenew: true
  csr:
    issuer: CN=Otoroshi Root
    hosts: 
    - otoroshi-service
    - otoroshi-service.${namespace}.svc.cluster.local
    - otoroshi-api-service.${namespace}.svc.cluster.local
    - otoroshi.${domain}
    - otoroshi-api.${domain}
    - privateapps.${domain}
    key:
      algo: rsa
      size: 2048
    subject: uid=otoroshi-service-cert, O=Otoroshi
    client: false
    ca: false
    duration: 31536000000
    signatureAlg: SHA256WithRSAEncryption
    digestAlg: SHA-256

dns.example

copyotoroshi.your.otoroshi.domain      IN CNAME generated.cname.for.leader.of.your.cluster.loadbalancer
otoroshi-api.your.otoroshi.domain  IN CNAME generated.cname.for.leader.of.your.cluster.loadbalancer
privateapps.your.otoroshi.domain   IN CNAME generated.cname.for.leader.of.your.cluster.loadbalancer

api1.another.domain                IN CNAME generated.cname.for.worker.of.your.cluster.loadbalancer
api2.another.domain                IN CNAME generated.cname.for.worker.of.your.cluster.loadbalancer
*.api.the.api.domain               IN CNAME generated.cname.for.worker.of.your.cluster.loadbalancer

Deploy an otoroshi cluster on a bare metal kubernetes cluster

Here we have 2 replicas of otoroshi leader connected to the same redis instance and 2 replicas for otoroshi worker. The otoroshi instances are exposed as nodePort so you’ll have to add a loadbalancer in front of your kubernetes nodes to route external traffic (TCP) to your otoroshi instances. You have to setup your DNS to bind otoroshi domain names to your loadbalancer (see the example below).

deployment.yaml

copy---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otoroshi-leader-deployment
spec:
  selector:
    matchLabels:
      run: otoroshi-leader-deployment
  template:
    metadata:
      labels:
        run: otoroshi-leader-deployment
    replicas: 2
    strategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: 1
        maxSurge: 1
    spec:
      serviceAccountName: otoroshi-admin-user
      terminationGracePeriodSeconds: 60
      hostNetwork: false
      restartPolicy: Always
      containers:
      - image: maif/otoroshi:16.16.1-jdk11
        imagePullPolicy: IfNotPresent
        name: otoroshi-leader
        ports:
          - containerPort: 8080
            name: "http"
            protocol: TCP
          - containerPort: 8443
            name: "https"
            protocol: TCP
        env:
          - name: APP_STORAGE_ROOT
            value: otoroshi
          - name: OTOROSHI_INITIAL_ADMIN_PASSWORD
            value: ${password}
          - name: APP_DOMAIN
            value: ${domain}
          - name: APP_STORAGE
            value: lettuce
          - name: REDIS_URL
            value: ${redisUrl}
            # value: redis://redis-leader-service:6379/0
          - name: CLUSTER_MODE
            value: Leader
          - name: CLUSTER_AUTO_UPDATE_STATE
            value: 'true'
          - name: CLUSTER_MTLS_ENABLED
            value: 'true'
          - name: CLUSTER_MTLS_LOOSE
            value: 'true'
          - name: CLUSTER_MTLS_TRUST_ALL
            value: 'true'
          - name: CLUSTER_LEADER_URL
            value: https://otoroshi-leader-api-service.${namespace}.svc.cluster.local:8443
          - name: CLUSTER_LEADER_HOST
            value: otoroshi-leader-api-service.${namespace}.svc.cluster.local
          - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN
            value: otoroshi-leader-api-service.${namespace}.svc.cluster.local
          - name: ADMIN_API_CLIENT_ID
            value: ${clientId}
          - name: CLUSTER_LEADER_CLIENT_ID
            value: ${clientId}
          - name: ADMIN_API_CLIENT_SECRET
            value: ${clientSecret}
          - name: CLUSTER_LEADER_CLIENT_SECRET
            value: ${clientSecret}
          - name: OTOROSHI_SECRET
            value: ${otoroshiSecret}
          - name: HEALTH_LIMIT
            value: "5000"
          - name: SSL_OUTSIDE_CLIENT_AUTH
            value: Want
          - name: HTTPS_WANT_CLIENT_AUTH
            value: "true"
          - name: OTOROSHI_INITIAL_CUSTOMIZATION
            value: >
              {
                "config":{
                  "tlsSettings": {
                    "defaultDomain": "www.${domain}",
                    "randomIfNotFound": false
                  },
                  "scripts":{
                    "enabled":true,
                    "sinkRefs":[
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator",
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector"
                    ],
                    "sinkConfig": {},
                    "jobRefs":[
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesOtoroshiCRDsControllerJob"
                    ],
                    "jobConfig":{
                      "KubernetesConfig": {
                        "trust": false,
                        "namespaces": [
                          "*"
                        ],
                        "labels": {},
                        "namespacesLabels": {},
                        "ingressClasses": [
                          "otoroshi"
                        ],
                        "defaultGroup": "default",
                        "ingresses": false,
                        "crds": true,
                        "coreDnsIntegration": false,
                        "coreDnsIntegrationDryRun": false,
                        "kubeLeader": false,
                        "restartDependantDeployments": false,
                        "watch": false,
                        "syncDaikokuApikeysOnly": false,
                        "kubeSystemNamespace": "kube-system",
                        "coreDnsConfigMapName": "coredns",
                        "coreDnsDeploymentName": "coredns",
                        "corednsPort": 53,
                        "otoroshiServiceName": "otoroshi-worker-service",
                        "otoroshiNamespace": "${namespace}",
                        "clusterDomain": "cluster.local",
                        "syncIntervalSeconds": 60,
                        "coreDnsEnv": null,
                        "watchTimeoutSeconds": 60,
                        "watchGracePeriodSeconds": 5,
                        "mutatingWebhookName": "otoroshi-admission-webhook-injector",
                        "validatingWebhookName": "otoroshi-admission-webhook-validation",
                        "templates": {
                          "service-group": {},
                          "service-descriptor": {},
                          "apikeys": {},
                          "global-config": {},
                          "jwt-verifier": {},
                          "tcp-service": {},
                          "certificate": {},
                          "auth-module": {},
                          "script": {},
                          "organizations": {},
                          "teams": {},
                          "webhooks": {
                            "flags": {
                              "requestCert": true,
                              "originCheck": true,
                              "tokensCheck": true,
                              "displayEnv": false,
                              "tlsTrace": false
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }  
          - name: JAVA_OPTS
            value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0'
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          failureThreshold: 1
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /live
            port: 8080
          failureThreshold: 3
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otoroshi-worker-deployment
spec:
  selector:
    matchLabels:
      run: otoroshi-worker-deployment
  template:
    metadata:
      labels:
        run: otoroshi-worker-deployment
    replicas: 2
    strategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: 1
        maxSurge: 1
    spec:
      serviceAccountName: otoroshi-admin-user
      terminationGracePeriodSeconds: 60
      hostNetwork: false
      restartPolicy: Always   
      containers:
      - image: maif/otoroshi:16.16.1-dev-dev
        imagePullPolicy: IfNotPresent
        name: otoroshi-worker
        ports:
          - containerPort: 8080
            name: "http"
            protocol: TCP
          - containerPort: 8443
            name: "https"
            protocol: TCP
        env:
          - name: APP_STORAGE_ROOT
            value: otoroshi
          - name: OTOROSHI_INITIAL_ADMIN_PASSWORD
            value: ${password}
          - name: APP_DOMAIN
            value: ${domain}
          - name: CLUSTER_MODE
            value: Worker
          - name: CLUSTER_AUTO_UPDATE_STATE
            value: 'true'
          - name: CLUSTER_MTLS_ENABLED
            value: 'true'
          - name: CLUSTER_MTLS_LOOSE
            value: 'true'
          - name: CLUSTER_MTLS_TRUST_ALL
            value: 'true'
          - name: CLUSTER_LEADER_URL
            value: https://otoroshi-leader-api-service.${namespace}.svc.cluster.local:8443
          - name: CLUSTER_LEADER_HOST
            value: otoroshi-leader-api-service.${namespace}.svc.cluster.local
          - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN
            value: otoroshi-leader-api-service.${namespace}.svc.cluster.local
          - name: ADMIN_API_CLIENT_ID
            value: ${clientId}
          - name: CLUSTER_LEADER_CLIENT_ID
            value: ${clientId}
          - name: ADMIN_API_CLIENT_SECRET
            value: ${clientSecret}
          - name: CLUSTER_LEADER_CLIENT_SECRET
            value: ${clientSecret}
          - name: HEALTH_LIMIT
            value: "5000"
          - name: SSL_OUTSIDE_CLIENT_AUTH
            value: Want
          - name: HTTPS_WANT_CLIENT_AUTH
            value: "true"
          - name: JAVA_OPTS
            value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0'
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          failureThreshold: 1
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /live
            port: 8080
          failureThreshold: 3
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-leader-api-service
spec:
  selector:
    run: otoroshi-leader-deployment
  ports:
  - port: 8080
    name: "http"
    targetPort: "http"
  - port: 8443
    name: "https"
    targetPort: "https"
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-leader-service
spec:
  selector:
    run: otoroshi-leader-deployment
  ports:
  - port: 8080
    nodePort: 31080
    name: "http"
    targetPort: "http"
  - port: 8443
    nodePort: 31443
    name: "https"
    targetPort: "https"
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-worker-service
spec:
  selector:
    run: otoroshi-worker-deployment
  ports:
  - port: 8080
    nodePort: 32080
    name: "http"
    targetPort: "http"
  - port: 8443
    nodePort: 32443
    name: "https"
    targetPort: "https"
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
  name: otoroshi-service-certificate
spec:
  description: certificate for otoroshi-service
  autoRenew: true
  csr:
    issuer: CN=Otoroshi Root
    hosts: 
    - otoroshi-service
    - otoroshi-service.${namespace}.svc.cluster.local
    - otoroshi-api-service.${namespace}.svc.cluster.local
    - otoroshi.${domain}
    - otoroshi-api.${domain}
    - privateapps.${domain}
    key:
      algo: rsa
      size: 2048
    subject: uid=otoroshi-service-cert, O=Otoroshi
    client: false
    ca: false
    duration: 31536000000
    signatureAlg: SHA256WithRSAEncryption
    digestAlg: SHA-256

nginx.example

copystream {

  upstream worker_http_nodes {
    zone worker_http_nodes 64k;
    server 10.2.2.40:32080 max_fails=1;
    server 10.2.2.41:32080 max_fails=1;
    server 10.2.2.42:32080 max_fails=1;
  }

  upstream worker_https_nodes {
    zone worker_https_nodes 64k;
    server 10.2.2.40:32443 max_fails=1;
    server 10.2.2.41:32443 max_fails=1;
    server 10.2.2.42:32443 max_fails=1;
  }

  upstream leader_http_nodes {
    zone leader_http_nodes 64k;
    server 10.2.2.40:31080 max_fails=1;
    server 10.2.2.41:31080 max_fails=1;
    server 10.2.2.42:31080 max_fails=1;
  }

  upstream leader_https_nodes {
    zone leader_https_nodes 64k;
    server 10.2.2.40:31443 max_fails=1;
    server 10.2.2.41:31443 max_fails=1;
    server 10.2.2.42:31443 max_fails=1;
  }

  server {
    listen     80;
    proxy_pass worker_http_nodes;
    health_check;
  }

  server {
    listen     443;
    proxy_pass worker_https_nodes;
    health_check;
  }

  server {
    listen     81;
    proxy_pass leader_http_nodes;
    health_check;
  }

  server {
    listen     444;
    proxy_pass leader_https_nodes;
    health_check;
  }
  
}

dns.example

copy# if your loadbalancer is at ip address 10.2.2.50

otoroshi.your.otoroshi.domain      IN A 10.2.2.50
otoroshi-api.your.otoroshi.domain  IN A 10.2.2.50
privateapps.your.otoroshi.domain   IN A 10.2.2.50
api1.another.domain                IN A 10.2.2.50
api2.another.domain                IN A 10.2.2.50
*.api.the.api.domain               IN A 10.2.2.50

dns.example

copy# if your loadbalancer is at ip address 10.2.2.50

otoroshi.your.otoroshi.domain      IN A 10.2.2.50
otoroshi-api.your.otoroshi.domain  IN A 10.2.2.50
privateapps.your.otoroshi.domain   IN A 10.2.2.50
api1.another.domain                IN A 10.2.2.50
api2.another.domain                IN A 10.2.2.50
*.api.the.api.domain               IN A 10.2.2.50

Deploy an otoroshi cluster on a bare metal kubernetes cluster using DaemonSet

Here we have 1 otoroshi leader instance on each kubernetes node (with the otoroshi-kind: leader label) connected to the same redis instance and 1 otoroshi worker instance on each kubernetes node (with the otoroshi-kind: worker label). The otoroshi instances are exposed as nodePort so you’ll have to add a loadbalancer in front of your kubernetes nodes to route external traffic (TCP) to your otoroshi instances. You have to setup your DNS to bind otoroshi domain names to your loadbalancer (see the example below).

deployment.yaml

copy---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otoroshi-leader-deployment
spec:
  selector:
    matchLabels:
      run: otoroshi-leader-deployment
  template:
    metadata:
      labels:
        run: otoroshi-leader-deployment
    strategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: 1
        maxSurge: 1
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: otoroshi-kind
                operator: In
                values:
                - leader
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      serviceAccountName: otoroshi-admin-user
      terminationGracePeriodSeconds: 60
      hostNetwork: false
      restartPolicy: Always
      containers:
      - image: maif/otoroshi:16.16.1-jdk11
        imagePullPolicy: IfNotPresent
        name: otoroshi-leader
        ports:
          - containerPort: 8080
            hostPort: 41080
            name: "http"
            protocol: TCP
          - containerPort: 8443
            hostPort: 41443
            name: "https"
            protocol: TCP
        env:
          - name: APP_STORAGE_ROOT
            value: otoroshi
          - name: OTOROSHI_INITIAL_ADMIN_PASSWORD
            value: ${password}
          - name: APP_DOMAIN
            value: ${domain}
          - name: APP_STORAGE
            value: lettuce
          - name: REDIS_URL
            value: ${redisUrl}
            # value: redis://redis-leader-service:6379/0
          - name: CLUSTER_MODE
            value: Leader
          - name: CLUSTER_AUTO_UPDATE_STATE
            value: 'true'
          - name: CLUSTER_MTLS_ENABLED
            value: 'true'
          - name: CLUSTER_MTLS_LOOSE
            value: 'true'
          - name: CLUSTER_MTLS_TRUST_ALL
            value: 'true'
          - name: CLUSTER_LEADER_URL
            value: https://otoroshi-leader-api-service.${namespace}.svc.cluster.local:8443
          - name: CLUSTER_LEADER_HOST
            value: otoroshi-leader-api-service.${namespace}.svc.cluster.local
          - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN
            value: otoroshi-leader-api-service.${namespace}.svc.cluster.local
          - name: ADMIN_API_CLIENT_ID
            value: ${clientId}
          - name: CLUSTER_LEADER_CLIENT_ID
            value: ${clientId}
          - name: ADMIN_API_CLIENT_SECRET
            value: ${clientSecret}
          - name: CLUSTER_LEADER_CLIENT_SECRET
            value: ${clientSecret}
          - name: OTOROSHI_SECRET
            value: ${otoroshiSecret}
          - name: HEALTH_LIMIT
            value: "5000"
          - name: SSL_OUTSIDE_CLIENT_AUTH
            value: Want
          - name: HTTPS_WANT_CLIENT_AUTH
            value: "true"
          - name: OTOROSHI_INITIAL_CUSTOMIZATION
            value: >
              {
                "config":{
                  "tlsSettings": {
                    "defaultDomain": "www.${domain}",
                    "randomIfNotFound": false
                  },
                  "scripts":{
                    "enabled":true,
                    "sinkRefs":[
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator",
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector"
                    ],
                    "sinkConfig": {},
                    "jobRefs":[
                      "cp:otoroshi.plugins.jobs.kubernetes.KubernetesOtoroshiCRDsControllerJob"
                    ],
                    "jobConfig":{
                      "KubernetesConfig": {
                        "trust": false,
                        "namespaces": [
                          "*"
                        ],
                        "labels": {},
                        "namespacesLabels": {},
                        "ingressClasses": [
                          "otoroshi"
                        ],
                        "defaultGroup": "default",
                        "ingresses": false,
                        "crds": true,
                        "coreDnsIntegration": false,
                        "coreDnsIntegrationDryRun": false,
                        "kubeLeader": false,
                        "restartDependantDeployments": false,
                        "watch": false,
                        "syncDaikokuApikeysOnly": false,
                        "kubeSystemNamespace": "kube-system",
                        "coreDnsConfigMapName": "coredns",
                        "coreDnsDeploymentName": "coredns",
                        "corednsPort": 53,
                        "otoroshiServiceName": "otoroshi-worker-service",
                        "otoroshiNamespace": "${namespace}",
                        "clusterDomain": "cluster.local",
                        "syncIntervalSeconds": 60,
                        "coreDnsEnv": null,
                        "watchTimeoutSeconds": 60,
                        "watchGracePeriodSeconds": 5,
                        "mutatingWebhookName": "otoroshi-admission-webhook-injector",
                        "validatingWebhookName": "otoroshi-admission-webhook-validation",
                        "templates": {
                          "service-group": {},
                          "service-descriptor": {},
                          "apikeys": {},
                          "global-config": {},
                          "jwt-verifier": {},
                          "tcp-service": {},
                          "certificate": {},
                          "auth-module": {},
                          "script": {},
                          "organizations": {},
                          "teams": {},
                          "webhooks": {
                            "flags": {
                              "requestCert": true,
                              "originCheck": true,
                              "tokensCheck": true,
                              "displayEnv": false,
                              "tlsTrace": false
                            }
                          }
                        }
                      }
                    }
                  }
                }
              }  
          - name: JAVA_OPTS
            value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0'
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          failureThreshold: 1
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /live
            port: 8080
          failureThreshold: 3
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: otoroshi-worker-deployment
spec:
  selector:
    matchLabels:
      run: otoroshi-worker-deployment
  template:
    metadata:
      labels:
        run: otoroshi-worker-deployment
    replicas: 2
    strategy:
      type: RollingUpdate
      rollingUpdate:
        maxUnavailable: 1
        maxSurge: 1
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: otoroshi-kind
                operator: In
                values:
                - worker
      tolerations:
      - key: node-role.kubernetes.io/master
        effect: NoSchedule
      serviceAccountName: otoroshi-admin-user
      terminationGracePeriodSeconds: 60
      hostNetwork: false
      restartPolicy: Always   
      containers:
      - image: maif/otoroshi:16.16.1-dev-dev
        imagePullPolicy: IfNotPresent
        name: otoroshi-worker
        ports:
          - containerPort: 8080
            hostPort: 42080
            name: "http"
            protocol: TCP
          - containerPort: 8443
            hostPort: 42443
            name: "https"
            protocol: TCP
        env:
          - name: APP_STORAGE_ROOT
            value: otoroshi
          - name: OTOROSHI_INITIAL_ADMIN_PASSWORD
            value: ${password}
          - name: APP_DOMAIN
            value: ${domain}
          - name: CLUSTER_MODE
            value: Worker
          - name: CLUSTER_AUTO_UPDATE_STATE
            value: 'true'
          - name: CLUSTER_MTLS_ENABLED
            value: 'true'
          - name: CLUSTER_MTLS_LOOSE
            value: 'true'
          - name: CLUSTER_MTLS_TRUST_ALL
            value: 'true'
          - name: CLUSTER_LEADER_URL
            value: https://otoroshi-leader-api-service.${namespace}.svc.cluster.local:8443
          - name: CLUSTER_LEADER_HOST
            value: otoroshi-leader-api-service.${namespace}.svc.cluster.local
          - name: ADMIN_API_ADDITIONAL_EXPOSED_DOMAIN
            value: otoroshi-leader-api-service.${namespace}.svc.cluster.local
          - name: ADMIN_API_CLIENT_ID
            value: ${clientId}
          - name: CLUSTER_LEADER_CLIENT_ID
            value: ${clientId}
          - name: ADMIN_API_CLIENT_SECRET
            value: ${clientSecret}
          - name: CLUSTER_LEADER_CLIENT_SECRET
            value: ${clientSecret}
          - name: HEALTH_LIMIT
            value: "5000"
          - name: SSL_OUTSIDE_CLIENT_AUTH
            value: Want
          - name: HTTPS_WANT_CLIENT_AUTH
            value: "true"
          - name: JAVA_OPTS
            value: '-Xms2g -Xmx4g -XX:+UseContainerSupport -XX:MaxRAMPercentage=80.0'
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          failureThreshold: 1
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
        livenessProbe:
          httpGet:
            path: /live
            port: 8080
          failureThreshold: 3
          initialDelaySeconds: 60
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 2
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-leader-api-service
spec:
  selector:
    run: otoroshi-leader-deployment
  ports:
  - port: 8080
    name: "http"
    targetPort: "http"
  - port: 8443
    name: "https"
    targetPort: "https"
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-leader-service
spec:
  selector:
    run: otoroshi-leader-deployment
  ports:
  - port: 8080
    name: "http"
    targetPort: "http"
  - port: 8443
    name: "https"
    targetPort: "https"
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-worker-service
spec:
  selector:
    run: otoroshi-worker-deployment
  ports:
  - port: 8080
    name: "http"
    targetPort: "http"
  - port: 8443
    name: "https"
    targetPort: "https"
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
  name: otoroshi-service-certificate
spec:
  description: certificate for otoroshi-service
  autoRenew: true
  csr:
    issuer: CN=Otoroshi Root
    hosts: 
    - otoroshi-service
    - otoroshi-service.${namespace}.svc.cluster.local
    - otoroshi-api-service.${namespace}.svc.cluster.local
    - otoroshi.${domain}
    - otoroshi-api.${domain}
    - privateapps.${domain}
    key:
      algo: rsa
      size: 2048
    subject: uid=otoroshi-service-cert, O=Otoroshi
    client: false
    ca: false
    duration: 31536000000
    signatureAlg: SHA256WithRSAEncryption
    digestAlg: SHA-256

nginx.example

copystream {

  upstream worker_http_nodes {
    zone worker_http_nodes 64k;
    server 10.2.2.40:42080 max_fails=1;
    server 10.2.2.41:42080 max_fails=1;
    server 10.2.2.42:42080 max_fails=1;
  }

  upstream worker_https_nodes {
    zone worker_https_nodes 64k;
    server 10.2.2.40:42443 max_fails=1;
    server 10.2.2.41:42443 max_fails=1;
    server 10.2.2.42:42443 max_fails=1;
  }

  upstream leader_http_nodes {
    zone leader_http_nodes 64k;
    server 10.2.2.40:41080 max_fails=1;
    server 10.2.2.41:41080 max_fails=1;
    server 10.2.2.42:41080 max_fails=1;
  }

  upstream leader_https_nodes {
    zone leader_https_nodes 64k;
    server 10.2.2.40:41443 max_fails=1;
    server 10.2.2.41:41443 max_fails=1;
    server 10.2.2.42:41443 max_fails=1;
  }

  server {
    listen     80;
    proxy_pass worker_http_nodes;
    health_check;
  }

  server {
    listen     443;
    proxy_pass worker_https_nodes;
    health_check;
  }

  server {
    listen     81;
    proxy_pass leader_http_nodes;
    health_check;
  }

  server {
    listen     444;
    proxy_pass leader_https_nodes;
    health_check;
  }
  
}

dns.example

copy# if your loadbalancer is at ip address 10.2.2.50

otoroshi.your.otoroshi.domain      IN A 10.2.2.50
otoroshi-api.your.otoroshi.domain  IN A 10.2.2.50
privateapps.your.otoroshi.domain   IN A 10.2.2.50
api1.another.domain                IN A 10.2.2.50
api2.another.domain                IN A 10.2.2.50
*.api.the.api.domain               IN A 10.2.2.50

dns.example

copy# if your loadbalancer is at ip address 10.2.2.50

otoroshi.your.otoroshi.domain      IN A 10.2.2.50
otoroshi-api.your.otoroshi.domain  IN A 10.2.2.50
privateapps.your.otoroshi.domain   IN A 10.2.2.50
api1.another.domain                IN A 10.2.2.50
api2.another.domain                IN A 10.2.2.50
*.api.the.api.domain               IN A 10.2.2.50

Using Otoroshi as an Ingress Controller

If you want to use Otoroshi as an Ingress Controller, just go to the danger zone, and in Global scripts add the job named Kubernetes Ingress Controller.

Then add the following configuration for the job (with your own tweaks of course)

{
  "KubernetesConfig": {
    "enabled": true,
    "endpoint": "https://127.0.0.1:6443",
    "token": "eyJhbGciOiJSUzI....F463SrpOehQRaQ",
    "namespaces": [
      "*"
    ]
  }
}

the configuration can have the following values

{
  "KubernetesConfig": {
    "endpoint": "https://127.0.0.1:6443", // the endpoint to talk to the kubernetes api, optional
    "token": "xxxx", // the bearer token to talk to the kubernetes api, optional
    "userPassword": "user:password", // the user password tuple to talk to the kubernetes api, optional
    "caCert": "/etc/ca.cert", // the ca cert file path to talk to the kubernetes api, optional
    "trust": false, // trust any cert to talk to the kubernetes api, optional
    "namespaces": ["*"], // the watched namespaces
    "labels": ["label"], // the watched namespaces
    "ingressClasses": ["otoroshi"], // the watched kubernetes.io/ingress.class annotations, can be *
    "defaultGroup": "default", // the group to put services in otoroshi
    "ingresses": true, // sync ingresses
    "crds": false, // sync crds
    "kubeLeader": false, // delegate leader election to kubernetes, to know where the sync job should run
    "restartDependantDeployments": true, // when a secret/cert changes from otoroshi sync, restart dependant deployments
    "templates": { // template for entities that will be merged with kubernetes entities. can be "default" to use otoroshi default templates
      "service-group": {},
      "service-descriptor": {},
      "apikeys": {},
      "global-config": {},
      "jwt-verifier": {},
      "tcp-service": {},
      "certificate": {},
      "auth-module": {},
      "data-exporter": {},
      "script": {},
      "organization": {},
      "team": {},
      "data-exporter": {},
      "routes": {},
      "route-compositions": {},
      "backends": {}
    }
  }
}

If endpoint is not defined, Otoroshi will try to get it from $KUBERNETES_SERVICE_HOST and $KUBERNETES_SERVICE_PORT. If token is not defined, Otoroshi will try to get it from the file at /var/run/secrets/kubernetes.io/serviceaccount/token. If caCert is not defined, Otoroshi will try to get it from the file at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt. If $KUBECONFIG is defined, endpoint, token and caCert will be read from the current context of the file referenced by it.

Now you can deploy your first service ;)

Deploy an ingress route

now let’s say you want to deploy an http service and route to the outside world through otoroshi

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-app-deployment
spec:
  selector:
    matchLabels:
      run: http-app-deployment
  replicas: 1
  template:
    metadata:
      labels:
        run: http-app-deployment
    spec:
      containers:
      - image: kennethreitz/httpbin
        imagePullPolicy: IfNotPresent
        name: otoroshi
        ports:
          - containerPort: 80
            name: "http"
---
apiVersion: v1
kind: Service
metadata:
  name: http-app-service
spec:
  ports:
    - port: 8080
      targetPort: http
      name: http
  selector:
    run: http-app-deployment
---
apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: http-app-ingress
  annotations:
    kubernetes.io/ingress.class: otoroshi
spec:
  tls:
  - hosts:
    - httpapp.foo.bar
    secretName: http-app-cert
  rules:
  - host: httpapp.foo.bar
    http:
      paths:
      - path: /
        backend:
          serviceName: http-app-service
          servicePort: 8080

once deployed, otoroshi will sync with kubernetes and create the corresponding service to route your app. You will be able to access your app with

curl -X GET https://httpapp.foo.bar/get

Support for Ingress Classes

Since Kubernetes 1.18, you can use IngressClass type of manifest to specify which ingress controller you want to use for a deployment (https://kubernetes.io/blog/2020/04/02/improvements-to-the-ingress-api-in-kubernetes-1.18/#extended-configuration-with-ingress-classes). Otoroshi is fully compatible with this new manifest kind. To use it, configure the Ingress job to match your controller

{
  "KubernetesConfig": {
    ...
    "ingressClasses": ["otoroshi.io/ingress-controller"],
    ...
  }
}

then you have to deploy an IngressClass to declare Otoroshi as an ingress controller

apiVersion: "networking.k8s.io/v1beta1"
kind: "IngressClass"
metadata:
  name: "otoroshi-ingress-controller"
spec:
  controller: "otoroshi.io/ingress-controller"
  parameters:
    apiGroup: "proxy.otoroshi.io/v1alpha"
    kind: "IngressParameters"
    name: "otoroshi-ingress-controller"

and use it in your Ingress

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: http-app-ingress
spec:
  ingressClassName: otoroshi-ingress-controller
  tls:
  - hosts:
    - httpapp.foo.bar
    secretName: http-app-cert
  rules:
  - host: httpapp.foo.bar
    http:
      paths:
      - path: /
        backend:
          serviceName: http-app-service
          servicePort: 8080

Use multiple ingress controllers

It is of course possible to use multiple ingress controller at the same time (https://kubernetes.io/docs/concepts/services-networking/ingress-controllers/#using-multiple-ingress-controllers) using the annotation kubernetes.io/ingress.class. By default, otoroshi reacts to the class otoroshi, but you can make it the default ingress controller with the following config

{
  "KubernetesConfig": {
    ...
    "ingressClass": "*",
    ...
  }
}

Supported annotations

if you need to customize the service descriptor behind an ingress rule, you can use some annotations. If you need better customisation, just go to the CRDs part. The following annotations are supported :

• ingress.otoroshi.io/groups
• ingress.otoroshi.io/group
• ingress.otoroshi.io/groupId
• ingress.otoroshi.io/name
• ingress.otoroshi.io/targetsLoadBalancing
• ingress.otoroshi.io/stripPath
• ingress.otoroshi.io/enabled
• ingress.otoroshi.io/userFacing
• ingress.otoroshi.io/privateApp
• ingress.otoroshi.io/forceHttps
• ingress.otoroshi.io/maintenanceMode
• ingress.otoroshi.io/buildMode
• ingress.otoroshi.io/strictlyPrivate
• ingress.otoroshi.io/sendOtoroshiHeadersBack
• ingress.otoroshi.io/readOnly
• ingress.otoroshi.io/xForwardedHeaders
• ingress.otoroshi.io/overrideHost
• ingress.otoroshi.io/allowHttp10
• ingress.otoroshi.io/logAnalyticsOnServer
• ingress.otoroshi.io/useAkkaHttpClient
• ingress.otoroshi.io/useNewWSClient
• ingress.otoroshi.io/tcpUdpTunneling
• ingress.otoroshi.io/detectApiKeySooner
• ingress.otoroshi.io/letsEncrypt
• ingress.otoroshi.io/publicPatterns
• ingress.otoroshi.io/privatePatterns
• ingress.otoroshi.io/additionalHeaders
• ingress.otoroshi.io/additionalHeadersOut
• ingress.otoroshi.io/missingOnlyHeadersIn
• ingress.otoroshi.io/missingOnlyHeadersOut
• ingress.otoroshi.io/removeHeadersIn
• ingress.otoroshi.io/removeHeadersOut
• ingress.otoroshi.io/headersVerification
• ingress.otoroshi.io/matchingHeaders
• ingress.otoroshi.io/ipFiltering.whitelist
• ingress.otoroshi.io/ipFiltering.blacklist
• ingress.otoroshi.io/api.exposeApi
• ingress.otoroshi.io/api.openApiDescriptorUrl
• ingress.otoroshi.io/healthCheck.enabled
• ingress.otoroshi.io/healthCheck.url
• ingress.otoroshi.io/jwtVerifier.ids
• ingress.otoroshi.io/jwtVerifier.enabled
• ingress.otoroshi.io/jwtVerifier.excludedPatterns
• ingress.otoroshi.io/authConfigRef
• ingress.otoroshi.io/redirection.enabled
• ingress.otoroshi.io/redirection.code
• ingress.otoroshi.io/redirection.to
• ingress.otoroshi.io/clientValidatorRef
• ingress.otoroshi.io/transformerRefs
• ingress.otoroshi.io/transformerConfig
• ingress.otoroshi.io/accessValidator.enabled
• ingress.otoroshi.io/accessValidator.excludedPatterns
• ingress.otoroshi.io/accessValidator.refs
• ingress.otoroshi.io/accessValidator.config
• ingress.otoroshi.io/preRouting.enabled
• ingress.otoroshi.io/preRouting.excludedPatterns
• ingress.otoroshi.io/preRouting.refs
• ingress.otoroshi.io/preRouting.config
• ingress.otoroshi.io/issueCert
• ingress.otoroshi.io/issueCertCA
• ingress.otoroshi.io/gzip.enabled
• ingress.otoroshi.io/gzip.excludedPatterns
• ingress.otoroshi.io/gzip.whiteList
• ingress.otoroshi.io/gzip.blackList
• ingress.otoroshi.io/gzip.bufferSize
• ingress.otoroshi.io/gzip.chunkedThreshold
• ingress.otoroshi.io/gzip.compressionLevel
• ingress.otoroshi.io/cors.enabled
• ingress.otoroshi.io/cors.allowOrigin
• ingress.otoroshi.io/cors.exposeHeaders
• ingress.otoroshi.io/cors.allowHeaders
• ingress.otoroshi.io/cors.allowMethods
• ingress.otoroshi.io/cors.excludedPatterns
• ingress.otoroshi.io/cors.maxAge
• ingress.otoroshi.io/cors.allowCredentials
• ingress.otoroshi.io/clientConfig.useCircuitBreaker
• ingress.otoroshi.io/clientConfig.retries
• ingress.otoroshi.io/clientConfig.maxErrors
• ingress.otoroshi.io/clientConfig.retryInitialDelay
• ingress.otoroshi.io/clientConfig.backoffFactor
• ingress.otoroshi.io/clientConfig.connectionTimeout
• ingress.otoroshi.io/clientConfig.idleTimeout
• ingress.otoroshi.io/clientConfig.callAndStreamTimeout
• ingress.otoroshi.io/clientConfig.callTimeout
• ingress.otoroshi.io/clientConfig.globalTimeout
• ingress.otoroshi.io/clientConfig.sampleInterval
• ingress.otoroshi.io/enforceSecureCommunication
• ingress.otoroshi.io/sendInfoToken
• ingress.otoroshi.io/sendStateChallenge
• ingress.otoroshi.io/secComHeaders.claimRequestName
• ingress.otoroshi.io/secComHeaders.stateRequestName
• ingress.otoroshi.io/secComHeaders.stateResponseName
• ingress.otoroshi.io/secComTtl
• ingress.otoroshi.io/secComVersion
• ingress.otoroshi.io/secComInfoTokenVersion
• ingress.otoroshi.io/secComExcludedPatterns
• ingress.otoroshi.io/secComSettings.size
• ingress.otoroshi.io/secComSettings.secret
• ingress.otoroshi.io/secComSettings.base64
• ingress.otoroshi.io/secComUseSameAlgo
• ingress.otoroshi.io/secComAlgoChallengeOtoToBack.size
• ingress.otoroshi.io/secComAlgoChallengeOtoToBack.secret
• ingress.otoroshi.io/secComAlgoChallengeOtoToBack.base64
• ingress.otoroshi.io/secComAlgoChallengeBackToOto.size
• ingress.otoroshi.io/secComAlgoChallengeBackToOto.secret
• ingress.otoroshi.io/secComAlgoChallengeBackToOto.base64
• ingress.otoroshi.io/secComAlgoInfoToken.size
• ingress.otoroshi.io/secComAlgoInfoToken.secret
• ingress.otoroshi.io/secComAlgoInfoToken.base64
• ingress.otoroshi.io/securityExcludedPatterns

for more informations about it, just go to https://maif.github.io/otoroshi/swagger-ui/index.html

with the previous example, the ingress does not define any apikey, so the route is public. If you want to enable apikeys on it, you can deploy the following descriptor

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: http-app-ingress
  annotations:
    kubernetes.io/ingress.class: otoroshi
    ingress.otoroshi.io/group: http-app-group
    ingress.otoroshi.io/forceHttps: 'true'
    ingress.otoroshi.io/sendOtoroshiHeadersBack: 'true'
    ingress.otoroshi.io/overrideHost: 'true'
    ingress.otoroshi.io/allowHttp10: 'false'
    ingress.otoroshi.io/publicPatterns: ''
spec:
  tls:
  - hosts:
    - httpapp.foo.bar
    secretName: http-app-cert
  rules:
  - host: httpapp.foo.bar
    http:
      paths:
      - path: /
        backend:
          serviceName: http-app-service
          servicePort: 8080

now you can use an existing apikey in the http-app-group to access your app

curl -X GET https://httpapp.foo.bar/get -u existing-apikey-1:secret-1

Use Otoroshi CRDs for a better/full integration

Otoroshi provides some Custom Resource Definitions for kubernetes in order to manage Otoroshi related entities in kubernetes

• routes
• backends
• route-compositions
• service-descriptors
• tcp-services
• error-templates
• apikeys
• certificates
• jwt-verifiers
• auth-modules
• admin-sessions
• admins
• auth-module-users
• service-groups
• organizations
• tenants
• teams
• data-exporters
• scripts
• wasm-plugins
• global-configs
• green-scores
• coraza-configs

using CRDs, you will be able to deploy and manager those entities from kubectl or the kubernetes api like

sudo kubectl get apikeys --all-namespaces
sudo kubectl get service-descriptors --all-namespaces
curl -X GET \
  -H 'Authorization: Bearer eyJhbGciOiJSUzI....F463SrpOehQRaQ' \
  -H 'Accept: application/json' -k \
  https://127.0.0.1:6443/apis/proxy.otoroshi.io/v1/apikeys | jq

You can see this as better Ingress resources. Like any Ingress resource can define which controller it uses (using the kubernetes.io/ingress.class annotation), you can chose another kind of resource instead of Ingress. With Otoroshi CRDs you can even define resources like Certificate, Apikey, AuthModules, JwtVerifier, etc. It will help you to use all the power of Otoroshi while using the deployment model of kubernetes.

Resources examples

group.yaml

copyapiVersion: proxy.otoroshi.io/v1
kind: ServiceGroup
metadata:
  name: http-app-group
  annotations:
    io.otoroshi/id: http-app-group
spec:
  description: a group to hold services about the http-app

apikey.yaml

copyapiVersion: proxy.otoroshi.io/v1
kind: ApiKey
metadata:
  name: http-app-2-apikey-1
# this apikey can be used to access another app in a different group
spec:
  # a secret name secret-1 will be created by otoroshi and can be used by containers
  exportSecret: true 
  secretName: secret-2
  authorizedEntities: 
  - http-app-2-group
  metadata:
    foo: bar
  rotation: # not mandatory
    enabled: true
    rotationEvery: 720 # hours
    gracePeriod: 168  # hours

service-descriptor.yaml

copyapiVersion: proxy.otoroshi.io/v1
kind: ServiceDescriptor
metadata:
  name: http-app-service-descriptor
spec:
  description: the service descriptor for the http app
  groups: 
  - http-app-group
  forceHttps: true
  hosts:
  - httpapp.foo.bar
  matchingRoot: /
  targets:
  - url: 'https://http-app-service:8443'
    # you can also use serviceName and servicePort to use pods ip addresses. Can be used without or in combination with url
    # serviceName: http-app-service
    # servicePort: https
    mtlsConfig: # not mandatory
      # use mtls to contact the backend
      mtls: true
      certs: 
        # reference the DN for the client cert
        - UID=httpapp-client, O=OtoroshiApps
      trustedCerts: 
        # reference the DN for the CA cert
        - CN=Otoroshi Root
  sendOtoroshiHeadersBack: true
  xForwardedHeaders: true
  overrideHost: true
  allowHttp10: false
  publicPatterns:
    - /health
  additionalHeaders:
    x-foo: bar

certificate.yaml

copyapiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
  name: http-app-certificate-client
spec:
  description: certificate for the http-app
  autoRenew: true
  csr:
    issuer: CN=Otoroshi Root
    key:
      algo: rsa
      size: 2048
    subject: UID=httpapp-client, O=OtoroshiApps
    client: false
    ca: false
    duration: 31536000000
    signatureAlg: SHA256WithRSAEncryption
    digestAlg: SHA-256

jwt.yaml

copyapiVersion: proxy.otoroshi.io/v1
kind: JwtVerifier
metadata:
  name: http-app-verifier
  annotations:
    io.otoroshi/id: http-app-verifier
spec:
  desc: verify that the jwt token in header jwt is ok
  strict: true
  source:
    type: InHeader
    name: jwt
    remove: ''
  algoSettings:
    type: HSAlgoSettings
    size: 512
    secret: secret
  strategy:
    type: PassThrough
    verificationSettings:
      fields: 
        foo: bar
      arrayFields: {}

auth.yaml

copyapiVersion: proxy.otoroshi.io/v1
kind: AuthModule
metadata:
  name: http-app-auth
  annotations:
    io.otoroshi/id: http-app-auth
spec:
  type: oauth2
  desc: Keycloak mTLS
  sessionMaxAge: 86400
  clientId: otoroshi
  clientSecret: ''
  authorizeUrl: 'https://keycloak.foo.bar/auth/realms/master/protocol/openid-connect/auth'
  tokenUrl: 'https://keycloak.foo.bar/auth/realms/master/protocol/openid-connect/token'
  userInfoUrl: 'https://keycloak.foo.bar/auth/realms/master/protocol/openid-connect/userinfo'
  introspectionUrl: 'https://keycloak.foo.bar/auth/realms/master/protocol/openid-connect/token/introspect'
  loginUrl: 'https://keycloak.foo.bar/auth/realms/master/protocol/openid-connect/auth'
  logoutUrl: 'https://keycloak.foo.bar/auth/realms/master/protocol/openid-connect/logout'
  scope: openid address email microprofile-jwt offline_access phone profile roles web-origins
  claims: ''
  useCookie: false
  useJson: false
  readProfileFromToken: false
  accessTokenField: access_token
  jwtVerifier:
    type: JWKSAlgoSettings
    url: 'http://keycloak.foo.bar/auth/realms/master/protocol/openid-connect/certs'
    timeout: 2000
    headers: {}
    ttl: 3600000
    kty: RSA
    proxy: 
    mtlsConfig:
      certs: []
      trustedCerts: []
      mtls: false
      loose: false
      trustAll: false
  nameField: email
  emailField: email
  apiKeyMetaField: apkMeta
  apiKeyTagsField: apkTags
  otoroshiDataField: app_metadata|otoroshi_data
  callbackUrl: 'https://privateapps.oto.tools/privateapps/generic/callback'
  oidConfig: 'http://keycloak.foo.bar/auth/realms/master/.well-known/openid-configuration'
  mtlsConfig:
    certs:
    - UID=httpapp-client, O=OtoroshiApps
    trustedCerts:
    - UID=httpapp-client, O=OtoroshiApps
    mtls: true
    loose: false
    trustAll: false
  proxy: 
  extraMetadata: {}
  refreshTokens: false

organization.yaml

copyapiVersion: proxy.otoroshi.io/v1
kind: Tenant
metadata:
  name: default-organization
spec:
  id: default
  name: Default organization
  description: Default organization created for any otoroshi instance
  metadata: {}

team.yaml

copyapiVersion: proxy.otoroshi.io/v1
kind: Team
metadata:
  name: default-team
spec:
  id: default
  tenant: default
  name: Default team
  description: Default team created for any otoroshi instance
  metadata: {}

Configuration

To configure it, just go to the danger zone, and in Global scripts add the job named Kubernetes Otoroshi CRDs Controller. Then add the following configuration for the job (with your own tweak of course)

{
  "KubernetesConfig": {
    "enabled": true,
    "crds": true,
    "endpoint": "https://127.0.0.1:6443",
    "token": "eyJhbGciOiJSUzI....F463SrpOehQRaQ",
    "namespaces": [
      "*"
    ]
  }
}

the configuration can have the following values

{
  "KubernetesConfig": {
    "endpoint": "https://127.0.0.1:6443", // the endpoint to talk to the kubernetes api, optional
    "token": "xxxx", // the bearer token to talk to the kubernetes api, optional
    "userPassword": "user:password", // the user password tuple to talk to the kubernetes api, optional
    "caCert": "/etc/ca.cert", // the ca cert file path to talk to the kubernetes api, optional
    "trust": false, // trust any cert to talk to the kubernetes api, optional
    "namespaces": ["*"], // the watched namespaces
    "labels": ["label"], // the watched namespaces
    "ingressClasses": ["otoroshi"], // the watched kubernetes.io/ingress.class annotations, can be *
    "defaultGroup": "default", // the group to put services in otoroshi
    "ingresses": false, // sync ingresses
    "crds": true, // sync crds
    "kubeLeader": false, // delegate leader election to kubernetes, to know where the sync job should run
    "restartDependantDeployments": true, // when a secret/cert changes from otoroshi sync, restart dependant deployments
    "templates": { // template for entities that will be merged with kubernetes entities. can be "default" to use otoroshi default templates
      "service-group": {},
      "service-descriptor": {},
      "apikeys": {},
      "global-config": {},
      "jwt-verifier": {},
      "tcp-service": {},
      "certificate": {},
      "auth-module": {},
      "data-exporter": {},
      "script": {},
      "organization": {},
      "team": {},
      "data-exporter": {}
    }
  }
}

If endpoint is not defined, Otoroshi will try to get it from $KUBERNETES_SERVICE_HOST and $KUBERNETES_SERVICE_PORT. If token is not defined, Otoroshi will try to get it from the file at /var/run/secrets/kubernetes.io/serviceaccount/token. If caCert is not defined, Otoroshi will try to get it from the file at /var/run/secrets/kubernetes.io/serviceaccount/ca.crt. If $KUBECONFIG is defined, endpoint, token and caCert will be read from the current context of the file referenced by it.

you can find a more complete example of the configuration object here

Note about apikeys and certificates resources

Apikeys and Certificates are a little bit different than the other resources. They have ability to be defined without their secret part, but with an export setting so otoroshi will generate the secret parts and export the apikey or the certificate to kubernetes secret. Then any app will be able to mount them as volumes (see the full example below)

In those resources you can define

exportSecret: true 
secretName: the-secret-name

and omit clientSecret for apikey or publicKey, privateKey for certificates. For certificate you will have to provide a csr for the certificate in order to generate it

csr:
  issuer: CN=Otoroshi Root
  hosts: 
  - httpapp.foo.bar
  - httpapps.foo.bar
  key:
    algo: rsa
    size: 2048
  subject: UID=httpapp-front, O=OtoroshiApps
  client: false
  ca: false
  duration: 31536000000
  signatureAlg: SHA256WithRSAEncryption
  digestAlg: SHA-256

when apikeys are exported as kubernetes secrets, they will have the type otoroshi.io/apikey-secret with values clientId and clientSecret

apiVersion: v1
kind: Secret
metadata:
  name: apikey-1
type: otoroshi.io/apikey-secret
data:
  clientId: TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdA==
  clientSecret: TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdA==

when certificates are exported as kubernetes secrets, they will have the type kubernetes.io/tls with the standard values tls.crt (the full cert chain) and tls.key (the private key). For more convenience, they will also have a cert.crt value containing the actual certificate without the ca chain and ca-chain.crt containing the ca chain without the certificate.

apiVersion: v1
kind: Secret
metadata:
  name: certificate-1
type: kubernetes.io/tls
data:
  tls.crt: TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdA==
  tls.key: TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdA==
  cert.crt: TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdA==
  ca-chain.crt: TG9yZW0gaXBzdW0gZG9sb3Igc2l0IGFtZXQsIGNvbnNlY3RldHVyIGFkaXBpc2NpbmcgZWxpdA== 

Full CRD example

then you can deploy the previous example with better configuration level, and using mtls, apikeys, etc

Let say the app looks like :

const fs = require('fs'); 
const https = require('https'); 

// here we read the apikey to access http-app-2 from files mounted from secrets
const clientId = fs.readFileSync('/var/run/secrets/kubernetes.io/apikeys/clientId').toString('utf8')
const clientSecret = fs.readFileSync('/var/run/secrets/kubernetes.io/apikeys/clientSecret').toString('utf8')

const backendKey = fs.readFileSync('/var/run/secrets/kubernetes.io/certs/backend/tls.key').toString('utf8')
const backendCert = fs.readFileSync('/var/run/secrets/kubernetes.io/certs/backend/cert.crt').toString('utf8')
const backendCa = fs.readFileSync('/var/run/secrets/kubernetes.io/certs/backend/ca-chain.crt').toString('utf8')

const clientKey = fs.readFileSync('/var/run/secrets/kubernetes.io/certs/client/tls.key').toString('utf8')
const clientCert = fs.readFileSync('/var/run/secrets/kubernetes.io/certs/client/cert.crt').toString('utf8')
const clientCa = fs.readFileSync('/var/run/secrets/kubernetes.io/certs/client/ca-chain.crt').toString('utf8')

function callApi2() {
  return new Promise((success, failure) => {
    const options = { 
      // using the implicit internal name (*.global.otoroshi.mesh) of the other service descriptor passing through otoroshi
      hostname: 'http-app-service-descriptor-2.global.otoroshi.mesh',  
      port: 433, 
      path: '/', 
      method: 'GET',
      headers: {
        'Accept': 'application/json',
        'Otoroshi-Client-Id': clientId,
        'Otoroshi-Client-Secret': clientSecret,
      },
      cert: clientCert,
      key: clientKey,
      ca: clientCa
    }; 
    let data = '';
    const req = https.request(options, (res) => { 
      res.on('data', (d) => { 
        data = data + d.toString('utf8');
      }); 
      res.on('end', () => { 
        success({ body: JSON.parse(data), res });
      }); 
      res.on('error', (e) => { 
        failure(e);
      }); 
    }); 
    req.end();
  })
}

const options = { 
  key: backendKey, 
  cert: backendCert, 
  ca: backendCa, 
  // we want mtls behavior
  requestCert: true, 
  rejectUnauthorized: true
}; 
https.createServer(options, (req, res) => { 
  res.writeHead(200, {'Content-Type': 'application/json'});
  callApi2().then(resp => {
    res.write(JSON.stringify{ ("message": `Hello to ${req.socket.getPeerCertificate().subject.CN}`, api2: resp.body })); 
  });
}).listen(433);

then, the descriptors will be :

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: http-app-deployment
spec:
  selector:
    matchLabels:
      run: http-app-deployment
  replicas: 1
  template:
    metadata:
      labels:
        run: http-app-deployment
    spec:
      containers:
      - image: foo/http-app
        imagePullPolicy: IfNotPresent
        name: otoroshi
        ports:
          - containerPort: 443
            name: "https"
        volumeMounts:
        - name: apikey-volume
          # here you will be able to read apikey from files 
          # - /var/run/secrets/kubernetes.io/apikeys/clientId
          # - /var/run/secrets/kubernetes.io/apikeys/clientSecret
          mountPath: "/var/run/secrets/kubernetes.io/apikeys"
          readOnly: true
        volumeMounts:
        - name: backend-cert-volume
          # here you will be able to read app cert from files 
          # - /var/run/secrets/kubernetes.io/certs/backend/tls.crt
          # - /var/run/secrets/kubernetes.io/certs/backend/tls.key
          mountPath: "/var/run/secrets/kubernetes.io/certs/backend"
          readOnly: true
        - name: client-cert-volume
          # here you will be able to read app cert from files 
          # - /var/run/secrets/kubernetes.io/certs/client/tls.crt
          # - /var/run/secrets/kubernetes.io/certs/client/tls.key
          mountPath: "/var/run/secrets/kubernetes.io/certs/client"
          readOnly: true
      volumes:
      - name: apikey-volume
        secret:
          # here we reference the secret name from apikey http-app-2-apikey-1
          secretName: secret-2
      - name: backend-cert-volume
        secret:
          # here we reference the secret name from cert http-app-certificate-backend
          secretName: http-app-certificate-backend-secret
      - name: client-cert-volume
        secret:
          # here we reference the secret name from cert http-app-certificate-client
          secretName: http-app-certificate-client-secret
---
apiVersion: v1
kind: Service
metadata:
  name: http-app-service
spec:
  ports:
    - port: 8443
      targetPort: https
      name: https
  selector:
    run: http-app-deployment
---
apiVersion: proxy.otoroshi.io/v1
kind: ServiceGroup
metadata:
  name: http-app-group
  annotations:
    otoroshi.io/id: http-app-group
spec:
  description: a group to hold services about the http-app
---
apiVersion: proxy.otoroshi.io/v1
kind: ApiKey
metadata:
  name: http-app-apikey-1
# this apikey can be used to access the app
spec:
  # a secret name secret-1 will be created by otoroshi and can be used by containers
  exportSecret: true 
  secretName: secret-1
  authorizedEntities: 
  - group_http-app-group
---
apiVersion: proxy.otoroshi.io/v1
kind: ApiKey
metadata:
  name: http-app-2-apikey-1
# this apikey can be used to access another app in a different group
spec:
  # a secret name secret-1 will be created by otoroshi and can be used by containers
  exportSecret: true 
  secretName: secret-2
  authorizedEntities: 
  - group_http-app-2-group
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
  name: http-app-certificate-frontend
spec:
  description: certificate for the http-app on otorshi frontend
  autoRenew: true
  csr:
    issuer: CN=Otoroshi Root
    hosts: 
    - httpapp.foo.bar
    key:
      algo: rsa
      size: 2048
    subject: UID=httpapp-front, O=OtoroshiApps
    client: false
    ca: false
    duration: 31536000000
    signatureAlg: SHA256WithRSAEncryption
    digestAlg: SHA-256
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
  name: http-app-certificate-backend
spec:
  description: certificate for the http-app deployed on pods
  autoRenew: true
  # a secret name http-app-certificate-backend-secret will be created by otoroshi and can be used by containers
  exportSecret: true 
  secretName: http-app-certificate-backend-secret
  csr:
    issuer: CN=Otoroshi Root
    hosts: 
    - http-app-service 
    key:
      algo: rsa
      size: 2048
    subject: UID=httpapp-back, O=OtoroshiApps
    client: false
    ca: false
    duration: 31536000000
    signatureAlg: SHA256WithRSAEncryption
    digestAlg: SHA-256
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
  name: http-app-certificate-client
spec:
  description: certificate for the http-app
  autoRenew: true
  secretName: http-app-certificate-client-secret
  csr:
    issuer: CN=Otoroshi Root
    key:
      algo: rsa
      size: 2048
    subject: UID=httpapp-client, O=OtoroshiApps
    client: false
    ca: false
    duration: 31536000000
    signatureAlg: SHA256WithRSAEncryption
    digestAlg: SHA-256
---
apiVersion: proxy.otoroshi.io/v1
kind: ServiceDescriptor
metadata:
  name: http-app-service-descriptor
spec:
  description: the service descriptor for the http app
  groups: 
  - http-app-group
  forceHttps: true
  hosts:
  - httpapp.foo.bar # hostname exposed oustide of the kubernetes cluster
  # - http-app-service-descriptor.global.otoroshi.mesh # implicit internal name inside the kubernetes cluster 
  matchingRoot: /
  targets:
  - url: https://http-app-service:8443
    # alternatively, you can use serviceName and servicePort to use pods ip addresses
    # serviceName: http-app-service
    # servicePort: https
    mtlsConfig:
      # use mtls to contact the backend
      mtls: true
      certs: 
        # reference the DN for the client cert
        - UID=httpapp-client, O=OtoroshiApps
      trustedCerts: 
        # reference the DN for the CA cert 
        - CN=Otoroshi Root
  sendOtoroshiHeadersBack: true
  xForwardedHeaders: true
  overrideHost: true
  allowHttp10: false
  publicPatterns:
    - /health
  additionalHeaders:
    x-foo: bar
# here you can specify everything supported by otoroshi like jwt-verifiers, auth config, etc ... for more informations about it, just go to https://maif.github.io/otoroshi/swagger-ui/index.html

now with this descriptor deployed, you can access your app with a command like

CLIENT_ID=`kubectl get secret secret-1 -o jsonpath="{.data.clientId}" | base64 --decode`
CLIENT_SECRET=`kubectl get secret secret-1 -o jsonpath="{.data.clientSecret}" | base64 --decode`
curl -X GET https://httpapp.foo.bar/get -u "$CLIENT_ID:$CLIENT_SECRET"

Expose Otoroshi to outside world

If you deploy Otoroshi on a kubernetes cluster, the Otoroshi service is deployed as a loadbalancer (service type: LoadBalancer). You’ll need to declare in your DNS settings any name that can be routed by otoroshi going to the loadbalancer endpoint (CNAME or ip addresses) of your kubernetes distribution. If you use a managed kubernetes cluster from a cloud provider, it will work seamlessly as they will provide external loadbalancers out of the box. However, if you use a bare metal kubernetes cluster, id doesn’t come with support for external loadbalancers (service of type LoadBalancer). So you will have to provide this feature in order to route external TCP traffic to Otoroshi containers running inside the kubernetes cluster. You can use projects like MetalLB that provide software LoadBalancer services to bare metal clusters or you can use and customize examples in the installation section.

Access a service from inside the k8s cluster

Using host header overriding

You can access any service referenced in otoroshi, through otoroshi from inside the kubernetes cluster by using the otoroshi service name (if you use a template based on https://github.com/MAIF/otoroshi/tree/master/kubernetes/base deployed in the otoroshi namespace) and the host header with the service domain like :

CLIENT_ID="xxx"
CLIENT_SECRET="xxx"
curl -X GET -H 'Host: httpapp.foo.bar' https://otoroshi-service.otoroshi.svc.cluster.local:8443/get -u "$CLIENT_ID:$CLIENT_SECRET"

Using dedicated services

it’s also possible to define services that targets otoroshi deployment (or otoroshi workers deployment) and use then as valid hosts in otoroshi services

apiVersion: v1
kind: Service
metadata:
  name: my-awesome-service
spec:
  selector:
    # run: otoroshi-deployment
    # or in cluster mode
    run: otoroshi-worker-deployment
  ports:
  - port: 8080
    name: "http"
    targetPort: "http"
  - port: 8443
    name: "https"
    targetPort: "https"

and access it like

CLIENT_ID="xxx"
CLIENT_SECRET="xxx"
curl -X GET https://my-awesome-service.my-namspace.svc.cluster.local:8443/get -u "$CLIENT_ID:$CLIENT_SECRET"

Using coredns integration

You can also enable the coredns integration to simplify the flow. You can use the the following keys in the plugin config :

{
  "KubernetesConfig": {
    ...
    "coreDnsIntegration": true,                // enable coredns integration for intra cluster calls
    "kubeSystemNamespace": "kube-system",      // the namespace where coredns is deployed
    "corednsConfigMap": "coredns",             // the name of the coredns configmap
    "otoroshiServiceName": "otoroshi-service", // the name of the otoroshi service, could be otoroshi-workers-service
    "otoroshiNamespace": "otoroshi",           // the namespace where otoroshi is deployed
    "clusterDomain": "cluster.local",          // the domain for cluster services
    ...
  }
}

otoroshi will patch coredns config at startup then you can call your services like

CLIENT_ID="xxx"
CLIENT_SECRET="xxx"
curl -X GET https://my-awesome-service.my-awesome-service-namespace.otoroshi.mesh:8443/get -u "$CLIENT_ID:$CLIENT_SECRET"

By default, all services created from CRDs service descriptors are exposed as ${service-name}.${service-namespace}.otoroshi.mesh or ${service-name}.${service-namespace}.svc.otoroshi.local

Using coredns with manual patching

you can also patch the coredns config manually

kubectl edit configmaps coredns -n kube-system # or your own custom config map

and change the Corefile data to add the following snippet in at the end of the file

otoroshi.mesh:53 {
    errors
    health
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
        pods insecure
        upstream
        fallthrough in-addr.arpa ip6.arpa
    }
    rewrite name regex (.*)\.otoroshi\.mesh otoroshi-worker-service.otoroshi.svc.cluster.local
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

you can also define simpler rewrite if it suits you use case better

rewrite name my-service.otoroshi.mesh otoroshi-worker-service.otoroshi.svc.cluster.local

do not hesitate to change otoroshi-worker-service.otoroshi according to your own setup. If otoroshi is not in cluster mode, change it to otoroshi-service.otoroshi. If otoroshi is not deployed in the otoroshi namespace, change it to otoroshi-service.the-namespace, etc.

By default, all services created from CRDs service descriptors are exposed as ${service-name}.${service-namespace}.otoroshi.mesh

then you can call your service like

CLIENT_ID="xxx"
CLIENT_SECRET="xxx"

curl -X GET https://my-awesome-service.my-awesome-service-namespace.otoroshi.mesh:8443/get -u "$CLIENT_ID:$CLIENT_SECRET"

Using old kube-dns system

if your stuck with an old version of kubernetes, it uses kube-dns that is not supported by otoroshi, so you will have to provide your own coredns deployment and declare it as a stubDomain in the old kube-dns system.

Here is an example of coredns deployment with otoroshi domain config

coredns.yaml

copy---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otoroshi-dns
  labels:
    app: otoroshi
    component: coredns
data:
  Corefile: |
    otoroshi.mesh:5353 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
        }
        rewrite name regex (.*)\.otoroshi\.mesh otoroshi-service.otoroshi.svc.cluster.local
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    .:5353 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-dns
  labels:
    app: otoroshi
    component: coredns
spec:
  # clusterIP: 1.1.1.1
  selector:
    app: otoroshi
    component: coredns
  type: ClusterIP
  ports:
    - name: dns
      port: 5353
      protocol: UDP
    - name: dns-tcp
      port: 5353
      protocol: TCP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otoroshi-dns
  labels:
    app: otoroshi
    component: coredns
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      app: otoroshi
      component: coredns
  template:
    metadata:
      labels:
        app: otoroshi
        component: coredns
    spec:
      serviceAccountName: otoroshi-admin-user
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - otoroshi
                    - key: component
                      operator: In
                      values:
                        - coredns
                topologyKey: "kubernetes.io/hostname"
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      containers:
        - name: coredns
          image: coredns/coredns:1.8.0
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              memory: 170Mi
            requests:
              cpu: 100m
              memory: 70Mi
          args: [ "-conf", "/etc/coredns/Corefile" ]
          volumeMounts:
            - name: config-volume
              mountPath: /etc/coredns
              readOnly: true
          ports:
            - containerPort: 5353
              name: dns
              protocol: UDP
            - containerPort: 5353
              name: dns-tcp
              protocol: TCP
          securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 30
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 5
      dnsPolicy: Default
      volumes:
        - name: config-volume
          configMap:
            name: otoroshi-dns
            items:
              - key: Corefile
                path: Corefile
  

then you can enable the kube-dns integration in the otoroshi kubernetes job

{
  "KubernetesConfig": {
    ...
    "kubeDnsOperatorIntegration": true,                // enable kube-dns integration for intra cluster calls
    "kubeDnsOperatorCoreDnsNamespace": "otoroshi",    // namespace where coredns is installed
    "kubeDnsOperatorCoreDnsName": "otoroshi-dns",     // name of the coredns service
    "kubeDnsOperatorCoreDnsPort": 5353,               // port of the coredns service
    ...
  }
}

Using Openshift DNS operator

Openshift DNS operator does not allow to customize DNS configuration a lot, so you will have to provide your own coredns deployment and declare it as a stub in the Openshift DNS operator.

Here is an example of coredns deployment with otoroshi domain config

coredns.yaml

copy---
apiVersion: v1
kind: ConfigMap
metadata:
  name: otoroshi-dns
  labels:
    app: otoroshi
    component: coredns
data:
  Corefile: |
    otoroshi.mesh:5353 {
        errors
        health
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
        }
        rewrite name regex (.*)\.otoroshi\.mesh otoroshi-service.otoroshi.svc.cluster.local
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
    .:5353 {
        errors
        health
        kubernetes cluster.local in-addr.arpa ip6.arpa {
          pods insecure
          fallthrough in-addr.arpa ip6.arpa
        }
        forward . /etc/resolv.conf
        cache 30
        loop
        reload
        loadbalance
    }
---
apiVersion: v1
kind: Service
metadata:
  name: otoroshi-dns
  labels:
    app: otoroshi
    component: coredns
spec:
  # clusterIP: 1.1.1.1
  selector:
    app: otoroshi
    component: coredns
  type: ClusterIP
  ports:
    - name: dns
      port: 5353
      protocol: UDP
    - name: dns-tcp
      port: 5353
      protocol: TCP
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otoroshi-dns
  labels:
    app: otoroshi
    component: coredns
spec:
  replicas: 2
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 1
  selector:
    matchLabels:
      app: otoroshi
      component: coredns
  template:
    metadata:
      labels:
        app: otoroshi
        component: coredns
    spec:
      serviceAccountName: otoroshi-admin-user
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - weight: 100
              podAffinityTerm:
                labelSelector:
                  matchExpressions:
                    - key: app
                      operator: In
                      values:
                        - otoroshi
                    - key: component
                      operator: In
                      values:
                        - coredns
                topologyKey: "kubernetes.io/hostname"
      tolerations:
        - key: "CriticalAddonsOnly"
          operator: "Exists"
      containers:
        - name: coredns
          image: coredns/coredns:1.8.0
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              memory: 170Mi
            requests:
              cpu: 100m
              memory: 70Mi
          args: [ "-conf", "/etc/coredns/Corefile" ]
          volumeMounts:
            - name: config-volume
              mountPath: /etc/coredns
              readOnly: true
          ports:
            - containerPort: 5353
              name: dns
              protocol: UDP
            - containerPort: 5353
              name: dns-tcp
              protocol: TCP
          securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            add:
            - NET_BIND_SERVICE
            drop:
            - all
          readOnlyRootFilesystem: true
          livenessProbe:
            httpGet:
              path: /health
              port: 8080
              scheme: HTTP
            initialDelaySeconds: 30
            timeoutSeconds: 5
            successThreshold: 1
            failureThreshold: 5
      dnsPolicy: Default
      volumes:
        - name: config-volume
          configMap:
            name: otoroshi-dns
            items:
              - key: Corefile
                path: Corefile
  

then you can enable the Openshift DNS operator integration in the otoroshi kubernetes job

{
  "KubernetesConfig": {
    ...
    "openshiftDnsOperatorIntegration": true,                // enable openshift dns operator integration for intra cluster calls
    "openshiftDnsOperatorCoreDnsNamespace": "otoroshi",    // namespace where coredns is installed
    "openshiftDnsOperatorCoreDnsName": "otoroshi-dns",     // name of the coredns service
    "openshiftDnsOperatorCoreDnsPort": 5353,               // port of the coredns service
    ...
  }
}

don’t forget to update the otoroshi ClusterRole

- apiGroups:
    - operator.openshift.io
  resources:
    - dnses
  verbs:
    - get
    - list
    - watch
    - update

CRD validation in kubectl

In order to get CRD validation before manifest deployments right inside kubectl, you can deploy a validation webhook that will do the trick. Also check that you have otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookCRDValidator request sink enabled.

validation-webhook.yaml

copyapiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: otoroshi-admission-webhook-validation
  labels:
    app: otoroshi
    component: otoroshi-validation-webhook
webhooks:
  - name: otoroshi-admission-webhook.otoroshi.io
    rules:
      - operations: 
          - "CREATE"
          - "UPDATE"
        apiGroups: 
          - "proxy.otoroshi.io"
        apiVersions: 
          - "*"
        resources: 
          - "*"
        scope: "Namespaced"
    clientConfig:
      # url: "https://otoroshi-kubernetes-admission-webhook.otoroshi.svc.cluster.local:8443/apis/webhooks/validation"
      service:
        name: otoroshi-service
        namespace: otoroshi
        path: "/apis/webhooks/validation"
        port: 8443
      caBundle: "" # injected at runtime
    failurePolicy: Ignore # inject at runtime
    sideEffects: None
    admissionReviewVersions: 
      - "v1"

Easier integration with otoroshi-sidecar

Otoroshi can help you to easily use existing services without modifications while gettings all the perks of otoroshi like apikeys, mTLS, exchange protocol, etc. To do so, otoroshi will inject a sidecar container in the pod of your deployment that will handle call coming from otoroshi and going to otoroshi. To enable otoroshi-sidecar, you need to deploy the following admission webhook. Also check that you have otoroshi.plugins.jobs.kubernetes.KubernetesAdmissionWebhookSidecarInjector request sink enabled.

sidecar-webhook.yaml

copyapiVersion: admissionregistration.k8s.io/v1
kind: MutatingWebhookConfiguration
metadata:
  name: otoroshi-admission-webhook-injector
  labels:
    app: otoroshi
    component: otoroshi-validation-webhook
webhooks:
  - name: otoroshi-admission-webhook-injector.otoroshi.io
    rules:
      - operations: 
          - "CREATE" 
        apiGroups: 
          - ""
        apiVersions: 
          - "v1"
        resources: 
          - "pods"
        scope: "Namespaced"
    # namespaceSelector:
    #   matchLabels:
    #     otoroshi.io/sidecar: inject
    objectSelector:
      matchLabels:
        otoroshi.io/sidecar: inject
    clientConfig:
      # url: "https://otoroshi-kubernetes-admission-webhook.otoroshi.svc.cluster.local:8443/apis/webhooks/inject"
      service:
        name: otoroshi-service
        namespace: otoroshi
        path: "/apis/webhooks/inject"
        port: 8443
      caBundle: "" # inject at runtime
    failurePolicy: Ignore # inject at runtime
    sideEffects: None
    admissionReviewVersions: 
      - "v1"

then it’s quite easy to add the sidecar, just add the following label to your pod otoroshi.io/sidecar: inject and some annotations to tell otoroshi what certificates and apikeys to use.

annotations:
  otoroshi.io/sidecar-apikey: backend-apikey
  otoroshi.io/sidecar-backend-cert: backend-cert
  otoroshi.io/sidecar-client-cert: oto-client-cert
  otoroshi.io/token-secret: secret
  otoroshi.io/expected-dn: UID=oto-client-cert, O=OtoroshiApps

now you can just call you otoroshi handled apis from inside your pod like curl http://my-service.namespace.otoroshi.mesh/api without passing any apikey or client certificate and the sidecar will handle everything for you. Same thing for call from otoroshi to your pod, everything will be done in mTLS fashion with apikeys and otoroshi exchange protocol

here is a full example

sidecar.yaml

copy---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: app-deployment
spec:
  selector:
    matchLabels:
      run: app-deployment
      app: node
  replicas: 1
  template:
    metadata:
      labels:
        run: app-deployment
        app: node
        foo: bar
        otoroshi.io/sidecar: inject
      annotations:
        otoroshi.io/sidecar-apikey: backend-apikey
        otoroshi.io/sidecar-backend-cert: backend-cert
        otoroshi.io/sidecar-client-cert: oto-client-cert
        otoroshi.io/token-secret: secret
        otoroshi.io/expected-dn: UID=oto-client-cert, O=OtoroshiApps
    spec:
      containers:
      - image: containous/whoami:latest
        name: whoami
        args: ["--port", "8081"]
        ports:
          - name: main-port
            containerPort: 8081
---
apiVersion: v1
kind: Service
metadata:
  name: app-service
spec:
  selector:
    run: app-deployment
  ports:
  - port: 8443
    name: "https"
    targetPort: "https"
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
  name: backend-cert
spec:
  description: backend-cert
  autoRenew: true
  exportSecret: true 
  secretName: backend-cert
  csr:
    hosts:
      - app-service.default.svc.cluster.local
    issuer: otoroshi-intermediate-ca
    key:
      algo: rsa
      size: 2048
    subject: UID=backend-cert, O=OtoroshiApps
    duration: 31536000000
    signatureAlg: SHA256WithRSAEncryption
    digestAlg: SHA-256
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
  name: client-cert
  annotations:
    otoroshi.io/id: client-cert
spec:
  description: client-cert
  autoRenew: true
  exportSecret: true
  client: true
  secretName: client-cert
  csr:
    client: true
    issuer: otoroshi-intermediate-ca
    key:
      algo: rsa
      size: 2048
    subject: UID=client-cert, O=OtoroshiApps
    duration: 31536000000
    signatureAlg: SHA256WithRSAEncryption
    digestAlg: SHA-256
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
  name: oto-client-cert
  annotations:
    otoroshi.io/id: oto-client-cert
spec:
  description: oto-client-cert
  autoRenew: true
  exportSecret: true
  client: true
  secretName: oto-client-cert
  csr:
    client: true
    issuer: otoroshi-intermediate-ca
    key:
      algo: rsa
      size: 2048
    subject: UID=oto-client-cert, O=OtoroshiApps
    duration: 31536000000
    signatureAlg: SHA256WithRSAEncryption
    digestAlg: SHA-256
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
  name: frontend-cert
spec:
  description: frontend-cert
  autoRenew: true
  csr:
    issuer: otoroshi-intermediate-ca
    hosts:
      - backend.oto.tools
    key:
      algo: rsa
      size: 2048
    subject: UID=frontend-cert, O=OtoroshiApps
    duration: 31536000000
    signatureAlg: SHA256WithRSAEncryption
    digestAlg: SHA-256
---
apiVersion: proxy.otoroshi.io/v1
kind: Certificate
metadata:
  name: mesh-cert
spec:
  description: mesh-cert
  autoRenew: true
  csr:
    issuer: otoroshi-intermediate-ca
    hosts:
      - '*.default.otoroshi.mesh'
    key:
      algo: rsa
      size: 2048
    subject: O=Otoroshi, OU=Otoroshi Certificates, CN=kubernetes-mesh
    duration: 31536000000
    signatureAlg: SHA256WithRSAEncryption
    digestAlg: SHA-256
---
apiVersion: proxy.otoroshi.io/v1
kind: ApiKey
metadata:
  name: backend-apikey
spec:
  exportSecret: true 
  secretName: backend-apikey
  authorizedEntities:
    - group_default
---
apiVersion: proxy.otoroshi.io/v1
kind: ServiceDescriptor
metadata:
  name: backend
spec:
  description: backend
  groups: 
  - default
  forceHttps: false
  hosts:
  - backend.oto.tools
  matchingRoot: /
  publicPatterns:
    - /.*
  secComUseSameAlgo: true
  secComVersion: 2
  secComInfoTokenVersion: Latest
  secComSettings:
    type: HSAlgoSettings
    size: 512
    secret: secret
    base64: false
  secComAlgoChallengeOtoToBack:
    type: HSAlgoSettings
    size: 512
    secret: secret
    base64: false
  secComAlgoChallengeBackToOto:
    type: HSAlgoSettings
    size: 512
    secret: secret
    base64: false
  secComAlgoInfoToken:
    type: HSAlgoSettings
    size: 512
    secret: secret
    base64: false
  targets:
  - url: https://app-service.default.svc.cluster.local:8443
    mtlsConfig:
      mtls: true
      certs: 
        - UID=oto-client-cert, O=OtoroshiApps
      trustedCerts: 
        - otoroshi-intermediate-ca

Daikoku integration

It is possible to easily integrate daikoku generated apikeys without any human interaction with the actual apikey secret. To do that, create a plan in Daikoku and setup the integration mode to Automatic

then when a user subscribe for an apikey, he will only see an integration token

then just create an ApiKey manifest with this token and your good to go

apiVersion: proxy.otoroshi.io/v1
kind: ApiKey
metadata:
  name: http-app-2-apikey-3
spec:
  exportSecret: true 
  secretName: secret-3
  daikokuToken: RShQrvINByiuieiaCBwIZfGFgdPu7tIJEN5gdV8N8YeH4RI9ErPYJzkuFyAkZ2xy