Kubernetes support?

[mperham]

What should Faktory look like in a world full of Kubernetes? My understanding is that Kubernetes could be very useful in scaling worker processes as queues grow. How can Faktory make this easy?

[jbielick]

The newer implementations of the Horizontal Pod Autoscaler (HPA 1.6+?) support scaling on Custom Metrics (k8s 1.8+ custom metrics), for which there are some preliminary implementations. I think one interesting one is the Prometheus Adapter—if I understand correctly, k8s can pull metrics from prometheus and then an HPA can be set to scale based on a metric in that set. Perhaps in this world the responsibility of Faktory would be to have a prometheus exporter (á la oliver006/redis_exporter) that can send metics to Prometheus.

Since an exporter might live separately, I believe at a fundamental level the basic necessity would be Faktory's API for exposing processing metrics. An obvious metric for scaling might be the size of the queues, but I have also found myself interested in the amount of time a job spends in the queue before processing as well, because I believe that's also an intelligent indicator of a need to scale up workers.

Are there current ideas / plans for what internal metrics will be gathered / recorded?

[mperham]

Yeah, queue size and latency are easily implementable.

What's the best approach to getting Kubernetes aware of Faktory? Are people using Helm? Would a Docker image or DEB/RPM binaries be most useful?

[syndbg]

@mperham Definitely Docker image would enable faktory to be used everywhere. Be that Kubernetes, Mesos Marathon/Aurora, Nomad and etc.

[rosskevin]

We run google container engine a.k.a. GKE (managed kubernetes) for all our services which include sidekiq workers. We have a situation where we may have high flash traffic (pdf rendering) where we are going to use google cloud functions a.k.a. GCF so we don't worry about scale. So sidekiq handles typical jobs, GCF handles high scale/flash traffic jobs like rendering.

With the introduction of faktory, I think one approach/architecture for pain-free scaling:

• kubernetes service + deployment for faktory
• kubernetes deployment for ruby worker (migrate sidekiq jobs)
• google cloud functions (GCP's answer for serverless) for something like faktory_worker_node_gcf worker (no scale/cluster management needed)

Faktory needs proper probes:

• readiness - ready for traffic
• liveness - lightest ping possible

Preparing for kubernetes:

• some people use helm, but I haven't found it very heavily trafficked. Nonetheless, a reference helm config helps others develop their own configs (and can be used as a quickstart)
• many are using spinnaker for continuous deployment (e.g. based on vanilla kubernetes yaml files)
• optimized docker image would be most useful - e.g. redis-alpine
• persistent storage of faktory itself and ensuring recovery/availability will be most important concern for most.

Last but not least, consider an equivalent service to redislabs.com. We use GKE because we want to offload as much management as possible to focus on app development. For that reason, we didn't even try to deploy redis inside our cluster, but we used redislabs.com to deploy in the same region as our GKE cluster. Feel free to ping me via email if you want to dig into any of these areas.

[mperham]

@rosskevin Thanks for the info, that's great detail. Running a Faktory SaaS is not in my immediate plans but like antirez and Redislabs, I've already had pings from people interested in partnering in building something and I'm always willing to chat: mike @ contribsys.com

I'm still trying to wrap my head around what an enterprise-ready Faktory would look like and require; this helps a lot.

[gmaliar]

@rosskevin @mperham a few notes to add about k8s support, let me know what you think and if I'm getting anything wrong.

is the master process that also holds the rocksdb database, remember that k8s might kill faktory pod at will, so there should be a service with both web and worker ports open connected to the faktory-master deployment.
the deployment itself should have a , so that the actual database file will be mounted on that volume, otherwise you'd lose all jobs waiting to be fulfilled,
this looks okay

faktory/cli/cli.go

Line 52 in 01dadde

// undocumented on purpose, we don't want people changing these if possible

as it allows you to specify in the deployment yaml under the command key but it might be a better approach to allow everything to be specified as environment variables, i.e password, port, storage directory, no-tls etc.

I am a heavy helm user, it's pretty easy to set up a basic chart that will take care of setting up the faktory server, but workers might need to be as a different chart, so upgrading of the worker chart won't require a redeploy of the faktory server.

let me know if you need any help with PRs around these areas ...

[mickume]

I will look into configuration & deployment for OpenShift ... similar to Kubernetes ...

[mperham]

Gonna close this because I'm not a fan of nebulous, open-ended issues. Please open a specific issue if there's something Faktory can do to make our k8s support better.

[kirs]

@mperham I was going to open an issue regarding Kubernetes support, and I discovered you have one already open!

Specifically, I'd like to see examples of Kubernetes YAMLs that people could copy, paste, kubectl apply and have Faktory running.

This would be especially cool to have for zero-downtime deploy of Faktory, as well as for Redis Gateway and replicated Faktory Pro.

[jbielick]

This could be a starting point.

Couple of things to note:

• I don't know if I would consider it production-ready, but it's a stateful set of 1 faktory server (embedded redis). A redis gateway would be a much better idea for production usage (redundancy, failover being proper concerns).
• It's configured for Google Cloud's Kubernetes offering (GKE)
• This definition expects and creates within a faktory namespace
• The container limits are super low so you'll want to tweak those: 300m CPU, 1Gi Memory
• It also depends on a secret being created for the faktory password. You can create it with this:
• It assumes a persistent disk in us-east1-b is ok (change this to your cluster's zone if necessary)

kubectl --namespace=faktory create secret generic faktory --from-literal=password=yoursecurepassword

apiVersion: v1
kind: Namespace
metadata:
  name: faktory
---
kind: StorageClass
apiVersion: storage.k8s.io/v1beta1
metadata:
  namespace: faktory
  name: fast
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
  zone: us-east1-b
---
apiVersion: v1
kind: Service
metadata:
  namespace: faktory
  labels:
    name: faktory
  name: faktory
spec:
  ports:
    - name: faktory
      protocol: TCP
      port: 7419
      targetPort: 7419
  selector:
    app: faktory
---
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: faktory
  name: faktory-conf
data:
  cron.toml: |
---
apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  namespace: faktory
  name: faktory
spec:
  serviceName: faktory
  replicas: 1
  template:
    metadata:
      namespace: faktory
      labels:
        app: faktory
    spec:
      terminationGracePeriodSeconds: 10
      containers:
        - name: faktory
          image: contribsys/faktory:0.9.6
          command:
            - /faktory
            - -b
            - :7419
            - -w
            - :7420
            - -e
            - production
          resources:
            requests:
              cpu: 300m
              memory: 1Gi
          env:
            - name: FAKTORY_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: faktory
                  key: password
          ports:
            - containerPort: 7419
              name: faktory
          volumeMounts:
            - name: faktory-data
              mountPath: /var/lib/faktory/db
            - name: faktory-conf
              mountPath: /etc/faktory/conf.d
      volumes:
        - name: faktory-conf
          configMap:
            name: faktory-conf
            items:
              - key: cron.toml
                path: cron.toml
  volumeClaimTemplates:
    - metadata:
        name: faktory-data
        namespace: faktory
        annotations:
          volume.beta.kubernetes.io/storage-class: fast
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 10Gi

A helm chart might be the right call for a more configurable and standardized deployment.