Connected with the above issue is that there is, as of now, no go-to way of versioning machine learning models. It’s qui

To avoid all this, we decided to use Consul, Vault and Consul Template for configuration management. We run Consul Template as an init container today and plan to run it as a side car in pods so that it can watch for configuration changes in Consul and refresh expiring secrets from Vault and gracefully reload application processes.

Logs have always been a big problem for us. We have struggled to create a stable logging platform using ELK. We find ELK full of features that are not realistically used by our team. Those features come at a cost. Also, we think there are inherent challenges in using Elasticsearch for logs, making it an expensive solution for logs. We finalized on Loki by Grafana. It’s simple. It has necessary features for our team’s needs. It’s extremely cost-effective. But most importantly, it has a superior UX owing to it’s query language being very similar to PromQL. Also, it works well with Grafana. So that brings the entire metrics monitoring and logging experience together in one user interface.

Even when using a managed Kubernetes service, invest early in infrastructure-as-code setup to make disaster recovery and upgrade process relatively less painful in the future and be able to recover fast in face of disasters.

An important aspect of Kubernetes is to think about how developers are going to interact with the cluster and deploy their workloads. We wanted to keep things simple and easy to scale. We are converging towards Kustomize, Skaffold along with a few home-grown CRDs as the way for developers to deploy and manage applications. Having said that, any team is free to use whatever tools they would like to use to interact with the cluster as long as they are open-source and built on open standards.

We are not doing distributed tracing just yet. However, we plan to invest into that area real soon. Like with logging, our desire is to have distributed tracing be available next to metrics and logging in Grafana to deliver a more integrated observability experience to our development teams.

We mostly operate out of the Singapore region on AWS. At the time we started our journey with Kubernetes, EKS was not available as a service in the Singapore region. So we had to set up our own Kubernetes cluster on EC2 using kops.

We were using Jenkins before migrating to Kubernetes. After migrating to Kubernetes, we decided to stick to Jenkins. Our experience so far has been that Jenkins is not the best solution for working with cloud-native infrastructure. We found ourselves doing a lot of plumbing using Python, Bash, Docker and scripted/declarative Jenkins pipelines to make it all work. Building and maintaining these tools and pipelines started to feel expensive. We are right now exploring Tekton and Argo Workflows as our new CI/CD platform. There are more options you can explore in the CI/CD landscape such as Jenkins X, Screwdriver, Keptn, etc.

Our learning is that operating Kubernetes is complex. There are a lot of moving parts. And learning how to operate Kubernetes is most likely not core to your business. Offload as much as possible to cloud service providers (EKS, GKE, AKS). There is no value in doing this yourself.

We finalized on Prometheus. Prometheus is almost a defacto metrics infrastructure today. CNCF and Kubernetes love it very much. It works really well within the Grafana ecosystem. And we love Grafana! Our only problem was that we were using InfluxDB. We have decided to migrate away from InfluxDB and totally commit to Prometheus.

http://team.vidrio.org/xpy/Video-Tecnyconta-Zaragoza-Nizhny-Novgorod-BC-v-en-gb-1svp-9.php

http://elta.actiup.com/cqn/v-ideos-Balzan-St.-Lucia-FC-v-en-gb-iuk-.php

http://www.ectp.org/kzz/Video-sibir-novorossiisk-v-lokomotiv-iaroslavl-v-ru-ru-1vhs-14.php

http://molos.bodasturias.com/qxz/video-torpedo-nn-v-neftekhimik-v-ru-ru-1rqi-21.php

http://team.vidrio.org/xpy/Video-Tecnyconta-Zaragoza-Nizhny-Novgorod-BC-v-en-gb-1tvx30122020-1.php

http://molos.bodasturias.com/qxz/Video-torpedo-nn-v-neftekhimik-v-ru-ru-1avf-18.php

http://www.ectp.org/kzz/videos-severstal-cherepovets-v-kunlun-red-star-v-ru-ru-1amy-13.php

http://startup.munich.es/dyn/Video-espana-v-croacia-v-es-es-1ahn-27.php

http://molos.bodasturias.com/qxz/v-ideos-torpedo-nn-v-neftekhimik-v-ru-ru-1wdd-8.php

http://www.ectp.org/kzz/Video-severstal-cherepovets-v-kunlun-red-star-v-ru-ru-1dij-19.php

http://startup.munich.es/dyn/videos-espana-v-croacia-v-es-es-1zye-8.php

http://team.vidrio.org/xpy/video-v-gr-gr-1kqk-3.php

http://www.ectp.org/kzz/v-ideos-severstal-cherepovets-v-kunlun-red-star-v-ru-ru-1los-18.php

http://molos.bodasturias.com/qxz/v-ideos-torpedo-nn-v-neftekhimik-v-ru-ru-1fyk-6.php

http://team.vidrio.org/xpy/video-v-gr-gr-1lrh-3.php

http://team.vidrio.org/xpy/videos-v-gr-gr-1ace-5.php

http://elta.actiup.com/cqn/Video-Balzan-St.-Lucia-FC-v-en-gb-yiu-.php

http://startup.munich.es/dyn/v-ideos-espana-v-croacia-v-es-es-1bva-3.php

http://elta.actiup.com/cqn/video-TOP-Oss-Jong-Ajax-v-en-gb-off-.php

http://elta.actiup.com/cqn/video-TOP-Oss-Jong-Ajax-v-en-gb-lcn-.php

occur in groups,” says Christy A. Denckla, PhD, an assistant professor at Harvard T.H. Chan School of Public Health and a clinical psychologist who specializes in grief and trauma. “Traditionally people gather in some way physically to mourn and grieve and collectively support one another. That’s of course impossible in Covid. So some of the very natural or typical ways that people might gather to commemorate loss and grief are impossible now.”

It’s all the more important that as soon as a project is started, a benchmark is established against which the model runs now and in the future. In combination with diligent version control, data scientists can get their models reproducible.

You can make an attempt to push towards GitOps if you will. If you can’t do that, reducing manual steps to a bare minimum is a great start. We use a combination of eksctl, terraform and our cluster configuration manifests (including manifests for platform services) to set up what we call the “Grofers Kubernetes Platform”. To make the setup and deployment process simpler and repeatable, we have built an automated pipeline to set up new clusters and deploy changes to existing ones.

You will find most articles use configmap and secret objects in Kubernetes. Our learning is that it can get you started but we found it barely enough for our use-cases. Using configmap with existing services comes at a certain cost. Configmap can be mounted into pods in a certain way — using environment variables is the most common way. If you have a ton of legacy microservices that read configuration from files rendered by a configuration management tool such as Puppet, Chef or Ansible, you will have to redo configuration handling in all your code bases to now read from environment variables. We didn’t find enough reason to do this where it made sense. Also, a change in configuration or secret means that you will have to redeploy your deployment by patching it. This would be additional imperative orchestration of kubectl commands.

Setting up a basic cluster is perhaps not as difficult. We were able to get up our first cluster running within a week. Most issues happen when you start deploying your workloads. From tuning cluster autoscaler to provisioning resources at the right time to configuring the network correctly for the right performance, you have to do research and configure it all yourself. Defaults don’t work most of the time (or at least they didn’t work for us back then) for production.

There are a number of ways to use Kubernetes in your development workflow. We mostly zeroed down to two options — Telepresence.io and Skaffold. Skaffold is capable of watching your local changes and constantly deploying them to your Kubernetes cluster. Telepresence, on the other hand, allows you to run a service locally while setting up a transparent network proxy with the Kubernetes cluster so that your local service can communicate with other services in Kubernetes as if it was deployed in the cluster. It is a matter of personal opinions and preferences. It has been hard to decide on one tool. We are mostly experimenting with Telepresence right now but we have not abandoned the possibility of Skaffold being a better tooling for us. Only time will tell what we decide to use, or perhaps we use both. There are other solutions as well such as Draft that deserve a mention.