Kubernetes is meant to unlock the cloud platform for developers, make them more independent and push the DevOps culture.

Migration to Nginx ingress was relatively simple for us and didn’t require a lot of changes because of our controller approach. More savings can come if we use ingress in production as well. It’s not a simple change. Several considerations have to go in configuring ingress for production the right way and needs to be looked at from the perspective of security and API management as well. This is an area we intend to work in the near future.

Deploying Open Policy Agent to build the right controls helped automate the entire change management process and build the right safety nets for our developers. With Open Policy Agent, we can restrict scenarios like one just mentioned before — it is possible to restrict service objects from getting created unless the right annotation is present so that developers don’t accidentally create public ELBs.

Pods can be provisioned on any node. Even if you control how pods are spread in your cluster, there is no easy way to control how services discover each other in a way that a pod of one service talks to the pod of another service in the same AZ to reduce cross-AZ data transfer.

After a lot of research and conversations with peers in other companies, we learned that something like this can be achieved by introducing a service mesh to control how traffic from a pod is routed to the destination pod. We were not ready to take the complexity of operating a service mesh ourselves just for the benefit of saving the cost of cross-AZ data transfer.

Using spot instances with Kubernetes is a lot easier than using spot instances with vanilla VMs. With VMs, you can manage spot instances yourself which might have some complexity of ensuring a proper uptime for your applications or use a service like SpotInst. The same applies to Kubernetes as well but the resource efficiency brought in by Kubernetes can leave you enough room for keeping some buffer so that even if a few instances in your cluster get interrupted, the containers scheduled on them can be quickly rescheduled elsewhere. There are a few options for efficiently managing spot interruptions.

http://svt.munich.es/fmi/videos-sivas-belediyespor-v-elaziğspor-v-tr-tr-1qov-28.php

http://stream88.colomboserboli.com/eca/v-ideos-karacabey-birlikspor-v-etimesgut-belediyespor-v-tr-tr-1enl-28.php

http://m.dentisalut.com/omy/videos-zbrojovka-brno-v-wsg-wattens-v-cs-cs-1qrw-13.php

http://svt.munich.es/fmi/v-ideos-sivas-belediyespor-v-elaziğspor-v-tr-tr-1gcz-24.php

http://svt.munich.es/fmi/Video-sivas-belediyespor-v-elaziğspor-v-tr-tr-1kan-19.php

http://agro.ruicasa.com/vtm/Video-TP-Mazembe-AS-Bouenguidi-v-en-gb-1dxj-.php

http://svt.munich.es/fmi/video-sivas-belediyespor-v-elaziğspor-v-tr-tr-1atm-21.php

http://m.dentisalut.com/omy/video-Apollon-Smyrnis-Panathinaikos-v-en-gb-pim30122020-.php

http://m.dentisalut.com/omy/video-Apollon-Smyrnis-Panathinaikos-v-en-gb-kuv-.php

http://m.dentisalut.com/omy/Video-Apollon-Smyrnis-Panathinaikos-v-en-gb-ksf30122020-.php

http://m.dentisalut.com/omy/v-ideos-Esae-FC-TAS-Casablanca-v-en-gb-caj-.php

http://m.dentisalut.com/omy/video-Esae-FC-TAS-Casablanca-v-en-gb-yzk-.php

http://m.dentisalut.com/omy/Video-Esae-FC-TAS-Casablanca-v-en-gb-qvk30122020-.php

http://agro.ruicasa.com/vtm/video-TP-Mazembe-AS-Bouenguidi-v-en-gb-1ttw30122020-1.php

http://m.dentisalut.com/omy/videos-Kampala-City-AS-Kigali-v-en-gb-nfz30122020-.php

http://m.dentisalut.com/omy/video-Kampala-City-AS-Kigali-v-en-gb-dzf30122020-.php

http://m.dentisalut.com/omy/videos-Kampala-City-AS-Kigali-v-en-gb-pfa30122020-.php

http://agro.ruicasa.com/vtm/videos-TP-Mazembe-AS-Bouenguidi-v-en-gb-1wdj-16.php

http://m.dentisalut.com/omy/Video-esae-fc-v-tas-casablanca-v-fr-fr-1dmt-5.php

http://m.dentisalut.com/omy/Video-esae-fc-v-tas-casablanca-v-fr-fr-1jke-10.php

s: if you think you deserve better, you do. If you’re happy with whom you’re with, then be happy and don’t listen to what others have to say. If it bites you in the end, it’s a lesson learned. Just like nobody can define your worth, nobody can tell you what you deserve.

We have started investing in a bunch of controllers and CRDs. For instance, LoadBalancer service type to ingress conversion is a controller operation. Similarly, we use controllers to automatically create CNAME records in our DNS provider whenever a new service is deployed. These are a few examples. We have 5 other separate use-cases where we are relying on our internal controller to simplify daily operations and reduce toil.

We have also built a few CRDs. One of them is widely used today to generate monitoring dashboards on Grafana by declaratively specifying what monitoring dashboards should be constructed with. This makes it possible for developers to check-in their monitoring dashboards next to their application code base and deploy everything using the same workflow — kubectl apply -f . .

Sometimes this independence could pose severe risks. For example, using the LoadBalancer type service in EKS provisions a public-network facing ELB by default. Adding a certain annotation would ensure that an internal ELB is provisioned.We made some of these mistakes early on.

High infrastructure cost due to large resource buffers was a big problem. We were not really able to realise any benefits of capacity utilisation due to Kubernetes that we should have. It was after migrating to EKS and observing the stability it brought helped us become more confident, which helped us take the necessary steps to correct resource requests and bring down resource wastage drastically.

We used Ingress to consolidate ELBs in our stage environment and reduce the fixed costs of ELBs drastically. To avoid this from becoming a cause of dev/prod disparity in code, we decided to implement a controller that would mutate LoadBalancer type services to NodePort type services along with an ingress object in our stage cluster.

In our two years of journey with Kubernetes, we learned that Kubernetes is great but it’s better when you are using its features such as controllers, operators and CRDs to simplify daily operations and provide a more integrated experience to your developers.

Spot instances helped us get massive savings. Today, our entire stage Kubernetes cluster runs on spot instances and 99% of our production Kubernetes cluster is covered by reserved instances, savings plan and spot instances.

However, initially we had an enormous amount of wastage of resources while we were migrating. Owing to our inability to tune our self-managed Kubernetes cluster the right way which led to a ton of performance issues, we ended up requesting a lot of resources in our pods as buffer and more like insurance to reduce chances of outages or performance issues due to lack of compute or memory.

This was the most obvious one. Our infrastructure today has far less compute, memory and storage provisioned than we had before. Apart from better capacity utilisation due to better packing of containers/processes, we were able to better utilise our shared services such as processes for observability (metrics, logs) than before.