☸️ Docker & Kubernetes
Auto-Scaling & Self Healing
Horizontal Pod Autoscaling and Kubernetes self-healing mechanisms
Kubernetes HPA watches CPU/memory metrics (or custom metrics via KEDA) and automatically adds or removes Pod replicas to match demand. Combined with self-healing, this gives your system automatic resilience without human intervention.
Key Concepts
HPA Scaling Logic
Desired replicas = ceil(current replicas × current metric / target metric). At 80% CPU target with 5 pods at 160% CPU: ceil(5 × 160/80) = 10 pods. Scale triggers within 15-30 seconds.
Self-Healing
K8s ReplicaSet controller continuously reconciles. If a Pod crashes, controller creates a replacement. If a Node fails, its Pods are evicted and recreated on healthy nodes within 60-300 seconds.
Rolling Updates
Deployment controller updates one Pod at a time (configurable). Old Pods serve traffic while new ones start. readinessProbe must pass before traffic is shifted. Zero downtime by default.
HPA Scaling Diagram
How Auto-Scaling Works
The Horizontal Pod Autoscaler (HPA) works in a continuous loop, ensuring that your application matches traffic demand without human intervention.
- **1. Metrics Collection:** The Kubernetes Metrics Server continuously gathers CPU and Memory utilization data from every Pod.
- **2. Evaluation:** The HPA Controller checks if the average utilization across all Pods exceeds the target (e.g. 70%).
- **3. Scaling Decision:** If demand is too high, it calculates how many additional Pods are needed and updates the Deployment's `replicas` field.
- **4. Traffic Distribution:** As soon as the new Pods pass their `readinessProbe`, the Kubernetes Service automatically starts sending them traffic, lowering the CPU usage of the original Pods.
HPA Configuration Example
hpa.yamlyaml
1apiVersion: autoscaling/v2
2kind: HorizontalPodAutoscaler
3metadata:
4 name: order-service-hpa
5spec:
6 scaleTargetRef:
7 apiVersion: apps/v1
8 kind: Deployment
9 name: order-service
10 minReplicas: 3 # Never go below 3 — availability guarantee (အနည်းဆုံး ၃ ခု အမြဲရှိရမည်)
11 maxReplicas: 50 # Never exceed 50 — cost control (ကုန်ကျစရိတ်ထိန်းချုပ်ရန် အများဆုံး ၅၀ ထက်မကျော်ရ)
12 metrics:
13 - type: Resource
14 resource:
15 name: cpu
16 target:
17 type: Utilization
18 averageUtilization: 70 # Scale up when avg CPU > 70% (CPU 70% ကျော်လျှင် တိုးချဲ့မည်)
19 - type: Resource
20 resource:
21 name: memory
22 target:
23 type: AverageValue
24 averageValue: 400Mi
25 behavior:
26 scaleUp:
27 stabilizationWindowSeconds: 30 # React to spikes quickly (ရုတ်တရက်တက်လာပါက မြန်မြန်တုံ့ပြန်မည်)
28 policies:
29 - type: Pods
30 value: 5 # Add max 5 pods per 60s
31 periodSeconds: 60
32 scaleDown:
33 stabilizationWindowSeconds: 300 # Wait 5 min before scaling down (ပြန်မလျှော့ခင် ၅ မိနစ် စောင့်မည်)💡
Senior Architect Insight: Set scaleDown.stabilizationWindowSeconds to 300+ seconds. Without this, a 10-second traffic spike causes pods to scale up then immediately back down, causing thrashing. The stabilization window prevents premature scale-down after a burst.