Fix Kubernetes scheduling issue by optimizing deployment configuration

This commit is contained in:
Automated Action 2025-06-05 23:54:57 +00:00
parent cfbbf9051f
commit 2ce57cbcbe
6 changed files with 185 additions and 51 deletions

112
README.md
View File

@ -55,34 +55,120 @@ The application can be deployed using Kubernetes with the provided configuration
- `pvc.yaml`: Persistent Volume Claim for storage - `pvc.yaml`: Persistent Volume Claim for storage
- `secrets.yaml`: Secret configuration for environment variables - `secrets.yaml`: Secret configuration for environment variables
### Deployment Notes ### Deployment Configuration
- Optimized for reliable scheduling on resource-constrained clusters: The deployment has been optimized for maximum scheduling flexibility and minimal resource requirements:
- Reduced replicas to 1 to ensure easier scheduling
- Reduced resource requests and limits
- Changed image pull policy to IfNotPresent to reduce pull issues
- Added flexible tolerations to run on nodes with any taints
- Configured node affinity for Linux but as a preference, not a requirement
- Set empty storageClassName to use the cluster's default storage class
- If you experience the "pod does not have a host assigned" error: - **Minimal Resource Requirements**:
- Check that your cluster has available nodes with sufficient resources - Single replica (no high availability) to ensure easier scheduling
- Verify that the default storage class exists and is working - Extremely low resource requests (10m CPU, 64Mi memory)
- Ensure the specified image exists and is accessible - No resource limits to allow for flexible resource usage
- Reduced PVC storage request (100Mi)
- Optional emptyDir volume configuration (commented out)
- **Maximum Scheduling Flexibility**:
- Uses Recreate deployment strategy instead of RollingUpdate
- Tolerations for all taints using `operator: "Exists"`
- No node selector or required affinity rules
- Default scheduler and DNS policy
- **Vault Integration (Commented Out)**:
- Added configuration templates for HashiCorp Vault integration
- Instructions for using Vault Agent Injector with the deployment
- Support for vault-agent and vault-agent-init sidecars
### Troubleshooting "pod does not have a host assigned" Error
If you encounter this error, follow these steps:
1. **Check Cluster Resources**:
```bash
kubectl get nodes
kubectl describe nodes
```
- Ensure nodes have available CPU, memory, and are in Ready state
2. **Check PVC Status**:
```bash
kubectl get pvc
kubectl describe pvc inventory-api-pvc
```
- If the PVC is stuck in Pending state, check storage class availability:
```bash
kubectl get storageclass
```
- If no storage class is available, modify deployment.yaml to use emptyDir:
```yaml
volumes:
- name: storage-volume
emptyDir: {}
# persistentVolumeClaim:
# claimName: inventory-api-pvc
```
3. **Check Pod Status**:
```bash
kubectl get pods
kubectl describe pod <pod-name>
kubectl get events --sort-by='.metadata.creationTimestamp'
```
- Look for scheduling errors or resource constraints
4. **Verify Vault Configuration** (if applicable):
- If your cluster uses HashiCorp Vault, uncomment the Vault configuration in deployment.yaml
- Check Vault service status:
```bash
kubectl get svc -n vault
```
5. **Extreme Resource Reduction**:
- If still having issues, you can uncomment and adjust these values in deployment.yaml:
```yaml
resources:
requests:
cpu: "1m" # Absolute minimum
memory: "32Mi" # Absolute minimum
```
### Deployment Commands
To deploy: To deploy:
```bash ```bash
# Apply secret first
kubectl apply -f kubernetes/secrets.yaml kubectl apply -f kubernetes/secrets.yaml
# Create PVC
kubectl apply -f kubernetes/pvc.yaml kubectl apply -f kubernetes/pvc.yaml
# Apply deployment
kubectl apply -f kubernetes/deployment.yaml kubectl apply -f kubernetes/deployment.yaml
# Apply service
kubectl apply -f kubernetes/service.yaml kubectl apply -f kubernetes/service.yaml
``` ```
To check the status: To check status:
```bash ```bash
# Get pod status
kubectl get pods kubectl get pods
# Describe pod for detailed information
kubectl describe pod <pod-name> kubectl describe pod <pod-name>
# Check recent events
kubectl get events --sort-by='.metadata.creationTimestamp' kubectl get events --sort-by='.metadata.creationTimestamp'
# Get logs from container
kubectl logs <pod-name> -c app
```
To delete deployment:
```bash
kubectl delete -f kubernetes/deployment.yaml
kubectl delete -f kubernetes/service.yaml
kubectl delete -f kubernetes/pvc.yaml
kubectl delete -f kubernetes/secrets.yaml
``` ```

View File

@ -3,4 +3,7 @@ from .crud_product import product
from .crud_category import category from .crud_category import category
from .crud_supplier import supplier from .crud_supplier import supplier
from .crud_inventory import inventory from .crud_inventory import inventory
from .crud_order import order from .crud_order import order
# Re-export all CRUD modules
__all__ = ["user", "product", "category", "supplier", "inventory", "order"]

View File

@ -27,7 +27,7 @@ class CRUDProduct(CRUDBase[Product, ProductCreate, ProductUpdate]):
def get_active( def get_active(
self, db: Session, *, skip: int = 0, limit: int = 100 self, db: Session, *, skip: int = 0, limit: int = 100
) -> List[Product]: ) -> List[Product]:
return db.query(Product).filter(Product.is_active == True).offset(skip).limit(limit).all() return db.query(Product).filter(Product.is_active).offset(skip).limit(limit).all()
product = CRUDProduct(Product) product = CRUDProduct(Product)

View File

@ -4,4 +4,16 @@ from .category import Category, CategoryCreate, CategoryInDB, CategoryUpdate
from .supplier import Supplier, SupplierCreate, SupplierInDB, SupplierUpdate from .supplier import Supplier, SupplierCreate, SupplierInDB, SupplierUpdate
from .product import Product, ProductCreate, ProductInDB, ProductUpdate from .product import Product, ProductCreate, ProductInDB, ProductUpdate
from .inventory import InventoryItem, InventoryItemCreate, InventoryItemInDB, InventoryItemUpdate from .inventory import InventoryItem, InventoryItemCreate, InventoryItemInDB, InventoryItemUpdate
from .order import Order, OrderCreate, OrderInDB, OrderUpdate, OrderItem, OrderItemCreate, OrderItemInDB, OrderItemUpdate from .order import Order, OrderCreate, OrderInDB, OrderUpdate, OrderItem, OrderItemCreate, OrderItemInDB, OrderItemUpdate
# Re-export all schema types
__all__ = [
"Token", "TokenPayload",
"User", "UserCreate", "UserInDB", "UserUpdate",
"Category", "CategoryCreate", "CategoryInDB", "CategoryUpdate",
"Supplier", "SupplierCreate", "SupplierInDB", "SupplierUpdate",
"Product", "ProductCreate", "ProductInDB", "ProductUpdate",
"InventoryItem", "InventoryItemCreate", "InventoryItemInDB", "InventoryItemUpdate",
"Order", "OrderCreate", "OrderInDB", "OrderUpdate",
"OrderItem", "OrderItemCreate", "OrderItemInDB", "OrderItemUpdate"
]

View File

@ -4,32 +4,57 @@ metadata:
name: prod-pod name: prod-pod
labels: labels:
app: inventory-api app: inventory-api
# Uncomment the following annotation if using Vault Agent Injector
# annotations:
# vault.hashicorp.com/agent-inject: "true"
# vault.hashicorp.com/agent-inject-status: "update"
# vault.hashicorp.com/role: "inventory-api"
# vault.hashicorp.com/agent-inject-secret-config: "secret/data/inventory-api/config"
# vault.hashicorp.com/agent-inject-template-config: |
# {{- with secret "secret/data/inventory-api/config" -}}
# export SECRET_KEY="{{ .Data.data.secret_key }}"
# export FIRST_SUPERUSER_PASSWORD="{{ .Data.data.admin_password }}"
# {{- end -}}
spec: spec:
replicas: 1 # Reduced replicas to 1 to ensure easier scheduling replicas: 1 # Use only 1 replica to minimize resource requirements
selector: selector:
matchLabels: matchLabels:
app: inventory-api app: inventory-api
strategy: strategy:
type: RollingUpdate type: Recreate # Changed from RollingUpdate to Recreate to avoid running multiple pods
rollingUpdate:
maxSurge: 1
maxUnavailable: 0
template: template:
metadata: metadata:
labels: labels:
app: inventory-api app: inventory-api
# Uncomment the following annotations if using Vault Agent Injector
# annotations:
# vault.hashicorp.com/agent-inject: "true"
# vault.hashicorp.com/agent-inject-status: "update"
# vault.hashicorp.com/role: "inventory-api"
# vault.hashicorp.com/agent-inject-secret-config: "secret/data/inventory-api/config"
# vault.hashicorp.com/agent-inject-template-config: |
# {{- with secret "secret/data/inventory-api/config" -}}
# export SECRET_KEY="{{ .Data.data.secret_key }}"
# export FIRST_SUPERUSER_PASSWORD="{{ .Data.data.admin_password }}"
# {{- end -}}
spec: spec:
# If using Vault, you might need to add an initContainer to wait for Vault secrets
# initContainers:
# - name: wait-for-vault
# image: busybox
# command: ['sh', '-c', 'until [ -f /vault/secrets/config ]; do echo "Waiting for Vault secrets..."; sleep 2; done']
# volumeMounts:
# - name: vault-secrets
# mountPath: /vault/secrets
containers: containers:
- name: app - name: app
image: ${IMAGE_REPOSITORY}:${IMAGE_TAG} image: ${IMAGE_REPOSITORY}:${IMAGE_TAG}
imagePullPolicy: IfNotPresent # Changed from Always to IfNotPresent to reduce pull issues imagePullPolicy: IfNotPresent # Use existing images when possible
resources: resources:
requests: requests:
cpu: "50m" # Reduced CPU request cpu: "10m" # Absolute minimum CPU request
memory: "128Mi" # Reduced memory request memory: "64Mi" # Absolute minimum memory request
limits: # Removed limits to allow the pod to use available resources if needed
cpu: "300m" # Reduced CPU limit
memory: "384Mi" # Reduced memory limit
ports: ports:
- containerPort: 8000 - containerPort: 8000
name: http name: http
@ -50,41 +75,45 @@ spec:
secretKeyRef: secretKeyRef:
name: inventory-api-secrets name: inventory-api-secrets
key: admin-password key: admin-password
# Simplified probes to reduce load during startup
livenessProbe: livenessProbe:
httpGet: httpGet:
path: /health path: /health
port: 8000 port: 8000
initialDelaySeconds: 30 initialDelaySeconds: 60 # Increased to give pod more time to start
periodSeconds: 10 periodSeconds: 30
timeoutSeconds: 5 timeoutSeconds: 10
failureThreshold: 3 failureThreshold: 5
readinessProbe: readinessProbe:
httpGet: httpGet:
path: /health path: /health
port: 8000 port: 8000
initialDelaySeconds: 5 initialDelaySeconds: 30 # Increased to give pod more time to start
periodSeconds: 10 periodSeconds: 30
timeoutSeconds: 3 timeoutSeconds: 10
failureThreshold: 3 failureThreshold: 5
volumeMounts: volumeMounts:
- name: storage-volume - name: storage-volume
mountPath: /app/storage mountPath: /app/storage
# If using Vault, add the following volume mount
# - name: vault-secrets
# mountPath: /vault/secrets
volumes: volumes:
- name: storage-volume - name: storage-volume
# Uncomment the following if PVC fails to provision
# emptyDir: {}
persistentVolumeClaim: persistentVolumeClaim:
claimName: inventory-api-pvc claimName: inventory-api-pvc
# Enhanced tolerations to allow more flexible scheduling # If using Vault, add this volume
# - name: vault-secrets
# emptyDir:
# medium: Memory
# Maximum scheduling flexibility
tolerations: tolerations:
- operator: "Exists" # This will tolerate all taints - operator: "Exists" # Tolerate all taints
# Affinity settings to help with scheduling dnsPolicy: ClusterFirst
affinity: restartPolicy: Always
nodeAffinity: securityContext: {}
preferredDuringSchedulingIgnoredDuringExecution: terminationGracePeriodSeconds: 30
- weight: 1 schedulerName: default-scheduler
preference: # No nodeSelector, no affinity to allow maximum scheduling flexibility
matchExpressions:
- key: "kubernetes.io/os"
operator: In
values:
- linux
# No nodeSelector to allow maximum scheduling flexibility

View File

@ -2,10 +2,14 @@ apiVersion: v1
kind: PersistentVolumeClaim kind: PersistentVolumeClaim
metadata: metadata:
name: inventory-api-pvc name: inventory-api-pvc
annotations:
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/no-provisioner
volume.kubernetes.io/storage-provisioner: kubernetes.io/no-provisioner
spec: spec:
accessModes: accessModes:
- ReadWriteOnce - ReadWriteOnce
resources: resources:
requests: requests:
storage: 1Gi storage: 100Mi # Reduced storage request
storageClassName: "" # Empty string to use the default storage class storageClassName: "" # Empty string to use default storage class
volumeMode: Filesystem