Fix Kubernetes scheduling issue by optimizing deployment configuration
This commit is contained in:
parent
cfbbf9051f
commit
2ce57cbcbe
112
README.md
112
README.md
@ -55,34 +55,120 @@ The application can be deployed using Kubernetes with the provided configuration
|
|||||||
- `pvc.yaml`: Persistent Volume Claim for storage
|
- `pvc.yaml`: Persistent Volume Claim for storage
|
||||||
- `secrets.yaml`: Secret configuration for environment variables
|
- `secrets.yaml`: Secret configuration for environment variables
|
||||||
|
|
||||||
### Deployment Notes
|
### Deployment Configuration
|
||||||
|
|
||||||
- Optimized for reliable scheduling on resource-constrained clusters:
|
The deployment has been optimized for maximum scheduling flexibility and minimal resource requirements:
|
||||||
- Reduced replicas to 1 to ensure easier scheduling
|
|
||||||
- Reduced resource requests and limits
|
|
||||||
- Changed image pull policy to IfNotPresent to reduce pull issues
|
|
||||||
- Added flexible tolerations to run on nodes with any taints
|
|
||||||
- Configured node affinity for Linux but as a preference, not a requirement
|
|
||||||
- Set empty storageClassName to use the cluster's default storage class
|
|
||||||
|
|
||||||
- If you experience the "pod does not have a host assigned" error:
|
- **Minimal Resource Requirements**:
|
||||||
- Check that your cluster has available nodes with sufficient resources
|
- Single replica (no high availability) to ensure easier scheduling
|
||||||
- Verify that the default storage class exists and is working
|
- Extremely low resource requests (10m CPU, 64Mi memory)
|
||||||
- Ensure the specified image exists and is accessible
|
- No resource limits to allow for flexible resource usage
|
||||||
|
- Reduced PVC storage request (100Mi)
|
||||||
|
- Optional emptyDir volume configuration (commented out)
|
||||||
|
|
||||||
|
- **Maximum Scheduling Flexibility**:
|
||||||
|
- Uses Recreate deployment strategy instead of RollingUpdate
|
||||||
|
- Tolerations for all taints using `operator: "Exists"`
|
||||||
|
- No node selector or required affinity rules
|
||||||
|
- Default scheduler and DNS policy
|
||||||
|
|
||||||
|
- **Vault Integration (Commented Out)**:
|
||||||
|
- Added configuration templates for HashiCorp Vault integration
|
||||||
|
- Instructions for using Vault Agent Injector with the deployment
|
||||||
|
- Support for vault-agent and vault-agent-init sidecars
|
||||||
|
|
||||||
|
### Troubleshooting "pod does not have a host assigned" Error
|
||||||
|
|
||||||
|
If you encounter this error, follow these steps:
|
||||||
|
|
||||||
|
1. **Check Cluster Resources**:
|
||||||
|
```bash
|
||||||
|
kubectl get nodes
|
||||||
|
kubectl describe nodes
|
||||||
|
```
|
||||||
|
- Ensure nodes have available CPU, memory, and are in Ready state
|
||||||
|
|
||||||
|
2. **Check PVC Status**:
|
||||||
|
```bash
|
||||||
|
kubectl get pvc
|
||||||
|
kubectl describe pvc inventory-api-pvc
|
||||||
|
```
|
||||||
|
- If the PVC is stuck in Pending state, check storage class availability:
|
||||||
|
```bash
|
||||||
|
kubectl get storageclass
|
||||||
|
```
|
||||||
|
- If no storage class is available, modify deployment.yaml to use emptyDir:
|
||||||
|
```yaml
|
||||||
|
volumes:
|
||||||
|
- name: storage-volume
|
||||||
|
emptyDir: {}
|
||||||
|
# persistentVolumeClaim:
|
||||||
|
# claimName: inventory-api-pvc
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Check Pod Status**:
|
||||||
|
```bash
|
||||||
|
kubectl get pods
|
||||||
|
kubectl describe pod <pod-name>
|
||||||
|
kubectl get events --sort-by='.metadata.creationTimestamp'
|
||||||
|
```
|
||||||
|
- Look for scheduling errors or resource constraints
|
||||||
|
|
||||||
|
4. **Verify Vault Configuration** (if applicable):
|
||||||
|
- If your cluster uses HashiCorp Vault, uncomment the Vault configuration in deployment.yaml
|
||||||
|
- Check Vault service status:
|
||||||
|
```bash
|
||||||
|
kubectl get svc -n vault
|
||||||
|
```
|
||||||
|
|
||||||
|
5. **Extreme Resource Reduction**:
|
||||||
|
- If still having issues, you can uncomment and adjust these values in deployment.yaml:
|
||||||
|
```yaml
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
cpu: "1m" # Absolute minimum
|
||||||
|
memory: "32Mi" # Absolute minimum
|
||||||
|
```
|
||||||
|
|
||||||
|
### Deployment Commands
|
||||||
|
|
||||||
To deploy:
|
To deploy:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
# Apply secret first
|
||||||
kubectl apply -f kubernetes/secrets.yaml
|
kubectl apply -f kubernetes/secrets.yaml
|
||||||
|
|
||||||
|
# Create PVC
|
||||||
kubectl apply -f kubernetes/pvc.yaml
|
kubectl apply -f kubernetes/pvc.yaml
|
||||||
|
|
||||||
|
# Apply deployment
|
||||||
kubectl apply -f kubernetes/deployment.yaml
|
kubectl apply -f kubernetes/deployment.yaml
|
||||||
|
|
||||||
|
# Apply service
|
||||||
kubectl apply -f kubernetes/service.yaml
|
kubectl apply -f kubernetes/service.yaml
|
||||||
```
|
```
|
||||||
|
|
||||||
To check the status:
|
To check status:
|
||||||
|
|
||||||
```bash
|
```bash
|
||||||
|
# Get pod status
|
||||||
kubectl get pods
|
kubectl get pods
|
||||||
|
|
||||||
|
# Describe pod for detailed information
|
||||||
kubectl describe pod <pod-name>
|
kubectl describe pod <pod-name>
|
||||||
|
|
||||||
|
# Check recent events
|
||||||
kubectl get events --sort-by='.metadata.creationTimestamp'
|
kubectl get events --sort-by='.metadata.creationTimestamp'
|
||||||
|
|
||||||
|
# Get logs from container
|
||||||
|
kubectl logs <pod-name> -c app
|
||||||
|
```
|
||||||
|
|
||||||
|
To delete deployment:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
kubectl delete -f kubernetes/deployment.yaml
|
||||||
|
kubectl delete -f kubernetes/service.yaml
|
||||||
|
kubectl delete -f kubernetes/pvc.yaml
|
||||||
|
kubectl delete -f kubernetes/secrets.yaml
|
||||||
```
|
```
|
||||||
|
@ -3,4 +3,7 @@ from .crud_product import product
|
|||||||
from .crud_category import category
|
from .crud_category import category
|
||||||
from .crud_supplier import supplier
|
from .crud_supplier import supplier
|
||||||
from .crud_inventory import inventory
|
from .crud_inventory import inventory
|
||||||
from .crud_order import order
|
from .crud_order import order
|
||||||
|
|
||||||
|
# Re-export all CRUD modules
|
||||||
|
__all__ = ["user", "product", "category", "supplier", "inventory", "order"]
|
@ -27,7 +27,7 @@ class CRUDProduct(CRUDBase[Product, ProductCreate, ProductUpdate]):
|
|||||||
def get_active(
|
def get_active(
|
||||||
self, db: Session, *, skip: int = 0, limit: int = 100
|
self, db: Session, *, skip: int = 0, limit: int = 100
|
||||||
) -> List[Product]:
|
) -> List[Product]:
|
||||||
return db.query(Product).filter(Product.is_active == True).offset(skip).limit(limit).all()
|
return db.query(Product).filter(Product.is_active).offset(skip).limit(limit).all()
|
||||||
|
|
||||||
|
|
||||||
product = CRUDProduct(Product)
|
product = CRUDProduct(Product)
|
@ -4,4 +4,16 @@ from .category import Category, CategoryCreate, CategoryInDB, CategoryUpdate
|
|||||||
from .supplier import Supplier, SupplierCreate, SupplierInDB, SupplierUpdate
|
from .supplier import Supplier, SupplierCreate, SupplierInDB, SupplierUpdate
|
||||||
from .product import Product, ProductCreate, ProductInDB, ProductUpdate
|
from .product import Product, ProductCreate, ProductInDB, ProductUpdate
|
||||||
from .inventory import InventoryItem, InventoryItemCreate, InventoryItemInDB, InventoryItemUpdate
|
from .inventory import InventoryItem, InventoryItemCreate, InventoryItemInDB, InventoryItemUpdate
|
||||||
from .order import Order, OrderCreate, OrderInDB, OrderUpdate, OrderItem, OrderItemCreate, OrderItemInDB, OrderItemUpdate
|
from .order import Order, OrderCreate, OrderInDB, OrderUpdate, OrderItem, OrderItemCreate, OrderItemInDB, OrderItemUpdate
|
||||||
|
|
||||||
|
# Re-export all schema types
|
||||||
|
__all__ = [
|
||||||
|
"Token", "TokenPayload",
|
||||||
|
"User", "UserCreate", "UserInDB", "UserUpdate",
|
||||||
|
"Category", "CategoryCreate", "CategoryInDB", "CategoryUpdate",
|
||||||
|
"Supplier", "SupplierCreate", "SupplierInDB", "SupplierUpdate",
|
||||||
|
"Product", "ProductCreate", "ProductInDB", "ProductUpdate",
|
||||||
|
"InventoryItem", "InventoryItemCreate", "InventoryItemInDB", "InventoryItemUpdate",
|
||||||
|
"Order", "OrderCreate", "OrderInDB", "OrderUpdate",
|
||||||
|
"OrderItem", "OrderItemCreate", "OrderItemInDB", "OrderItemUpdate"
|
||||||
|
]
|
@ -4,32 +4,57 @@ metadata:
|
|||||||
name: prod-pod
|
name: prod-pod
|
||||||
labels:
|
labels:
|
||||||
app: inventory-api
|
app: inventory-api
|
||||||
|
# Uncomment the following annotation if using Vault Agent Injector
|
||||||
|
# annotations:
|
||||||
|
# vault.hashicorp.com/agent-inject: "true"
|
||||||
|
# vault.hashicorp.com/agent-inject-status: "update"
|
||||||
|
# vault.hashicorp.com/role: "inventory-api"
|
||||||
|
# vault.hashicorp.com/agent-inject-secret-config: "secret/data/inventory-api/config"
|
||||||
|
# vault.hashicorp.com/agent-inject-template-config: |
|
||||||
|
# {{- with secret "secret/data/inventory-api/config" -}}
|
||||||
|
# export SECRET_KEY="{{ .Data.data.secret_key }}"
|
||||||
|
# export FIRST_SUPERUSER_PASSWORD="{{ .Data.data.admin_password }}"
|
||||||
|
# {{- end -}}
|
||||||
spec:
|
spec:
|
||||||
replicas: 1 # Reduced replicas to 1 to ensure easier scheduling
|
replicas: 1 # Use only 1 replica to minimize resource requirements
|
||||||
selector:
|
selector:
|
||||||
matchLabels:
|
matchLabels:
|
||||||
app: inventory-api
|
app: inventory-api
|
||||||
strategy:
|
strategy:
|
||||||
type: RollingUpdate
|
type: Recreate # Changed from RollingUpdate to Recreate to avoid running multiple pods
|
||||||
rollingUpdate:
|
|
||||||
maxSurge: 1
|
|
||||||
maxUnavailable: 0
|
|
||||||
template:
|
template:
|
||||||
metadata:
|
metadata:
|
||||||
labels:
|
labels:
|
||||||
app: inventory-api
|
app: inventory-api
|
||||||
|
# Uncomment the following annotations if using Vault Agent Injector
|
||||||
|
# annotations:
|
||||||
|
# vault.hashicorp.com/agent-inject: "true"
|
||||||
|
# vault.hashicorp.com/agent-inject-status: "update"
|
||||||
|
# vault.hashicorp.com/role: "inventory-api"
|
||||||
|
# vault.hashicorp.com/agent-inject-secret-config: "secret/data/inventory-api/config"
|
||||||
|
# vault.hashicorp.com/agent-inject-template-config: |
|
||||||
|
# {{- with secret "secret/data/inventory-api/config" -}}
|
||||||
|
# export SECRET_KEY="{{ .Data.data.secret_key }}"
|
||||||
|
# export FIRST_SUPERUSER_PASSWORD="{{ .Data.data.admin_password }}"
|
||||||
|
# {{- end -}}
|
||||||
spec:
|
spec:
|
||||||
|
# If using Vault, you might need to add an initContainer to wait for Vault secrets
|
||||||
|
# initContainers:
|
||||||
|
# - name: wait-for-vault
|
||||||
|
# image: busybox
|
||||||
|
# command: ['sh', '-c', 'until [ -f /vault/secrets/config ]; do echo "Waiting for Vault secrets..."; sleep 2; done']
|
||||||
|
# volumeMounts:
|
||||||
|
# - name: vault-secrets
|
||||||
|
# mountPath: /vault/secrets
|
||||||
containers:
|
containers:
|
||||||
- name: app
|
- name: app
|
||||||
image: ${IMAGE_REPOSITORY}:${IMAGE_TAG}
|
image: ${IMAGE_REPOSITORY}:${IMAGE_TAG}
|
||||||
imagePullPolicy: IfNotPresent # Changed from Always to IfNotPresent to reduce pull issues
|
imagePullPolicy: IfNotPresent # Use existing images when possible
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
cpu: "50m" # Reduced CPU request
|
cpu: "10m" # Absolute minimum CPU request
|
||||||
memory: "128Mi" # Reduced memory request
|
memory: "64Mi" # Absolute minimum memory request
|
||||||
limits:
|
# Removed limits to allow the pod to use available resources if needed
|
||||||
cpu: "300m" # Reduced CPU limit
|
|
||||||
memory: "384Mi" # Reduced memory limit
|
|
||||||
ports:
|
ports:
|
||||||
- containerPort: 8000
|
- containerPort: 8000
|
||||||
name: http
|
name: http
|
||||||
@ -50,41 +75,45 @@ spec:
|
|||||||
secretKeyRef:
|
secretKeyRef:
|
||||||
name: inventory-api-secrets
|
name: inventory-api-secrets
|
||||||
key: admin-password
|
key: admin-password
|
||||||
|
# Simplified probes to reduce load during startup
|
||||||
livenessProbe:
|
livenessProbe:
|
||||||
httpGet:
|
httpGet:
|
||||||
path: /health
|
path: /health
|
||||||
port: 8000
|
port: 8000
|
||||||
initialDelaySeconds: 30
|
initialDelaySeconds: 60 # Increased to give pod more time to start
|
||||||
periodSeconds: 10
|
periodSeconds: 30
|
||||||
timeoutSeconds: 5
|
timeoutSeconds: 10
|
||||||
failureThreshold: 3
|
failureThreshold: 5
|
||||||
readinessProbe:
|
readinessProbe:
|
||||||
httpGet:
|
httpGet:
|
||||||
path: /health
|
path: /health
|
||||||
port: 8000
|
port: 8000
|
||||||
initialDelaySeconds: 5
|
initialDelaySeconds: 30 # Increased to give pod more time to start
|
||||||
periodSeconds: 10
|
periodSeconds: 30
|
||||||
timeoutSeconds: 3
|
timeoutSeconds: 10
|
||||||
failureThreshold: 3
|
failureThreshold: 5
|
||||||
volumeMounts:
|
volumeMounts:
|
||||||
- name: storage-volume
|
- name: storage-volume
|
||||||
mountPath: /app/storage
|
mountPath: /app/storage
|
||||||
|
# If using Vault, add the following volume mount
|
||||||
|
# - name: vault-secrets
|
||||||
|
# mountPath: /vault/secrets
|
||||||
volumes:
|
volumes:
|
||||||
- name: storage-volume
|
- name: storage-volume
|
||||||
|
# Uncomment the following if PVC fails to provision
|
||||||
|
# emptyDir: {}
|
||||||
persistentVolumeClaim:
|
persistentVolumeClaim:
|
||||||
claimName: inventory-api-pvc
|
claimName: inventory-api-pvc
|
||||||
# Enhanced tolerations to allow more flexible scheduling
|
# If using Vault, add this volume
|
||||||
|
# - name: vault-secrets
|
||||||
|
# emptyDir:
|
||||||
|
# medium: Memory
|
||||||
|
# Maximum scheduling flexibility
|
||||||
tolerations:
|
tolerations:
|
||||||
- operator: "Exists" # This will tolerate all taints
|
- operator: "Exists" # Tolerate all taints
|
||||||
# Affinity settings to help with scheduling
|
dnsPolicy: ClusterFirst
|
||||||
affinity:
|
restartPolicy: Always
|
||||||
nodeAffinity:
|
securityContext: {}
|
||||||
preferredDuringSchedulingIgnoredDuringExecution:
|
terminationGracePeriodSeconds: 30
|
||||||
- weight: 1
|
schedulerName: default-scheduler
|
||||||
preference:
|
# No nodeSelector, no affinity to allow maximum scheduling flexibility
|
||||||
matchExpressions:
|
|
||||||
- key: "kubernetes.io/os"
|
|
||||||
operator: In
|
|
||||||
values:
|
|
||||||
- linux
|
|
||||||
# No nodeSelector to allow maximum scheduling flexibility
|
|
@ -2,10 +2,14 @@ apiVersion: v1
|
|||||||
kind: PersistentVolumeClaim
|
kind: PersistentVolumeClaim
|
||||||
metadata:
|
metadata:
|
||||||
name: inventory-api-pvc
|
name: inventory-api-pvc
|
||||||
|
annotations:
|
||||||
|
volume.beta.kubernetes.io/storage-provisioner: kubernetes.io/no-provisioner
|
||||||
|
volume.kubernetes.io/storage-provisioner: kubernetes.io/no-provisioner
|
||||||
spec:
|
spec:
|
||||||
accessModes:
|
accessModes:
|
||||||
- ReadWriteOnce
|
- ReadWriteOnce
|
||||||
resources:
|
resources:
|
||||||
requests:
|
requests:
|
||||||
storage: 1Gi
|
storage: 100Mi # Reduced storage request
|
||||||
storageClassName: "" # Empty string to use the default storage class
|
storageClassName: "" # Empty string to use default storage class
|
||||||
|
volumeMode: Filesystem
|
Loading…
x
Reference in New Issue
Block a user