k3s subset not found
December 20, 2024
While I was working on an app in my k3s cluster, I was seeing a 404 not found in the web browser. This error message was not coming from my web application. The error was coming from Traefik ingress, indicating subset not found.
Looking in the Traefik pod logs I found a few errors.
> kubectl logs traefik-57b79cf995-cfpf9 -n kube-system
time="2024-09-16T21:04:36Z" level=info msg="Configuration loaded from flags."
time="2024-09-16T22:15:37Z" level=error msg="endpoints not found for default/test-app" ingress=test-app namespace=default providerName=kubernetescrd
time="2024-09-16T22:15:37Z" level=error msg="subset not found for default/test-app" namespace=default providerName=kubernetescrd ingress=test-app
You can see errors in Traefik for example level=error msg="endpoints not found for...
Then if you check your endpoints in k8s
> kubectl get endpoints
NAME ENDPOINTS AGE
kubernetes 192.100.10.01:6443 7d2h
test-app <none> 5s
You will see there is <none> endpoints for the app. This indicates your k8s service cannot find the correct pods to use.
Kubernetes endpoints not found and subset not found errors are because your pod port names don’t match your port names
in your service definition.
Example deployment with naming your pod ports.
kind: Deployment
metadata:
name: my-app
labels:
app: my-app
spec:
replicas: 3
selector:
matchLabels:
app: my-app
template:
metadata:
namespace: default
labels:
app: my-app
spec:
containers:
- name: my-app
image: nginx
ports:
- containerPort: 8080
# pod port name must match in the service
name: my-port-name
The service definition example with port named the same as the deployment definition port names.
apiVersion: v1
kind: Service
metadata:
name: my-app-service
labels:
app: my-app
spec:
type: ClusterIP
selector:
app: my-app
ports:
# port name must match what is in your pods
- name: my-port-name
port: 80
targetPort: my-port-name
Now the pod port names are defined in the service ports.
Verify by checking your endpoints. Here you will see an IP for each pod in your deployment.
> kubectl get endpoints
NAME ENDPOINTS AGE
kubernetes 192.100.10.01:6443 7d2h
my-app 10.42.0.10:80,10.42.0.11:80,10.42.0.12:80 8s
When troubleshooting and debugging issues in kubernetes, it usually helps to look into your logs for your pod or cluster.