Monitoring with kubectl
PureLB attempts to provide all of the information necessary to monitor and troubleshoot the health of Load Balancer services via kubectl without resorting to inspecting PureLB’s POD logging. PureLB’s operational status and events are updated in the Load Balancer Services. PureLB annotates services where it was the source of the allocated address. If allocated-by is not present, PureLB did not allocate the address.
The simplest way to check the status of services if using the kubectl describe command.
$kubectl describe service specific-address2
Name: specific-address2
Namespace: adamd
Labels: app=echoserver3
Annotations: purelb.io/service-group: virtualsg
purelb.io/allocated-by: PureLB
purelb.io/allocated-from: virtualsub
Selector: app=echoserver3
Type: LoadBalancer
IP: 10.104.193.121
IP: 172.31.1.225
LoadBalancer Ingress: 172.31.1.225
Port: <unset> 80/TCP
TargetPort: 8080/TCP
NodePort: <unset> 31377/TCP
Endpoints: 10.129.3.151:8080,10.129.4.33:8080
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AnnouncingNonLocal 52s (x2 over 22h) purelb-lbnodeagent Announcing 172.31.1.225 from node purelb2-4 interface kube-lb0
Normal AnnouncingNonLocal 37s (x5 over 43s) purelb-lbnodeagent Announcing 172.31.1.225 from node purelb2-5 interface kube-lb0
Normal AnnouncingNonLocal 37s (x4 over 41s) purelb-lbnodeagent Announcing 172.31.1.225 from node purelb2-3 interface kube-lb0
Normal AnnouncingNonLocal 37s purelb-lbnodeagent Announcing 172.31.1.225 from node purelb2-2 interface kube-lb0
Normal AnnouncingNonLocal 37s purelb-lbnodeagent Announcing 172.31.1.225 from node purelb2-4 interface kube-lb0
Normal AnnouncingNonLocal 36s purelb-lbnodeagent Announcing 172.31.1.225 from node purelb2-1 interface kube-lb0
The example above shows that PureLB allocated the address from the requested service group virtualsg, this information was added by the allocator. The event messages are added by lbnodeagent and show the nodes where the address was added. As the address was added to multiple nodes, it is a virtual address as local addresses can only be added to a single node.
$ kubectl describe service echoserver21
Name: echoserver21
Namespace: toby
Labels: app=echoserver1
Annotations: purelb.io/service-group: default
purelb.io/allocated-by: PureLB
purelb.io/allocated-from: default
purelb.io/announcing-interface: enp1s0
purelb.io/announcing-node: purelb2-1
Selector: app=echoserver1
Type: LoadBalancer
IP: 10.110.8.48
LoadBalancer Ingress: 172.30.250.53
Port: <unset> 80/TCP
TargetPort: 8080/TCP
NodePort: <unset> 32380/TCP
Endpoints: <none>
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal AnnouncingLocal 19m purelb-lbnodeagent Node purelb2-4 announcing 172.30.250.53 on interface enp1s0
This example shows that the default pool is part of the local address. PureLB annotates these services with the node and interface where the address was announced as well us updating the event.
This useful command command will show all nodes that are advertizing addresses locally. The annotations make it easier find information in larger k8s clusters.
$ kubectl get services -A -o json | jq ' .items[].metadata.annotations' | grep announcing-node
"purelb.io/announcing-node": "purelb2-1"
"purelb.io/announcing-node": "purelb2-1"