PureLB

Monitoring with kubectl

PureLB attempts to provide all of the information necessary to monitor and troubleshoot the health of Load Balancer services via kubectl without resorting to inspecting PureLB’s POD logging. PureLB’s operational status and events are updated in the Load Balancer Services. PureLB annotates services where it was the source of the allocated address. If allocated-by is not present, PureLB did not allocate the address.

The simplest way to check the status of services if using the kubectl describe command.

$ kubectl describe service kuard-svc-dual-remote 
Name:                     kuard-svc-dual-remote
Namespace:                adamd
Labels:                   <none>
Annotations:              purelb.io/allocated-by: PureLB
                          purelb.io/allocated-from: remotedual
                          purelb.io/service-group: remotedual
Selector:                 app=kuard
Type:                     LoadBalancer
IP Family Policy:         RequireDualStack
IP Families:              IPv4,IPv6
IP:                       10.152.183.170
IPs:                      10.152.183.170,fd98::4078
LoadBalancer Ingress:     172.32.100.225, fc00:370:155:0:8000::
Port:                     <unset>  80/TCP
TargetPort:               8080/TCP
Endpoints:                10.1.217.204:8080,10.1.217.205:8080,10.1.238.137:8080
Session Affinity:         None
External Traffic Policy:  Local
HealthCheck NodePort:     30439
Events:
  Type    Reason              Age                From                Message
  ----    ------              ----               ----                -------
  Normal  AddressAssigned     11s                purelb-allocator    Assigned {Ingress:[{IP:172.32.100.225 Hostname: Ports:[]} {IP:fc00:370:155:0:8000:: Hostname: Ports:[]}]} from pool remotedual
  Normal  AnnouncingNonLocal  10s (x2 over 10s)  purelb-lbnodeagent  Announcing 172.32.100.225 from node mk8s3 interface kube-lb0
  Normal  AnnouncingNonLocal  10s (x2 over 10s)  purelb-lbnodeagent  Announcing 172.32.100.225 from node mk8s1 interface kube-lb0
  Normal  AnnouncingNonLocal  10s (x2 over 10s)  purelb-lbnodeagent  Announcing fc00:370:155:0:8000:: from node mk8s1 interface kube-lb0
  Normal  AnnouncingNonLocal  10s (x2 over 10s)  purelb-lbnodeagent  Announcing fc00:370:155:0:8000:: from node mk8s3 interface kube-lb0

The example above shows that PureLB allocated the address from the requested service group virtualsg, this information was added by the allocator. The event messages are added by lbnodeagent and show the nodes where the address was added. As the address was added to multiple nodes, it is a virtual address as local addresses can only be added to a single node.

k describe service kuard-service 
Name:                     kuard-service
Namespace:                adamd
Labels:                   app=kuard
Annotations:              purelb.io/allocated-by: PureLB
                          purelb.io/allocated-from: default
                          purelb.io/announcing-IPv4: mk8s2,enp1s0
Selector:                 app=kuard
Type:                     LoadBalancer
IP Family Policy:         SingleStack
IP Families:              IPv4
IP:                       10.152.183.155
IPs:                      10.152.183.155
LoadBalancer Ingress:     192.168.10.240
Port:                     <unset>  80/TCP
TargetPort:               8080/TCP
NodePort:                 <unset>  30310/TCP
Endpoints:                10.1.217.204:8080,10.1.217.205:8080,10.1.238.137:8080
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason           Age                From                Message
  ----    ------           ----               ----                -------
  Normal  AddressAssigned  27m                purelb-allocator    Assigned {Ingress:[{IP:192.168.10.240 Hostname: Ports:[]}]} from pool default
  Normal  AnnouncingLocal  27m (x4 over 27m)  purelb-lbnodeagent  Node mk8s2 announcing 192.168.10.240 on interface enp1s0

This example shows that the default pool is part of the local address. PureLB annotates these services with the node and interface where the address was announced as well us updating the event.

This useful command command will show all nodes that are advertizing addresses locally. The annotations make it easier find information in larger k8s clusters.

$ kubectl get services -A -o json | jq '.items[].metadata.annotations' | grep announcing
  "purelb.io/announcing-IPv4": "mk8s2,enp1s0",
  "purelb.io/announcing-IPv6": "mk8s2,enp1s0",
  "purelb.io/announcing-IPv4": "mk8s2,enp1s0"