Metrics

This section describes the MQTT messages and topics the Management Service handles to perform the "run-stats" and "stop-stats" actions. When using AnythingDB in Studio all of this is taken care of within the product.

Metrics / Statistics Data

The Management Service publishes metrics for the nodes and deployments of the cluster at a regular interval (default: 30 seconds). This data is obtained from the Kubernetes Metrics Server which is installed by default in a K3s installation.

Start the Metrics / Update the Interval

The metrics publishing can be started, or the interval can be updated, by sending a message to the following topic:
status/<space>/collections/cluster/things/<thing-id>/actions/run-stats

The message format for publishing is as follows:

{
  "run-stats": {
    "input": {
      "interval": {in-seconds}
    },
    "status": "pending",
    "timeRequested": "<datetime>",
    "href": "<space>/collections/cluster/things/<thing-id>/actions/run-stats/<action-id>"
  }
}

For example:

{
  "run-stats": {
    "input": {
      "interval": 15
    },
    "status": "pending",
    "timeRequested": "2020-06-18 10:32:46+0000",
    "href": "martin/things/ag150/actions/run-stats/1234"
  }
}

Stop the Metrics

The metrics publishing can be stopped by sending a message to the following topic:
status/<space>/collections/cluster/things/<thing-id>/actions/stop-stats

The message format for publishing is as follows:

{
  "stop-stats": {
    "status": "pending",
    "timeRequested": "<datetime>",
    "href": "<space>/collections/cluster/things/<thing-id>/actions/stop-stats/<action-id>"
  }
}

For example:

{
  "stop-stats": {
    "status": "pending",
    "timeRequested": "2020-06-18 10:36:46+0000",
    "href": "martin/things/ag150/actions/stop-stats/1234"
  }
}

Example of Node-stats

{
  "nodes-stats": {
    "data": [
      {
        "capacities": {
          "cpu": 4000000000,
          "memory": 1937969152,
          "pods": 110,
          "storage": 31084183552
        },
        "conditions": [
          {
            "lastHeartbeatTime": 1607879049,
            "lastTransitionTime": 1607879049,
            "message": "Flannel is running on this node",
            "reason": "FlannelIsUp",
            "status": "False",
            "type": "NetworkUnavailable"
          },
          {
            "lastHeartbeatTime": 1608055458,
            "lastTransitionTime": 1607592799,
            "message": "kubelet has sufficient memory available",
            "reason": "KubeletHasSufficientMemory",
            "status": "False",
            "type": "MemoryPressure"
          },
          {
            "lastHeartbeatTime": 1608055458,
            "lastTransitionTime": 1607592799,
            "message": "kubelet has no disk pressure",
            "reason": "KubeletHasNoDiskPressure",
            "status": "False",
            "type": "DiskPressure"
          },
          {
            "lastHeartbeatTime": 1608055458,
            "lastTransitionTime": 1607592799,
            "message": "kubelet has sufficient PID available",
            "reason": "KubeletHasSufficientPID",
            "status": "False",
            "type": "PIDPressure"
          },
          {
            "lastHeartbeatTime": 1608055458,
            "lastTransitionTime": 1607879059,
            "message": "kubelet is posting ready status. AppArmor enabled",
            "reason": "KubeletReady",
            "status": "True",
            "type": "Ready"
          }
        ],
        "name": "ubuntu",
        "stats": {
          "cpu": 867753808,
          "cpuPercentage": 21.693845200000002,
          "memory": 1239392256,
          "memoryPercentage": 63.95314676298831,
          "timestamp": 1608143932,
          "window": 30
        }
      }
    ]
  }
}

Example of Deployments-stats

{
  "deployments-stats": {
    "data": {
      "nodes": [
        {
          "deployments": [
            {
              "name": "ase-core-message",
              "pods": [
                {
                  "containers": [
                    {
                      "name": "ase-core-message",
                      "stats": {
                        "cpu": 105314661,
                        "cpuPercentage": 2.6328665250000003,
                        "memory": 140861440,
                        "memoryPercentage": 7.268507852905184,
                        "timestamp": 1608143929,
                        "window": 30
                      }
                    }
                  ],
                  "name": "ase-core-message-866947bfb6-59g4f"
                }
              ],
              "status": {
                "availableReplicas": 1,
                "conditions": [
                  {
                    "lastTransitionTime": 1605289453,
                    "lastUpdateTime": 1607448834,
                    "message": "ReplicaSet \"ase-core-message-866947bfb6\" has successfully progressed.",
                    "reason": "NewReplicaSetAvailable",
                    "status": "True",
                    "type": "Progressing"
                  },
                  {
                    "lastTransitionTime": 1607879075,
                    "lastUpdateTime": 1607879075,
                    "message": "Deployment has minimum availability.",
                    "reason": "MinimumReplicasAvailable",
                    "status": "True",
                    "type": "Available"
                  }
                ],
                "observedGeneration": 4,
                "readyReplicas": 1,
                "replicas": 1,
                "updatedReplicas": 1
              }
            },
            {
              "name": "ase-core-management",
              "pods": [
                {
                  "containers": [
                    {
                      "name": "ase-core-management",
                      "stats": {
                        "cpu": 804966,
                        "cpuPercentage": 0.02012415,
                        "memory": 15163392,
                        "memoryPercentage": 0.7824372221999125,
                        "timestamp": 1608143927,
                        "window": 30
                      }
                    }
                  ],
                  "name": "ase-core-management-764f8d8b66-pv2gf"
                }
              ],
              "status": {
                "availableReplicas": 1,
                "conditions": [
                  {
                    "lastTransitionTime": 1605795610,
                    "lastUpdateTime": 1607678997,
                    "message": "ReplicaSet \"ase-core-management-764f8d8b66\" has successfully progressed.",
                    "reason": "NewReplicaSetAvailable",
                    "status": "True",
                    "type": "Progressing"
                  },
                  {
                    "lastTransitionTime": 1607879075,
                    "lastUpdateTime": 1607879075,
                    "message": "Deployment has minimum availability.",
                    "reason": "MinimumReplicasAvailable",
                    "status": "True",
                    "type": "Available"
                  }
                ],
                "observedGeneration": 23,
                "readyReplicas": 1,
                "replicas": 1,
                "updatedReplicas": 1
              }
            },
            {
              "name": "ase-core-cache",
              "pods": [
                {
                  "containers": [
                    {
                      "name": "ase-core-cache",
                      "stats": {
                        "cpu": 3803112,
                        "cpuPercentage": 0.0950778,
                        "memory": 2957312,
                        "memoryPercentage": 0.15259850740905911,
                        "timestamp": 1608143924,
                        "window": 30
                      }
                    }
                  ],
                  "name": "ase-core-cache-676b54585d-jrncd"
                }
              ],
              "status": {
                "availableReplicas": 1,
                "conditions": [
                  {
                    "lastTransitionTime": 1605289503,
                    "lastUpdateTime": 1607452737,
                    "message": "ReplicaSet \"ase-core-cache-676b54585d\" has successfully progressed.",
                    "reason": "NewReplicaSetAvailable",
                    "status": "True",
                    "type": "Progressing"
                  },
                  {
                    "lastTransitionTime": 1607879075,
                    "lastUpdateTime": 1607879075,
                    "message": "Deployment has minimum availability.",
                    "reason": "MinimumReplicasAvailable",
                    "status": "True",
                    "type": "Available"
                  }
                ],
                "observedGeneration": 2,
                "readyReplicas": 1,
                "replicas": 1,
                "updatedReplicas": 1
              }
            },
            {
              "name": "ase-virtual-meter",
              "pods": [
                {
                  "containers": [
                    {
                      "name": "ase-virtual-meter",
                      "stats": {
                        "cpu": 1428236,
                        "cpuPercentage": 0.0357059,
                        "memory": 7585792,
                        "memoryPercentage": 0.391429966373376,
                        "timestamp": 1608143930,
                        "window": 30
                      }
                    }
                  ],
                  "name": "ase-virtual-meter-655b56d958-vw7cn"
                }
              ],
              "status": {
                "availableReplicas": 1,
                "conditions": [
                  {
                    "lastTransitionTime": 1607879074,
                    "lastUpdateTime": 1607879074,
                    "message": "Deployment has minimum availability.",
                    "reason": "MinimumReplicasAvailable",
                    "status": "True",
                    "type": "Available"
                  },
                  {
                    "lastTransitionTime": 1606748304,
                    "lastUpdateTime": 1607972823,
                    "message": "ReplicaSet \"ase-virtual-meter-655b56d958\" has successfully progressed.",
                    "reason": "NewReplicaSetAvailable",
                    "status": "True",
                    "type": "Progressing"
                  }
                ],
                "observedGeneration": 9,
                "readyReplicas": 1,
                "replicas": 1,
                "updatedReplicas": 1
              }
            },
            {
              "name": "ase-core-cloud",
              "pods": [
                {
                  "containers": [
                    {
                      "name": "ase-core-cloud",
                      "stats": {
                        "cpu": 861411,
                        "cpuPercentage": 0.021535275,
                        "memory": 6254592,
                        "memoryPercentage": 0.3227395025119574,
                        "timestamp": 1608143924,
                        "window": 30
                      }
                    }
                  ],
                  "name": "ase-core-cloud-57486fd5f5-5drcv"
                }
              ],
              "status": {
                "availableReplicas": 1,
                "conditions": [
                  {
                    "lastTransitionTime": 1607695991,
                    "lastUpdateTime": 1607696007,
                    "message": "ReplicaSet \"ase-core-cloud-57486fd5f5\" has successfully progressed.",
                    "reason": "NewReplicaSetAvailable",
                    "status": "True",
                    "type": "Progressing"
                  },
                  {
                    "lastTransitionTime": 1607879068,
                    "lastUpdateTime": 1607879068,
                    "message": "Deployment has minimum availability.",
                    "reason": "MinimumReplicasAvailable",
                    "status": "True",
                    "type": "Available"
                  }
                ],
                "observedGeneration": 1,
                "readyReplicas": 1,
                "replicas": 1,
                "updatedReplicas": 1
              }
            },
            {
              "name": "ase-core-health",
              "pods": [
                {
                  "containers": [
                    {
                      "name": "ase-core-health",
                      "stats": {
                        "cpu": 2052389,
                        "cpuPercentage": 0.051309725,
                        "memory": 7409664,
                        "memoryPercentage": 0.38234168961632675,
                        "timestamp": 1608143930,
                        "window": 30
                      }
                    }
                  ],
                  "name": "ase-core-health-7c4f9d966d-vw9mt"
                }
              ],
              "status": {
                "availableReplicas": 1,
                "conditions": [
                  {
                    "lastTransitionTime": 1605289626,
                    "lastUpdateTime": 1607696276,
                    "message": "ReplicaSet \"ase-core-health-7c4f9d966d\" has successfully progressed.",
                    "reason": "NewReplicaSetAvailable",
                    "status": "True",
                    "type": "Progressing"
                  },
                  {
                    "lastTransitionTime": 1607879074,
                    "lastUpdateTime": 1607879074,
                    "message": "Deployment has minimum availability.",
                    "reason": "MinimumReplicasAvailable",
                    "status": "True",
                    "type": "Available"
                  }
                ],
                "observedGeneration": 5,
                "readyReplicas": 1,
                "replicas": 1,
                "updatedReplicas": 1
              }
            }
          ],
          "name": "ubuntu"
        }
      ]
    }
  }
}