Management Service MQTT Messaging

Send kubectl

This section describes the MQTT messages and topics the Management Service handles to perform the "send-kubectl" action. When using AnythingDB in Studio all of this is taken care of within the product.

Sending a Message to the Management Service

The MQTT topic the Management Service subscribes to, for running kubectl commands is:
spaces/<space>/things/<thing-id>/actions/send-kubectl

The message format for publishing is as follows:

{
  "send-kubectl": {
    "input": {
      "correlationId": "{create-a-unique-id-for-each-message}",
      "command": "kubectl",
      "arguments": [
        "{each}",
        "{new}",
        "{argument}",
        "{needs}",
        "{its}",
        "{own}",
        "{array}",
        "{entry}"
      ]
    },
    "status": "pending",
    "timeRequested": "<datetime>",
    "href": "/spaces/<space>/things/<thing-id>/actions/send-kubectl/<action-id>"
  }
}

For example:

{
  "send-kubectl": {
    "input": {
      "correlationId": "666",
      "command": "kubectl",
      "arguments": [
        "get",
        "nodes",
        "-o",
        "wide"
      ]
    },
    "status": "pending",
    "timeRequested": "2020-05-26 15:37:46+0000",
    "href": "/spaces/my-space/things/my-thing-id/actions/send-kubectl/1234"
  }
}

It is important that each argument is put in a new array entry. If there is a space in the complete command then that means a new argument.

Status Update After Receiving Action Request

MQTT topic the message is sent to:
spaces/<space>/things/<thing-id>/actions

Message:

{
  "send-kubectl": {
    "href": "/spaces/<space>/things/<thing-id>/actions/send-kubectl/<action-id>",
    "status": "received"
  }
}

Event Publish After Running Action

MQTT topic the message is sent to:
spaces/<space>/things/<thing-id>/events

Message:

{
    "kubectl-logs": {
        "data": {
            "correlationId": "{the-unique-id-from-the-request}",
            "statusCode": "{an-http-status-code}",
            "statusMessage": "{optional descriptive message}",
            "response": "{the output in plain text format}"
        }
    }
}

Example:

{
    "kubectl-logs": {
        "data": {
            "correlationId": "666",
            "statusCode": 200,
            "statusMessage": "Command executed successfully",
            "response": "NAME           STATUS   ROLES    AGE   VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME\nmartin-ag150   Ready    master   50d   v1.18.2+k3s1   10.10.10.100   \u003cnone\u003e        Ubuntu 20.04 LTS   5.4.0-40-generic   containerd://1.3.3-k3s2\n"
        }
    }
}

You need to use the correlationId to match the response to the request.

Status Update After Completing Action

MQTT topic the message is sent to:
spaces/<space>/things/<thing-id>/actions

Message:

{
  "send-kubectl": {
    "href": "/spaces/<space>/things/<thing-id>/actions/send-kubectl/<action-id>",
    "status": "completed"
  }
}

Send kubernetes API

This section describes the MQTT messages and topics the Management Service handles to perform the send-kubernetes-api action. When using AnythingDB in Studio all of this is taken care of within the product.

Sending a Message to the Management Service

The MQTT topic the Management Service subscribes to, for running kubernetes API requests is:
spaces/<space>/things/<thing-id>/actions/send-kubernetes-api

The message format for publishing is as follows:

{
  "send-kubernetes-api": {
    "input": {
      "correlationId": "{create-a-unique-id-for-each-message}",
      "requestMethod": "{GET|POST|PUT|PATCH|DELETE}",
      "href": "{see API docs}",
      "requestBody": "{body if required}"
    },
    "status": "pending",
    "timeRequested": "<datetime>",
    "href": "/spaces/<space>/things/<thing-id>/actions/send-kubernetes-api/<action-id>"
  }
}

The requestBody is a string. For a JSON requestBody this means it has to be stringified first. This results in "double JSON-encoding".

For example:

{
    "send-kubernetes-api":{
        "input":{
            "correlationId": "666",
            "requestMethod": "GET",
            "href": "/api/v1/nodes",
            "requestBody": ""
        },
        "status": "pending",
        "timeRequested": "2020-05-26 15:37:46+0000",
        "href": "/spaces/may-space/things/my-thing-id/actions/send-kubernetes-api/1234"
    }
}

Status Update After Receiving Action Request

MQTT topic the message is sent to:
spaces/<space>/things/<thing-id>/actions

Message:

{
  "send-kubernetes-api": {
    "href": "/spaces/<space>/things/<thing-id>/actions/send-kubernetes-api/<action-id>",
    "status": "received"
  }
}

Event Publish After Running Action

MQTT topic the message is sent to:
spaces/<space>/things/<thing-id>/events

Message:

{
    "send-kubernetes-api": {
        "data": {
            "correlationId": "{the-unique-id-from-the-request}",
            "statusCode": "{an-http-status-code}",
            "statusMessage": "{optional descriptive message}",
            "response": "{the output of the API request}"
        }
    }
}

Example:

{
    "send-kubernetes-api": {
        "data": {
            "correlationId": "666",
            "statusCode": 200,
            "statusMessage": "Command executed successfully",
            "response": ""
        }
    }
}

You need to use the correlationId to match the response to the request.

Status Update After Completing Action

MQTT topic the message is sent to:
spaces/<space>/things/<thing-id>/actions

Message:

{
  "send-kubernetes-api": {
    "href": "/spaces/<space>/things/<thing-id>/actions/send-kubernetes-api/<action-id>",
    "status": "completed"
  }
}

Metrics

This section describes the MQTT messages and topics the Management Service handles to perform the "run-stats" and "stop-stats" actions. When using AnythingDB in Studio all of this is taken care of within the product.

Metrics / Statistics Data

The Management Service publishes metrics for the nodes and deployments of the cluster at a regular interval (default: 30 seconds). This data is obtained from the Kubernetes Metrics Server which is installed by default in a K3s installation.

Start the Metrics / Update the Interval

The metrics publishing can be started, or the interval can be updated, by sending a message to the following topic:
status/<space>/things/<thing-id>/actions/run-stats

The message format for publishing is as follows:

{
  "run-stats": {
    "input": {
      "interval": {in-seconds}
    },
    "status": "pending",
    "timeRequested": "<datetime>",
    "href": "<space>/things/<thing-id>/actions/run-stats/<action-id>"
  }
}

For example:

{
  "run-stats": {
    "input": {
      "interval": 15
    },
    "status": "pending",
    "timeRequested": "2020-06-18 10:32:46+0000",
    "href": "martin/things/ag150/actions/run-stats/1234"
  }
}

Stop the Metrics

The metrics publishing can be stopped by sending a message to the following topic:
status/<space>/things/<thing-id>/actions/stop-stats

The message format for publishing is as follows:

{
  "stop-stats": {
    "status": "pending",
    "timeRequested": "<datetime>",
    "href": "<space>/things/<thing-id>/actions/stop-stats/<action-id>"
  }
}

For example:

{
  "stop-stats": {
    "status": "pending",
    "timeRequested": "2020-06-18 10:36:46+0000",
    "href": "martin/things/ag150/actions/stop-stats/1234"
  }
}

Example of Metrics

{
  "nodes": [
    {
      "capacities": {
        "cpu": 4000000000,
        "memory": 16608059392,
        "pods": 110,
        "storage": 105089261568
      },
      "conditions": [
        {
          "lastHeartbeatTime": 1724660414,
          "lastTransitionTime": 1715072919,
          "message": "kubelet has sufficient memory available",
          "reason": "KubeletHasSufficientMemory",
          "status": "False",
          "type": "MemoryPressure"
        },
        {
          "lastHeartbeatTime": 1724660414,
          "lastTransitionTime": 1715072919,
          "message": "kubelet has no disk pressure",
          "reason": "KubeletHasNoDiskPressure",
          "status": "False",
          "type": "DiskPressure"
        },
        {
          "lastHeartbeatTime": 1724660414,
          "lastTransitionTime": 1715072919,
          "message": "kubelet has sufficient PID available",
          "reason": "KubeletHasSufficientPID",
          "status": "False",
          "type": "PIDPressure"
        },
        {
          "lastHeartbeatTime": 1724660414,
          "lastTransitionTime": 1715072919,
          "message": "kubelet is posting ready status",
          "reason": "KubeletReady",
          "status": "True",
          "type": "Ready"
        }
      ],
      "daemonsets": [
        {
          "name": "ase-support-cloud-storage",
          "namespace": "ecp",
          "pods": [
            {
              "containers": [
                {
                  "name": "ase-support-cloud-storage",
                  "stats": {
                    "cpu": 265892,
                    "cpuPercentage": 0.0066473,
                    "memory": 9846784,
                    "memoryPercentage": 0.05928919067295205,
                    "timestamp": 1724660596,
                    "window": 11
                  }
                }
              ],
              "name": "ase-support-cloud-storage-pcr9t",
              "namespace": "ecp"
            }
          ],
          "status": {
            "currentNumberScheduled": 1,
            "desiredNumberScheduled": 1,
            "numberAvailable": 1,
            "numberReady": 1,
            "observedGeneration": 1,
            "updatedNumberScheduled": 1
          }
        }
      ],
      "deployments": [
        {
          "name": "ase-core-export-mqtt",
          "namespace": "ecp",
          "pods": [
            {
              "containers": [
                {
                  "name": "ase-core-export-mqtt",
                  "stats": {
                    "cpu": 2021938,
                    "cpuPercentage": 0.05054845,
                    "memory": 33472512,
                    "memoryPercentage": 0.20154378792818806,
                    "timestamp": 1724660600,
                    "window": 12
                  }
                }
              ],
              "name": "ase-core-export-mqtt-844c67d4f7-878kz",
              "namespace": "ecp"
            }
          ],
          "status": {
            "availableReplicas": 1,
            "conditions": [
              {
                "lastTransitionTime": 1715073169,
                "lastUpdateTime": 1720074799,
                "message": "ReplicaSet \"ase-core-export-mqtt-844c67d4f7\" has successfully progressed.",
                "reason": "NewReplicaSetAvailable",
                "status": "True",
                "type": "Progressing"
              },
              {
                "lastTransitionTime": 1723539614,
                "lastUpdateTime": 1723539614,
                "message": "Deployment has minimum availability.",
                "reason": "MinimumReplicasAvailable",
                "status": "True",
                "type": "Available"
              }
            ],
            "observedGeneration": 3,
            "readyReplicas": 1,
            "replicas": 1,
            "updatedReplicas": 1
          }
        },
        {
          "name": "ase-virtual-meter",
          "namespace": "ecp",
          "pods": [
            {
              "containers": [
                {
                  "name": "ase-virtual-meter",
                  "stats": {
                    "cpu": 605035,
                    "cpuPercentage": 0.015125874999999999,
                    "memory": 7507968,
                    "memoryPercentage": 0.04520677475188066,
                    "timestamp": 1724660599,
                    "window": 17
                  }
                }
              ],
              "name": "ase-virtual-meter-5f487f7787-fltpl",
              "namespace": "ecp"
            }
          ],
          "status": {
            "availableReplicas": 1,
            "conditions": [
              {
                "lastTransitionTime": 1715073162,
                "lastUpdateTime": 1715073190,
                "message": "ReplicaSet \"ase-virtual-meter-5f487f7787\" has successfully progressed.",
                "reason": "NewReplicaSetAvailable",
                "status": "True",
                "type": "Progressing"
              },
              {
                "lastTransitionTime": 1722360862,
                "lastUpdateTime": 1722360862,
                "message": "Deployment has minimum availability.",
                "reason": "MinimumReplicasAvailable",
                "status": "True",
                "type": "Available"
              }
            ],
            "observedGeneration": 1,
            "readyReplicas": 1,
            "replicas": 1,
            "updatedReplicas": 1
          }
        },
        {
          "name": "ase-cluster-upgrade",
          "namespace": "ecp",
          "pods": [
            {
              "containers": [
                {
                  "name": "ase-cluster-upgrade",
                  "stats": {
                    "cpu": 0,
                    "memory": 12972032,
                    "memoryPercentage": 0.07810684977588982,
                    "timestamp": 1724660597,
                    "window": 29
                  }
                }
              ],
              "name": "ase-cluster-upgrade-765b94587c-w2cfc",
              "namespace": "ecp"
            }
          ],
          "status": {
            "availableReplicas": 1,
            "conditions": [
              {
                "lastTransitionTime": 1715073165,
                "lastUpdateTime": 1715073173,
                "message": "ReplicaSet \"ase-cluster-upgrade-765b94587c\" has successfully progressed.",
                "reason": "NewReplicaSetAvailable",
                "status": "True",
                "type": "Progressing"
              },
              {
                "lastTransitionTime": 1722360558,
                "lastUpdateTime": 1722360558,
                "message": "Deployment has minimum availability.",
                "reason": "MinimumReplicasAvailable",
                "status": "True",
                "type": "Available"
              }
            ],
            "observedGeneration": 1,
            "readyReplicas": 1,
            "replicas": 1,
            "updatedReplicas": 1
          }
        },
        {
          "name": "ase-core-updater",
          "namespace": "ecp",
          "pods": [
            {
              "containers": [
                {
                  "name": "ase-core-updater",
                  "stats": {
                    "cpu": 115982,
                    "cpuPercentage": 0.00289955,
                    "memory": 9703424,
                    "memoryPercentage": 0.058425995301257645,
                    "timestamp": 1724660600,
                    "window": 14
                  }
                }
              ],
              "name": "ase-core-updater-55b47fbb74-5txhn",
              "namespace": "ecp"
            }
          ],
          "status": {
            "availableReplicas": 1,
            "conditions": [
              {
                "lastTransitionTime": 1723507248,
                "lastUpdateTime": 1723507248,
                "message": "Deployment has minimum availability.",
                "reason": "MinimumReplicasAvailable",
                "status": "True",
                "type": "Available"
              },
              {
                "lastTransitionTime": 1715073143,
                "lastUpdateTime": 1724284830,
                "message": "ReplicaSet \"ase-core-updater-55b47fbb74\" has successfully progressed.",
                "reason": "NewReplicaSetAvailable",
                "status": "True",
                "type": "Progressing"
              }
            ],
            "observedGeneration": 8,
            "readyReplicas": 1,
            "replicas": 1,
            "updatedReplicas": 1
          }
        },
        {
          "name": "ase-core-cloud",
          "namespace": "ecp",
          "pods": [
            {
              "containers": [
                {
                  "name": "ase-core-cloud",
                  "stats": {
                    "cpu": 2493175,
                    "cpuPercentage": 0.06232937499999999,
                    "memory": 21626880,
                    "memoryPercentage": 0.1302191875013256,
                    "timestamp": 1724660598,
                    "window": 11
                  }
                }
              ],
              "name": "ase-core-cloud-6bf4dcbf7b-nbdpr",
              "namespace": "ecp"
            }
          ],
          "status": {
            "availableReplicas": 1,
            "conditions": [
              {
                "lastTransitionTime": 1715073169,
                "lastUpdateTime": 1720074798,
                "message": "ReplicaSet \"ase-core-cloud-6bf4dcbf7b\" has successfully progressed.",
                "reason": "NewReplicaSetAvailable",
                "status": "True",
                "type": "Progressing"
              },
              {
                "lastTransitionTime": 1723746270,
                "lastUpdateTime": 1723746270,
                "message": "Deployment has minimum availability.",
                "reason": "MinimumReplicasAvailable",
                "status": "True",
                "type": "Available"
              }
            ],
            "observedGeneration": 3,
            "readyReplicas": 1,
            "replicas": 1,
            "updatedReplicas": 1
          }
        },
        {
          "name": "ase-core-meta",
          "namespace": "ecp",
          "pods": [
            {
              "containers": [
                {
                  "name": "ase-core-meta",
                  "stats": {
                    "cpu": 3273851,
                    "cpuPercentage": 0.081846275,
                    "memory": 18644992,
                    "memoryPercentage": 0.11226472377008224,
                    "timestamp": 1724660609,
                    "window": 14
                  }
                }
              ],
              "name": "ase-core-meta-648f5bb74c-zrjbs",
              "namespace": "ecp"
            }
          ],
          "status": {
            "availableReplicas": 1,
            "conditions": [
              {
                "lastTransitionTime": 1715073169,
                "lastUpdateTime": 1720074800,
                "message": "ReplicaSet \"ase-core-meta-648f5bb74c\" has successfully progressed.",
                "reason": "NewReplicaSetAvailable",
                "status": "True",
                "type": "Progressing"
              },
              {
                "lastTransitionTime": 1722360937,
                "lastUpdateTime": 1722360937,
                "message": "Deployment has minimum availability.",
                "reason": "MinimumReplicasAvailable",
                "status": "True",
                "type": "Available"
              }
            ],
            "observedGeneration": 3,
            "readyReplicas": 1,
            "replicas": 1,
            "updatedReplicas": 1
          }
        },
        {
          "name": "ase-core-cache",
          "namespace": "ecp",
          "pods": [
            {
              "containers": [
                {
                  "name": "ase-core-cache",
                  "stats": {
                    "cpu": 2071880,
                    "cpuPercentage": 0.051796999999999996,
                    "memory": 4472832,
                    "memoryPercentage": 0.026931695596865073,
                    "timestamp": 1724660602,
                    "window": 13
                  }
                }
              ],
              "name": "ase-core-cache-76dbfbb696-hdflc",
              "namespace": "ecp"
            }
          ],
          "status": {
            "availableReplicas": 1,
            "conditions": [
              {
                "lastTransitionTime": 1715073169,
                "lastUpdateTime": 1715073180,
                "message": "ReplicaSet \"ase-core-cache-76dbfbb696\" has successfully progressed.",
                "reason": "NewReplicaSetAvailable",
                "status": "True",
                "type": "Progressing"
              },
              {
                "lastTransitionTime": 1722360587,
                "lastUpdateTime": 1722360587,
                "message": "Deployment has minimum availability.",
                "reason": "MinimumReplicasAvailable",
                "status": "True",
                "type": "Available"
              }
            ],
            "observedGeneration": 1,
            "readyReplicas": 1,
            "replicas": 1,
            "updatedReplicas": 1
          }
        },
        {
          "name": "ase-core-health",
          "namespace": "ecp",
          "pods": [
            {
              "containers": [
                {
                  "name": "ase-core-health",
                  "stats": {
                    "cpu": 886369,
                    "cpuPercentage": 0.022159225,
                    "memory": 17387520,
                    "memoryPercentage": 0.10469326722407712,
                    "timestamp": 1724660604,
                    "window": 19
                  }
                }
              ],
              "name": "ase-core-health-797fdf9d77-gmgb9",
              "namespace": "ecp"
            }
          ],
          "status": {
            "availableReplicas": 1,
            "conditions": [
              {
                "lastTransitionTime": 1715073169,
                "lastUpdateTime": 1720074789,
                "message": "ReplicaSet \"ase-core-health-797fdf9d77\" has successfully progressed.",
                "reason": "NewReplicaSetAvailable",
                "status": "True",
                "type": "Progressing"
              },
              {
                "lastTransitionTime": 1722360937,
                "lastUpdateTime": 1722360937,
                "message": "Deployment has minimum availability.",
                "reason": "MinimumReplicasAvailable",
                "status": "True",
                "type": "Available"
              }
            ],
            "observedGeneration": 3,
            "readyReplicas": 1,
            "replicas": 1,
            "updatedReplicas": 1
          }
        },
        {
          "name": "ase-core-management",
          "namespace": "ecp",
          "pods": [
            {
              "containers": [
                {
                  "name": "ase-core-management",
                  "stats": {
                    "cpu": 172030,
                    "cpuPercentage": 0.00430075,
                    "memory": 14680064,
                    "memoryPercentage": 0.08839120606150588,
                    "timestamp": 1724660601,
                    "window": 11
                  }
                }
              ],
              "name": "ase-core-management-587f89b769-vk8h4",
              "namespace": "ecp"
            }
          ],
          "status": {
            "availableReplicas": 1,
            "conditions": [
              {
                "lastTransitionTime": 1722360557,
                "lastUpdateTime": 1722360557,
                "message": "Deployment has minimum availability.",
                "reason": "MinimumReplicasAvailable",
                "status": "True",
                "type": "Available"
              },
              {
                "lastTransitionTime": 1715073144,
                "lastUpdateTime": 1724284825,
                "message": "ReplicaSet \"ase-core-management-587f89b769\" has successfully progressed.",
                "reason": "NewReplicaSetAvailable",
                "status": "True",
                "type": "Progressing"
              }
            ],
            "observedGeneration": 4,
            "readyReplicas": 1,
            "replicas": 1,
            "updatedReplicas": 1
          }
        },
        {
          "name": "ase-core-message",
          "namespace": "ecp",
          "pods": [
            {
              "containers": [
                {
                  "name": "ase-core-message",
                  "stats": {
                    "cpu": 13169991,
                    "cpuPercentage": 0.329249775,
                    "memory": 141828096,
                    "memoryPercentage": 0.853971512579716,
                    "timestamp": 1724660602,
                    "window": 13
                  }
                }
              ],
              "name": "ase-core-message-849668cf76-55l48",
              "namespace": "ecp"
            }
          ],
          "status": {
            "availableReplicas": 1,
            "conditions": [
              {
                "lastTransitionTime": 1715073169,
                "lastUpdateTime": 1715073223,
                "message": "ReplicaSet \"ase-core-message-849668cf76\" has successfully progressed.",
                "reason": "NewReplicaSetAvailable",
                "status": "True",
                "type": "Progressing"
              },
              {
                "lastTransitionTime": 1722360926,
                "lastUpdateTime": 1722360926,
                "message": "Deployment has minimum availability.",
                "reason": "MinimumReplicasAvailable",
                "status": "True",
                "type": "Available"
              }
            ],
            "observedGeneration": 1,
            "readyReplicas": 1,
            "replicas": 1,
            "updatedReplicas": 1
          }
        },
        {
          "name": "ase-support-iot-studio",
          "namespace": "ecp",
          "pods": [
            {
              "containers": [
                {
                  "name": "ase-support-iot-studio",
                  "stats": {
                    "cpu": 2701762,
                    "cpuPercentage": 0.06754405,
                    "memory": 21422080,
                    "memoryPercentage": 0.12898605125604792,
                    "timestamp": 1724660601,
                    "window": 13
                  }
                }
              ],
              "name": "ase-support-iot-studio-658f7d8f95-56t5x",
              "namespace": "ecp"
            }
          ],
          "status": {
            "availableReplicas": 1,
            "conditions": [
              {
                "lastTransitionTime": 1715073164,
                "lastUpdateTime": 1715073251,
                "message": "ReplicaSet \"ase-support-iot-studio-658f7d8f95\" has successfully progressed.",
                "reason": "NewReplicaSetAvailable",
                "status": "True",
                "type": "Progressing"
              },
              {
                "lastTransitionTime": 1722360967,
                "lastUpdateTime": 1722360967,
                "message": "Deployment has minimum availability.",
                "reason": "MinimumReplicasAvailable",
                "status": "True",
                "type": "Available"
              }
            ],
            "observedGeneration": 1,
            "readyReplicas": 1,
            "replicas": 1,
            "updatedReplicas": 1
          }
        },
        {
          "name": "ase-core-api",
          "namespace": "ecp",
          "pods": [
            {
              "containers": [
                {
                  "name": "ase-core-api",
                  "stats": {
                    "cpu": 957308,
                    "cpuPercentage": 0.0239327,
                    "memory": 14315520,
                    "memoryPercentage": 0.08619622354491156,
                    "timestamp": 1724660604,
                    "window": 17
                  }
                }
              ],
              "name": "ase-core-api-5cbf864fd-dq9dg",
              "namespace": "ecp"
            }
          ],
          "status": {
            "availableReplicas": 1,
            "conditions": [
              {
                "lastTransitionTime": 1715073169,
                "lastUpdateTime": 1720074798,
                "message": "ReplicaSet \"ase-core-api-5cbf864fd\" has successfully progressed.",
                "reason": "NewReplicaSetAvailable",
                "status": "True",
                "type": "Progressing"
              },
              {
                "lastTransitionTime": 1722360937,
                "lastUpdateTime": 1722360937,
                "message": "Deployment has minimum availability.",
                "reason": "MinimumReplicasAvailable",
                "status": "True",
                "type": "Available"
              }
            ],
            "observedGeneration": 3,
            "readyReplicas": 1,
            "replicas": 1,
            "updatedReplicas": 1
          }
        }
      ],
      "name": "k3d-dev-01-server-0",
      "role": [
        "control-plane",
        "master"
      ],
      "stats": {
        "cpu": 208512731,
        "cpuPercentage": 5.212818275,
        "memory": 1401962496,
        "memoryPercentage": 8.441458829773433,
        "timestamp": 1724660605,
        "window": 20
      }
    }
  ]
}