Fleet Assets in AnythingDB

AnythingDB provides the backbone for EdgeOps, just like it would in your applications. Each Asset in EdgeOps Fleet Management is represented in AnythingDB automatically, so you can interact with it.

In AnythingDB, for every Asset you provision, you will find a corresponding record in the Cluster category.

This section describes Actions defined on the Thing schema and how they allow interaction with the Edge.

Asset Thing Details in AnythingDB

Each Asset record in AnythingDB has a few preconfigured actions which you can use to control and configure your Asset remotely.

The following actions are defined automatically:

Action Description
Run statistics/metrics gathering Start and update the interval to receive the metrics.
Send kubectl Send kubectl commands from SmartWorks IoT
Stop statistics/metrics gathering Stop the reception of metrics.
Send Kubernetes API Send a RESTful request to the Kubernetes API server running on the Cluster.
Note: The Management Service publishes metrics for the Nodes and Deployments of the Cluster at a regular interval (default: 30 seconds). This data is obtained from the Kubernetes Metrics Server which is installed by default in a K3s installation. The published data can be accessed on the Raw History pane.

If an Action has a red settings icon and a greyed out Run button, then the Action needs to have parameters filled in before it can be executed. Click on the Settings icon to display the form.

When running an Action, the status updates for the Action are shown below the form. The final results of the Action are set up as an Event.

Action: Send kubectl

The Send kubectl form has three fields:

Name Description
CorrelationId Any alphanumeric text string specified by the user in order to match the resulting Event message to the originating Action.
Command Must be "kubectl".
Arguments Takes an array of strings based on the command line arguments for a kubectl command. This array can be entered using JSON notation.
Note: It is important that each argument is put in a new array entry. If there is a space in the complete command, then that means a new argument.

Useful kubectl commands translated into "command" and "arguments" parts:

Description Command Arguments
Get Nodes kubectl [ "get", "nodes"]
Get Nodes (wide) kubectl [ "get", "nodes", "-o", "wide"]
Top Node {node-name} kubectl [ "top", "node", "{node-name}"]
Get Services kubectl [ "get", "services"]
Get Config Maps kubectl [ "get", "configmaps"]
Get Deployments kubectl [ "get", "deployments"]
Restart Deployment {{deployment-name}} kubectl [ "rollout", "restart", "deployment", "{deployment-name}"]
Get Pods kubectl [ "get", "pods"]
Top Pod {pod-name} kubectl [ "top", "pod", "{pod-name}"]
Describe Pod {pod-name} kubectl [ "describe", "pod", "{pod-name}"]
Apply YAML {configuration.yaml} kubectl [ "apply", "-f", "{configuration.yaml}"]
Apply Multiple YAMLs kubectl [ "apply", "-f", "{configuration1.yaml}", "-f", "{configuration2.yaml}"]
Tip: Useful documentation for the kubectl CLI, including a kubectl cheat sheet can be found on the kubernetes website.

Action: Send kubernetes API

Available Endpoints

API calls to the Kubernetes API server at the edge cluster can be made using the Send Kubernetes API action.

The form fields show the following parameters: CorrelationId, RequestMethod, Href, and RequestBody. This is a RESTful API.

The CorrelationId is any alphanumeric text string specified by the user in order to match the resulting Event message to the originating Action.

Implemented kubernetes API endpoints are a subset of https://kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/

Currently Available
GET /api/v1/namespaces/{namespace}/pods/{name}/status
GET /api/v1/namespaces/{namespace}/services/{name}/status
GET /api/v1/namespaces/{namespace}/configmaps/{name}
GET /api/v1/namespaces/{namespace}/pods/{name}
GET /api/v1/namespaces/{namespace}/secrets/{name}
GET /api/v1/namespaces/{namespace}/serviceaccounts/{name}
GET /api/v1/namespaces/{namespace}/services/{name}
GET /api/v1/namespaces/{namespace}/configmaps
GET /api/v1/namespaces/{namespace}/pods
GET /api/v1/namespaces/{namespace}/secrets
GET /api/v1/namespaces/{namespace}/serviceaccounts
GET /api/v1/namespaces/{namespace}/services
GET /api/v1/namespaces/{name}/status
GET /api/v1/namespaces/{name}
GET /api/v1/nodes/{name}/status
GET /api/v1/nodes/{name}

PATCH /api/v1/namespaces/{namespace}/pods/{name}/status
PATCH /api/v1/namespaces/{namespace}/services/{name}/status
PATCH /api/v1/namespaces/{namespace}/configmaps/{name}
PATCH /api/v1/namespaces/{namespace}/pods/{name}
PATCH /api/v1/namespaces/{namespace}/secrets/{name}
PATCH /api/v1/namespaces/{namespace}/serviceaccounts/{name}
PATCH /api/v1/namespaces/{namespace}/services/{name}
PATCH /api/v1/namespaces/{name}
PATCH /api/v1/nodes/{name}/status
PATCH /api/v1/nodes/{name}

PUT /api/v1/namespaces/{namespace}/pods/{name}/status
PUT /api/v1/namespaces/{namespace}/services/{name}/status
PUT /api/v1/namespaces/{namespace}/configmaps/{name}
PUT /api/v1/namespaces/{namespace}/pods/{name}
PUT /api/v1/namespaces/{namespace}/secrets/{name}
PUT /api/v1/namespaces/{namespace}/serviceaccounts/{name}
PUT /api/v1/namespaces/{namespace}/services/{name}
PUT /api/v1/namespaces/{name}
PUT /api/v1/nodes/{name}/status
PUT /api/v1/nodes/{name}

DELETE /api/v1/namespaces/{namespace}/configmaps/{name}
DELETE /api/v1/namespaces/{namespace}/pods/{name}
DELETE /api/v1/namespaces/{namespace}/secrets/{name}
DELETE /api/v1/namespaces/{namespace}/serviceaccounts/{name}
DELETE /api/v1/namespaces/{namespace}/services/{name}
DELETE /api/v1/namespaces/{name}
DELETE /api/v1/nodes/{name}

POST /api/v1/namespaces/{namespace}/configmaps
POST /api/v1/namespaces/{namespace}/pods
POST /api/v1/namespaces/{namespace}/secrets
POST /api/v1/namespaces/{namespace}/serviceaccounts
POST /api/v1/namespaces/{namespace}/services

GET /api/v1/configmaps
GET /api/v1/namespaces
GET /api/v1/nodes
GET /api/v1/pods
GET /api/v1/secrets
GET /api/v1/serviceaccounts
GET /api/v1/services

GET /apis/apps/v1/namespaces/{namespace}/deployments/{name}/status
GET /apis/apps/v1/namespaces/{namespace}/deployments/{name}
GET /apis/apps/v1/namespaces/{namespace}/deployments

PATCH /apis/apps/v1/namespaces/{namespace}/deployments/{name}/status
PATCH /apis/apps/v1/namespaces/{namespace}/deployments/{name}

PUT /apis/apps/v1/namespaces/{namespace}/deployments/{name}/status
PUT /apis/apps/v1/namespaces/{namespace}/deployments/{name}

DELETE /apis/apps/v1/namespaces/{namespace}/deployments/{name}

POST /apis/apps/v1/namespaces/{namespace}/deployments

GET /apis/apps/v1/deployments

GET /apis/rbac.authorization.k8s.io/v1/clusterroles/{name}
GET /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/{name}
GET /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/roles/{name}
GET /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/rolebindings/{name}
GET /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/roles
GET /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/rolebindings

PATCH /apis/rbac.authorization.k8s.io/v1/clusterroles/{name}
PATCH /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/{name}
PATCH /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/roles/{name}
PATCH /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/rolebindings/{name}

PUT /apis/rbac.authorization.k8s.io/v1/clusterroles/{name}
PUT /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/{name}
PUT /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/roles/{name}
PUT /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/rolebindings/{name}

DELETE /apis/rbac.authorization.k8s.io/v1/clusterroles/{name}
DELETE /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/{name}
DELETE /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/roles/{name}
DELETE /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/rolebindings/{name}

GET /apis/rbac.authorization.k8s.io/v1/clusterroles
GET /apis/rbac.authorization.k8s.io/v1/clusterrolebindings
GET /apis/rbac.authorization.k8s.io/v1/roles
GET /apis/rbac.authorization.k8s.io/v1/rolebindings

POST /apis/rbac.authorization.k8s.io/v1/clusterroles
POST /apis/rbac.authorization.k8s.io/v1/clusterrolebindings
POST /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/roles
POST /apis/rbac.authorization.k8s.io/v1/namespaces/{namespace}/rolebindings
Custom
GET /apis/swx/v1/edge-apps
POST /apis/swx/v1/edge-apps
GET /apis/swx/v1/edge-apps/{edge-app-id}
PUT /apis/swx/v1/edge-apps/{edge-app-id}
DELETE /apis/swx/v1/edge-apps/{edge-app-id}

PUT /apis/swx/v1/apply
Request Body
The custom API endpoint is the equivalent of "kubectl apply" and accepts a URL as the request body or the actual yaml (string).
The other endpoints use JSON (stringified).
See the kubernetes reference documentation for details.

Developer Documentation

Send kubectl

This section describes the MQTT messages and topics the Management Service handles to perform the "send-kubectl" action. When using AnythingDB in Studio all of this is taken care of within the product.

Sending a Message to the Management Service
The MQTT topic the Management Service subscribes to, for running kubectl commands is:
status/<space>/categories/cluster/things/<thing-id>/actions/send-kubectl

The message format for publishing is as follows:

{
  "send-kubectl": {
    "input": {
      "correlationId": "{create-a-unique-id-for-each-message}",
      "command": "kubectl",
      "arguments": [
        "{each}",
        "{new}",
        "{argument}",
        "{needs}",
        "{its}",
        "{own}",
        "{array}",
        "{entry}"
      ]
    },
    "status": "pending",
    "timeRequested": "<datetime>",
    "href": "/spaces/<space>/categories/cluster/things/<thing-id>/actions/send-kubectl/<action-id>"
  }
}

For example:

{
  "send-kubectl": {
    "input": {
      "correlationId": "666",
      "command": "kubectl",
      "arguments": [
        "get",
        "nodes",
        "-o",
        "wide"
      ]
    },
    "status": "pending",
    "timeRequested": "2020-05-26 15:37:46+0000",
    "href": "/spaces/my-space/categories/cluster/things/my-thing-id/actions/send-kubectl/1234"
  }
}

It is important that each argument is put in a new array entry. If there is a space in the complete command then that means a new argument.

Status Update After Receiving Action Request
MQTT topic the message is sent to:
spaces/<space>/categories/cluster/things/<thing-id>/actions

Message:

{
  "send-kubectl": {
    "href": "/spaces/<space>/categories/cluster/things/<thing-id>/actions/send-kubectl/<action-id>",
    "status": "received"
  }
}
Event Publish After Running Action
MQTT topic the message is sent to:
/spaces/<space>/categories/cluster/things/<thing-id>/events

Message:

{
    "kubectl-logs": {
        "data": {
            "correlationId": "{the-unique-id-from-the-request}",
            "statusCode": "{an-http-status-code}",
            "statusMessage": "{optional descriptive message}",
            "response": "{the output in plain text format}"
        }
    }
}

Example:

{
    "kubectl-logs": {
        "data": {
            "correlationId": "666",
            "statusCode": 200,
            "statusMessage": "Command executed successfully",
            "response": "NAME           STATUS   ROLES    AGE   VERSION        INTERNAL-IP    EXTERNAL-IP   OS-IMAGE           KERNEL-VERSION     CONTAINER-RUNTIME\nmartin-ag150   Ready    master   50d   v1.18.2+k3s1   10.10.10.100   \u003cnone\u003e        Ubuntu 20.04 LTS   5.4.0-40-generic   containerd://1.3.3-k3s2\n"
        }
    }
}

You need to use the correlationId to match the response to the request.

Status Update After Completing Action
MQTT topic the message is sent to:
spaces/<space>/categories/cluster/things/<thing-id>/actions

Message:

{
  "send-kubectl": {
    "href": "/spaces/<space>/categories/cluster/things/<thing-id>/actions/send-kubectl/<action-id>",
    "status": "completed"
  }
}

Send kubernetes API

This section describes the MQTT messages and topics the Management Service handles to perform the send-kubernetes-api action. When using AnythingDB in Studio all of this is taken care of within the product.

Sending a Message to the Management Service
The MQTT topic the Management Service subscribes to, for running kubernetes API requests is:
spaces/<space>/categories/cluster/things/<thing-id>/actions/send-kubernetes-api

The message format for publishing is as follows:

{
  "send-kubernetes-api": {
    "input": {
      "correlationId": "{create-a-unique-id-for-each-message}",
      "requestMethod": "{GET|POST|PUT|PATCH|DELETE}",
      "href": "{see API docs}",
      "requestBody": "{body if required}"
    },
    "status": "pending",
    "timeRequested": "<datetime>",
    "href": "/spaces/<space>/categories/cluster/things/<thing-id>/actions/send-kubernetes-api/<action-id>"
  }
}

The requestBody is a string. For a JSON requestBody this means it has to be stringified first. This results in "double JSON-encoding".

For example:

{
    "send-kubernetes-api":{
        "input":{
            "correlationId": "666",
            "requestMethod": "GET",
            "href": "/api/v1/nodes",
            "requestBody": ""
        },
        "status": "pending",
        "timeRequested": "2020-05-26 15:37:46+0000",
        "href": "/spaces/may-space/categories/cluster/things/my-thing-id/actions/send-kubernetes-api/1234"
    }
}
Status Update After Receiving Action Request
MQTT topic the message is sent to:
spaces/<space>/categories/cluster/things/<thing-id>/actions

Message:

{
  "send-kubernetes-api": {
    "href": "/spaces/<space>/categories/cluster/things/<thing-id>/actions/send-kubernetes-api/<action-id>",
    "status": "received"
  }
}
Event Publish After Running Action
MQTT topic the message is sent to:
spaces/<space>/categories/cluster/things/<thing-id>/events

Message:

{
    "send-kubernetes-api": {
        "data": {
            "correlationId": "{the-unique-id-from-the-request}",
            "statusCode": "{an-http-status-code}",
            "statusMessage": "{optional descriptive message}",
            "response": "{the output of the API request}"
        }
    }
}

Example:

{
    "send-kubernetes-api": {
        "data": {
            "correlationId": "666",
            "statusCode": 200,
            "statusMessage": "Command executed successfully",
            "response": ""
        }
    }
}

You need to use the correlationId to match the response to the request.

Status Update After Completing Action
MQTT topic the message is sent to:
spaces/<space>/categories/cluster/things/<thing-id>/actions

Message:

{
  "send-kubernetes-api": {
    "href": "/spaces/<space>/categories/cluster/things/<thing-id>/actions/send-kubernetes-api/<action-id>",
    "status": "completed"
  }
}

Metrics

This section describes the MQTT messages and topics the Management Service handles to perform the "run-stats" and "stop-stats" actions. When using AnythingDB in Studio all of this is taken care of within the product.

Metrics / Statistics Data

The Management Service publishes metrics for the nodes and deployments of the cluster at a regular interval (default: 30 seconds). This data is obtained from the Kubernetes Metrics Server which is installed by default in a K3s installation.

Start the Metrics / Update the Interval
The metrics publishing can be started, or the interval can be updated, by sending a message to the following topic:
status/<space>/categories/cluster/things/<thing-id>/actions/run-stats

The message format for publishing is as follows:

{
  "run-stats": {
    "input": {
      "interval": {in-seconds}
    },
    "status": "pending",
    "timeRequested": "<datetime>",
    "href": "<space>/categories/cluster/things/<thing-id>/actions/run-stats/<action-id>"
  }
}

For example:

{
  "run-stats": {
    "input": {
      "interval": 15
    },
    "status": "pending",
    "timeRequested": "2020-06-18 10:32:46+0000",
    "href": "martin/things/ag150/actions/run-stats/1234"
  }
}
Stop the Metrics
The metrics publishing can be stopped by sending a message to the following topic:
status/<space>/categories/cluster/things/<thing-id>/actions/stop-stats

The message format for publishing is as follows:

{
  "stop-stats": {
    "status": "pending",
    "timeRequested": "<datetime>",
    "href": "<space>/categories/cluster/things/<thing-id>/actions/stop-stats/<action-id>"
  }
}

For example:

{
  "stop-stats": {
    "status": "pending",
    "timeRequested": "2020-06-18 10:36:46+0000",
    "href": "martin/things/ag150/actions/stop-stats/1234"
  }
}
Example of Node-stats
{
  "nodes-stats": {
    "data": [
      {
        "capacities": {
          "cpu": 4000000000,
          "memory": 1937969152,
          "pods": 110,
          "storage": 31084183552
        },
        "conditions": [
          {
            "lastHeartbeatTime": 1607879049,
            "lastTransitionTime": 1607879049,
            "message": "Flannel is running on this node",
            "reason": "FlannelIsUp",
            "status": "False",
            "type": "NetworkUnavailable"
          },
          {
            "lastHeartbeatTime": 1608055458,
            "lastTransitionTime": 1607592799,
            "message": "kubelet has sufficient memory available",
            "reason": "KubeletHasSufficientMemory",
            "status": "False",
            "type": "MemoryPressure"
          },
          {
            "lastHeartbeatTime": 1608055458,
            "lastTransitionTime": 1607592799,
            "message": "kubelet has no disk pressure",
            "reason": "KubeletHasNoDiskPressure",
            "status": "False",
            "type": "DiskPressure"
          },
          {
            "lastHeartbeatTime": 1608055458,
            "lastTransitionTime": 1607592799,
            "message": "kubelet has sufficient PID available",
            "reason": "KubeletHasSufficientPID",
            "status": "False",
            "type": "PIDPressure"
          },
          {
            "lastHeartbeatTime": 1608055458,
            "lastTransitionTime": 1607879059,
            "message": "kubelet is posting ready status. AppArmor enabled",
            "reason": "KubeletReady",
            "status": "True",
            "type": "Ready"
          }
        ],
        "name": "ubuntu",
        "stats": {
          "cpu": 867753808,
          "cpuPercentage": 21.693845200000002,
          "memory": 1239392256,
          "memoryPercentage": 63.95314676298831,
          "timestamp": 1608143932,
          "window": 30
        }
      }
    ]
  }
}
Example of Deployments-stats
{
  "deployments-stats": {
    "data": {
      "nodes": [
        {
          "deployments": [
            {
              "name": "ase-core-message",
              "pods": [
                {
                  "containers": [
                    {
                      "name": "ase-core-message",
                      "stats": {
                        "cpu": 105314661,
                        "cpuPercentage": 2.6328665250000003,
                        "memory": 140861440,
                        "memoryPercentage": 7.268507852905184,
                        "timestamp": 1608143929,
                        "window": 30
                      }
                    }
                  ],
                  "name": "ase-core-message-866947bfb6-59g4f"
                }
              ],
              "status": {
                "availableReplicas": 1,
                "conditions": [
                  {
                    "lastTransitionTime": 1605289453,
                    "lastUpdateTime": 1607448834,
                    "message": "ReplicaSet \"ase-core-message-866947bfb6\" has successfully progressed.",
                    "reason": "NewReplicaSetAvailable",
                    "status": "True",
                    "type": "Progressing"
                  },
                  {
                    "lastTransitionTime": 1607879075,
                    "lastUpdateTime": 1607879075,
                    "message": "Deployment has minimum availability.",
                    "reason": "MinimumReplicasAvailable",
                    "status": "True",
                    "type": "Available"
                  }
                ],
                "observedGeneration": 4,
                "readyReplicas": 1,
                "replicas": 1,
                "updatedReplicas": 1
              }
            },
            {
              "name": "ase-core-management",
              "pods": [
                {
                  "containers": [
                    {
                      "name": "ase-core-management",
                      "stats": {
                        "cpu": 804966,
                        "cpuPercentage": 0.02012415,
                        "memory": 15163392,
                        "memoryPercentage": 0.7824372221999125,
                        "timestamp": 1608143927,
                        "window": 30
                      }
                    }
                  ],
                  "name": "ase-core-management-764f8d8b66-pv2gf"
                }
              ],
              "status": {
                "availableReplicas": 1,
                "conditions": [
                  {
                    "lastTransitionTime": 1605795610,
                    "lastUpdateTime": 1607678997,
                    "message": "ReplicaSet \"ase-core-management-764f8d8b66\" has successfully progressed.",
                    "reason": "NewReplicaSetAvailable",
                    "status": "True",
                    "type": "Progressing"
                  },
                  {
                    "lastTransitionTime": 1607879075,
                    "lastUpdateTime": 1607879075,
                    "message": "Deployment has minimum availability.",
                    "reason": "MinimumReplicasAvailable",
                    "status": "True",
                    "type": "Available"
                  }
                ],
                "observedGeneration": 23,
                "readyReplicas": 1,
                "replicas": 1,
                "updatedReplicas": 1
              }
            },
            {
              "name": "ase-core-cache",
              "pods": [
                {
                  "containers": [
                    {
                      "name": "ase-core-cache",
                      "stats": {
                        "cpu": 3803112,
                        "cpuPercentage": 0.0950778,
                        "memory": 2957312,
                        "memoryPercentage": 0.15259850740905911,
                        "timestamp": 1608143924,
                        "window": 30
                      }
                    }
                  ],
                  "name": "ase-core-cache-676b54585d-jrncd"
                }
              ],
              "status": {
                "availableReplicas": 1,
                "conditions": [
                  {
                    "lastTransitionTime": 1605289503,
                    "lastUpdateTime": 1607452737,
                    "message": "ReplicaSet \"ase-core-cache-676b54585d\" has successfully progressed.",
                    "reason": "NewReplicaSetAvailable",
                    "status": "True",
                    "type": "Progressing"
                  },
                  {
                    "lastTransitionTime": 1607879075,
                    "lastUpdateTime": 1607879075,
                    "message": "Deployment has minimum availability.",
                    "reason": "MinimumReplicasAvailable",
                    "status": "True",
                    "type": "Available"
                  }
                ],
                "observedGeneration": 2,
                "readyReplicas": 1,
                "replicas": 1,
                "updatedReplicas": 1
              }
            },
            {
              "name": "ase-virtual-meter",
              "pods": [
                {
                  "containers": [
                    {
                      "name": "ase-virtual-meter",
                      "stats": {
                        "cpu": 1428236,
                        "cpuPercentage": 0.0357059,
                        "memory": 7585792,
                        "memoryPercentage": 0.391429966373376,
                        "timestamp": 1608143930,
                        "window": 30
                      }
                    }
                  ],
                  "name": "ase-virtual-meter-655b56d958-vw7cn"
                }
              ],
              "status": {
                "availableReplicas": 1,
                "conditions": [
                  {
                    "lastTransitionTime": 1607879074,
                    "lastUpdateTime": 1607879074,
                    "message": "Deployment has minimum availability.",
                    "reason": "MinimumReplicasAvailable",
                    "status": "True",
                    "type": "Available"
                  },
                  {
                    "lastTransitionTime": 1606748304,
                    "lastUpdateTime": 1607972823,
                    "message": "ReplicaSet \"ase-virtual-meter-655b56d958\" has successfully progressed.",
                    "reason": "NewReplicaSetAvailable",
                    "status": "True",
                    "type": "Progressing"
                  }
                ],
                "observedGeneration": 9,
                "readyReplicas": 1,
                "replicas": 1,
                "updatedReplicas": 1
              }
            },
            {
              "name": "ase-core-cloud",
              "pods": [
                {
                  "containers": [
                    {
                      "name": "ase-core-cloud",
                      "stats": {
                        "cpu": 861411,
                        "cpuPercentage": 0.021535275,
                        "memory": 6254592,
                        "memoryPercentage": 0.3227395025119574,
                        "timestamp": 1608143924,
                        "window": 30
                      }
                    }
                  ],
                  "name": "ase-core-cloud-57486fd5f5-5drcv"
                }
              ],
              "status": {
                "availableReplicas": 1,
                "conditions": [
                  {
                    "lastTransitionTime": 1607695991,
                    "lastUpdateTime": 1607696007,
                    "message": "ReplicaSet \"ase-core-cloud-57486fd5f5\" has successfully progressed.",
                    "reason": "NewReplicaSetAvailable",
                    "status": "True",
                    "type": "Progressing"
                  },
                  {
                    "lastTransitionTime": 1607879068,
                    "lastUpdateTime": 1607879068,
                    "message": "Deployment has minimum availability.",
                    "reason": "MinimumReplicasAvailable",
                    "status": "True",
                    "type": "Available"
                  }
                ],
                "observedGeneration": 1,
                "readyReplicas": 1,
                "replicas": 1,
                "updatedReplicas": 1
              }
            },
            {
              "name": "ase-core-health",
              "pods": [
                {
                  "containers": [
                    {
                      "name": "ase-core-health",
                      "stats": {
                        "cpu": 2052389,
                        "cpuPercentage": 0.051309725,
                        "memory": 7409664,
                        "memoryPercentage": 0.38234168961632675,
                        "timestamp": 1608143930,
                        "window": 30
                      }
                    }
                  ],
                  "name": "ase-core-health-7c4f9d966d-vw9mt"
                }
              ],
              "status": {
                "availableReplicas": 1,
                "conditions": [
                  {
                    "lastTransitionTime": 1605289626,
                    "lastUpdateTime": 1607696276,
                    "message": "ReplicaSet \"ase-core-health-7c4f9d966d\" has successfully progressed.",
                    "reason": "NewReplicaSetAvailable",
                    "status": "True",
                    "type": "Progressing"
                  },
                  {
                    "lastTransitionTime": 1607879074,
                    "lastUpdateTime": 1607879074,
                    "message": "Deployment has minimum availability.",
                    "reason": "MinimumReplicasAvailable",
                    "status": "True",
                    "type": "Available"
                  }
                ],
                "observedGeneration": 5,
                "readyReplicas": 1,
                "replicas": 1,
                "updatedReplicas": 1
              }
            }
          ],
          "name": "ubuntu"
        }
      ]
    }
  }
}