Running ADx, Conversion and Platform Health Checks via REST API

This document explains how to run health checks on a running ADx and Conversion instances and how to check the status of the host itself.

For information about legacy checks, see Running Deep Health Checks on Legacy Endpoints

Scripted Platform Health Check

ADx and Conversion packages contain a scripted health check which you can use to verify the status of the hosting platform. To run the scripted health check:

In the folder where you unzipped the deployment package, open the terminal and run the ./check-health.sh --url https://[HOST]:[PORT] command.

Change the command accordingly so that it's referring to your server (Conversion or ADx), port, and transfer protocol (http/https). You can reference the environment variable script while running health check by adding --environment ../environment.sh to the command (adapt the path accordingly if you saved your environment script elsewhere).

After the health check has finished (it may take a few minutes), there are two possible outcomes:

Successfully completed health check. Deployed services are available. - everything is OK
Health check failed! For more information see health-check-result.json and/or run health check directly via http://localhost:8080/tribefire-services/healthz - there is a problem that needs to be fixed

Conversion Status Check

Send a request to the /tribefire-services/api/v1/check endpoint of your Conversion instance to make sure everything is running as expected.

Sample cURL Call

curl --request GET https://[CONVERSION_HOST]:[PORT]/tribefire-services/api/v1/check?sessionId=[CONVERSION_SESSION_ID] --header "Accept: application/json"

Please replace the following items in your call:

[CONVERSION_HOST]:[PORT] - replace with the domain name hosting your Conversion instance (for example conversion-host.com or conversion-host:8080.com)
[CONVERSION_SESSION_ID] - replace with a valid session ID. See Authentication for information on how to get a session ID.
--header "Accept: application/json" - you can also use text/html for HTML output.

Response and Interpretation

You will get a long response as a result, which includes a number of checks related to base platform and Conversion itself. Database connectivity, thread pools, and Conversion functionality are checked, among others. It's most important to verify the check status of all components, which should read ok, as in "checkStatus": "ok". If you get a warn or fail status, it indicates a problem within the reporting component.

{"_type": "com.braintribe.model.deploymentapi.check.data.CheckBundlesResponse", "_id": "0",
 "elapsedTimeInMs": 626.82,
 "elements": [
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "1",
   "check": {"_type": "com.braintribe.model.extensiondeployment.check.HardwiredCheckProcessor", "_id": "2",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "externalId": "checkProcessor.hardwired.BaseConnectivityCheckProcessor",
    "globalId": "hardwired:check/BaseConnectivityCheckProcessor",
    "id": "hardwired:check/BaseConnectivityCheckProcessor",
    "module": {"_type": "com.braintribe.model.deployment.Module", "_id": "3",
     "bindsDeployables": false,
     "bindsHardwired": false,
     "bindsInitializers": false,
     "bindsWireContracts": false,
     "globalId": "module://platform",
     "id": "module://platform",
     "name": "tribefire-web-platform",
     "partition": "cortex"
    },
    "name": "Base Connectivity Check Processor",
    "partition": "cortex"
   },
   "coverage": "connectivity",
   "isPlatformRelevant": false,
   "name": "Base Platform Connectivity Checks",
   "node": "conversion@DESKTOP-J07TD08#200720151401978a268ac9350b4e7f9d",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "4",
    "elapsedTimeInMs": 8.72,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "5",
      "checkStatus": "ok",
      "details": "Type | Valid | Check duration\n--- | --- | ---\nLeadership | true | 0 ms\nLock | true | 0 ms\n",
      "detailsAsMarkdown": true,
      "name": "System Databases"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "6",
      "checkStatus": "ok",
      "details": "Name | Idle Connections | Threads Waiting | Active Connections | Total Connections | Max Pool Size\n--- | --- | --- | --- | --- | ---\nconnection.conversion|3|0|0|3|20\nconnection.user-sessions|1|0|0|1|20\nconnection.default|2|0|1|3|20\nconnection.auth|1|0|0|1|10\nconnection.locking|2|0|0|2|10\nconnection.leadership|1|0|1|2|5\nconnection.user-session-statistics|1|0|0|1|10\n",
      "detailsAsMarkdown": true,
      "name": "Connection Pools"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "7",
      "checkStatus": "ok",
      "details": "Name | Message Transit Duration\n--- | ---\nDMB Messaging | 304700 ns\n",
      "detailsAsMarkdown": true,
      "name": "Messaging"
     }
    ]
   },
   "status": "ok"
  },
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "8",
   "check": {"_type": "com.braintribe.model.extensiondeployment.check.HardwiredCheckProcessor", "_id": "9",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "externalId": "checkProcessor.hardwired.BaseFunctionalityCheckProcessor",
    "globalId": "hardwired:check/BaseFunctionalityCheckProcessor",
    "id": "hardwired:check/BaseFunctionalityCheckProcessor",
    "module": {"_ref": "3"},
    "name": "Base Functionality Check Processor",
    "partition": "cortex"
   },
   "coverage": "functional",
   "isPlatformRelevant": false,
   "name": "Base Platform Functionality Checks",
   "node": "conversion@DESKTOP-J07TD08#200720151401978a268ac9350b4e7f9d",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "10",
    "elapsedTimeInMs": 6.1765,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "11",
      "checkStatus": "ok",
      "details": "---|---\nLock Manager|DB Lock Manager\n",
      "detailsAsMarkdown": true,
      "name": "Lock Check"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "12",
      "checkStatus": "ok",
      "details": "---|---\nLeadership Manager|DB Leadership Manager\n",
      "detailsAsMarkdown": true,
      "name": "Leadership Check"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "13",
      "checkStatus": "ok",
      "details": "---|---\nIssued to|Braintribe Technology GmbH\nIssued by|Braintribe\nExpiration|01 Apr 2021\n",
      "detailsAsMarkdown": true,
      "name": "License"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "14",
      "checkStatus": "ok",
      "details": "Name|Active Threads|Total Executions|Average Execution Time|Pool Size|Core Pool Size|Maximum Pool Size|Pending Tasks in Queue|Average Pending Time in Queue|Time Since Last Execution\n---|---|---|---|---|---|---|---|---|---\nActivation|0|1|254 ms|0|5|5|0|0.65 ms|2 d\nCartridge Evaluator|0|0|0 ms|0|40|40|0|0 ms|0 ms\nCheckBundleExecution|3|107|154 ms|5|5|20|0|0.23 ms|0 ms\nConversion Cleanup|0|0|0 ms|0|5|5|0|0 ms|0 ms\nConversion Document Creation|0|0|0 ms|0|3|3|0|0 ms|0 ms\nConversion Document Creation Auxiliary|0|0|0 ms|0|6|6|0|0 ms|0 ms\nConversion Job Executor|0|0|0 ms|0|3|3|0|0 ms|0 ms\nDatabase Connection Check|0|0|0 ms|0|1|10|0|0 ms|0 ms\nDmb Message Consumer (conversion.ConversionJobNotification)|1|1|0 ms|1|1|1|0|0 ms|2 d\nDmb Message Consumer (conversion.tf.queue.trustedRequest)|1|1|0 ms|1|1|1|0|0 ms|2 d\nDmb Message Consumer (conversion.tf.topic.heartbeat)|1|1|0 ms|1|1|1|0|0 ms|2 d\nDmb Message Consumer (conversion.tf.topic.multicastRequest)|1|1|0 ms|1|1|1|0|0 ms|2 d\nDmb Message Consumer (conversion.tf.topic.multicastResponse)|1|1|0 ms|1|1|1|0|0 ms|2 d\nJava Common Pool|0|-1|-1 ms|0|0|0|0| |0 ms\nJob Cleanup|0|0|0 ms|0|3|3|0|0 ms|0 ms\nLeadership Daemon|0|141794|3 ms|1|1|unbound|1|0 ms|860 ms\nMBean Message Reaper|1|1|0 ms|1|1|1|0|0 ms|2 d\nMaster Heartbeat Sender|0|14235|0 ms|1|1|unbound|1|0 ms|1 s\nMaster Messaging Service|0|0|0 ms|0|20|20|0|0 ms|0 ms\nMaster Multicast Consumer|1|1|0 ms|1|20|20|0|0 ms|2 d\nMaster RPC Service|0|0|0 ms|0|20|unbound|0|0 ms|0 ms\nPlatform Thread-Pool|6|7|180 ms|6|100|100|0|0.49 ms|2 d\nScheduled System Executor|2|14342|11 ms|10|10|unbound|2|0.27 ms|0 ms\nState Change Processing (Aspect)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Audit Workbench)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Audit)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Authentication and Authorization Workbench)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Cortex Workbench)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Cortex)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Incremental Access Listener)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Platform Setup Workbench)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Platform Setup)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (User Sessions Workbench)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (User Statistics Workbench)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Workbench)|0|0|0 ms|0|20|20|0|0 ms|0 ms\n",
      "detailsAsMarkdown": true,
      "name": "Thread Pools"
     }
    ]
   },
   "status": "ok"
  },
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "15",
   "check": {"_type": "com.braintribe.model.extensiondeployment.check.HardwiredCheckProcessor", "_id": "16",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "externalId": "checkProcessor.hardwired.BaseVitalityCheckProcessor",
    "globalId": "hardwired:check/BaseVitalityCheckProcessor",
    "id": "hardwired:check/BaseVitalityCheckProcessor",
    "module": {"_ref": "3"},
    "name": "Base Vitality Check Processor",
    "partition": "cortex"
   },
   "coverage": "vitality",
   "isPlatformRelevant": true,
   "name": "Base Platform Vitality Checks",
   "node": "conversion@DESKTOP-J07TD08#200720151401978a268ac9350b4e7f9d",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "17",
    "elapsedTimeInMs": 0.0077,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "18",
      "checkStatus": "ok",
      "details": "Check infrastructure is ok",
      "detailsAsMarkdown": false,
      "name": "Base Check"
     }
    ]
   },
   "status": "ok"
  },
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "19",
   "check": {"_type": "com.braintribe.model.conversion.HealthCheckProcessor", "_id": "20",
    "autoDeploy": true,
    "conversionAccessId": "conversion.access",
    "conversionProcessorId": "serviceProcessor.conversion.serviceProcessor",
    "deploymentStatus": "deployed",
    "externalId": "conversion.healthcheck",
    "globalId": "wire://ConversionInitializerModuleWireModule/ConversionInitializerModuleCommonsSpace/healthCheckProcessor",
    "id": "wire://ConversionInitializerModuleWireModule/ConversionInitializerModuleCommonsSpace/healthCheckProcessor",
    "maxDurationPerJob": 3600000,
    "module": {"_type": "com.braintribe.model.deployment.Module", "_id": "21",
     "bindsDeployables": true,
     "bindsHardwired": false,
     "bindsInitializers": false,
     "bindsWireContracts": false,
     "globalId": "module://tribefire.extension.conversion:conversion-module",
     "id": "module://tribefire.extension.conversion:conversion-module",
     "name": "conversion-module",
     "partition": "cortex"
    },
    "name": "Conversion Check Processor",
    "partition": "cortex",
    "workerThreads": 3
   },
   "coverage": "functional",
   "isPlatformRelevant": false,
   "module": {"_ref": "21"},
   "name": "Conversion Checks",
   "node": "conversion@DESKTOP-J07TD08#200720151401978a268ac9350b4e7f9d",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "22",
    "elapsedTimeInMs": 618.4326,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "23",
      "checkStatus": "ok",
      "details": "Checking Conversion Access configuration.",
      "detailsAsMarkdown": false,
      "name": "Conversion Access Check"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "24",
      "checkStatus": "ok",
      "details": "Checking Conversion Processor configuration.",
      "detailsAsMarkdown": false,
      "name": "Conversion Processor Check"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "25",
      "checkStatus": "ok",
      "details": "Jobs | Pending | Worker Pool\n--- | --- | ---\n0 | 0 | 3\n\nNo conversions currently active.",
      "detailsAsMarkdown": true,
      "name": "Current Job"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "26",
      "checkStatus": "ok",
      "details": "Jobs | Success | Failures\n--- | --- | ---\n0 | 0 | 0\n",
      "detailsAsMarkdown": true,
      "name": "Jobs Processed"
     }
    ]
   },
   "status": "ok",
   "weight": "under1s"
  }
 ],
 "status": "ok"

ADx Status Check

Monitoring ADx Status with the /check Endpoint

Send a request to the /tribefire-services/api/v1/check endpoint of your ADx instance to make sure everything is running as expected.

Sample cURL Call

curl --request GET http://[ADX_HOST]:[PORT]/tribefire-services/api/v1/check?sessionId=ADX_SESSION_ID --header "Accept: application/json"

Please replace the following items in the above call:

[ADX_HOST]:[PORT] - replace with the domain name hosting your Conversion instance (for example adx-host.com or adx-host:8080.com)
[ADX_SESSION_ID] - replace with a valid ADx session ID. See Authentication for information on how to get a session ID.
--header "Accept: application/json" - this part of the call controls how the output is returned. You can also use text/html or text/plain for HTML and plain text output, respectively.

Response and Interpretation

You will get a long response as a result, which includes a number of checks related to base platform and ADx itself. Database connectivity, thread pools, and ADx functionality are checked, among others. It's most important to verify the check status of all components, which should read ok, as in "checkStatus": "ok". If you get a warn or fail status, it indicates a problem within the reporting component.

{"_type": "com.braintribe.model.deploymentapi.check.data.CheckBundlesResponse", "_id": "0",
 "elapsedTimeInMs": 1250.6271,
 "elements": [
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "1",
   "check": {"_type": "tribefire.adx.model.deployment.cmis.check.CmisConnectivityCheckProcessor", "_id": "2",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "externalId": "processor.adx.cmis-connectivity-check",
    "globalId": "wire://AdxCmisInitializerWireModule/AdxCmisInitializerDataSpace/connectivityCheckProcessor",
    "id": "wire://AdxCmisInitializerWireModule/AdxCmisInitializerDataSpace/connectivityCheckProcessor",
    "name": "ADx CMIS Connectivity Check Processor",
    "partition": "cortex"
   },
   "coverage": "connectivity",
   "isPlatformRelevant": false,
   "module": {"_type": "com.braintribe.model.deployment.Module", "_id": "3",
    "bindsDeployables": true,
    "bindsHardwired": false,
    "bindsInitializers": false,
    "bindsWireContracts": false,
    "globalId": "module://tribefire.adx.phoenix:adx-cmis-module",
    "id": "module://tribefire.adx.phoenix:adx-cmis-module",
    "name": "adx-cmis-module",
    "partition": "cortex"
   },
   "name": "ADx CMIS Connectivity Checks",
   "node": "adx@DESKTOP-J07TD08#200720151955598c4700f3a48d468c9f",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "4",
    "elapsedTimeInMs": 8.6313,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "5",
      "checkStatus": "ok",
      "details": "No deployed connections found.",
      "detailsAsMarkdown": false,
      "name": "ADx CMIS Connections"
     }
    ]
   },
   "status": "ok",
   "weight": "under10s"
  },
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "6",
   "check": {"_type": "tribefire.adx.model.deployment.healthcheck.ConnectivityCheckProcessor", "_id": "7",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "externalId": "adx.check.connectivity",
    "globalId": "wire://AdxInitializerWireModule/AdxInitializerDataSpace/connectivityCheckProcessor",
    "id": "wire://AdxInitializerWireModule/AdxInitializerDataSpace/connectivityCheckProcessor",
    "name": "ADx Connectivity Check Processor",
    "partition": "cortex"
   },
   "coverage": "connectivity",
   "isPlatformRelevant": false,
   "module": {"_type": "com.braintribe.model.deployment.Module", "_id": "8",
    "bindsDeployables": true,
    "bindsHardwired": false,
    "bindsInitializers": false,
    "bindsWireContracts": false,
    "globalId": "module://tribefire.adx.phoenix:adx-module",
    "id": "module://tribefire.adx.phoenix:adx-module",
    "name": "adx-module",
    "partition": "cortex"
   },
   "name": "ADx Connectivity Checks",
   "node": "adx@DESKTOP-J07TD08#200720151955598c4700f3a48d468c9f",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "9",
    "elapsedTimeInMs": 387.917,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "10",
      "checkStatus": "ok",
      "details": "Type | Repository | Accessible | Response Time | Session Created\n--- | --- | --- | --- | ---\nSTANDARD | Default | true | 0 ms | true\n",
      "detailsAsMarkdown": true,
      "name": "Active Repositories"
     }
    ]
   },
   "status": "ok",
   "weight": "under10s"
  },
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "11",
   "check": {"_type": "tribefire.adx.model.deployment.documentum.check.DctmConnectivityCheckProcessor", "_id": "12",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "externalId": "processor.adx.documentum-connectivity-check",
    "globalId": "wire://AdxDocumentumInitializerWireModule/AdxDocumentumInitializerDataSpace/connectivityCheckProcessor",
    "id": "wire://AdxDocumentumInitializerWireModule/AdxDocumentumInitializerDataSpace/connectivityCheckProcessor",
    "name": "ADx Documentum Connectivity Check Processor",
    "partition": "cortex"
   },
   "coverage": "connectivity",
   "isPlatformRelevant": false,
   "module": {"_type": "com.braintribe.model.deployment.Module", "_id": "13",
    "bindsDeployables": true,
    "bindsHardwired": false,
    "bindsInitializers": false,
    "bindsWireContracts": false,
    "globalId": "module://tribefire.adx.phoenix:adx-documentum-module",
    "id": "module://tribefire.adx.phoenix:adx-documentum-module",
    "name": "adx-documentum-module",
    "partition": "cortex"
   },
   "name": "ADx Documentum Connectivity Checks",
   "node": "adx@DESKTOP-J07TD08#200720151955598c4700f3a48d468c9f",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "14",
    "elapsedTimeInMs": 11.4634,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "15",
      "checkStatus": "ok",
      "details": "No deployed connections found.",
      "detailsAsMarkdown": false,
      "name": "ADx CMIS Connections"
     }
    ]
   },
   "status": "ok",
   "weight": "under10s"
  },
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "16",
   "check": {"_type": "tribefire.adx.model.deployment.healthcheck.FunctionalityCheckProcessor", "_id": "17",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "externalId": "adx.check.functionality",
    "globalId": "wire://AdxInitializerWireModule/AdxInitializerDataSpace/functionalityCheckProcessor",
    "id": "wire://AdxInitializerWireModule/AdxInitializerDataSpace/functionalityCheckProcessor",
    "name": "ADx Functionality Check Processor",
    "partition": "cortex"
   },
   "coverage": "functional",
   "isPlatformRelevant": false,
   "module": {"_ref": "8"},
   "name": "ADx Functionality Checks",
   "node": "adx@DESKTOP-J07TD08#200720151955598c4700f3a48d468c9f",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "18",
    "elapsedTimeInMs": 16.1388,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "19",
      "checkStatus": "ok",
      "details": "Repository | Info\n--- | ---\nTotal | 2\nActive | 1\nInactive | 1\nSTANDARD (active) | 1\n",
      "detailsAsMarkdown": true,
      "name": "Repositories"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "20",
      "checkStatus": "ok",
      "details": "Number of Hibernate Accesses of active repositories: 2",
      "detailsAsMarkdown": false,
      "name": "Hibernate Accesses"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "21",
      "checkStatus": "ok",
      "details": "Type | Repository | Path | Exists\n--- | --- | --- | ---\nStandard | Default | C:\\Users\\Grzeynick\\Documents\\adx23\\repository\\content | true\nStandard (Cache) | Default | C:\\Users\\Grzeynick\\Documents\\adx23\\repository\\cache | true\n",
      "detailsAsMarkdown": true,
      "name": "File Systems"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "22",
      "checkStatus": "ok",
      "details": "Repository | Deployment Status | Details\n--- | --- | ---\nDefault |  ok | 35 Deployables\n",
      "detailsAsMarkdown": true,
      "name": "Deployment Status"
     }
    ]
   },
   "status": "ok",
   "weight": "under1s"
  },
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "23",
   "check": {"_type": "tribefire.adx.model.deployment.repository.check.AdxGcpConnectivityCheckProcessor", "_id": "24",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "externalId": "processor.adx.gcp-connectivity-check",
    "globalId": "wire://AdxGcpInitializerWireModule/AdxGcpInitializerDataSpace/connectivityCheckProcessor",
    "id": "wire://AdxGcpInitializerWireModule/AdxGcpInitializerDataSpace/connectivityCheckProcessor",
    "module": {"_type": "com.braintribe.model.deployment.Module", "_id": "25",
     "bindsDeployables": true,
     "bindsHardwired": false,
     "bindsInitializers": false,
     "bindsWireContracts": false,
     "globalId": "module://tribefire.adx.phoenix:adx-gcp-module",
     "id": "module://tribefire.adx.phoenix:adx-gcp-module",
     "name": "adx-gcp-module",
     "partition": "cortex"
    },
    "name": "ADx GCP Connectivity Check Processor",
    "partition": "cortex"
   },
   "coverage": "connectivity",
   "isPlatformRelevant": false,
   "module": {"_ref": "25"},
   "name": "ADx GCP Connectivity Checks",
   "node": "adx@DESKTOP-J07TD08#200720151955598c4700f3a48d468c9f",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "26",
    "elapsedTimeInMs": 339.5783,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "27",
      "checkStatus": "ok",
      "details": "No deployed connections found.",
      "detailsAsMarkdown": false,
      "name": "ADx GCP Connections"
     }
    ]
   },
   "status": "ok",
   "weight": "under10s"
  },
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "28",
   "check": {"_type": "com.braintribe.model.aws.deployment.processor.HealthCheckProcessor", "_id": "29",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "externalId": "aws.healthcheck",
    "globalId": "wire://AwsInitializerModuleWireModule/AwsInitializerModuleSpace/healthCheckProcessor",
    "id": "wire://AwsInitializerModuleWireModule/AwsInitializerModuleSpace/healthCheckProcessor",
    "module": {"_type": "com.braintribe.model.deployment.Module", "_id": "30",
     "bindsDeployables": true,
     "bindsHardwired": false,
     "bindsInitializers": false,
     "bindsWireContracts": false,
     "globalId": "module://tribefire.extension.aws:aws-module",
     "id": "module://tribefire.extension.aws:aws-module",
     "name": "aws-module",
     "partition": "cortex"
    },
    "name": "AWS Check Processor",
    "partition": "cortex"
   },
   "coverage": "connectivity",
   "isPlatformRelevant": false,
   "module": {"_ref": "30"},
   "name": "AWS Checks",
   "node": "adx@DESKTOP-J07TD08#200720151955598c4700f3a48d468c9f",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "31",
    "elapsedTimeInMs": 0.9973,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "32",
      "checkStatus": "ok",
      "details": "No deployed connections found.",
      "detailsAsMarkdown": false,
      "name": "S3 Connections"
     }
    ]
   },
   "status": "ok",
   "weight": "under1s"
  },
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "33",
   "check": {"_type": "com.braintribe.model.extensiondeployment.check.HardwiredCheckProcessor", "_id": "34",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "externalId": "checkProcessor.hardwired.BaseConnectivityCheckProcessor",
    "globalId": "hardwired:check/BaseConnectivityCheckProcessor",
    "id": "hardwired:check/BaseConnectivityCheckProcessor",
    "module": {"_type": "com.braintribe.model.deployment.Module", "_id": "35",
     "bindsDeployables": false,
     "bindsHardwired": false,
     "bindsInitializers": false,
     "bindsWireContracts": false,
     "globalId": "module://platform",
     "id": "module://platform",
     "name": "tribefire-web-platform",
     "partition": "cortex"
    },
    "name": "Base Connectivity Check Processor",
    "partition": "cortex"
   },
   "coverage": "connectivity",
   "isPlatformRelevant": false,
   "name": "Base Platform Connectivity Checks",
   "node": "adx@DESKTOP-J07TD08#200720151955598c4700f3a48d468c9f",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "36",
    "elapsedTimeInMs": 11.1001,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "37",
      "checkStatus": "ok",
      "details": "Type | Valid | Check duration\n--- | --- | ---\nLeadership | true | 1 ms\nLock | true | 0 ms\n",
      "detailsAsMarkdown": true,
      "name": "System Databases"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "38",
      "checkStatus": "ok",
      "details": "Name | Idle Connections | Threads Waiting | Active Connections | Total Connections | Max Pool Size\n--- | --- | --- | --- | --- | ---\nconnection.user-sessions|1|0|0|1|20\nconnection.default|2|0|1|3|20\nconnection.auth|1|0|0|1|10\nconnection.locking|1|0|0|1|10\nconnection.adx.cache.default|0|1|0|0|20\nconnection.leadership|1|0|0|1|5\nconnection.user-session-statistics|1|0|0|1|10\nconnection.adx.content.default|0|0|0|0|20\n",
      "detailsAsMarkdown": true,
      "name": "Connection Pools"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "39",
      "checkStatus": "ok",
      "details": "Name | Message Transit Duration\n--- | ---\nDMB Messaging | 6 ms 26100 ns\n",
      "detailsAsMarkdown": true,
      "name": "Messaging"
     }
    ]
   },
   "status": "ok"
  },
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "40",
   "check": {"_type": "com.braintribe.model.extensiondeployment.check.HardwiredCheckProcessor", "_id": "41",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "externalId": "checkProcessor.hardwired.BaseFunctionalityCheckProcessor",
    "globalId": "hardwired:check/BaseFunctionalityCheckProcessor",
    "id": "hardwired:check/BaseFunctionalityCheckProcessor",
    "module": {"_ref": "35"},
    "name": "Base Functionality Check Processor",
    "partition": "cortex"
   },
   "coverage": "functional",
   "isPlatformRelevant": false,
   "name": "Base Platform Functionality Checks",
   "node": "adx@DESKTOP-J07TD08#200720151955598c4700f3a48d468c9f",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "42",
    "elapsedTimeInMs": 5.1022,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "43",
      "checkStatus": "ok",
      "details": "---|---\nLock Manager|DB Lock Manager\n",
      "detailsAsMarkdown": true,
      "name": "Lock Check"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "44",
      "checkStatus": "ok",
      "details": "---|---\nLeadership Manager|DB Leadership Manager\n",
      "detailsAsMarkdown": true,
      "name": "Leadership Check"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "45",
      "checkStatus": "ok",
      "details": "---|---\nIssued to|Braintribe Technology GmbH\nIssued by|Braintribe\nExpiration|01 Apr 2021\n",
      "detailsAsMarkdown": true,
      "name": "License"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "46",
      "checkStatus": "ok",
      "details": "Name|Active Threads|Total Executions|Average Execution Time|Pool Size|Core Pool Size|Maximum Pool Size|Pending Tasks in Queue|Average Pending Time in Queue|Time Since Last Execution\n---|---|---|---|---|---|---|---|---|---\nADx Documents Cleanup|0|0|0 ms|0|3|3|0|0 ms|0 ms\nADx Generic|0|0|0 ms|0|10|10|0|0 ms|0 ms\nActivation|0|1|550 ms|0|5|5|0|0.53 ms|2 d\nCartridge Evaluator|0|0|0 ms|0|40|40|0|0 ms|0 ms\nCheckBundleExecution|5|324|83 ms|5|5|20|3|16.53 ms|2 ms\nDatabase Connection Check|0|0|0 ms|0|1|10|0|0 ms|0 ms\nDmb Message Consumer (adx.tf.queue.trustedRequest)|1|1|0 ms|1|1|1|0|0 ms|2 d\nDmb Message Consumer (adx.tf.topic.heartbeat)|1|1|0 ms|1|1|1|0|0 ms|2 d\nDmb Message Consumer (adx.tf.topic.multicastRequest)|1|1|0 ms|1|1|1|0|0 ms|2 d\nDmb Message Consumer (adx.tf.topic.multicastResponse)|1|1|0 ms|1|1|1|0|0 ms|2 d\nDocument Retrieval|0|0|0 ms|0|50|50|0|0 ms|0 ms\nDocuments Cleanup|0|0|0 ms|0|3|3|0|0 ms|0 ms\nDocuments Download|0|0|0 ms|0|50|50|0|0 ms|0 ms\nDocuments-Conversion Upload|0|0|0 ms|0|30|30|0|0 ms|0 ms\nJava Common Pool|0|-1|-1 ms|0|0|0|0| |0 ms\nJob Cleanup|0|0|0 ms|0|3|3|0|0 ms|0 ms\nLeadership Daemon|0|145511|255 ms|1|1|unbound|1|0 ms|89 ms\nMBean Message Reaper|1|1|0 ms|1|1|1|0|0 ms|2 d\nMaster Heartbeat Sender|0|14636|0 ms|1|1|unbound|1|0 ms|5 s\nMaster Messaging Service|0|0|0 ms|0|20|20|0|0 ms|0 ms\nMaster Multicast Consumer|1|1|0 ms|1|20|20|0|0 ms|2 d\nMaster RPC Service|0|40|737 ms|20|20|unbound|0|0.38 ms|4 h\nPlatform Thread-Pool|7|21|1408 ms|7|100|100|0|0.61 ms|2 d\nRemote Evaluator|0|1|642 ms|0|100|100|0|0.38 ms|1 d\nScheduled System Executor|2|14742|8 ms|10|10|unbound|2|0.68 ms|1 ms\nScheduled Thread Pool|0|245|1 ms|2|5|unbound|1|0 ms|27 s\nState Change Processing (Aspect)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Audit Workbench)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Audit)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Authentication and Authorization Workbench)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Cortex Workbench)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Cortex)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Incremental Access Listener)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Platform Setup Workbench)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Platform Setup)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (User Sessions Workbench)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (User Statistics Workbench)|0|0|0 ms|0|20|20|0|0 ms|0 ms\nState Change Processing (Workbench)|0|0|0 ms|0|20|20|0|0 ms|0 ms\n",
      "detailsAsMarkdown": true,
      "name": "Thread Pools"
     }
    ]
   },
   "status": "ok"
  },
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "47",
   "check": {"_type": "com.braintribe.model.extensiondeployment.check.HardwiredCheckProcessor", "_id": "48",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "externalId": "checkProcessor.hardwired.BaseVitalityCheckProcessor",
    "globalId": "hardwired:check/BaseVitalityCheckProcessor",
    "id": "hardwired:check/BaseVitalityCheckProcessor",
    "module": {"_ref": "35"},
    "name": "Base Vitality Check Processor",
    "partition": "cortex"
   },
   "coverage": "vitality",
   "isPlatformRelevant": true,
   "name": "Base Platform Vitality Checks",
   "node": "adx@DESKTOP-J07TD08#200720151955598c4700f3a48d468c9f",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "49",
    "elapsedTimeInMs": 0.131,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "50",
      "checkStatus": "ok",
      "details": "Check infrastructure is ok",
      "detailsAsMarkdown": false,
      "name": "Base Check"
     }
    ]
   },
   "status": "ok"
  },
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "51",
   "check": {"_type": "com.braintribe.model.documents.HealthCheckProcessor", "_id": "52",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "documentsAccessId": "access.adx.cache.default",
    "documentsProcessorId": "custom:documents.serviceProcessor",
    "externalId": "health-check-processor.tribefire.adx.phoenix.adx-cartridge/-cache-space/documents-acces-template-context/repository/default.documents-templates-space.health-check-processor",
    "globalId": "wire://tribefire.adx.phoenix.adx-cartridge/CacheSpace/documentsAccesTemplateContext/repository/default/DocumentsTemplatesSpace/healthCheckProcessor",
    "id": "wire://tribefire.adx.phoenix.adx-cartridge/CacheSpace/documentsAccesTemplateContext/repository/default/DocumentsTemplatesSpace/healthCheckProcessor",
    "module": {"_type": "com.braintribe.model.deployment.Module", "_id": "53",
     "bindsDeployables": true,
     "bindsHardwired": false,
     "bindsInitializers": false,
     "bindsWireContracts": false,
     "globalId": "module://tribefire.extension.documents:documents-module",
     "id": "module://tribefire.extension.documents:documents-module",
     "name": "documents-module",
     "partition": "cortex"
    },
    "name": "Documents Check Processor default",
    "partition": "cortex"
   },
   "coverage": "functional",
   "isPlatformRelevant": false,
   "module": {"_ref": "53"},
   "name": "Documents Checks",
   "node": "adx@DESKTOP-J07TD08#200720151955598c4700f3a48d468c9f",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "54",
    "elapsedTimeInMs": 78.3109,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "55",
      "checkStatus": "ok",
      "details": "Checking availability of Documents Access access.adx.cache.default.",
      "detailsAsMarkdown": false,
      "name": "Documents Access Check"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "56",
      "checkStatus": "ok",
      "details": "Jobs currently processing/waiting: 0",
      "detailsAsMarkdown": false,
      "name": "Open Jobs"
     }
    ]
   },
   "status": "ok",
   "weight": "under1s"
  },
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "57",
   "check": {"_type": "com.braintribe.model.elasticsearchdeployment.service.HealthCheckProcessor", "_id": "58",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "externalId": "elasticsearch.healthzProcessor",
    "globalId": "wire://ElasticsearchInitializerModuleWireModule/ElasticsearchInitializerModuleSpace/healthCheckProcessor",
    "id": "wire://ElasticsearchInitializerModuleWireModule/ElasticsearchInitializerModuleSpace/healthCheckProcessor",
    "module": {"_type": "com.braintribe.model.deployment.Module", "_id": "59",
     "bindsDeployables": true,
     "bindsHardwired": false,
     "bindsInitializers": false,
     "bindsWireContracts": false,
     "globalId": "module://tribefire.extension.elastic:elasticsearch-module",
     "id": "module://tribefire.extension.elastic:elasticsearch-module",
     "name": "elasticsearch-module",
     "partition": "cortex"
    },
    "name": "Elastic Check Processor",
    "partition": "cortex"
   },
   "coverage": "functional",
   "isPlatformRelevant": false,
   "module": {"_ref": "59"},
   "name": "Elastic Checks",
   "node": "adx@DESKTOP-J07TD08#200720151955598c4700f3a48d468c9f",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "60",
    "elapsedTimeInMs": 3.8001,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "61",
      "checkStatus": "ok",
      "details": "Name | Address | Connectivity\n--- | --- | ---\nADx Content ElasticSearch Connection: \tDefault | 127.0.0.1:9300 | true\n",
      "detailsAsMarkdown": true,
      "name": "Check Elastic Service Availability"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "62",
      "checkStatus": "ok",
      "details": "Name | Index | Active Workers | Enqueued | Max. Active Runtime | Packages Indexed | Entities Indexed\n--- | --- | --- | --- | --- | --- | ---\nworker.adx.content-elasticsearch.default | elastic-index-default | 0/2 | 0/1000 | 0 ms | 2 | 2\n",
      "detailsAsMarkdown": true,
      "name": "Elastic Indexing Workers"
     }
    ]
   },
   "status": "ok",
   "weight": "under1s"
  },
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "63",
   "check": {"_type": "com.braintribe.model.gcp.deployment.HealthCheckProcessor", "_id": "64",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "externalId": "gcp.healthcheck",
    "globalId": "wire://GcpInitializerModuleWireModule/GcpInitializerModuleSpace/healthCheckProcessor",
    "id": "wire://GcpInitializerModuleWireModule/GcpInitializerModuleSpace/healthCheckProcessor",
    "module": {"_type": "com.braintribe.model.deployment.Module", "_id": "65",
     "bindsDeployables": true,
     "bindsHardwired": false,
     "bindsInitializers": false,
     "bindsWireContracts": false,
     "globalId": "module://tribefire.extension.gcp:gcp-module",
     "id": "module://tribefire.extension.gcp:gcp-module",
     "name": "gcp-module",
     "partition": "cortex"
    },
    "name": "GCP Check Processor",
    "partition": "cortex"
   },
   "coverage": "connectivity",
   "isPlatformRelevant": false,
   "module": {"_ref": "65"},
   "name": "GCP Checks",
   "node": "adx@DESKTOP-J07TD08#200720151955598c4700f3a48d468c9f",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "66",
    "elapsedTimeInMs": 2.615,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "67",
      "checkStatus": "ok",
      "details": "No deployed connections found.",
      "detailsAsMarkdown": false,
      "name": "GCP Connections"
     }
    ]
   },
   "status": "ok",
   "weight": "under1s"
  },
  {"_type": "com.braintribe.model.deploymentapi.check.data.aggr.CheckBundleResult", "_id": "68",
   "check": {"_type": "com.braintribe.model.wopi.service.WacHealthCheckProcessor", "_id": "69",
    "autoDeploy": true,
    "deploymentStatus": "deployed",
    "externalId": "wac-health-check-processor.repository-default.wopi-templates-space.wac-health-check-processor",
    "globalId": "wire://WopiTemplatesSpace/wacHealthCheckProcessor/repository/default",
    "id": "wire://WopiTemplatesSpace/wacHealthCheckProcessor/repository/default",
    "module": {"_type": "com.braintribe.model.deployment.Module", "_id": "70",
     "bindsDeployables": true,
     "bindsHardwired": false,
     "bindsInitializers": false,
     "bindsWireContracts": false,
     "globalId": "module://tribefire.extension.wopi:wopi-module",
     "id": "module://tribefire.extension.wopi:wopi-module",
     "name": "wopi-module",
     "partition": "cortex"
    },
    "name": "WOPI WAC Health Check - Repository/default",
    "partition": "cortex",
    "wopiWacConnector": {"_type": "com.braintribe.model.wopi.connector.WopiWacConnector", "_id": "71",
     "autoDeploy": true,
     "connectTimeoutInMs": 2000,
     "connectionRequestTimeoutInMs": 2000,
     "connectionRetries": 3,
     "delayOnRetryInMs": 1000,
     "deploymentStatus": "deployed",
     "externalId": "wopi-wac-connector.repository-default.wopi-templates-space.wopi-wac-connector",
     "globalId": "wire://WopiTemplatesSpace/wopiWacConnector/repository/default",
     "id": "wire://WopiTemplatesSpace/wopiWacConnector/repository/default",
     "module": {"_ref": "70"},
     "name": "WOPI WAC Connector - Repository/default",
     "partition": "cortex",
     "socketTimeoutInMs": 2000,
     "wacDiscoveryEndpoint": "http://wopi.agile-documents.com/hosting/discovery"
    }
   },
   "coverage": "connectivity",
   "isPlatformRelevant": false,
   "module": {"_ref": "70"},
   "name": "WOPI Checks",
   "node": "adx@DESKTOP-J07TD08#200720151955598c4700f3a48d468c9f",
   "result": {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "72",
    "elapsedTimeInMs": 1225.1377,
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "73",
      "checkStatus": "ok",
      "details": "Checks connectivity to 'http://wopi.agile-documents.com/hosting/discovery' endpoint",
      "detailsAsMarkdown": false,
      "name": "WacHealthCheckProcessor"
     }
    ]
   },
   "status": "ok",
   "weight": "under1s"
  }
 ],
 "status": "ok"

Additional Endpoints for Monitoring ADx Status

ADx currently offers the following ways to gain internal information about ADx functioning.

Note: These calls are resource-intensive and should not be executed periodically too frequently (e.g., every second would create too much load on the system, but maybe once every 1-10 min should be fine).

Every call to the following REST URLs requires a valid session ID that can be obtained prior to that using the /authenticate endpoint.

curl -X POST "https://[host]:[port]/services/api/v1/authenticate" -H  "accept: application/json" -H  "Content-Type: multipart/form-data" -F "user=[username]" -F "password=[password]"

If you need information from a single node in a cluster, the following REST calls (depicted as cURL commands) can be used:

For ADx versions prior to 2.5:

curl -H "Cookie: tfsessionId=[sessionId]" -X POST 'https://[host]:[port]/services/api/v1/serviceDomain%3Aplatform/com.braintribe.model.platformreflection.request.ReflectPlatform'

For ADx version 2.6:

curl -H "Cookie: tfsessionId=[sessionId]" -X GET "https://[host]:[port]/services/api/v1/monitoring/platform-reflect" -H  "accept: application/json"

There is also a way to ask any node in the cluster (e.g., via the load balancer) to get the same information for all nodes in the cluster in a single JSON combined. This is only supported in ADx 2.6 and later versions.

curl -H "Cookie: tfsessionId=[sessionId]" -X GET https://[host]:[port]/services/about?type=json

Response Interpretation

The services mentioned above return information about the internal status of ADx in a JSON structure. This JSON structure is big and it might help to understand it's structure by taking a look at the About page, where the same information is presented in a user-friendlier HTML page. See Monitoring ADx Status - ADx About Page.

To monitor ADx status on the ADx About page, log in to ADx, and then click About under Runtime.

In the figure below, the About page shows information about the CPU, memory, and hardware. You can click the links on the left side to show information about the system or threads. You can also use these links to download the Diagnostics package.

Here are a few guidelines:

For each node in the JSON structure, three main sub-sections are provided:

System (operational information - e.g., information about the hardware and operating system ADx is running on, disks, filesystems, network, database connections, memory usage, thread pools, etc)
Host (information about the application server that ADx/TF is running on; e.g., active connections, connection pools, thread pools, etc.)
Tribefire (mostly static information about the ADx/TF installation; e.g., version numbers, runtime properties, etc)

Each of the entries in the following JSON can be used to monitor the liveliness of the system. Here are some examples:

Hardware / OS / System:

CPU Load: <node>.system.cpu.cpuLoad.totalCpu
Available Memory: <node>.system.memory.available
Available disk space: <node>.system.fileSystems [*] .freeSpaceInBytes
Active DB connections: <node>.system.databaseInformation.combinedConnectionInfo [*] .metrics.activeConnections
Thread count: <node>.system.javaEnvironment.threadCount
Virtual Memory size: <node>.system.javaEnvironment.virtualSize
RSS (Resident Set Size): <node>.system.javaEnvironment.residentSetSize
Active threads in Threadpools: <node>.system.threadPools.threadPools [*] .activeThreads
Average pending time in thread pools: <node>.system.threadPools.threadPools [*] .averagePendingTimeInMs

Example:

"cpu": {"_type": "com.braintribe.model.platformreflection.check.cpu.Cpu", "_id": "15",
   "cpu64bit": true,
   "cpuLoad": {"_type": "com.braintribe.model.platformreflection.check.cpu.CpuLoad", "_id": "16",
    "contextSwitches": 673244914363,
    "globalId": null,
    "id": null,
    "idle": 65.9,
    "interrupts": 36371309142,
    "ioWait": 0.0,
    "irq": 0.0,
    "nice": 0.13,
    "partition": null,
    "softIrq": 0.26,
    "steal": 0.0,
    "sys": 3.45,
    "systemCpuLoad": null,
    "systemLoadAverage15Minutes": 2.35,
    "systemLoadAverage1Minute": 3.28,
    "systemLoadAverage5Minutes": 3.01,
    "systemLoadPerProcessor": [
     20.83,
     19.59,
     29.29,
     17.35,
     50.0,
     100.0,
     25.0,
     8.33
    ],
    "totalCpu": 34.1,
    "user": 30.27
   },
   "cpuTemperature": 0.0,
   "cpuVoltage": 0.0,
   "currentFrequencies": [
    "2.2 GHz",
    "2.2 GHz",
    "2.2 GHz",
    "2.2 GHz",
    "2.2 GHz",
    "2.2 GHz",
    "2.2 GHz"
   ],
   "family": "6",
   "fanSpeeds": null,
   "globalId": null,
   "id": null,
   "identifier": "Intel64 Family 6 Model 79 Stepping 0",
   "logicalProcessorCount": 8,
   "maxFreq": 2200000000,
   "maxFreqInGh": 2.2,
   "model": "79",
   "name": "Intel(R) Xeon(R) CPU @ 2.20GHz",
   "partition": null,
   "physicalProcessorCount": 4,
   "processorId": "0F81FBFF004006F0",
   "systemBootTime": "2020-07-29T22:58:20.044+0000",
   "systemSerialNumber": "unknown",
   "vendor": "GenuineIntel",
   "vendorFreq": 2200000000,
   "vendorFreqInGh": 2.2
  },
  "databaseInformation": {"_type": "com.braintribe.model.platformreflection.db.DatabaseInformation", "_id": "17",
   "combinedConnectionInfo": [
    {"_type": "com.braintribe.model.platformreflection.db.DatabaseConnectionPool", "_id": "18",
     "connectionDescription": "connection.adx.content.gerry, connection.adx.cache.cmis-momcilo, connection.adx.content.mydemorepo, connection.adx.cache.gcs-marko, connection.adx.content.rku-playground-gcp, connection.adx.cache.case-management, connection.adx.cache.swissre-claims, connection.adx.cache.rverepo, connection.adx.content.gkpoliciesdemo, connection.adx.content.removeme3, connection.adx.cache.nor-test, connection.adx.content.migration, connection.adx.cache.spr-test-repo, connection.adx.content.gk, connection.adx.cache.gsc-db, connection.adx.cache.radotestrepo, connection.adx.cache.rku-playground-gcp, connection.adx.cache.renditiontest, connection.adx.content.spr-test-repo, connection.adx.content.swissre-claims, connection.adx.content.radotestrepo, connection.adx.cache.dctm, connection.adx.cache.nikolacmis, connection.adx.cache.migration, connection.adx.cache.media-test, connection.adx.content.rku-playground, connection.adx.cache.default, connection.adx.content.case-management, connection.adx.cache.gkpoliciesdemo, connection.adx.content.default, connection.adx.content.gcs-marko, connection.adx.content.media-test, connection.adx.content.rverepo, connection.adx.content.nor-test, connection.adx.cache.s3-marko, connection.adx.cache.mydemorepo, storage.db.connection, connection.adx.cache.cmis, connection.adx.cache.gerry, connection.adx.content.gsc-db, connection.adx.content.gktrace, connection.adx.cache.gktrace, connection.adx.cache.removeme3, connection.adx.content.s3-marko, connection.adx.cache.rku-playground, connection.adx.cache.gk",
     "connectionTimeout": null,
     "globalId": null,
     "id": null,
     "idleTimeout": null,
     "maxLifetime": null,
     "maximumPoolSize": 950,
     "metrics": {"_type": "com.braintribe.model.platformreflection.db.DatabaseConnectionPoolMetrics", "_id": "19",
      "activeConnections": 0,
      "globalId": null,
      "id": null,
      "idleConnections": 2,
      "leaseCount": 7907,
      "partition": null,
      "threadsAwaitingConnections": 0,
      "totalConnections": 2,
      "usageMaxTime": null,
      "usageMeanTime": null,
      "usageMedianTime": null,
      "usageMinTime": null,
      "waitTimeFifteenMinutesRate": null,
      "waitTimeFiveMinutesRate": null,
      "waitTimeMeanRate": null,
      "waitTimeOneMinuteRate": null
     },
     "minimumPoolSize": 2,
     "name": "35.198.165.135:5432",
     "partition": null
    }

Host:

Connection count: <node>.host.threadPools.threadPools [*] .connectionCount
Busy threads: <node>.host.threadPools.threadPools [*] .currentThreadsBusy

Example:

[
{  "master@phoenix-dev-tribefire-master-f49d8c48c-cbbrd--RND-eXMF0dh4G3" :
{"_type": "com.braintribe.model.platformreflection.PlatformReflection", "_id": "0",
 "globalId": null,
 "host": {"_type": "com.braintribe.model.platformreflection.host.tomcat.TomcatHostInfo", "_id": "1",
  "connectors": [
   {"_type": "com.braintribe.model.platformreflection.host.tomcat.Connector", "_id": "2",
    "acceptCount": 100,
    "bytesReceived": 0,
    "bytesSent": 0,
    "connectionLinger": -1,
    "connectionTimeout": 120000,
    "errorCount": 0,
    "globalId": null,
    "id": null,
    "keepAliveTimeout": 120000,
    "localPort": 8443,
    "maxHeaderCount": 100,
    "maxParameterCount": 10000,
    "maxPostSize": 2097152,
    "maxSavePostSize": 4096,
    "maxSwallowSize": 2097152,
    "maxThreads": 5000,
    "maxTime": 0,
    "minSpareThreads": 10,
    "packetSize": null,
    "partition": null,
    "port": 8443,
    "processingTime": 0,
    "processorCache": 200,
    "protocol": "HTTP/1.1",
    "proxyName": null,
    "proxyPort": 0,
    "redirectPort": 443,
    "requestCount": 0,
    "scheme": "https",
    "secure": true,
    "sslProtocols": null,
    "stateName": "Started",
    "tcpNoDelay": false,
    "threadPriority": 5,
    "uriEncoding": "UTF-8"
   },
   {"_type": "com.braintribe.model.platformreflection.host.tomcat.Connector", "_id": "3",
    "acceptCount": 100,
    "bytesReceived": 204909,
    "bytesSent": 16363779,
    "connectionLinger": -1,
    "connectionTimeout": 120000,
    "errorCount": 64,
    "globalId": null,
    "id": null,
    "keepAliveTimeout": 120000,
    "localPort": 8080,
    "maxHeaderCount": 100,
    "maxParameterCount": 10000,
    "maxPostSize": 2097152,
    "maxSavePostSize": 4096,
    "maxSwallowSize": 2097152,
    "maxThreads": 5000,
    "maxTime": 8903,
    "minSpareThreads": 10,
    "packetSize": null,
    "partition": null,
    "port": 8080,
    "processingTime": 78665,
    "processorCache": 200,
    "protocol": "HTTP/1.1",
    "proxyName": null,
    "proxyPort": 0,
    "redirectPort": 443,
    "requestCount": 4997,
    "scheme": "http",
    "secure": false,
    "sslProtocols": null,
    "stateName": "Started",
    "tcpNoDelay": false,
    "threadPriority": 5,
    "uriEncoding": "UTF-8"
   }
  ],
  "engine": {"_type": "com.braintribe.model.platformreflection.host.tomcat.Engine", "_id": "4",
   "baseDir": null,
   "defaultHost": "localhost",
   "globalId": null,
   "id": null,
   "name": "Catalina",
   "partition": null,
   "startStopThreads": 1,
   "stateName": "Started"
  },
  "globalId": null,
  "hostBasePath": "/opt/tribefire/runtime/host",
  "hostHomePath": "/opt/tribefire/runtime/host",
  "hostType": "Apache Tomcat",
  "hostVersion": "9.0.43",
  "id": null,
  "partition": null,
  "ssl": [
   {"_type": "com.braintribe.model.platformreflection.host.tomcat.Ssl", "_id": "5",
    "ciphers": [
     "HIGH:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!kRSA"
    ],
    "globalId": null,
    "id": null,
    "keyAlias": null,
    "keyStoreFile": "/opt/tribefire/runtime/host/conf/keystore.p12",
    "keyStoreProvider": null,
    "keyStoreType": "PKCS12",
    "partition": null,
    "port": 8443,
    "protocol": "TLS",
    "trustStoreFile": null,
    "trustStoreProvider": null,
    "trustStoreType": "JKS"
   }
  ],
  "threadPools": {"_type": "com.braintribe.model.platformreflection.host.tomcat.ThreadPools", "_id": "6",
   "busyThreadCount": 1,
   "globalId": null,
   "id": null,
   "partition": null,
   "threadPools": [
    {"_type": "com.braintribe.model.platformreflection.host.tomcat.ThreadPool", "_id": "7",
     "connectionCount": 1,
     "currentThreadCount": 10,
     "currentThreadsBusy": 0,
     "globalId": null,
     "id": null,
     "maxThreadCount": 5000,
     "name": "https-jsse-nio-8443",
     "partition": null
    },
    {"_type": "com.braintribe.model.platformreflection.host.tomcat.ThreadPool", "_id": "8",
     "connectionCount": 10,
     "currentThreadCount": 23,
     "currentThreadsBusy": 1,
     "globalId": null,
     "id": null,
     "maxThreadCount": 5000,
     "name": "http-nio-8080",
     "partition": null
    }
   ],
   "totalConnectionCount": 11,
   "totalThreadCount": 33
  },
  "webapps": [
   {"_type": "com.braintribe.model.platformreflection.host.Webapp", "_id": "9",
    "globalId": null,
    "host": "localhost",
    "id": null,
    "name": "/",
    "partition": null,
    "stateName": "Started"
   }
  ]
 }
]

Note that there might be changes to this structure in future versions.

Basic Platform Health Check via /healthz Endpoint

This check is deprecated - we recommend to run ADx or Conversion-specific checks instead.

This check returns a response showing the status of the ADx or Conversion host, either in JSON, HTML or plain text.

Either in a local or distributed setup the status of an instance needs to be checked to decide if the instance is still alive or if it must be deactivated/replaced. The check API provides a standardized procedure to define and implement checks in order to obtain information about the status of a tribefire instance. Such checks may include:

Availability of database connections
Reachability of extensions
Thread Pools
Status of deployables, especially accesses

The platform provides the /healthz endpoint which is responsible for the execution of checks. The full URL is https://[host]:[port]/tribefire-services/healthz. By default, this prints an HTML report:

If you render the output, you get a human-readable report:

You can change the output to either JSON or plain text by passing a HTTP Accept header, as explained below in the sample call.

Sample Call using cURL

curl --request GET https://hostname:port/tribefire-services/healthz --header "Accept: application/json"

You might want to use --insecure when using HTTPS with self-signed certificates. Keep in mind that this option disables certificate validation.

HTTP Accept Header	Description
`application/json`	Prints out the report in JSON
`text/html`	Prints out the report in HTML
`text/plain`	Prints out the report in plain text

JSON Response

In case the accept header is set to application/json, the endpoint answers with a JSON structured as check results per node.

Sample output:

{"_type": "flatmap", "value":[
 {"_type": "com.braintribe.model.service.api.InstanceId", "_id": "0",
   "applicationId": "master",
   "nodeId": "tf@NB-VIE01-CWI02#200128101854615f41f7f263064628ab"
  },[
   {"_type": "com.braintribe.model.check.service.CheckResult", "_id": "1",
    "entries": [
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "2",
      "checkStatus": "ok",
      "details": "Check infrastructure is ok",
      "name": "Base Check"
     },
     {"_type": "com.braintribe.model.check.service.CheckResultEntry", "_id": "3",
      "checkStatus": "ok",
      "details": "Active Threads: 0, Total Executions: 1, Average Execution Time: 182 ms, Pool Size: 0, Core Pool Size: 5",
      "name": "Thread Pool: Activation"
     },
   }
}

The maps's key value defines the type InstanceId which reflects the node (applicationId@nodeId) that was responsible for the check execution.
The map's value defines the list of CheckResults. A CheckResult returns a list of CheckResultEntry.

A CheckResultEntry is qualified by:

Value	Description
`status`	Set to one of the following values: `ok`, `warn` or `fail`
`name`	The name of the executed check e.g. "DB Connectivity Check"
`message`	The summarized check result message.
`details`	Contains check result details like an exception stacktrace or further information of the check result.

HTTP Status Codes

If you're using a monitoring system, you might be interested in the different HTTP status codes returned.

A 200 OK is returned when all checks have passed, while a 503 Service Unavailable is returned when at least one check has failed.
If check result entries contain at least one warn or fail, the returned HTTP status code is always 503 Service Unavailable. For example, if you have 4 checks:

[ok, ok, ok, ok] the status is 200 OK
[ok, ok, ok, warn] the status is 503 Service Unavailable

Custom HTTP Status Codes

You can define a custom status code for when a warn is thrown. You can do this by adding the warnStatusCode=123 parameter to the URL when calling the endpoint. Sample call:

https://hostname:port/tribefire-services/healthz?warnStatusCode=123

If the parameter is set, the defined status code is returned in case a CheckResult results in a warn. If this parameter is not defined, the default HTTP status code 503 is used.

Running Checks From User Interface

For information on how to run Health Checks from ADx UI, see one of the following pages:

Running ADx, Conversion and Platform Health Checks via REST API

On this page

Scripted Platform Health Check

Conversion Status Check

Sample cURL Call

Response and Interpretation

ADx Status Check

Monitoring ADx Status with the /check Endpoint

Sample cURL Call

Response and Interpretation

Additional Endpoints for Monitoring ADx Status

Response Interpretation

Basic Platform Health Check via /healthz Endpoint

Sample Call using cURL

JSON Response

HTTP Status Codes

Custom HTTP Status Codes

Running Checks From User Interface