Metrics

The server keeps per-agent metrics in memory from startup until shutdown. Metrics are not written to the SQLite database.

Prerequisites

The caller’s source address must match allowed-ips, and requests must carry a valid admin token in the Authorization header. Query the endpoint from the server itself or from an authorized monitoring system.

Query Prometheus metrics

The Prometheus exposition endpoint is:

curl -H "Authorization: ADMIN_TOKEN_REPLACE_ME" https://goauld.example.com/metrics

It is protected by the server’s allowed-ips and admin-token settings.

Query JSON metrics

The equivalent JSON endpoint is:

curl -H "Authorization: ADMIN_TOKEN_REPLACE_ME" https://goauld.example.com/admin/metrics/

It returns one entry per agent observed since startup, including disconnected agents.

Recorded values and units

Metrics include:

• Bytes received from and sent to the agent on each agent-facing transport.
• Active transport and SSH-channel counts.
• Successful reconnect count.
• Transport failures exposed as separate per-transport and per-reason counts.
• The latest authenticated control-channel heartbeat round-trip time.

Byte counters measure bytes crossing the Goauld agent-facing SSH transport boundary, including SSH, multiplexing, keepalive, and channel payload traffic handled by that transport. Control-plane health is represented by heartbeat and reconnect metrics; Socket.IO framing is not included in the byte counters. These are application/transport byte counts rather than raw Ethernet or IP packet sizes.

Prometheus reference

Failure reasons and transports use different metric names because they are independent breakdowns. The metrics do not expose every transport/reason combination, and summing one family does not double-count the other.

A response contains lines such as:

goauld_agent_bytes_in_total{agent_id="agent-7f3a"} 182420
goauld_agent_active_channels{agent_id="agent-7f3a"} 2
goauld_agent_heartbeat_rtt_seconds{agent_id="agent-7f3a"} 0.042500
goauld_agent_transport_bytes_in_total{agent_id="agent-7f3a",transport="websocket"} 182420
goauld_agent_transport_active{agent_id="agent-7f3a",transport="websocket"} 1

Useful PromQL checks include:

rate(goauld_agent_reconnects_total[5m]) > 0
goauld_agent_heartbeat_rtt_seconds > 1
sum by (transport) (increase(goauld_agent_transport_failures_by_transport_total[15m])) > 0

JSON reference

The admin endpoint wraps its per-agent entries in an agents array. For example:

{
  "agents": [
    {
      "agent_id": "agent-7f3a",
      "agent_name": "operator@target",
      "connected": true,
      "active_channels": 2,
      "active_transports": 2,
      "bytes_in": 182420,
      "bytes_out": 94510,
      "reconnects": 1,
      "heartbeat_count": 86,
      "last_heartbeat_rtt": 42500000,
      "transport_failures": {
        "timeout": 1
      },
      "transports": {
        "websocket": {
          "bytes_in": 182420,
          "bytes_out": 94510,
          "failures": 1,
          "active": true
        },
        "control": {
          "bytes_in": 0,
          "bytes_out": 0,
          "failures": 0,
          "active": true
        }
      }
    }
  ]
}

The JSON byte and count fields are integers. last_heartbeat_rtt is a Go duration encoded in nanoseconds, so 42500000 represents 42.5 milliseconds. heartbeat_count is the number of authenticated heartbeat responses observed since server startup. connected represents the current agent control state; it is not inferred from byte activity. The control transport can appear in the transports object, but its byte fields remain zero because raw Socket.IO framing is not counted.

Lifetime and limitations

• Counters reset when the server process restarts.
• Disconnected agents remain in the in-memory snapshot until restart.
• Byte counters cover the SSH data transports, including channel payload and SSH protocol overhead; they do not include Socket.IO control framing.
• Heartbeat RTT measures the most recent authenticated Socket.IO ping/pong, not the latency of an interactive shell or file transfer.
• These values are application-level counters, not Ethernet or IP byte totals.