You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Issue observed
After installing the Oracle Integration, we do receive data from the instance via the Agent and the reported metrics are shown in the Datadog-provided "DBM Oracle Database Overview"-Dashboard:
For example, the very important metric User Transaction Per Sec is not used by the integration.
Our Oracle database can report the following metrics via GV$SYSMETRIC:
Execute Without Parse Ratio
Soft Parse Ratio
User Calls Ratio
Host CPU Utilization (%)
Network Traffic Volume Per Sec
Enqueue Timeouts Per Sec
Enqueue Timeouts Per Txn
Enqueue Waits Per Sec
Enqueue Waits Per Txn
Enqueue Deadlocks Per Sec
Enqueue Deadlocks Per Txn
Enqueue Requests Per Sec
Enqueue Requests Per Txn
DB Block Gets Per Sec
DB Block Gets Per Txn
Consistent Read Gets Per Sec
Consistent Read Gets Per Txn
DB Block Changes Per Sec
DB Block Changes Per Txn
Consistent Read Changes Per Sec
Consistent Read Changes Per Txn
CPU Usage Per Sec
CPU Usage Per Txn
CR Blocks Created Per Sec
CR Blocks Created Per Txn
CR Undo Records Applied Per Sec
CR Undo Records Applied Per Txn
User Rollback UndoRec Applied Per Sec
User Rollback Undo Records Applied Per Txn
Leaf Node Splits Per Sec
Leaf Node Splits Per Txn
Branch Node Splits Per Sec
Branch Node Splits Per Txn
PX downgraded 1 to 25% Per Sec
PX downgraded 25 to 50% Per Sec
PX downgraded 50 to 75% Per Sec
PX downgraded 75 to 99% Per Sec
PX downgraded to serial Per Sec
Physical Read Total IO Requests Per Sec
Physical Read Total Bytes Per Sec
GC CR Block Received Per Second
GC CR Block Received Per Txn
GC Current Block Received Per Second
GC Current Block Received Per Txn
Global Cache Average CR Get Time
Global Cache Average Current Get Time
Physical Write Total IO Requests Per Sec
Global Cache Blocks Corrupted
Global Cache Blocks Lost
Current Logons Count
Current Open Cursors Count
User Limit %
SQL Service Response Time
Database Wait Time Ratio
Database CPU Time Ratio
Response Time Per Txn
Row Cache Hit Ratio
Row Cache Miss Ratio
Library Cache Hit Ratio
Library Cache Miss Ratio
Shared Pool Free %
PGA Cache Hit %
Process Limit %
Session Limit %
Executions Per Txn
Executions Per Sec
Txns Per Logon
Database Time Per Sec
Physical Write Total Bytes Per Sec
Physical Read IO Requests Per Sec
Physical Read Bytes Per Sec
Physical Write IO Requests Per Sec
Physical Write Bytes Per Sec
DB Block Changes Per User Call
DB Block Gets Per User Call
Executions Per User Call
Logical Reads Per User Call
Total Sorts Per User Call
Total Table Scans Per User Call
Current OS Load
Streams Pool Usage Percentage
PQ QC Session Count
PQ Slave Session Count
Queries parallelized Per Sec
DML statements parallelized Per Sec
DDL statements parallelized Per Sec
PX operations not downgraded Per Sec
Session Count
Average Synchronous Single-Block Read Latency
I/O Megabytes per Second
I/O Requests per Second
Average Active Sessions
Active Serial Sessions
Active Parallel Sessions
Captured user calls
Replayed user calls
Workload Capture and Replay status
Background CPU Usage Per Sec
Background Time Per Sec
Host CPU Usage Per Sec
Cell Physical IO Interconnect Bytes
Temp Space Used
Total PGA Allocated
Total PGA Used by SQL Workareas
Run Queue Per Sec
VM in bytes Per Sec
VM out bytes Per Sec
Buffer Cache Hit Ratio
Total PGA Used by SQL Workareas
User Transaction Per Sec
Physical Reads Per Sec
Physical Reads Per Txn
Physical Writes Per Sec
Physical Writes Per Txn
Physical Reads Direct Per Sec
Physical Reads Direct Per Txn
Redo Generated Per Sec
Redo Generated Per Txn
Logons Per Sec
Logons Per Txn
User Calls Per Sec
User Calls Per Txn
Logical Reads Per Sec
Logical Reads Per Txn
Redo Writes Per Sec
Redo Writes Per Txn
Total Table Scans Per Sec
Total Table Scans Per Txn
Full Index Scans Per Sec
Full Index Scans Per Txn
Execute Without Parse Ratio
Soft Parse Ratio
Host CPU Utilization (%)
DB Block Gets Per Sec
DB Block Gets Per Txn
Consistent Read Gets Per Sec
Consistent Read Gets Per Txn
DB Block Changes Per Sec
DB Block Changes Per Txn
Consistent Read Changes Per Sec
Consistent Read Changes Per Txn
Database CPU Time Ratio
Library Cache Hit Ratio
Shared Pool Free %
Executions Per Txn
Executions Per Sec
Txns Per Logon
Database Time Per Sec
Average Active Sessions
Host CPU Usage Per Sec
Cell Physical IO Interconnect Bytes
Temp Space Used
Total PGA Allocated
Memory Sorts Ratio
It's probably easy to incorporate the additional metrics into the reporting.
Output of the info page Note: Our Agent in Kubernetes can't execute s6-svstat:
s6-svstat: command not found
$ agent status
2024-01-09 09:19:36 UTC | CLUSTER | WARN | (pkg/util/log/log.go:666 in func1) | Agent configuration relax permissions constraint on the secret backend cmd, Group can read and exec
2024-01-09 09:19:36 UTC | CLUSTER | INFO | (pkg/util/log/log.go:626 in func1) | 2 Features detected from environment: kubernetes,orchestratorexplorer
Getting the status from the agent.
===============================
Datadog Cluster Agent (v7.50.2)
===============================
Status date: 2024-01-09 09:19:36.358 UTC (1704791976358)
Agent start: 2024-01-08 19:59:50.748 UTC (1704743990748)
Pid: 1
Go Version: go1.20.12
Build arch: amd64
Agent flavor: cluster_agent
Check Runners: 4
Log Level: INFO
Paths
=====
Config File: /etc/datadog-agent/datadog-cluster.yaml
conf.d: /etc/datadog-agent/conf.d
Clocks
======
System time: 2024-01-09 09:19:36.358 UTC (1704791976358)
Hostnames
=========
host_aliases: [a573212b-6d81-4052-93aa-671bef947fca]
hostname: aks-system-27839219-vmss000001-aks-k8s-cluster-live
socket-fqdn: datadog-agent-cluster-agent-58cdf78bb6-724p8
socket-hostname: datadog-agent-cluster-agent-58cdf78bb6-724p8
hostname provider: container
unused hostname providers:
'hostname' configuration/environment: hostname is empty
'hostname_file' configuration/environment: 'hostname_file' configuration is not enabled
aws: not retrieving hostname from AWS: the host is not an ECS instance and other providers already retrieve non-default hostnames
azure: azure_hostname_style is set to 'os'
fargate: agent is not runnning on Fargate
fqdn: FQDN hostname is not usable
gce: unable to retrieve hostname from GCE: GCE metadata API error: status code 404 trying to GET http://169.254.169.254/computeMetadata/v1/instance/hostname
os: OS hostname is not usable
Metadata
========
Leader Election
===============
Leader Election Status: Running
Leader Name is: datadog-agent-cluster-agent-58cdf78bb6-724p8
Last Acquisition of the lease: Mon, 08 Jan 2024 20:00:19 UTC
Renewed leadership: Tue, 09 Jan 2024 09:19:35 UTC
Number of leader transitions: 622 transitions
Custom Metrics Server
=====================
Disabled: The external metrics provider is not enabled on the Cluster Agent
Cluster Checks Dispatching
==========================
Status: Leader, serving requests
Active agents: 1
Check Configurations: 3
- Dispatched: 3
- Unassigned: 0
Admission Controller
====================
Disabled: The admission controller is not enabled on the Cluster Agent
=========
Collector
=========
Running Checks
==============
kubernetes_apiserver
--------------------
Instance ID: kubernetes_apiserver [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/kubernetes_apiserver.yaml
Total Runs: 3,199
Metric Samples: Last Run: 0, Total: 0
Events: Last Run: 1, Total: 839
Service Checks: Last Run: 4, Total: 12,788
Average Execution Time : 1.968s
Last Execution Date : 2024-01-09 09:19:22 UTC (1704791962000)
Last Successful Execution Date : 2024-01-09 09:19:22 UTC (1704791962000)
orchestrator
------------
Instance ID: orchestrator:c640d4e943da6c1d [OK]
Configuration Source: file:/etc/datadog-agent/conf.d/orchestrator.d/conf.yaml.default
Total Runs: 4,799
Metric Samples: Last Run: 0, Total: 0
Events: Last Run: 0, Total: 0
Service Checks: Last Run: 0, Total: 0
Average Execution Time : 10ms
Last Execution Date : 2024-01-09 09:19:33 UTC (1704791973000)
Last Successful Execution Date : 2024-01-09 09:19:33 UTC (1704791973000)
=========
Forwarder
=========
Transactions
============
Cluster: 4,796
ClusterRole: 643
ClusterRoleBinding: 297
CronJob: 0
CustomResource: 0
CustomResourceDefinition: 0
DaemonSet: 481
Deployment: 660
Dropped: 0
HighPriorityQueueFull: 0
HorizontalPodAutoscaler: 0
Ingress: 1,653
Job: 0
Namespace: 4,796
Node: 3,238
OrchestratorManifest: 6,752
PersistentVolume: 456
PersistentVolumeClaim: 772
Pod: 3
ReplicaSet: 2,138
Requeued: 0
Retried: 0
RetryQueueSize: 0
Role: 389
RoleBinding: 339
Service: 818
ServiceAccount: 521
StatefulSet: 323
VerticalPodAutoscaler: 0
Transaction Successes
=====================
Total number: 35879
Successes By Endpoint:
check_run_v1: 3,199
intake: 406
orchestrator: 22,323
orchmanifest: 6,752
series_v2: 3,199
On-disk storage
===============
On-disk storage is disabled. Configure `forwarder_storage_max_size_in_bytes` to enable it.
==========
Endpoints
==========
https://app.datadoghq.eu - API Key ending with:
- 74b2d
=============
Autodiscovery
=============
Enabled Features
================
kubernetes
orchestratorexplorer
=====================
Orchestrator Explorer
=====================
Collection Status: The collection is at least partially running since the cache has been populated.
Cluster Name: aks-k8s-cluster-live
Cluster ID: 43d27346-bae4-4e3a-998a-e8b0e6cfcb9e
Container scrubbing: enabled
Manifest collection: enabled
======================
Orchestrator Endpoints
======================
https://orchestrator.datadoghq.eu - API Key ending with: 74b2d
===========
Cache Stats
===========
Elements in the cache: 1114
ClusterRoleBinding
Last Run: (Hits: 96 Miss: 0) | Total: (Hits: 435744 Miss: 24672)
ClusterRole
Last Run: (Hits: 114 Miss: 0) | Total: (Hits: 517446 Miss: 29298)
Cluster
Last Run: (Hits: 0 Miss: 1) | Total: (Hits: 0 Miss: 4796)
CronJob
Last Run: (Hits: 0 Miss: 0) | Total: (Hits: 0 Miss: 0)
DaemonSet
Last Run: (Hits: 10 Miss: 0) | Total: (Hits: 45331 Miss: 2629)
Deployment
Last Run: (Hits: 36 Miss: 0) | Total: (Hits: 163362 Miss: 9294)
Ingress
Last Run: (Hits: 40 Miss: 0) | Total: (Hits: 181522 Miss: 10318)
Job
Last Run: (Hits: 0 Miss: 0) | Total: (Hits: 0 Miss: 0)
Namespace
Last Run: (Hits: 24 Miss: 1) | Total: (Hits: 108917 Miss: 10983)
Node
Last Run: (Hits: 10 Miss: 1) | Total: (Hits: 52164 Miss: 5171)
PersistentVolumeClaim
Last Run: (Hits: 21 Miss: 0) | Total: (Hits: 95296 Miss: 5420)
PersistentVolume
Last Run: (Hits: 26 Miss: 0) | Total: (Hits: 117988 Miss: 6708)
Pod
Last Run: (Hits: 0 Miss: 0) | Total: (Hits: 31 Miss: 7)
ReplicaSet
Last Run: (Hits: 424 Miss: 0) | Total: (Hits: 1.928726e+06 Miss: 109576)
RoleBinding
Last Run: (Hits: 45 Miss: 0) | Total: (Hits: 204255 Miss: 11565)
Role
Last Run: (Hits: 44 Miss: 0) | Total: (Hits: 199716 Miss: 11308)
ServiceAccount
Last Run: (Hits: 99 Miss: 0) | Total: (Hits: 449262 Miss: 25542)
Service
Last Run: (Hits: 91 Miss: 1) | Total: (Hits: 417497 Miss: 23735)
StatefulSet
Last Run: (Hits: 11 Miss: 0) | Total: (Hits: 49928 Miss: 2828)
=====================
Manifest Buffer Stats
=====================
Buffer Flushed : 6752 times
Last Time Flushed Manifests : 5
==============================
Manifests Flushed Per Resource
==============================
ClusterRole : 29298
ClusterRoleBinding : 24672
DaemonSet : 2629
Deployment : 9294
Ingress : 10318
Namespace : 10983
Node : 5171
PersistentVolume : 6708
PersistentVolumeClaim : 5420
Pod : 7
ReplicaSet : 109576
Role : 11308
RoleBinding : 11565
Service : 23735
ServiceAccount : 25542
StatefulSet : 2828
Additional environment details (Operating System, Cloud provider, etc):
We're running Agent (v7.50.2) in Azure Kubernetes, installed via Helm (Chart v3.50.2).
We're using Oracle 19 for our RDBMS.
Steps to reproduce the issue:
Have an Oracle 19
Install the Agent via Kubernetes
Configure the Agent to collect Oracle metrics
See a half-empty dashboard
Describe the results you received:
The Oracle integration reports a few metrics
Describe the results you expected:
The Oracle integration reports all metrics.
Additional information you deem important (e.g. issue happens only occasionally):
The text was updated successfully, but these errors were encountered:
Got an update on this one:
Apparently, there is a "new" Oracle monitoring integration (oracle-dbm). This integration records way more metrics than the oracle integration written in Python.
After moving to the oracle-dbm integration and enabling dbm: true in the configuration, we see all reported values.
The dd_session view requires reading from the "x"-tables x$ksuse, x$kslwt and x$ksled but it's impossible to grant a select from those tables to the datadog-user. Hence, it's necessary to create views onto those tables and grant select permissions to the datadog user with the sysdba-user:
createviewv_x$ksuse asselect*from x$ksuse;
createviewv_x$kslwt asselect*from x$kslwt;
createviewv_x$ksled asselect*from x$ksled;
create synonym datadog.v_x$ksuse for v_x$ksuse;
create synonym datadog.v_x$kslwt for v_x$kslwt;
create synonym datadog.v_x$ksled for v_x$ksled;
and then create the dd_session-view with these new views instead:
FROM
v_x$ksuse s,
v_x$kslwt w,
v_x$ksled e,
v$sql sq,
v$sql sq_prev,
v$containers c,
v$sqlcommand comm
WHERE
Then, the datadog-user can successfully report the metrics.
Issue observed
After installing the Oracle Integration, we do receive data from the instance via the Agent and the reported metrics are shown in the Datadog-provided "DBM Oracle Database Overview"-Dashboard:
However, we're missing data in these reports, as the dashboards show that some metrics are missing. In fact, the integration does query DBMS statistics, but does not report all of them back to the agent:
https://github.com/DataDog/integrations-core/blob/master/oracle/datadog_checks/oracle/queries.py#L19
For example, the very important metric
User Transaction Per Sec
is not used by the integration.Our Oracle database can report the following metrics via
GV$SYSMETRIC
:It's probably easy to incorporate the additional metrics into the reporting.
Output of the info page
Note: Our Agent in Kubernetes can't execute
s6-svstat
:s6-svstat: command not found
Additional environment details (Operating System, Cloud provider, etc):
We're running Agent (v7.50.2) in Azure Kubernetes, installed via Helm (Chart v3.50.2).
We're using Oracle 19 for our RDBMS.
Steps to reproduce the issue:
Describe the results you received:
The Oracle integration reports a few metrics
Describe the results you expected:
The Oracle integration reports all metrics.
Additional information you deem important (e.g. issue happens only occasionally):
The text was updated successfully, but these errors were encountered: