Internal Data Source
The Internal data source monitors Mango's own performance and operational metrics. It provides access to a comprehensive set of internal monitors covering JVM memory usage, CPU load, database connections, work item queues, disk space, thread counts, and per-data-source polling statistics. These metrics are essential for capacity planning, performance tuning, and setting up alerts for system health issues before they affect data collection.
Because Mango dynamically creates monitors for certain subsystems (such as polling data sources and Persistent TCP connections), the Internal data source can automatically discover and create points for new monitors as they appear.
Overview
| Property | Value |
|---|---|
| Module | mangoAutomation-InternalDS |
| Protocol | N/A (internal metrics) |
| Direction | Polling |
| Typical Use | Monitoring Mango system performance and health |
Prerequisites
No external hardware or network configuration is required. The Internal data source is built into Mango and is available immediately. However, some metrics (such as OS load average) are only available on certain operating systems.
Configuration
Data Source Settings
| Setting | Description |
|---|---|
| Name | A descriptive name for the data source. |
| Update period | How frequently Mango samples its internal monitors. |
| Create points pattern | (Optional) A Java regular expression pattern. When new internal monitors are registered that match this pattern, the data source automatically creates data points for them. If blank, no automatic point creation occurs. |
To combine multiple patterns, use the regex alternation syntax:
(?:pattern1|pattern2|pattern3)
For example, to auto-create points for all data source polling monitors and all JVM monitors:
(?:com\.serotonin\.m2m2\.rt\.dataSource.*|java\.lang.*)
Data Point Configuration
Each data point is associated with a specific internal monitor by selecting the Monitor ID from the available list. The available monitors span several categories.
System and JVM Metrics
| Monitor | Description |
|---|---|
| JVM free memory (MB) | Amount of memory remaining before the heap attempts to grow. |
| JVM used memory (MB) | Current memory consumed by the JVM. |
| JVM maximum available memory (MB) | Maximum heap size the JVM can potentially acquire. |
| JVM available processors | Number of CPU cores accessible to the JVM. |
| JVM uptime (seconds) | Time since the JVM started. |
| JVM thread count | Current number of active threads. |
| JVM peak thread count | Highest thread count observed since JVM startup. |
| JVM daemon thread count | Number of daemon threads. |
| JVM classes loaded | Number of currently loaded classes. |
| System CPU load (%) | Total CPU utilization across all processes. |
| Mango process CPU load (%) | CPU utilization by the Mango process only. |
| OS load average (1 minute) (%) | One-minute system load average. Not available on Windows. |
| System uptime (Hrs) | Time since the operating system booted. |
Mango Process Metrics
| Monitor | Description |
|---|---|
| Mango process resident set size (RSS) | Memory allocated to Mango that is currently in RAM (bytes). |
| Mango process Virtual Memory Size | Total virtual memory accessible by the Mango process (bytes). |
| Mango process open files | Number of open file handles or network connections (-1 if unknown). |
| Mango process bytes read | Total bytes read from disk by the Mango process. |
| Mango process bytes written | Total bytes written to disk by the Mango process. |
| Mango process user cpu time | Milliseconds of user-mode CPU time consumed. |
| Mango process kernel cpu time | Milliseconds of kernel-mode CPU time consumed. |
Database and Storage Metrics
| Monitor | Description |
|---|---|
| Active database connections | Active connections in the JDBC pool (useful only with JDBC pooling). |
| Idle database connections | Idle connections in the pool. |
| SQL database size (GB) | Size of the SQL database on disk. |
| SQL database partition total/usable/used space (GB) | Disk space metrics for the SQL database partition. |
| NoSQL database size (GB) | Size of the NoSQL (time-series) database. |
| NoSQL partition total/usable/used space (GB) | Disk space metrics for the NoSQL partition. |
| Mango partition total/usable/used space (GB) | Disk space for the Mango installation directory. |
| Filestore partition total/usable/used space (GB) | Disk space for the filestore partition. |
Work Item Queues
| Monitor | Description |
|---|---|
| High priority work items currently running | Number of high-priority tasks executing now. |
| High priority tasks queued | Tasks waiting for a free thread in the high-priority pool. |
| Currently scheduled high priority tasks | Tasks scheduled to run in the future. |
| Medium priority tasks currently running | Number of medium-priority tasks executing. |
| Medium priority tasks queued | Tasks in the unbounded medium-priority queue. |
| Low priority tasks currently running | Number of low-priority tasks executing. |
| Low priority tasks queued | Tasks waiting for a free thread in the low-priority pool. |
| Point values to be written | Number of point values queued for database write. |
| Point value write threads | Threads concurrently writing point values. |
| Writes per second during database batches | Write throughput when values are queued. |
Entity Counts
| Monitor | Description |
|---|---|
| Data points | Total number of data points configured. |
| Data sources | Total number of data sources configured. |
| Event detectors | Total number of event detectors. |
| Event handlers | Total number of event handlers. |
| Users | Total number of user accounts. |
| Publishers | Total number of publishers. |
| Mailing lists | Total number of mailing lists. |
Per-Data-Source Polling Metrics
Every polling data source in Mango automatically registers these monitors:
| Monitor | Description |
|---|---|
| Last poll duration (ms) | How long the most recent poll took. |
| Poll success percentage | Percentage of successful polls. |
| Previous sequential successful polls | Number of consecutive successful polls before the last failure. |
HTTP Server Metrics
| Monitor | Description |
|---|---|
| HTTP server threads | Number of active HTTP server threads. |
| HTTP server idle threads | Number of idle threads in the HTTP thread pool. |
| HTTP server queue size | Number of HTTP requests waiting to be processed. |
Common Patterns
System Health Dashboard
Create an Internal data source with points for JVM memory, CPU load, disk space, and work item queue depths. Build a dashboard that displays these metrics with trend charts. Configure event detectors to alert when memory usage exceeds 80%, disk space drops below a threshold, or work item queues grow beyond normal levels.
Database Performance Monitoring
Monitor the Point values to be written and Writes per second metrics to track database write performance. A steadily growing write queue indicates that the database cannot keep up with the incoming data rate. This is an early warning signal that you may need to optimize your database, reduce point logging frequency, or add hardware resources.
Data Source Health Tracking
Use the per-data-source polling metrics (Last poll duration, Poll success percentage) to monitor the health of all data sources in the system. Create event detectors that fire when poll success drops below 95% or when poll duration exceeds the poll period, indicating an overloaded data source.
Capacity Planning
Track entity counts (data points, data sources, users) over time to understand growth trends. Combine this with JVM memory and CPU metrics to determine when the system is approaching its capacity limits and when hardware upgrades or configuration changes will be needed.
Troubleshooting
Missing Monitors
- Some monitors are dynamically created when their corresponding subsystems start. For example, per-data-source polling monitors only appear after the data source has been started at least once.
- OS-specific monitors (such as OS load average) are not available on all platforms. Windows does not support the 1-minute load average.
- JDBC pool metrics (Active/Idle database connections) are only meaningful when JDBC connection pooling is configured.
Unexpected Values
- JVM free memory fluctuates rapidly -- this is normal and expected due to Java garbage collection cycles.
- Process CPU load shows 0% -- on some operating systems, process-level CPU metrics require specific permissions or are not supported.
- Disk space metrics show unexpected values -- verify that the Mango configuration points to the correct directories. The partition metrics reflect the partition where the configured directory resides, which may differ from expected if symbolic links are used.
Auto-Created Points Not Appearing
- Verify the Create points pattern regex matches the monitor ID you expect. Test the regex against the monitor ID string.
- New monitors are only detected when the Internal data source polls. If the target data source starts between polls, the new monitor will be detected on the next poll cycle.
Related Pages
- Data Sources Overview — General data source and data point concepts
- Virtual Data Source — Generate simulated data for testing without external hardware
- Data Source Performance — Use internal metrics to tune poll intervals and detect bottlenecks
- High CPU Usage — Diagnose CPU issues using internal performance metrics
- Out of Memory — Detect and resolve memory issues using internal heap monitoring