Skip to main content

Out of Memory Errors

Java OutOfMemoryError is one of the most common performance-related issues in Mango deployments, particularly in systems with large numbers of data points, aggressive caching configurations, or insufficient JVM heap allocation. This page covers how to identify, diagnose, and resolve memory-related problems.

Symptoms​

  • Mango becomes unresponsive or extremely slow, then crashes.
  • The log file (ma.log) contains java.lang.OutOfMemoryError: Java heap space.
  • The log file contains java.lang.OutOfMemoryError: GC overhead limit exceeded, indicating the garbage collector is spending nearly all CPU time trying to reclaim memory.
  • The log file contains java.lang.OutOfMemoryError: Metaspace, indicating class metadata has exhausted its allocation.
  • Internal metrics show JVM free memory consistently near zero.
  • Data point value logging falls behind or stops, with "NoSQL Task Queue Full" events appearing.
  • The UI becomes unresponsive while the backend process continues running.

Common Causes​

1. Insufficient JVM Heap Allocation​

The default JVM heap size may be too small for your deployment. A system with thousands of data points, large point value caches, and multiple active dashboards requires significantly more heap than the default.

2. Excessive Point Value Cache Sizes​

Each data point has a Default cache size that determines how many recent values are held in memory. If hundreds or thousands of points each have large cache sizes (e.g., 500 or 1000 values), the cumulative memory consumption can be enormous.

Example: 5,000 numeric data points with a cache size of 200 each = 1,000,000 values held in memory simultaneously.

3. Too Many Data Points on a Single Instance​

Mango maintains runtime state for every enabled data point, including the point value cache, event detector state, and metadata. Systems with tens of thousands of points on a single instance may exceed available memory.

4. Large Bulk Data Exports or Queries​

REST API queries or CSV exports that request large time ranges of point value data can temporarily consume large amounts of heap memory while assembling the response.

5. Memory Leaks in Custom Scripts or Modules​

Custom meta data source scripts, scripting data source scripts, or third-party modules may inadvertently accumulate objects in memory over time without releasing them.

6. Too Many Concurrent User Sessions​

Each active UI session (especially with live-updating dashboards, charts, and watch lists) maintains WebSocket connections and server-side state that consume memory.

Diagnosis​

Check the Mango Log​

Search the log file for memory-related errors:

grep -i "OutOfMemoryError\|heap space\|GC overhead" MA_HOME/logs/ma.log

Monitor JVM Memory via Internal Metrics​

If Mango is still running, the Internal Metrics data source (when enabled) provides data points for:

  • JVM Free Memory: Amount of free heap memory in bytes.
  • JVM Used Memory: Amount of used heap memory in bytes.
  • JVM Max Memory: Maximum heap size the JVM can use.

Add these points to a watch list and chart them over time to identify memory consumption trends.

Check Current JVM Settings​

Review the startup script or systemd unit file to find the current heap settings:

grep -i "Xmx\|Xms\|MaxMetaspaceSize" MA_HOME/bin/start-mango.sh
# Or check the running process
ps aux | grep java | grep -oP '\-Xm[xs]\S+'

Use JVisualVM or JConsole for Live Analysis​

For detailed memory analysis on a running system:

# Find the Mango Java PID
jps -l | grep mango

# Connect with JVisualVM (requires JDK and GUI access)
jvisualvm --openpid <PID>

In JVisualVM, check:

  • Heap Usage: The sawtooth pattern should show memory being reclaimed by GC. If the baseline keeps rising, there is a leak.
  • Classes: A steadily increasing class count may indicate a Metaspace issue.
  • Threads: An excessive thread count can also contribute to memory pressure.

Generate a Heap Dump​

If the issue is intermittent, configure the JVM to automatically generate a heap dump on OOM:

# Add to JVM arguments in the startup script
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=MA_HOME/logs/

Analyze the resulting .hprof file with Eclipse MAT (Memory Analyzer Tool) or JVisualVM to identify which objects are consuming the most memory.

Solutions​

Solution 1: Increase JVM Heap Size​

Edit the Mango startup script (typically MA_HOME/bin/start-mango.sh) and adjust the heap parameters:

# Recommended starting points based on deployment size:
# Small (< 500 points):   -Xms256m -Xmx1g
# Medium (500-5000 points): -Xms512m -Xmx2g
# Large (5000-20000 points): -Xms1g -Xmx4g
# Very large (20000+ points): -Xms2g -Xmx8g

JAVA_OPTS="-Xms1g -Xmx4g -XX:MaxMetaspaceSize=512m"

Restart Mango after changing these settings.

tip

Set -Xms (initial heap) to the same value as -Xmx (max heap) for production systems. This avoids the overhead of heap resizing at runtime and ensures the memory is available from startup.

Solution 2: Reduce Point Value Cache Sizes​

Review and reduce the Default cache size for data points that do not need large caches:

  1. Navigate to the Bulk Data Point Edit page.
  2. Filter for points with large cache sizes.
  3. Reduce the cache size to the minimum needed (often 1-5 values is sufficient for most points).

For points that log on change or at intervals, a small cache (1-5 values) is usually adequate. Only increase the cache for points that need rapid access to recent history (e.g., for meta data source calculations or scripting).

Solution 3: Reduce the Number of Enabled Points​

If the system has more points than available memory can support:

  • Disable points that are not actively needed for monitoring, alarming, or automation.
  • Consider splitting the workload across multiple Mango instances and using the Persistent TCP or gRPC publisher to consolidate data.

Solution 4: Optimize Logging Settings​

Points configured with All data logging and high-frequency data sources generate continuous memory pressure. Switch to When point value changes or Interval logging where full-resolution logging is not required.

Solution 5: Limit Bulk Query Sizes​

When using the REST API for data exports, use pagination and limit the time range:

GET /rest/latest/point-values/{xid}?from=2025-01-01T00:00:00Z&to=2025-01-02T00:00:00Z&limit=10000

Avoid requesting months of high-frequency data in a single API call.

Solution 6: Enable GC Logging for Ongoing Monitoring​

Add GC logging to the JVM arguments to track garbage collection behavior:

# Java 11+
-Xlog:gc*:file=MA_HOME/logs/gc.log:time,uptime,level,tags:filecount=5,filesize=10m

Review the GC log to identify full GC pauses, promotion failures, or allocation failures that precede OOM events.

Prevention​

  • Right-size the JVM heap during initial deployment based on the expected number of data points and concurrent users.
  • Monitor JVM memory continuously using Internal Metrics data points with alarm thresholds (e.g., alert when free memory drops below 10% of max).
  • Audit point value cache sizes periodically, especially after adding large batches of new points.
  • Use interval logging instead of "All data" logging for high-frequency data sources where full-resolution history is not needed.
  • Configure heap dump on OOM (-XX:+HeapDumpOnOutOfMemoryError) so that diagnostic data is always available after a crash.
  • Plan capacity before scaling: each additional 1,000 numeric data points with a cache size of 10 consumes approximately 50-100 MB of heap depending on value types and metadata.