Skip to main content

Managing Disk Space

There is no guaranteed system to prevent Mango from running out of disk space. If a MangoES appliance or server running Mango does run out of disk space, it will cause serious problems including database corruption, JVM crashes, and potential data loss. The system will need manual intervention to recover.

As part of the default MangoES setup, a data point monitors remaining disk space with event detectors configured to trigger alarms at 2,000 MB and 1,000 MB remaining. It is strongly recommended to configure email event handlers for these detectors so the system administrator is notified before the disk is full.

Four Primary Consumers of Disk Space​

In Mango, there are four primary things that can fill the disk:

SourceLocationNotes
Historical dataMA_HOME/databases/mangoTSDB/Largest consumer in most systems
Backup filesMA_HOME/backup/SQL, NoSQL, and config backups
Event messagesSQL database (MA_HOME/databases/mah2.h2.db)Events table can grow large
Log filesMA_HOME/logs/Debug logging and module-specific logs
note

The file paths in this document use /opt/mango as the installation directory. Your MA_HOME path may be different depending on your installation.

Historical Data Purging​

Historical data is the primary consumer of disk space. The amount of space needed depends on:

  • How many data points are being logged
  • How frequently they are sampled
  • How long data is retained before purging

By default, Mango keeps all historical data and events for 12 months. You can adjust purge settings under System Settings > Purge Settings. Individual purge overrides can be set on each data source or data point.

Storage estimate: Approximately 200 million historical values consume about 4 GB of disk space.

If Mango is operational, adjust your purge settings to a shorter retention period and use the Purge Now button on the System Settings page to immediately free disk space.

Manual Purge of NoSQL Shard Files​

NoSQL data is stored in shard files under MA_HOME/databases/mangoTSDB/. Each shard file contains approximately 24 days of data. Files are named with a numeric shard ID (e.g., 685.data.rev).

To manually delete old shards on Linux:

Navigate to the TSDB directory:

cd /opt/mango/databases/mangoTSDB

Count files for a specific shard ID:

find . -name "686.data.rev" | wc -l

Create a manifest of files to delete:

find . -name '686.data.rev' > 686.manifest

Delete all files in the manifest:

for f in $(cat 686.manifest); do rm "$f"; done

Script to count all shard files (useful for identifying the oldest shards):

#!/bin/bash
directories="/opt/mango/databases/mangoTSDB"
for directory in ${directories}; do
    echo "${directory}: "
    find ${directory} -type f -exec basename {} \; | sort | uniq -c
done

Save this script as countDBFiles.sh, make it executable with chmod u+x countDBFiles.sh, and pipe the output to a file for analysis: ./countDBFiles.sh > DBFileCount.txt

Backup Files​

Backup files are stored in MA_HOME/backup/ and can be searched for and manually deleted. There are three types of automated backups:

  1. Mango NoSQL Backups (under System Settings > Mango NoSQL Configuration) -- These can be very large for systems with extensive historical data.
  2. SQL Database Backup -- SQL dump of the relational database (H2 or MySQL).
  3. Configuration Backup -- JSON export of configuration objects. These files are generally small.

Reduce the Versions to keep setting for each backup type to limit disk usage. On large production systems, consider disabling NoSQL auto-backup and using filesystem-level snapshots instead.

Event Messages in the Database​

Event messages are stored in the SQL database events table. Misconfigured systems can generate excessive events that cause the database to grow rapidly. Common causes include:

  • Data source exception events logged at a level other than "Ignore."
  • Event detectors with thresholds that trigger frequently.

Best practices for managing event storage:

  • Set data source exception events to Ignore level once the data source is functioning properly. Note that a "None" level event is still stored in the database.
  • Configure event purging under System Settings > Purge Settings (default is 1 year).
  • Use the SQL Console to count events: SELECT COUNT(*) FROM events

Log Files​

Log files are stored in MA_HOME/logs/ and can be generated by various Mango subsystems, data sources, and publishers. The default system logs are configured with rolling file appenders that automatically rotate and delete old files, so they should not consume significant disk space.

However, check the logs directory for:

  • Debug log files enabled for troubleshooting that were never disabled.
  • Module-specific log files from data sources or publishers.
  • NoSQL corruption or compression logs that may grow if the database encounters errors.

See Debug Log Settings for information on managing log file configuration.