Skip to main content

Data Points Disappear After Upgrade

Symptoms​

Data points that were previously visible in graphical views, watchlists, or charts are no longer displayed. This can manifest in several ways:

  • Points vanish from graphical views when entering edit mode after an upgrade.
  • Historical point values are missing from charts or data exports.
  • Points appear in the data source configuration but show no values.
  • The point value count for a data point drops to zero or a much lower number than expected.

Common Causes​

  1. Outdated Graphical Views module -- After upgrading Mango core, the Graphical Views module may not have been updated to a compatible version. The module version mismatch causes points to fail to render in the view editor.

  2. Purge settings too aggressive -- Mango's purge system automatically deletes old point values on a daily schedule (runs at 3:05 AM local time). If purge settings are configured with a very short retention period, historical data disappears as expected by the system but unexpectedly for the user. Purge settings exist at three levels: system-wide defaults, data source overrides, and individual data point overrides.

  3. Database corruption -- Unclean shutdowns (power failures, process kills) can corrupt the SQL database or the NoSQL point value store, causing point values to become unreadable even though they may still exist on disk.

  4. Clock skew or time zone issues -- If the system clock on the Mango server drifts significantly, or if the time zone is changed, point values may appear to be missing because the query time range no longer aligns with the timestamps stored in the database. Values with future-dated timestamps can also be discarded by Mango's built-in protections.

  5. Point logging configuration -- If a data point's logging type is set to "Do not log," no historical values are stored. A configuration change (accidental or via import) could have switched the logging type.

  6. Permissions change -- After an upgrade, permission models may have changed. A user who previously had access to view certain data points may no longer have the required read permissions.

Diagnosis​

Check the Module Versions​

Navigate to Administration > Modules and verify that all installed modules are compatible with the current Mango core version. Click "Check for upgrades" to see if newer module versions are available. Pay particular attention to the Graphical Views module version if points are disappearing specifically in graphical views.

Check Purge Settings​

Review purge settings at all three levels:

  1. System-wide: Navigate to Administration > System Settings > Purge Settings and check the default purge period for point values.
  2. Data source level: Open the data source editor and check if a purge override is configured.
  3. Data point level: Open the data point properties and check if "Override data source purge settings" is enabled. If so, review the point-level purge period.

A purge period of "1 day" will delete all point values older than 24 hours. Ensure the purge periods are long enough to retain the data you need.

Check Point Value Count​

Navigate to the data point details page or use the REST API to check the number of stored values for the affected point:

GET /rest/latest/point-values/count/{xid}?from=2024-01-01T00:00:00Z&to=2025-01-01T00:00:00Z

If the count is zero or unexpectedly low, values have been purged or were never logged.

Check Purge Logs​

In ma.log, search for purge-related messages. The purge process logs the number of values deleted:

INFO  - Purge completed: deleted X point values

If the deleted count is unexpectedly high, the purge settings are likely misconfigured.

Verify the System Clock​

Check the Mango server's system time and time zone. Compare it with the expected time. On Linux:

timedatectl

If the clock has drifted or the time zone was changed, point values may appear to be in the wrong time range.

Check Logging Type​

Open the data point properties editor and verify the Logging Type setting. If it is set to "Do not log," no values are being stored.

Solutions​

Solution for Outdated Modules​

  1. Navigate to Administration > Modules.
  2. Click Check for upgrades.
  3. Update the Graphical Views module and any other modules that have newer versions available.
  4. Restart Mango to apply the module updates.

Solution for Aggressive Purge Settings​

  1. Adjust the purge period at the appropriate level (system, data source, or data point) to retain data for the desired duration.
  2. Note that purged data cannot be recovered. Going forward, the new settings will prevent further premature data loss.
  3. If you need to retain different amounts of history for different points, use the per-point purge override to set a longer retention period on critical data points.

Solution for Database Corruption​

  1. Stop Mango.
  2. If using H2, try starting Mango with the db.recover=true property set in env.properties. Mango will attempt to repair the database on the next startup.
  3. If using MySQL or MariaDB, run the database repair utilities provided by the database server.
  4. For NoSQL corruption, allow Mango's automatic repair scan to complete. If it cannot repair the data, restore from a backup.
  5. If no backup is available, the corrupted data may be permanently lost. Delete the corrupted database files and let Mango create a fresh database.

Solution for Clock Skew​

  1. Synchronize the system clock using NTP or a similar time synchronization service.
  2. Verify the time zone is set correctly.
  3. Restart Mango after correcting the time.
  4. If point values were recorded with incorrect timestamps, they may need to be manually corrected or re-imported.

Solution for Permissions Issues​

  1. Navigate to Administration > Users and check the permissions assigned to the affected user account.
  2. Verify that the user has read permission on the data points and data sources in question.
  3. After an upgrade, review any permission model changes documented in the release notes.

Prevention​

  • Always update all modules when upgrading Mango core. Use the "Check for upgrades" feature on the Modules page to ensure compatibility.
  • Set purge periods deliberately and review them periodically. Use per-point overrides for data that requires longer retention.
  • Always shut down Mango cleanly before powering off the server. Use the proper shutdown command or stop the Mango service gracefully.
  • Configure NTP time synchronization on the Mango server to prevent clock drift.
  • Maintain regular backups of both the SQL database and the NoSQL point value store.
  • Periodically review the ma.log file for purge activity and unexpected warnings.