Skip to main content

Data File Data Source

The Data File data source polls a directory for incoming files and imports their contents as Mango data point values. It supports XML, CSV, Excel, and binary file formats through a customizable template system. Each file format requires a compiled Java template class that defines how to parse the file and map its contents to data points. The data source can either create new data points automatically from imported data or map imported values to existing points elsewhere in Mango.

This data source is ideal for batch data ingestion workflows where external systems periodically deposit files into a known directory -- for example, laboratory instruments exporting test results as CSV files, building management systems generating daily Excel reports, or SCADA historians dumping binary data files.

Overview​

PropertyValue
ModulemangoAutomation-DataFile
ProtocolHTTP + File
DirectionPolling
Typical UseRetrieving and parsing structured data from HTTP endpoints or files

Prerequisites​

  • A directory on the Mango server's local file system where import files will be placed.
  • A template class (custom Java importer) compiled and available for the file type you intend to import. Mango provides a mechanism to upload and compile these templates from the data source editor.
  • Knowledge of the file structure -- column headers, XML element names, or binary format specifications -- so you can write or configure the appropriate template.

Configuration​

Data Source Settings​

SettingDescription
NameA descriptive name for the data source.
Update periodHow frequently Mango polls the directory for new files.
Data File PathThe directory to search for import files. If pointed at a directory, all files within it are processed (non-recursive). Can also point to a single file.
File TypeThe document format: XML, CSV, Excel, or Binary.
TemplateThe name of the compiled .class file that implements the importer for this file type. Do not include the .class extension.
Filename match regex(Optional) A case-sensitive regular expression to filter which files are processed. For example, .*\.csv processes only CSV files.
Poll class(Optional) A custom Java class extending AbstractPollClass that executes on every poll cycle. This can be used to generate files, download files from remote systems, or perform pre-processing before import.
Poll arguments(Optional) A single string passed to the poll class on each execution.
Additional classpath(Optional) Additional Java classpath locations for compilation and runtime. Use fully qualified file paths. To include all JAR files in a directory, use a trailing asterisk: /path/to/lib/*.
Create PointsWhen checked, the data source automatically creates new data points assigned to itself using identifiers from the imported file.
Sort valuesWhen checked, imported values are sorted in ascending time order before processing. This ensures events fire correctly when the source data is not chronologically ordered.
Delete After ImportWhen checked, the imported file is deleted after processing. When unchecked, the file is renamed with the configured prefix.
Imported PrefixThe string prepended to file names after import (when Delete After Import is unchecked).

Data Point Configuration​

Data File data points serve two purposes: they can represent imported data directly as standard Mango data points, or they can act as mapping points that route imported values to other existing data points.

SettingDescription
Identifier in import documentThe string identifier that matches the identifier returned by the import template class for this point. Must match exactly.
Maps to external pointWhen checked, this point forwards imported values to another data point in Mango rather than storing them locally.
Point to import values to(When mapping is enabled) The target data point that receives the imported values. Appears in the point's JSON as toXid.
SettableWhen checked and the data source has a poll class defined, setting a value on this point passes the value into the poll class's setPointValueImpl method.
note

If two points on the same data source share the same identifier, only one will receive the update. Ensure identifiers are unique within a single data source.

Automatic Point Creation​

When Create Points is enabled, the data source creates new points from identifiers found in the imported file. The import template can provide the following properties for each new point:

PropertyDescription
xidThe XID for the new point. If set to the string "generate", a unique XID is auto-generated with the prefix DFDP_.
deviceNameThe device name assigned to the new point.
nameThe display name of the new point.
identifierThe point locator's in-document identifier. This is useful when the XID is auto-generated or when the natural identifier exceeds the 40-character XID limit.

Compiling Templates​

Custom importer templates must be uploaded and compiled before the data source can use them.

  1. Open the Data File Tools tab in the data source editor and click Upload Template/Poll Class.
  2. Select the Java source file (.java) from your file system and upload it.
  3. Ensure the file is in the root folder of the Data File data source template filestore.
  4. Click Compile New Templates to compile the uploaded source.
  5. A compilation notification appears at the bottom of the page. If compilation fails, review error messages and correct the source file.
important

After recompiling a template class, you must restart the data source. The Java class loader caches the previously loaded class, so changes do not take effect until the data source runtime is restarted.

Template types include:

  • CSV importers -- for comma-separated value files
  • Excel importers -- for .xls and .xlsx spreadsheets
  • XML importers -- for XML documents
  • Binary importers -- for custom binary file formats
  • Poll classes -- for pre-processing or file generation logic

Common Patterns​

Laboratory Data Ingestion​

Laboratory instruments often export test results as CSV or Excel files. Configure the Data File data source to watch the instrument's export directory. Create a CSV template that maps column headers to data point identifiers. Enable automatic point creation for the initial setup, then disable it once all expected points exist.

Building Management Reports​

Building management systems frequently produce daily or hourly Excel reports with energy consumption, temperature readings, and equipment status. Use an Excel template to parse these reports and import the values with their original timestamps (using the Sort values option to maintain chronological order).

Binary Protocol Dumps​

Some industrial devices produce binary data dumps in proprietary formats. Write a custom binary importer template that understands the byte layout, and configure the Data File data source to process these files. This approach bridges Mango to devices that lack standard protocol support.

Mapping to Existing Points​

When you want imported file data to update existing data points on other data sources, enable the Maps to external point option. This is useful when a file provides calibration data, setpoints, or configuration values that should be applied to points managed by a different data source.

Troubleshooting​

Template Compilation Failures​

  1. Verify the Java source file is in the root folder of the Data File template filestore, not in a subdirectory.
  2. Check compilation error messages for missing imports or syntax errors.
  3. If the template depends on external libraries, add them to the Additional classpath field.

Files Not Being Processed​

  1. Verify the Data File Path points to the correct directory and that Mango has read permission.
  2. If using a Filename match regex, test the regex against your actual file names. Remember that the regex is case-sensitive.
  3. Check that the data source is enabled and polling.

Points Not Receiving Values​

  1. Verify the Identifier in import document matches exactly what the template returns as the point identifier.
  2. If using mapping points, confirm the target point exists and is enabled.
  3. Enable data source event logging to see import results and any errors during file processing.

Data Source Must Be Restarted After Template Changes​

If you modify and recompile a template class, the data source must be restarted to pick up the new class. This is a consequence of the Java class loading mechanism -- the runtime caches the loaded class until the data source is stopped and started again.