6. Appendix¶

6.1. Base time¶

The base time is information that represents the start time of measurement. The Agent gets the base time as needed and sends it to intdash.

Note

"Measurement" refers to a collection of time series data sent from a particular edge. For details, refer to the separate document on iSCP v1.

There are two types of base times.

Base time by EdgeRTC

The real-time clock (RTC) base time of the system running the Agent.

Base time by NTP

The base time by the clock synchronized with the NTP server. The Status plugin must be enabled to use the NTP based base time.

To enable the Status plugin, write the Status plugin setting to loggers in the manager configuration file. See the configuration file example /opt/vm2m/etc/manager.conf (the setting with details.plugin as status is the Status plugin setting).

(The Status plugin is enabled in the configuration file /etc/opt/intdash/manager.conf after installation.)

Base time type	Timing to determine the base time	Timing to send the base time to intdash
EdgeRTC	When the Agent starts	At the start of data transmission
NTP	After the Status plugin has communicated with the NTP server	Fixed cycle (1-minute interval for 10 minutes after startup, then 10-minute interval)

6.2. Filter at the sender's side¶

The time series data collected by the manager through the Device Connector is filtered by the manager and distributed to each client such as the Realtime client and the Bulk client, and sent to the intdash server.

Filters give you the flexibility to switch between data transmission methods.

6.2.1. Filtering process on the sending side¶

For each time series data, the manager decides whether to send the data using each client (pass) or not (drop).

Unless otherwise specified, time series data is sent to the intdash server via the Realtime client.
Data that was not sent by the Realtime client because it was "dropped" by the filter for the Realtime client is sent by the Bulk client.
Data that is "dropped" by the filter for the Bulk client will not be sent to the intdash server.

In either case, the time series data is dumped as RAW data by the manager.

Realtime client	Bulk client	How the data is handled
Passed	-	Data is sent in real time to the intdash server and saved on the intdash server.
Dropped	Passed	Data is sent to the intdash server by batch transmission at regular intervals and saved on the intdash server.
Dropped	Dropped	Data is not sent to the intdash server in real time. It is not saved on the intdash server. It is saved as RAW data in the local storage on the edge device.

6.2.2. Filter configuration¶

Filtering is performed by multiple filters. Multiple filters are evaluated sequentially to determine which client should send each time series data.

For example, if the following filter configuration has been set up, data is sent as follows.

Data X does not correspond to any filter and is sent by the Realtime client.
Data Y is dropped by the Realtime client by the first filter and is sent by the Bulk client.
Data Z is not sent because the first filter drops it from the Realtime client and the fourth filter also drops it from the Bulk client.

6.2.3. Common filter settings¶

The settings for the filter are made in manager.filters[] of the configuration file. The contents of the objects stored in the "filters" array should be as follows.

Key	Type	Description
`name`	string	See the table for each filter in Filter type.
`channel`	string	Set the channel to which the filter is applied. You can set "-1", "0" to "255". In case of "-1", all channels are targeted.
`target`	string	Sets the client to which the filter is applied. (`realtime\|guaranteed\|both`)
`setting`	object[]	Filter setting. See the description of each filter in Filter type.

For target, specify one of the following.

realtime: Filter for the Realtime client.
guaranteed: Filter for the Bulk client.
both: Filter for both the Realtime client and the Bulk client.

6.2.4. Filter type¶

There are the following types of filters.

Some filters apply only to data of a specific data type, while others apply to all data types.

Filter name	Data types to which you can apply filters	Description
sampling filter	CAN, NMEA, Motion JPEG, String, Float, Int, Bytes only	Sampling filter. Thins out the data.
can_id filter	CAN only	Allow-list type CAN ID filter. Passes the data with the specified CAN ID.
can_mask filter	CAN only	Block-list type CAN ID filter. Drops the data with the specified CAN ID.
channel filter	Any	Block-list type channel filter. Drops the data for the specified channel.
duplicate filter	Any	Data replication filter. Duplicate the same data to the Realtime and Bulk clients.

6.2.4.1. sampling filter¶

When a sampling filter is set for a channel, data is extracted so that there is one data point for each data ID within the time range specified as the sampling interval.

In each sampling interval, data points are processed as follows.

The first data point of each data ID will pass.
The second and subsequent data points with the same data ID will be discarded.
If there is no distinction by data ID, as in MotionJPEG, the first data point will pass.

The following figure shows how CAN data with three different frequencies (ID 1, 2, and 3) are filtered by a sampling filter. In each sampling interval, the first data point of the data with ID 1 passes the filter. Similarly, in each sampling interval, the first data point with ID 2 and the first data point with ID 3 pass the filter.

Contents of the "setting" object of the sampling filter

Key	Value
rate	Sampling interval [msec]

Example (Sampling real-time data of channel 1 with a sampling interval of 1000 [msec]):

{
  "manager": {
    ...
    "filters": [{
      "name": "sampling",
      "channel": "1",
      "target": "realtime",
      "setting": [{"key": "rate", "value": "1000"}]
    }]
   },
   ...
 }

6.2.4.2. can_id filter¶

An allow-list type filter that passes data with a specific CAN ID and drops other data.

Passes only the data with the specified CAN ID.
- Multiple CAN IDs can be set.
- Drops data that does not have a specified CAN ID
- When setting the extended CAN ID, enter a number with the first bit set to 1, such as 0x80000000.

Contents of "setting" object of can_id filter

Key	Value
id	CAN ID to pass

Example (Only CAN IDs 0x00000010 and 0x00000020 in Realtime channel 2 pass):

{
  "manager": {
    ...
    "filters": [{
      "name": "can_id",
      "channel": "2",
      "target": "realtime",
      "setting": [
        {"key": "id", "value": "16"},
        {"key": "id", "value": "32"}
      ]
    }]
  },
  ...
}

6.2.4.3. can_mask filter¶

A block list type filter that drops data of a specific CAN ID.

Drops the data with the specified CAN ID.
- Multiple CAN IDs can be set.
- Passes data that does not have a specified CAN ID
- When setting the extended CAN ID, enter a number with the first bit set to 1, such as 0x80000000.

Contents of the filter "setting" object

Key	Value
id	CAN ID to be dropped

Example (Data with extended CAN ID 0x00000010 in Bulk channel 1 are dropped):

{
  "manager": {
    ...
    "filters": [{
      "name": "can_mask",
      "channel": "1",
      "target": "guaranteed",
      "setting": [
        {"key": "id", "value": "2147483664"}
      ]
    }]
  },
  ...
}

2147483664 is a decimal notation with the first bit of 0x00000010 set to 1.

6.2.4.4. channel filter¶

A block list type filter that drops a specific channel.

Drops the data of the specified channel

Contents of the filter "setting" object

None

Example (Data in Realtime channel 1 are dropped):

{
  "manager": {
    ...
    "filters": [{
      "name": "channel",
      "channel": "1",
      "target": "realtime",
      "setting": []
    }]
  },
  ...
}

6.2.4.5. duplicate filter¶

Duplicate the data.

Duplicate the data and send the same data to the Realtime and Bulk clients

Contents of the filter "setting" object

None

Example (copy Realtime channel 1 data and send it to the Bulk client):

{
  "manager": {
    ...
    "filters": [{
      "name": "channel",
      "channel": "1",
      "target": "realtime",
      "setting": []
    }]
  },
  ...
}

6.3. RAW data¶

All the data input to the Agent from the Device Connectors can be saved as RAW data in the dump file.

6.3.1. Saving and automatic deletion of RAW data¶

It is possible to set whether to save RAW data. By default, the data is saved.

To prevent your disk from filling up by storing your RAW data, you can automatically delete your RAW data.

{
  "manager": {
    "rawdir": "/var/lib/intdash/raw",
    "raw_autodelete": true,
    "raw_autodelete_th": 85,
    ...
  },
  ...
}

If raw_autodelete is set to true, the RAW data will be automatically deleted when the usage of the partition storing the RAW data reaches a certain level. The auto-delete feature deletes older RAW data first, until the partition containing rawdir uses less than raw_autodelete_th of disk space.

6.3.2. Destination and file structure of the RAW data¶

A directory for each measurement is created in the RAW data storage directory, and the dump files are saved in that directory.

+ /opt/vm2m/var/lib/intdash
  + raw
    + 1562549837.123456789     # (1)
      - 001_000.raw            # (2)
      - 002_000.raw            # (3)
    + 1562549837.999999999
      - 001_000.raw
      - 001_001.raw

Number	Description
(1)	Measurement directory
(2)	Dump of data from channel 1 Device Connector
(3)	Dump of data from channel 2 Device Connector

6.3.2.1. Measurement directory¶

A new measurement directory is created for each measurement. The directory name is the Edge RTC base time of the measurement in Unix time down to nanoseconds. (Example: The directory name 1562549837.123456789 means the base time 2019-07-08T01:37:17.123456789 UTC.)

6.3.2.2. Dump file¶

A dump file is created for each Device Connector. The format of the file name is as follows:

XXX_NNN.raw

XXX: Device connector channel number (decimal number: 000-255)
NNN: Counter number (decimal number: 000-999)

If the size of the dump file is 512MB or more, a new file will be created with the next counter number.

6.3.2.3. Dump file format¶

The dump file is a direct dump of data in FIFO data format used between Agent and Device Connector.

Sample (NMEA)

6.3.3. Tool for analyzing RAW data¶

Tools for analyzing RAW data are installed with the Agent.

Usage example:

/opt/vm2m/bin/rawutil -P hexdump /opt/vm2m/var/lib/intdash/raw/1562551394.826226991/000_000.raw

For more information on how to use the tool, see the rawutil help.

/opt/vm2m/bin/rawutil -h

6.4. Retransmission data¶

Data that is tried to be sent by a Realtime or Bulk client but fails is retained as retransmission data. The time-series data held as retransmission data is sequentially retransmitted by the Resend client.

6.4.1. Automatic stop when disk space is low¶

If the disk usage exceeds the threshold, the Agent stops automatically. The conditions are as follows ("[]" represents the key in the configuration file.):

[manager.required_space] < Usage of the partition where retransmission data is stored (%)
[manager.required_space_raw] < Usage of the partition containing [manager.rawdir] (%)

6.4.2. Save destination and file structure¶

A measurement directory is created under the retransmission data directory, and the retransmission section file is saved in it.

+ /opt/vm2m/var/lib/intdash
  + meas
    + <SERVER_NAME>                             # (1)
      + CCCCCCCCC_TTTTTTTTTT.NNNNNNNNN          # (2)
        - .meas.<MEAS_UUID>                     # (3)
        - .meta                                 # (4)
        - SSSSSSSSSS_TTTTTTTTTT.NNNNNNNNNF.EXT  # (5)
        - SSSSSSSSSS_TTTTTTTTTT.NNNNNNNNNF.EXT
    + <SERVER_NAME>
      + CCCCCCCCC_TTTTTTTTTT.NNNNNNNNN
        - .meas.<MEAS_UUID>
        - .meta
        - SSSSSSSSSS_TTTTTTTTTT.NNNNNNNNNF.EXT
        - SSSSSSSSSS_TTTTTTTTTT.NNNNNNNNNF.EXT

Number	Description
(1)	Server directory
(2)	Measurement directory
(3)	UUID of measurement
(4)	Measurement metadata
(5)	Retransmission section

6.4.2.1. Server directory¶

The server directory stores the measurement data to be sent to this server. The directory name is the name of the server.

6.4.2.2. Measurement directory¶

A new measurement directory is created for each measurement. The format of the directory name is as follows:

CCCCCCCCC_TTTTTTTTTT.NNNNNNNNN

CCCCCCCCC: Number of measurements
TTTTTTTTTT.NNNNNNNNN: EdgeRTC base time of the measurement (Unix timestamp down to nanoseconds)

6.4.2.3. Measurement UUID file¶

The measurement UUID file is an empty file that has the measurement UUID in the filename. This file is created to make it easier for users to find measurements. The file name is .meas.<first 8 digits of measurement UUID>.

6.4.2.4. Measurement metadata file¶

The following information about measurement is stored in the measurement metadata file.

Serial number (Serial number of the last section)
Unit count (Total number of units)
Number of retransmission section files
Retransmission file size (Total retransmission file size)
Measurement UUID flag (Indicates whether the measurement UUID was obtained)
End flag (Indicates whether the edge notified the server that the measurement is finished.)
Measurement tag flag (Indicates whether the measurement tag was sent to the server)
Measurement count (Counter number given to the measurements created in this terminal)
EdgeRTC base time
Measurement duration

The file name is .meta.

6.4.2.5. Retransmission section file¶

A retransmission section file is a file that dumps only the units in a particular intdash Section.

The file name format is SSSSSSSSSS_TTTTTTTTTT.NNNNNNNNNF.EXT.

S: Section serial number
T: Relative time (seconds) from EdgeRTC
N: Relative time from Edge RTC (nanoseconds)
F: Flag (B: Includes base time, None: The base time is not included.)
EXT: Extension (bin: retransmission data, store: data for the Bulk client, store.bin: retransmission data from the Bulk client)

Example:

0000000000_0000000013.517448529B.bin
0000000001_0000000014.002924135.bin
0000000002_0000000015.000175599.store
0000000003_0000000016.002819513.store.bin

6.5. FIFO data format used between Agent and Device Connector¶

The data format for the Agent FIFO consists of a common header part and a data-type specific part.

See Common header for the common header part.
See Data type-specific part for the specific parts of each data type.

The following is an example of the primitive string type "Hello".

Important

The data format and the data type used by the FIFO between the Agent and the Device Connector is different from those of iSCP v1.

6.5.1. Common header¶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+---------------+
| Type          | Length                                        |
+---------------+---------------+---------------+---------------+
| Time Sec                                                      |
+---------------+---------------+---------------+---------------+
| Time Nano                                                     |
+---------------+---------------+---------------+---------------+
| DataType      | SeqNo         | Data                          |
+---------------+---------------+---------------+-----------//--+

Field name	Byte length	Endian	Signed	Value	Description
Type	1	—	No	1	Message type (fixed at 1)
Length	3	LE	No	10-16777216	Size after Time Sec (including Time Sec)
Time Sec	4	LE	No	0-4294967295	Monotonically increasing system time (seconds) 1
Time Nano	4	LE	No	0-999999999	Monotonically increasing system time (nanoseconds)
DataType	1	—	No	—	FIFO data type codes (see data type table below)
SeqNo	1	—	No	0-255	Sequential number (data can be sent even if fixed to 0) (When sending data using upstream, the Agent receives this field from the device connector, but does not send it to the intdash server. Also, this field will always be 0 in data received by the device connector from the Agent when using downstream.)
Data	0-16777208	—	—	—	Data (see section for each FIFO data type)

1: In programming languages that can use POSIX, you can get this data by specifying CLOCK_MONOTONIC_RAW in clock_gettime() function.

Data types

Status (data type: 0x03)
Basetime (data type: 0x19)
NMEA (data type: 0x10)
CAN / CAN-FD (data type: 0x11)
JPEG (data type: 0x12)
H.264 (data type: 0x1C)
String (Data type: 0x1D) Primitive string type
Float (data type: 0x01E) Primitive Float64 type
Int (data type: 0x1F) Primitive Int64 type
Bytes (data type: 0x20) Primitive byte array type
PCM (data type: 0x22)
Generic (data type: 0x7F)

6.5.2. Data type-specific part¶

6.5.2.1. Status (data type: 0x03)¶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+---------------+
| StatusId      | Reserved      | Size                          |
+---------------+---------------+---------------+---------------+
| Data                                                          |
+---------------+---------------+---------------+-----------//--+

Field name	Byte length	Endian	Signed	Value	Description
StatusId	1	—	No	0x90	Status type (fixed at 0x90)
Size	2	LE	No	0-32767	Data size
Data	0-32767	—	—	—	JSON string

6.5.2.2. Basetime (data type: 0x19)¶

For Agents that receive data using a downstream, if receive_basetime is set to true in the device connector settings, base time data is passed from the Agent to the device connector. This is the data format used in this case.

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+---------------+
| Source        | Reserved                                      |
+---------------+---------------+---------------+---------------+
| Sec                                                           |
+---------------+---------------+---------------+---------------+
| Nano Sec                                                      |
+---------------+---------------+---------------+---------------+

Field name	Byte length	Endian	Signed	Value	Description
Source	1	—	U	Source (see Source below)	Source of time
Sec	4	LE	U	0-4294967295	Time at the start of the measurement, elapsed from the UNIX epoch (seconds).
Nano sec	4	LE	U	0-999999999	Time at the start of the measurement, elapsed from the UNIX epoch (nanoseconds).

Source	Source of time
1	Time obtained with EdgeRTC
2	Time obtained with NTP
3	Time obtained with GPS
255	Manually set time

6.5.2.3. NMEA (data type: 0x10)¶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+---------------+
| Size                                                          |
+---------------+---------------+---------------+---------------+
| String                                                        |
+---------------+---------------+---------------+-----------//--+

Field name	Byte length	Endian	Signed	Value	Description
Size	4	LE	No	0-4294967295	Data size
Data	0-4294967295	—	—	—	NMEA string

6.5.2.4. CAN / CAN-FD (data type: 0x11)¶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+---------------+
| ID                                                            |
+---------------+---------------+---------------+---------------+
| DLC           | Data                                          |
+---------------+---------------+---------------+-----------//--+

Field name	Byte length	Endian	Signed	Value	Description
ID	4	LE	No	0-4294967295	CAN ID (set the first bit to 1 in the case of extended CAN)
DLC	1	—	No	0-255	Data size
Data	0-255	—	—	—	data

6.5.2.5. JPEG (data type: 0x12)¶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+---------------+
| JPEG                                                          |
+---------------+---------------+---------------+-----------//--+

Field name	Byte length	Endian	Signed	Value	Description
JPEG	0-16777216	—	—	—	JPEG (ISO / IEC 10918-1, Annex B) binary data

6.5.2.6. H.264 (data type: 0x1C)¶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+---------------+
| NALType       | NALUnits                                      |
+---------------+---------------+---------------+-----------//--+

Field name	Byte length	Endian	Signed	Value	Description
NALType	1	—	—	—	NALType (see NAL Type below)
NALUnits	—	—	—	—	NALUnits

NALType

NALType	Data to be stored in NAL Units
0x00	Concatenation of NAL Units in 1 frame 2 which are IDR slices (nal_unit_type == 5) 3
0x01	Concatenation of NAL Units in 1 frame which are non-IDR slices (nal_unit_type == 1)
0x08	Concatenation of NAL Units 4 required for H.264 decoding. It is assumed that one is generated in advance for each NALType 0x00.

2: The sequence of NAL units of the same nal_unit_type 1-5 (Coded slice), starting from the unit where first_mb_in_slice_header in slice_header is 0, ending with the unit before the next 0.
3: According to “Byte stream format (Annex B)” in “ITU-T Rec. H.264 | ISO/IEC 14496-10 Advanced Video Coding”, concatenate as follows: start code prefix + NAL unit + start code prefix + NAL unit ... start code prefix + NAL unit
4: The NAL units required for H.264 decoding are SPS (nal_unit_type == 7) and PPS (nal_unit_type == 8).

6.5.2.7. String (Data type: 0x1D) Primitive string type¶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+---------------+
| ID Length     | ID                                            |
+---------------+---------------+---------------+-----------//--+
| Data                                                          |
+---------------+---------------+---------------+-----------//--+

Field name	Byte length	Endian	Signed	Value	Description
ID Length	1	—	—	0-255	ID length
ID	0-255	—	—	0-16777215	UTF-8 encoded ID
Data	—	—	—	—	String

6.5.2.8. Float (data type: 0x01E) Primitive Float64 type¶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+---------------+
| ID Length     | ID                                            |
+---------------+---------------+---------------+-----------//--+
| Data                                                          |
+---------------+---------------+---------------+---------------+
|                                                               |
+---------------+---------------+---------------+---------------+

Field name	Byte length	Endian	Signed	Value	Description
ID Length	1	—	—	0-255	ID length
ID	0-255	—	—	0-16777215	UTF-8 encoded ID
Data	8	LE	—	—	Byte string containing double-precision floating point number (based on IEEE 754)

6.5.2.9. Int (data type: 0x1F) Primitive Int64 type¶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+---------------+
| ID Length     | ID                                            |
+---------------+---------------+---------------+-----------//--+
| Data                                                          |
+---------------+---------------+---------------+---------------+
|                                                               |
+---------------+---------------+---------------+---------------+

Field name	Byte length	Endian	Signed	Value	Description
ID Length	1	—	—	0-255	ID length
ID	0-255	—	—	0-16777215	UTF-8 encoded ID
Data	8	LE	Yes	—	Int64

6.5.2.10. Bytes (data type: 0x20) Primitive byte array type¶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+---------------+
| ID Length     | ID                                            |
+---------------+---------------+---------------+-----------//--+
| Data                                                          |
+---------------+---------------+---------------+-----------//--+

Field name	Byte length	Endian	Signed	Value	Description
ID Length	1	—	—	0-255	ID length
ID	0-255	—	—	0-16777215	UTF-8 encoded ID
Data	—	—	—	—	Binary data

6.5.2.11. PCM (data type: 0x22)¶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+---------------+
| FormatID                      | Channels                      |
+---------------+---------------+---------------+---------------+
| SampleRate                                                    |
+---------------+---------------+---------------+---------------+
| BitsPerSample                 | Data                          |
+---------------+---------------+---------------+-----------//--+

Field name	Byte length	Endian	Signed	Value	Description
FormatID	2	LE	No	0-65535	Format ID defined in WAVE 5
Channels	2	LE	No	0-65535	Number of audio channels stored
SampleRate	4	LE	No	0-4294967295	Sampling frequency [Hz]
BitsPerSample	2	LE	No	0-65535	Bit rate [bit/sample]
Data	—	—	—	—	Waveform information conforming to WAVE 5

5(1,2): WAVE (RIFF waveform Audio Format), a container format for audio data

6.5.2.12. Generic (data type: 0x7F)¶

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+---------------+---------------+---------------+---------------+
| ID                                                            |
+---------------+---------------+---------------+---------------+
| Data                                                          |
+---------------+---------------+---------------+-----------//--+

Field name	Byte length	Endian	Signed	Value	Description
ID	4	LE	No	0-4294967295	Numeric ID
Data	0-65531	—	—	—	Arbitrary data

6.6. All settings for Agent¶

This section describes all items in the Agent configuration file.

The configuration file has the following structure. Refer to the following sections for details.

Note

The "logger" in the configuration file refers to the Device Connector.

Key	Type	Description
manager	object	Manager settings
clients	object[]	Client settings
loggers	object[]	Device connector settings

Example

{
  "manager": {
    ...
  },
  "clients": [{
    ...
  }],
  "loggers": [{
    ...
  }]
}

6.6.1. Replacing settings with environment variables¶

The string type setting values in the configuration file can be given by the environment variables. By giving the settings in the environment variables, you can flexibly change the settings at startup without rewriting the configuration file.

For example, to give a value to clients.my_secret using an environment variable, use the variable in the configuration file as follows. In the example below, the variable $SECRET is used.

{
  ...
  "clients": [
    {
      ...
      "my_secret": "$SECRET",
    }],
   ...
}

Then, set the environment variable with a prefix INTDASH_EDGE_ added to the above variable name. In this example, set a variable INTDASH_EDGE_SECRET.

When intdash-edge-manager is started using this configuration file, $SECRET in the configuration file will be expanded to the value of the environment variable INTDASH_EDGE_SECRET.

Note

Only string-type values can be replaced by environment variables.

6.6.2. Manager settings¶

Settings related to the manager are set in the manager field of the configuration file as an object. The items are as follows.

Key	Type	Default value	Description
`workdirs`	string[]	`["/opt/vm2m/var/lib/intdash/meas", "/opt/vm2m/var/run/intdash"]`	Directory path to be created at startup
`basetime`	string	`"/opt/vm2m/var/run/intdash/basetime"`	Path of the file that stores the base time information
`ntpserver`	string	`"ntp.intdash.jp"`	Default value of the NTP server name used by the Status plugin to obtain the NTP based base time. The value listed here will be used as the default value for `loggers[].details.plugin_arg.ntpserver`.
`meas_root`	string	`"/opt/vm2m/var/lib/intdash/meas"`	Directory path to store the retransmission data
`rawdir`	string	`"/opt/vm2m/var/lib/intdash/raw"`	Directory path to save RAW data
`raw_autodelete`	bool	`true`	Automatic deletion of RAW data
`raw_autodelete_th`	number	`85`	Threshold for automatically deleting RAW data [%]
`required_space`	number	`90`	Threshold at which the Agent stops measurement [%]. The measurement is automatically stopped when the free space ratio of the partition that stores the retransmission data becomes larger than this value. If you set it to 100, it will not stop automatically.
`required_space_raw`	number	`90`	Threshold at which the Agent stops measurement [%]. The measurement is automatically stopped when the free space ratio of the partition that stores RAW becomes larger than this value. If you set it to 100, it will not stop automatically.
`stat`	string	`"/opt/vm2m/var/run/intdash/manager.stat"`	Path to the file that records the status of the Manager
`process_stat`	string	`"/opt/vm2m/var/run/intdash/process.stat"`	Path of the file that records the status of the Process
`wwan_stat`	string	`"/opt/vm2m/var/run/intdash/wwan.stat"`	Path of the file that records the WWAN status
`logger_stat`	string	`"/opt/vm2m/var/run/intdash/logger_%03hhu.stat"`	Format of the file path to record the status of the Device Connector. The first format specifier (`%03hhu` in the default value) is replaced with the channel number.
`system_stat`	string	`"/opt/vm2m/var/run/intdash/system.stat"`	Path to the file that records the status of the System
`filters`	object[]	`[]`	Filter settings. See Filter at the sender's side for more information.

6.6.3. Client settings¶

Configuration of clients is done in the clients field of the configuration file. The configuration is set as an array of configuration (objects) for the clients to be used. Objects for unused clients are not needed. The following items can be set.

Key	Type	Default value	Description
`protocol`	string	`""`	Name of communication library to use (`mod_websocket.v2\|mod_http`)
`type`	string	`""`	Operating mode of the client (`realtime\|bulk\|resend\|control`). If the protocol is mod_http, only resend can be selected.
`mode`	number	`1`	Whether to persist data to the server (0: do not persist, 1: persist). This setting is only used if type is `realtime\|bulk\|resend` and protocol is `mod_websocket.v2`.
`my_id`	string	`""`	Edge UUID used to connect to intdash
`my_secret`	string	`""`	Client secret used to connect to intdash
`auth_path`	string	`"/opt/vm2m/var/lib/intdash/.auth"`	The path to the file where the connection information is saved. You must have write access to this file.
`my_token`	string	`""`	Edge token used to connect to intdash
`dst_id`	string[]	`[]`	UUID of the destination edge. Used when "type" is `realtime\|bulk\|resend`. (optional)
`down_dst_id`	string	`""`	Only data with the destination UUID specified here will be received from the server. However, if `00000000-0000-0000- 0000-000000000000` is specified, data for any destination will be received. If the specified string cannot be parsed as a UUID (including the default `""` case), the UUID set in `my_id` is assumed and only data for `my_id` is received from the server. Used when "type" is "control".
`ctlr_id`	string	`""`	UUID of the edge sending the data you want to receive, used if type is "control"; if you want to receive data from two edges, set `ctlr_ids` instead of this setting.
`ctlr_ids`	string[]	`[]`	UUIDs of the edges sending the data you want to receive, used if type is "control". You can specify up to two UUIDs. If `ctlr_id` is set to a valid value, the UUIDs specified here will be ignored.
`ctlr_flts`	object[]	`[]`	Filters to be applied to data received from the server. See ctlr_flts.channel, ctlr_flts.dtype, ctlr_flts.ids for filter content.
`ctlr_flts.channel`	number	`0`	Channel to receive data from the server. Used when "type" is "control".
`ctlr_flts.dtype`	number	`0`	iSCP data type code of data to be received from the server. Used when "type" is "control".
`ctlr_flts.ids`	number[] or string[] (Data IDs for CAN and Generic are numbers, so use number[]; data IDs for String, Float, Int, and Bytes are strings, so use string[].)	`[]`	Data ID of the data to be received from the server (optional). Used when "type" is "control". In the case of CAN data, if you set the value to an empty array `[]`, data of any ID will be received. For data types other than CAN, setting `[]` will cause nothing to be received.
`unit_flush_cycle`	number	`5`	Flush interval [msec]. Used when "type" is "realtime".
`resend_cycle`	number	`1000`	Retransmission cycle [msec]. Used when "type" is "resend".
`noresend`	bool	`false`	Sets whether the realtime client will create data for retransmission. Used when "type" is "realtime". If `true`, data that could not be sent in realtime will not be stored for retransmission, and therefore will not be retransmitted.
`store_flushtime`	number	`3000`	Flush interval [msec]. Used when "type" is "bulk".
`store_flushsize`	number	`10000`	Flush interval [number of units]. Used when "type" is "bulk".
`store_cycle`	number	`1000`	Transmission interval [msec]. Used when "type" is "bulk".
`http_client_count`	number	`1`	Number of simultaneous retransmissions. Used when the protocol is "mod_http".
`connection.host`	string	`""`	The hostname + domain name (FQDN) of the intdash server used by the edge (e.g., `dummy.intdash.jp`).
`connection.path`	string	`"/"`	Path to the server resource. (If the protocol is `mod_websocket*`, use `/api/v1/ws/measurements`. If the protocol is `mod_http`, use `/api/v1/measurements`.)
`connection.ssl`	string	`"secure"`	Security settings for SSL connections (`none\|lax\|secure`).
`connection.port`	number	`443`	The port of the intdash server to which the edge connects.
`connection.cert`	string	`"" (Use a certificate installed in the OS.)`	Server certificate file path (optional).
`connection.client_cert`	string	`""`	Certificate when using a client certificate (optional).
`connection.client_key`	string	`""`	Private key when using a client certificate (optional).
`project_uuid`	string	`""`	UUID of the project to be used by the edge; if no project is specified, Global Project is used.
`user_agent`	string	`"IntDash-Edge/unknown (Unknown; Unknown)"`	User agent (optional).

6.6.4. Device connector settings¶

The settings for Device Connectors are in the loggers field of the configuration file as an array of objects. The items are as follows.

Key	Type	Default value	Description
`path`	string	`""`	Full path of the Device Connector you want to start automatically.
`conf`	string	`""`	String to be passed as the second argument when the Device Connector is started automatically. The first argument is `-C`.
`details.plugin`	string	`""`	Name of the plugin to use (`fifo\|status`).
`details.plugin_dir`	string	`"/opt/vm2m/lib/plugins"`	Directory where the executable file of the plug-in is stored.
`details.plugin_arg`	string	`{}`	JSON object for plugin settings. Settings differ depending on the plug-in.
`details.plugin_arg.ntpserver`	string	Value set in `manager.ntpserver`.	The name of the NTP server that the Status plugin uses to obtain the NTP based base time.
`details.plugin_with_process`	bool	`false`	Whether to start the Device Connector automatically when using the plug-in.
`connections[].channel`	number	`-1`	Channel (0-255) to set for the Device Connector.
`connections[].channel_rx`	number	`-1` (If set to -1, the same value as the "channel" is used)	Channel used to send data to the Device Connector when using downstream
`connections[].receive_basetime`	bool	`false`	Whether base time data is sent to the Device Connector when using downstream.
`connections[].fifo_tx`	string	`"/opt/vm2m/var/run/intdash/logger_%03hhu.tx"`	FIFO file path used for communication. The first format specifier (`%03hhu` in the default value) is replaced with the channel number.
`connections[].fifo_rx`	string	`"/opt/vm2m/var/run/intdash/logger_%03hhu.rx"`	FIFO file path used for communication. The first format specifier (`% 03hhu` in the default value) is replaced with the channel number,
`connections[].disable_raw`	number	`0`	Whether to save the data of this channel as RAW data (0: save, non-zero: do not save).

6.7. Agent logs¶

The Agent outputs log messages to the standard output.

By redirecting the standard output, log messages can be saved to a file. For example, if you start the Agent as follows, log messages will be output to /var/run/intdash/intdash.log (${CONF_PATH} is the full path of the configuration file.

/opt/vm2m/sbin/intdash-edge-manager -C ${CONF_PATH} >/var/run/intdash/intdash.log 2>&1

6.7.1. Log message format¶

Log messages are output in a format similar to the following example.

01/13 01:51:49 intdash-edge-manager(1532): INFO  : procedure(): CREATED
01/13 01:51:49 intdash-edge-manager(1532): INFO  : start(): Manager thread STARTING
01/13 01:51:49 intdash-edge-manager(1532): INFO  : procedure(): STARTED
01/13 01:51:49 intdash-edge-manager(1532): INFO  : base_proc(): Manager thread STARTED
01/13 01:51:49 intdash-edge-manager(1532): INFO  : proc(): Basetime (RTC monotonic) : 58469.155277440
01/13 01:51:49 intdash-edge-manager(1532): INFO  : proc(): Basetime (RTC realtime) : 1610502709.225218700
01/13 01:51:49 intdash-edge-manager(1532): INFO  : start(): RawDataHandler thread STARTING
01/13 01:51:49 intdash-edge-manager(1532): INFO  : start(): DataSaver thread STARTING
 ^              ^                           ^       ^
(1)            (2)                         (3)     (4)

Number	Description
(1)	Date and time of occurrence
(2)	Module name (process number)
(3)	Prefix (`INFO`: information about normal operation, `WARN`: information about recoverable errors, `ERR`: information about non-recoverable errors)
(4)	Function name and log content

6.7.2. Log messages about the Agent application¶

The main log messages related to the Agent application are as follows.

6.7.2.1. Creating a FIFO to connect to the device connector¶

INFO  : procDataReader(): DeviceProcess create pipe : /opt/vm2m/var/run/intdash/logger_000.tx

The path in the log message is the path of the FIFO created by intdash-edge-manager.

6.7.2.2. Opening FIFO¶

INFO  : procDataReader(): DeviceProcess open pipe : /opt/vm2m/var/run/intdash/logger_000.tx

The path in the log message is the path of the FIFO opened by intdash-edge-manager.

6.7.2.3. Closing FIFO¶

INFO  : procDataReader(): close pipe:/opt/vm2m/var/run/intdash/logger_004.tx

The path in the log message is the path of the FIFO that was closed by intdash-edge-manager.

6.7.3. Log messages about real-time transmission via WebSocket¶

The following are the main log messages related to real-time transmission via WebSocket.

6.7.3.1. Establishing a connection with the server¶

INFO  : callbackPacketEstablish(): REALTIME Upstream request succeed

6.7.3.2. Sending data of one section¶

This message indicates that the data of one section has been sent. At this point, processing on the server side has not been completed.

INFO  : callbackSndSectionEnd(): REALTIME 46 units and 1 basetimes with ID:2
                                           ^           ^                   ^
                                          (1)         (2)                  (3)

Number	Description
(1)	Number of units in the section (excluding base time)
(2)	Number of base time units included in the section
(3)	Serial number given to the section

6.7.3.3. Completing transmission of a single section¶

This message indicates that the processing of one section has been completed on the server side. When an ACK is received from the server, this message is output.

INFO  : callbackRcvPacket(): REALTIME(c4a0e287) ACK 2 (45/45 units)
                                      ^             ^  ^  ^
                                     (1)           (2)(3)(4)

Number	Description
(1)	Measurement ID
(2)	Serial number given to the section
(3)	Number of units in this section
(4)	Total number of units sent in this measurement

6.7.3.4. Saving data of one section as a file for retransmission¶

INFO  : saveSection(): Store   2 units to /opt/vm2m/var/lib/intdash/meas/<SERVER_NAME>/<MEASUREMENT>/<SECTION>.bin
                               ^           ^
                              (1)         (2)

Number	Description
(1)	Number of units in the section
(2)	Path of the retransmission section file

6.7.3.5. Closing the connection to the server¶

INFO  : callbackPacketEstablish(): REALTIME Upstream closed

6.7.4. Log messages about retransmissions via HTTP¶

The main log messages related to retransmissions via HTTP are as follows

6.7.4.1. Completing retransmission of one section¶

INFO  : postSection(): RESEND 32187 units and ? basetimes with ID:80891
                              ^                                   ^
                             (1)                                 (2)

Number	Description
(1)	Number of units in the section
(2)	Serial number given to the section