How to log Time Series Data

This guide describes how to configure an AuterionOS app to store time series data in flight logs. It assumes you have the following file structure for your app.

├── auterion-app.yml
├── Dockerfile
├── msgs_ws/
│   └── src/
│       ├── custom_msgs/
│       │   └── msg/
│               └── CustomTopic.msg
│       ├── px4_msgs/
│       │   └── msg/
│               └── VehicleOdometry.msg
│       └── std_msgs/
│           └── msg/
│               └── ColorRGBA.msg
├── README.md
├── ros_ws/
│   └── src/
│       └── fake_visual_odometry/
│           └── src/
│               └── fake_visual_odometry_node.cpp
└── settings.default.env

Enable logging in the auterion-app.yml file

The first step is to add the logging key to the auterion-app.yml file. This should come after the services key.

Three keys can be nested below logging:

1. msg-paths

2. subscriptions - used for simple configuration.

3. profiles - used for advanced configuration.

auterion-api-version: 3
services:
  test-app:
    ...

logging:
  msg-paths:
    ...
  subscriptions:
    ...
  profiles:
    ...

The 'msg-paths' key

The msg-paths key defines one or more file paths to the type definitions of any messages to be logged. This must be specified when using custom message types.

This example app publishes messages over DDS or ROS 2 to log custom app data. It must include paths to the three ROS 2 message packages, as shown below.

logging:
  # Paths to message definitions of logged topics
  msg-paths: [msgs_ws/src/custom_msgs. msgs_ws/src/px4_msgs, msgs_ws/src/std_msgs] 

If a message doesn't contain a timestamp field, the logger will add it.

The message definitions specified in msg-paths can be found under /data/app-config/logging/msgs after the app is installed on AOS.

Selecting the data to log

By default, app data is logged from when the vehicle arms until it disarms. The subscriptions key is used to specify which data is logged.

For apps with more advanced logging requirements, the profiles key gives more flexibility about the conditions in which data is logged.

Basic configuration: the 'subscriptions' key

The subscriptions key specifies a list of messages or topics to be logged. Each subscription entry must contain a key specifying the topic name and either

• a simple string specifying the message type, or

• a detailed object with additional properties, such as logging rates.

This example app will have some simple and some detailed subscriptions.

logging:
  subscriptions:
    # Simple subscriptions
    /custom_topic: custom_msgs/CustomTopic
    /fmu/in/vehicle_visual_odometry: px4_msgs/VehicleOdometry
    # Detailed subscription
    /test_color:
      type: std_msgs/ColorRGBA  # the message type
      max_rate_hz: 2            # log this topic at 2Hz or less

Advanced configuration: the 'profiles' key

The profiles key allows for the customisation of which data is logged and under which conditions. Recall that the default behaviour is to log only while armed.

Two profiles can be configured:

1. default overrides or extends the default logging behaviour, and

2. verbose allows for logging additional data and/or logging in different conditions.

The available run conditions are: always, never, while-armed and while-disarmed.

When the run conditions are met, each profile generates its own ULog file in a defined directory, with its own logging rate and set of logged messages.

The default profile is streamed to Auterion Suite.

The logging rate of this profile must be kept low, in the order of a few KB/s.

Profile	default	verbose
Default condition	while-armed	never
Log directory	/data/log/flight-stack	/data/log/flight-stack-verbose
Streamed to Suite	Yes	No
Constrained log rate	Yes	No

The logging profiles are customised with a list of different conditions under the when key. Each entry in this list must have two keys:

1. condition: specifying when that particular configuration applies, and

2. subscriptions: the list of messages to be logged in that condition. The contents of this key have the same format as described in the basic configuration above.

Profile extensions will be placed in /data/app-config/logging/profiles.

Examples of different logging profiles are given below.

Example 1: Add a verbose log profile

The verbose profile is enabled and set to always log a custom topic at 1Hz.

logging:
  profiles:
    verbose:
      when:
        - condition: 'always'
          subscriptions:
            # Subscriptions specific to the 'always' condition.
            /custom_topic:
              type: custom_msgs/CustomTopic
              max_rate_hz: 1

Example 2: Add a verbose profile with multiple conditions

The verbose profile is enabled and set to always log a custom topic at 1Hz while disarmed and 20Hz while armed. It additionally logs a visual odometry topic, but only while armed.

logging:
  profiles:
    verbose:
      when:
        - condition: 'while-disarmed'
          subscriptions:
            # Subscriptions specific to the 'while-disarmed' condition.
            /custom_topic:
              type: custom_msgs/CustomTopic
              max_rate_hz: 1
        - condition: 'while-armed'
          subscriptions:
            # Subscriptions specific to the 'while-armed' condition.
            /custom_topic:
              type: custom_msgs/CustomTopic
              max_rate_hz: 20
            /fmu/in/vehicle_visual_odometry: px4_msgs/VehicleOdometry

Example 3: Override the default log profile

In the basic configuration example above, the custom and visual odometry topics were logged at full rate while armed.

In this example, that default profile is overridden and set to additionally log the custom and visual odometry messages at 1Hz while disarmed.

logging:
  profiles:
    verbose:
      when:
        # The original default profile
        - condition: 'while-armed'
          subscriptions:
            /custom_topic: custom_msgs/CustomTopic
            /fmu/in/vehicle_visual_odometry: px4_msgs/VehicleOdometry
            /test_color:
              type: std_msgs/ColorRGBA
              max_rate_hz: 2
        # Extend the profile to log while disarmed
        - condition: 'while-disarmed'
          subscriptions:
            /custom_topic:
              type: custom_msgs/CustomTopic
              max_rate_hz: 1
            /fmu/in/vehicle_visual_odometry:
              type: px4_msgs/VehicleOdometry
              max_rate_hz: 1
            /custom_topic:
              type: custom_msgs/CustomTopic
              max_rate_hz: 2

Monitoring and Troubleshooting

To monitor the logging activity and troubleshoot potential issues, use:

journalctl -u data-logger -b -f

If the message is logged successfully, running journalctl -u data-logger -b -f should contain: [dds_source.cpp:111] [dds] Subscriber matched update.

Key Limitations

• Maximum Topic Size: 65536 bytes. Ensure that messages do not exceed this size to prevent logging failures.

• No Support for Dynamically-Sized Messages: Static message definitions are required due to current logging limitations.