How Auterion tested

Summary of Auterion's test setup for Remote ID implemented in AuterionOS and AMC

Known issue: AMC does not show a warning if the GCS position fix is lost during flight

The main reference used for our tests is the document "Standard Practice for Remote ID Means of Compliance to Federal Aviation Administration Regulation 14 CFR Part 89" published by the American Society for Testing and Materials (ASTM), which specifies the Means of Compliance (MOC) for Remote ID.

Test methods for Designation F3586-22

Section 8 of the MOC covers the suggested test methods. The following is not a complete reproduction of the contents of the MOC. Only the information relevant to Auterion for testing Remote ID has been summarised below, accompanied by Auterion's test results.

MOC 8.7 specifies that all but the message interval test are to be conducted outdoors and with the use of a mobile device that has a receiver app installed. In our tests we used the Android Application OpenDroneID OSM. Auterion cannot guarantee that the use of this app sufficiently validates the Remote ID.

MOC 8.6 - Message interval inspection using Network sniffing

MOC 8.6.1 - 8.6.2.4 outline a test procedure using network sniffing and inspection of the recorded network traffic to make sure, that the following criterion is fulfilled:

The maximum observed time interval between two messages of a specific type is below the threshold specified in Specification F3411.

Test result by Auterion

This experiment was conducted on a bench test setup. We used Wireshark to analyze the received Wifi beacon frames with the Remote ID data.

The measurement results are based on the subsequent capture:

49KB

capture_result-v2.5.5.csv

• The inspected message headers show the required Remote ID data

• The interval between subsequent WiFi beacon frames is <= 0.337 seconds

• The interval between updated Wifi beacon frames is <= 1.001 seconds

• The maximum elapsed time since the time of applicability of the dynamic fields in the Location/Vector message as measured by the difference between the message timestamp and the received epoch time (synchronized to GPS time) is 0.939 seconds

MOC 8.8.1 - Power on and Pre-flight

The pilot should see all required information of the Remote ID system and should be able to observe the data on the receiver app mentioned before.

Test result by Auterion

• All Remote ID Data is shown in the receiver app while the vehicle is on the ground

• The timestamp is updated over time

• In AMC 1.14.6 the GCS altitude is reported correctly in as WGA84 altitude. In AMC 1.14.5 the GCS altitude is slightly off since AMC reports its altidude as MSL instead of the required WGA84 altitude. It's advised to use patched AMC 1.14.6 for the evaluation.

MOC 8.8.2 - Failure Mode Operational Test

The pilot should introduce a failure in Remote ID and verify that the vehicle rejects a takeoff command.

Test result by Auterion

• When denying the position fix for vehicle and GCS, Remote ID will report that it is not operational and the vehicle will reject arming commands

• When denying the position fix only for the GCS, the vehicle will report that it is not ready to fly and will reject arming commands

• When denying the position fix only for the vehicle, the pilot is able to arm and take off in manual flight mode, which does not require a GPS position. This is fine as according to note 6 in MOC 7.4.2 the Preflight self-test does not need to require a location fix in order to pass the check.

MOC 8.9.1: Nominal Operations

Step 1: The pilot should perform a take off, fly to an altitude of 10m above surface. Once airborne the Remote ID receiver application should receive and display the correct values for the required data fields, such as latitude and longitude of the operator's ground station for example.

Step 2: Fly to certain waypoints at a relative distance and altitude from the take off location. For Multirotors this is a cube spanning an altitude from 100ft (30.5m) to 200ft (61m). Fixed Wing aircraft are expected to loiter at 100ft altitude and 200ft altitude. with a loiter radius of 100ft. As in step 1 the information transmitted via Remote ID should be present and accurate during these flight maneouvers.

Test result by Auterion

• Performed a take off and flew to 10m above surface. The information in OpenDroneID reflects the altitude change accordingly.

• Fly in a circle at a constant speed of 1 m/s. OpenDroneID indicated that the vehicle is moving at a horizontal speed of 1 m/s

• During takeoff and landing, as well as during manual altitude changes, the vertical speed in OpenDroneID showed realistic numbers. A negative vertical speed indicates a descent.

MOC 8.9.2 - Emergency Status

If the Autopilot implements emergency status', as is the case with APX4, this needs to be tested and the information received from Remote ID verified. The receiver app should show the type or name of the emergency if one is present. The test procedure should be done for all automatic and manual emergency states.

Test result by Auterion

• When powering off the Remote Control or closing the AMC app during a flight, APX4 executes a failsafe and returns to home. The "Status" field in OpenDroneID changes from "Airborne" to "Emergency".

MOC 8.9.3 - Single (known/surveyed) Point Accuracy Measurement

This test is specific to each vehicle and its hardware. As such Auterion did not conduct this test.

The accuracy of the vehicle's GPS is to be determined and compared to a maximum value outlined in the specifications. For this test the vehicle is to be placed on a known surveyed physical position, which is then used to compare the data from Remote ID against.

MOC 8.9.5 - Omnidirectionality of the broadcast module

This test is specific to each vehicle and its hardware. As such Auterion did not conduct this test.

The broadcast module used for Remote ID should have an omnidirectional radiation. For this test the vehicle should be placed at a distance of 100m to the receiver device. Measurements are conducted while the vehicle is rotated at 9 degree intervals.

MOC 8.9.6 - Failure mode monitoring

Induce a real or an artificial failure in one of the components that are critical for a nominal function of the vehicle. The pilot should observe the failure indication received by the Remote ID app. Critical components are defined in MOC 7.2.3 and include positioning sensors as well as transmitters for the radio link.

Test result by Auterion

• Connecting AMC to the vehicle, then waiting for Remote ID to indicate that it's ready, then turning off the vehicle will result in AMC showing a failure with OpenDroneID

• Turning off the location service in Android after Remote ID indicated that it's ready does currently not trigger a warning or failure message in AMC. This is a known issue in AMC 1.14.5 / AMC 1.14.6 and will be patched in the future.