> For the complete documentation index, see [llms.txt](https://docs.auterion.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.auterion.com/app-development/auterion-sdk/object-detection-api.md).

# Object Detection API

Publish 2D object detections from your app. The SDK delivers them to the ground station, where they are drawn on top of the live video, and optionally to onboard consumers as well.

{% hint style="info" %}
Related C++ Headers in the Auterion SDK:

* [`<auterion_sdk/object_detection/object_detection_provider.hpp>`](https://auterion.github.io/auterion-sdk/object__detection__provider_8hpp_source.html)
* [`<auterion_sdk/object_detection/detections_from_image.hpp>`](https://auterion.github.io/auterion-sdk/detections__from__image_8hpp_source.html)
* [`<auterion_sdk/object_detection/image_detections_2d.hpp>`](https://auterion.github.io/auterion-sdk/image__detections__2d_8hpp_source.html)
* [`<auterion_sdk/object_detection/detection_2d.hpp>`](https://auterion.github.io/auterion-sdk/detection__2d_8hpp_source.html)
* [`<auterion_sdk/object_detection/class_catalog.hpp>`](https://auterion.github.io/auterion-sdk/class__catalog_8hpp_source.html)
  {% endhint %}

<figure><img src="https://3329189600-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FFW1Ge1p1f6WHyiYCb146%2Fuploads%2FD2LGdAigk1Nr1qP4Bw9T%2Fimage.png?alt=media&amp;token=ee2de1e6-c2fc-4d46-9f0b-22a970f54363" alt="Detections published by your app, drawn over the live video in Auterion Mission Control."><figcaption><p>Detections published by your app, drawn over the live video in Auterion Mission Control.</p></figcaption></figure>

Available from Auterion SDK 1.27.0 and newer. This page assumes an app that already builds and runs on the vehicle: see [Development environment Setup](https://docs.auterion.com/app-development/app-development/application-development) and [Building your First App](https://docs.auterion.com/app-development/app-framework/app-framework).

### Quick start

```cpp
#include <auterion_sdk/auterion.hpp>
#include <auterion_sdk/camera/camera.hpp>
#include <auterion_sdk/object_detection/detections_from_image.hpp>
#include <auterion_sdk/object_detection/object_detection_provider.hpp>

int main(int argc, char* argv[]) {
    auterion::SDK sdk{argc, argv, "my_detector"};

    const std::vector<std::string> class_names{"person", "vehicle", "boat"};
    auterion::ObjectDetectionProvider provider{
        sdk, auterion::DetectionProviderOptions{}.withClassCatalog(
                 auterion::ClassCatalog::fromModelOrder(class_names))};

    auto camera = auterion::Camera::openFirst(sdk);
    if (!camera) {
        return 1;  // no camera appeared within the timeout
    }
    camera->subscribeImage([&](const auterion::Image& image) {
        auto batch = auterion::detectionsForImage(image);

        for (const auto& result : runInference(image)) {
            auterion::Detection2D detection;
            detection.bbox.center = Eigen::Vector2d{result.cx, result.cy};
            detection.bbox.size_x = result.width;
            detection.bbox.size_y = result.height;
            detection.object_hypotheses.push_back(
                {class_names.at(result.class_index), result.score});
            detection.track_id = result.track_id;
            batch.detections.push_back(std::move(detection));
        }

        provider.updateDetections2D(batch);
    });

    sdk.run();
    return 0;
}
```

`runInference` stands in for your detector: the example consumes a box center and size in source-image pixels, a class index, a score and a track id. Everything else is the whole API for the common case, and the rest of this page is detail. Camera selection and image access are covered by the [Camera API](https://docs.auterion.com/app-development/auterion-sdk/system-state-api-1).

### Detections

```cpp
struct ImageDetections2D {
    ImageDetections2D(uint16_t frame_id, const rclcpp::Time& image_timestamp,
                      uint32_t image_width, uint32_t image_height);

    std::vector<Detection2D> detections;    // the only field you assign; may be empty
};

struct Detection2D {
    BoundingBox2D bbox;                             // center, size_x, size_y [px]
    std::vector<ObjectHypothesis> object_hypotheses;  // {class_id, score}
    std::optional<uint64_t> track_id;
    std::optional<Eigen::Vector3d> position_lla;    // lat [deg], lon [deg], alt [m, EGM96]
    std::optional<Eigen::Vector3d> velocity_ned;    // [m/s]
};
```

Call `updateDetections2D` once per processed frame, including frames with no detections: that is how the overlay clears.

Every batch needs the four frame fields, and `detectionsForImage` fills all four from an SDK image so you never restate them:

* **`frame_id`** ties your boxes to the exact frame they came from. The video the operator sees may be cropped, zoomed or rotated further down the pipeline, and the SDK uses `frame_id` to place your boxes on it correctly.
* **`image_width` / `image_height`** are required for the ground station, because boxes are normalized against the source image. A batch that passes 0 is skipped with a warning.

Boxes are in source-image pixels. If you run inference on a resized copy of the frame, map them back yourself: only you know whether the resize letterboxed, and the SDK does not guess.

Set **`track_id`** to the same value for the same object across frames if you track. The ground station keys its rendering by that id, so a stable id gives a box that follows the object. Leaving it unset is fine, the SDK fills in per-batch ids, but the overlay then has no identity between frames.

`position_lla` and `velocity_ned` are optional. Unset is transmitted as "unknown", not as zero.

The ground station draws every box labelled `<class name> #<object id> <confidence>%`; object id and confidence are always visible and cannot be suppressed. Only the highest-scoring hypothesis is shown.

Published detections can be selected for tracking: in tracking mode the operator taps a box and the tracker takes over that object, with nothing to implement in the detecting app. The operator workflow is described in [Tracking a detected object](https://docs.auterion.com/vehicle-operation/auterion-mission-control/strike-ui#tracking-a-detected-object). The selection reaches the tracking app through the [Visual Tracking API](https://docs.auterion.com/app-development/auterion-sdk/visual-tracking-api).

### Class names

Class ids go out exactly as you number them: nothing in the chain applies an offset, and the ground station resolves them against the catalog your app publishes.

```cpp
auto catalog = auterion::ClassCatalog::fromModelOrder({"person", "vehicle", "boat"});
// person -> 0, vehicle -> 1, boat -> 2
```

Use `ClassCatalog::fromClassIds({{1, "person"}, {4, "boat"}})` if your detector already numbers its classes. Without a catalog, hypothesis strings resolve numerically (`"3"` becomes class 3) and the ground station shows raw numbers. A class the SDK cannot resolve goes out as `auterion::kUnknownClassId`, so no ordinary class number is reserved and you can use 0.

After a model switch call `provider.setClassCatalog(new_catalog)` before publishing the first batch from the new model. It is safe to call while publishing.

#### Colour and category

`ClassCatalog::fromEntries` carries more than a name per class:

```cpp
auto catalog = auterion::ClassCatalog::fromEntries({
    {0, {"person", "#00FF00"}},
    {1, {"vehicle", "#FF8800", "neutral"}},
});
```

`category` selects how the ground station styles a class. It recognises `friend`, `friendly`, `neutral` and `hostile` (trimmed and lowercased) and draws any other value, including an unset one, in the default yellow. The derived colours are cyan for friend, green for neutral and red for hostile. A per-class `color` (`"#RRGGBB"`) overrides the derived colour.

### Stopping work when detection is off

Object detection can be requested to be switched off by other apps needing compute resources.

```cpp
provider.subscribeObjectDetectionEnabled([&](bool enabled) {
    if (enabled) {
        startInference();
    } else {
        stopInference();
    }
});
```

To poll instead, call `subscribeObjectDetectionEnabled()` without a callback, then read `provider.objectDetectionEnabledSubscription().last()`, guarded by `isLastValid()`.

### Output options

`DetectionProviderOptions` selects the channels and carries the catalog. Both channels are on by default, so an app that configures nothing already reaches on-board consumers and the ground station.

```cpp
auterion::DetectionProviderOptions{}.withRos(true).withMavlink(true).withClassCatalog(catalog);
```

Any number of providers on one vehicle may enable the ROS output. **At most one may enable the MAVLink output**, because the ground station sees a single detection source: one catalog, one object-id space. A second detector alongside an existing one takes `.withMavlink(false)`; two MAVLink providers give the operator interleaved, flickering boxes.

An integrator can flip either channel at launch through the parameter overrides `object_detection_ros_output` and `object_detection_mavlink_output`, without rebuilding the app.

The ROS output is `vision_msgs/msg/Detection2DArray` on `object_detection/detections_2d`, always in source-image pixels.