Object Detection API

Related C++ Headers in the Auterion SDK:

• <auterion_sdk/object_detection/object_detection_client.hpp>

• <auterion_sdk/object_detection/object_detection_provider.hpp>

The Object Detection API provides interfaces for both consuming and providing object detection results in the Auterion ecosystem. It consists of two main components:

1. Object Detection Client: Used by applications that want to receive object detection results

2. Object Detection Provider: Used by applications that perform object detection and want to publish their results

Object Detection Client

The Object Detection Client allows applications to subscribe to object detections and control the object detection service. This is useful for applications that need to react to detected objects or visualize detection results. Controlling the object detection service is particularly useful for resource management in cases where the detection result is used by another resource heavy application for which the resources of the object detection service should be freed.

Subscribing to Object Detections

To receive object detection results, instantiate an ObjectDetectionClient and subscribe to detections:

auterion::ObjectDetectionClient object_detection_client{sdk};

// Subscribe to detection updates with a callback
object_detection_client.subscribeDetections2D([](const auterion::ImageDetections2D& detections) {
    std::cout << "Received " << detections.detections.size() << " detections" << std::endl;
    
    // Process each detection
    for (const auto& detection : detections.detections) {
        // Access detection properties
        const auto& bbox = detection.bbox;
        std::cout << "Detection at position (" << bbox.center.x() << ", " << bbox.center.y() 
                  << ") with size " << bbox.size_x << "x" << bbox.size_y << std::endl;
        
        // Access object hypotheses (class labels and confidence scores)
        for (const auto& hypothesis : detection.object_hypotheses) {
            std::cout << "  Class: " << hypothesis.class_id 
                      << ", Score: " << hypothesis.score << std::endl;
        }
    }
});

Querying Latest Detections

You can also query the latest detection results without using callbacks:

auterion::ObjectDetectionClient object_detection_client{sdk};

// Activate subscription without callback
object_detection_client.subscribeDetections2D();

// Later, check if we have valid detections and process them
if (object_detection_client.detections2D().isLastValid()) {
    const auto detections = object_detection_client.detections2D().last();
    // Process detections...
}

Controlling the Object Detection Service

The client can enable or disable the object detection service:

auterion::ObjectDetectionClient object_detection_client{sdk};

// Enable object detection
object_detection_client.enableObjectDetection();

// ... perform operations that require object detection ...

// Disable object detection when no longer needed
object_detection_client.disableObjectDetection();

Object Detection Provider

The Object Detection Provider is used by applications that perform object detection and want to publish their results to other applications in the system. This is typically used by computer vision applications that process camera images and detect objects.

Publishing Object Detections

To publish object detection results, instantiate an ObjectDetectionProvider and publish detections:

auterion::ObjectDetectionProvider object_detection_provider{sdk};

// Create detections from an input image
auterion::ImageDetections2D detections;
// The timestamp must match the input image that was used to generate these detections
detections.image_timestamp = input_image_timestamp;  // e.g. from camera timestamp

// Add a detection with bounding box and class information
auterion::Detection2D detection;
// Set bounding box center and size
detection.bbox.center = Eigen::Vector2d{100.0, 200.0};  // x, y coordinates in pixels
detection.bbox.size_x = 50.0;  // width in pixels
detection.bbox.size_y = 30.0;  // height in pixels

// Add object hypotheses (class labels and confidence scores)
detection.object_hypotheses.emplace_back("person", 0.95);  // class name and confidence
detection.object_hypotheses.emplace_back("pedestrian", 0.92);  // multiple hypotheses possible

// Add the detection to the list
detections.detections.push_back(std::move(detection));

// Publish the detections
object_detection_provider.updateDetections2D(detections);

Controlling Provider Operation Based on Service State

The provider must respond to the object detection service's enabled/disabled state by starting or stopping its computational work. This prevents unnecessary processing when the service is disabled and ensures the provider is ready when enabled.

Important: Providers should implement proper enable/disable logic to:

• Stop heavy computational work (image processing, inference) when enable = false

• Resume processing and publishing detections when enable = true

• Avoid wasting system resources when the service is not needed

auterion::ObjectDetectionProvider object_detection_provider{sdk};
// Subscribe to object detection enabled changes and control processing accordingly
object_detection_provider.subscribeObjectDetectionEnabled([&](bool enable) {
    if (enabled) {
        std::cout << "Object detection service enabled - starting processing pipeline" << std::endl;
        // REQUIRED: Start your heavy computational work here
        startImageProcessing();
    } else {
        std::cout << "Object detection service disabled - stopping processing pipeline" << std::endl;
        // REQUIRED: Stop heavy computational work to save resources
        stopImageProcessing();
    }
});

You can also check the service state programmatically, but the immediate action via callback is encouraged:

auterion::ObjectDetectionProvider object_detection_provider{sdk};

// Activate subscription to object detection enabled state
object_detection_provider.subscribeObjectDetectionEnabled();

// Later, check if object detection is enabled
if (object_detection_provider.objectDetectionEnabled().isLastValid()) {
    bool enable = object_detection_provider.objectDetectionEnabled().last();
    if (enable) {
        // Object detection service is active
        ensureProcessingIsActive();
    } else {
        // Object detection service is inactive
        stopProcessingToSaveResources();
    }
}

Data Structures

The structure below is intentionally similar to ROS vision messages, but kept simpler.

ImageDetections2D

The ImageDetections2D structure contains a timestamp and a list of detections:

struct ImageDetections2D {
    rclcpp::Time image_timestamp;  // Timestamp of the image these detections are from
    std::vector<Detection2D> detections;  // List of detections in the image
};

The timestamp's purpose is to be able to match detections to the images were the originated when both are used in other applications.

Detection2D

Each Detection2D contains a bounding box and a list of object hypotheses:

struct Detection2D {
    BoundingBox2D bbox;  // The 2D bounding box of the detected object.
    std::vector<ObjectHypothesis>
        object_hypotheses;  // A list of hypotheses for the object in the bounding box.
    std::optional<uint64_t> track_id;  // Optional track ID for the object.
};

BoundingBox2D

The BoundingBox2D structure defines a 2D bounding box using a center point and size:

struct BoundingBox2D {
    Eigen::Vector2d center;  // Center of the bounding box in pixels.
    double size_x;           // Width of the bounding box in pixels.
    double size_y;           // Height of the bounding box in pixels.
};

ObjectHypothesis

Each ObjectHypothesis contains a class name and confidence score:

struct ObjectHypothesis {
    std::string class_id;  // Identifier for the detected class (e.g., "car", "person").
    double score{;          // Confidence score of the hypothesis, typically in [0, 1].

};

The lower three of these structs can be found separately in <auterion_sdk/object_detection/object_detection_client.hpp> to also be available idependenly from any ROS headers.