> For the complete documentation index, see [llms.txt](https://docs.auterion.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.auterion.com/app-development/auterion-sdk/object-detection-api.md). # Object Detection API {% hint style="info" %} Related C++ Headers in the Auterion SDK: * [``](https://auterion.github.io/auterion-sdk/object__detection__client_8hpp_source.html) * [``](#object-detection-provider) {% endhint %} The Object Detection API provides interfaces for both consuming and providing object detection results in the Auterion ecosystem. It consists of two main components: 1. **Object Detection Client**: Used by applications that want to receive object detection results 2. **Object Detection Provider**: Used by applications that perform object detection and want to publish their results ## Object Detection Client The Object Detection Client allows applications to subscribe to object detections and control the object detection service. This is useful for applications that need to react to detected objects or visualize detection results. Controlling the object detection service is particularly useful for resource management in cases where the detection result is used by another resource heavy application for which the resources of the object detection service should be freed. ### Subscribing to Object Detections To receive object detection results, instantiate an `ObjectDetectionClient` and subscribe to detections: ```cpp auterion::ObjectDetectionClient object_detection_client{sdk}; // Subscribe to detection updates with a callback object_detection_client.subscribeDetections2D([](const auterion::ImageDetections2D& detections) { std::cout << "Received " << detections.detections.size() << " detections" << std::endl; // Process each detection for (const auto& detection : detections.detections) { // Access detection properties const auto& bbox = detection.bbox; std::cout << "Detection at position (" << bbox.center.x() << ", " << bbox.center.y() << ") with size " << bbox.size_x << "x" << bbox.size_y << std::endl; // Access object hypotheses (class labels and confidence scores) for (const auto& hypothesis : detection.object_hypotheses) { std::cout << " Class: " << hypothesis.class_id << ", Score: " << hypothesis.score << std::endl; } } }); ``` ### Querying Latest Detections You can also query the latest detection results without using callbacks: ```cpp auterion::ObjectDetectionClient object_detection_client{sdk}; // Activate subscription without callback object_detection_client.subscribeDetections2D(); // Later, check if we have valid detections and process them if (object_detection_client.detections2D().isLastValid()) { const auto detections = object_detection_client.detections2D().last(); // Process detections... } ``` ### Controlling the Object Detection Service The client can enable or disable the object detection service: ```cpp auterion::ObjectDetectionClient object_detection_client{sdk}; // Enable object detection object_detection_client.enableObjectDetection(); // ... perform operations that require object detection ... // Disable object detection when no longer needed object_detection_client.disableObjectDetection(); ``` ## Object Detection Provider The Object Detection Provider is used by applications that perform object detection and want to publish their results to other applications in the system. This is typically used by computer vision applications that process camera images and detect objects. ### Publishing Object Detections To publish object detection results, instantiate an `ObjectDetectionProvider` and publish detections: ```cpp auterion::ObjectDetectionProvider object_detection_provider{sdk}; // Create detections from an input image auterion::ImageDetections2D detections; // The timestamp must match the input image that was used to generate these detections detections.image_timestamp = input_image_timestamp; // e.g. from camera timestamp // Add a detection with bounding box and class information auterion::Detection2D detection; // Set bounding box center and size detection.bbox.center = Eigen::Vector2d{100.0, 200.0}; // x, y coordinates in pixels detection.bbox.size_x = 50.0; // width in pixels detection.bbox.size_y = 30.0; // height in pixels // Add object hypotheses (class labels and confidence scores) detection.object_hypotheses.emplace_back("person", 0.95); // class name and confidence detection.object_hypotheses.emplace_back("pedestrian", 0.92); // multiple hypotheses possible // Add the detection to the list detections.detections.push_back(std::move(detection)); // Publish the detections object_detection_provider.updateDetections2D(detections); ``` ### Controlling Provider Operation Based on Service State The provider must respond to the object detection service's enabled/disabled state by starting or stopping its computational work. This prevents unnecessary processing when the service is disabled and ensures the provider is ready when enabled. **Important:** Providers should implement proper enable/disable logic to: * Stop heavy computational work (image processing, inference) when `enable = false` * Resume processing and publishing detections when `enable = true` * Avoid wasting system resources when the service is not needed ```cpp auterion::ObjectDetectionProvider object_detection_provider{sdk}; // Subscribe to object detection enabled changes and control processing accordingly object_detection_provider.subscribeObjectDetectionEnabled([&](bool enable) { if (enabled) { std::cout << "Object detection service enabled - starting processing pipeline" << std::endl; // REQUIRED: Start your heavy computational work here startImageProcessing(); } else { std::cout << "Object detection service disabled - stopping processing pipeline" << std::endl; // REQUIRED: Stop heavy computational work to save resources stopImageProcessing(); } }); ``` You can also check the service state programmatically, but the immediate action via callback is encouraged: ```cpp auterion::ObjectDetectionProvider object_detection_provider{sdk}; // Activate subscription to object detection enabled state object_detection_provider.subscribeObjectDetectionEnabled(); // Later, check if object detection is enabled if (object_detection_provider.objectDetectionEnabled().isLastValid()) { bool enable = object_detection_provider.objectDetectionEnabled().last(); if (enable) { // Object detection service is active ensureProcessingIsActive(); } else { // Object detection service is inactive stopProcessingToSaveResources(); } } ``` ## Data Structures The structure below is intentionally similar to ROS vision messages, but kept simpler. ### ImageDetections2D The `ImageDetections2D` structure contains a timestamp and a list of detections: ```cpp struct ImageDetections2D { rclcpp::Time image_timestamp; // Timestamp of the image these detections are from std::vector detections; // List of detections in the image }; ``` The timestamp's purpose is to be able to match detections to the images were the originated when both are used in other applications. ### Detection2D Each `Detection2D` contains a bounding box and a list of object hypotheses: ```cpp struct Detection2D { BoundingBox2D bbox; // The 2D bounding box of the detected object. std::vector object_hypotheses; // A list of hypotheses for the object in the bounding box. std::optional track_id; // Optional track ID for the object. }; ``` ### BoundingBox2D The `BoundingBox2D` structure defines a 2D bounding box using a center point and size: ```cpp struct BoundingBox2D { Eigen::Vector2d center; // Center of the bounding box in pixels. double size_x; // Width of the bounding box in pixels. double size_y; // Height of the bounding box in pixels. }; ``` ### ObjectHypothesis Each `ObjectHypothesis` contains a class name and confidence score: ```cpp struct ObjectHypothesis { std::string class_id; // Identifier for the detected class (e.g., "car", "person"). double score{; // Confidence score of the hypothesis, typically in [0, 1]. }; ``` The lower three of these structs can be found separately in `` to also be available idependenly from any ROS headers. --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter: ``` GET https://docs.auterion.com/app-development/auterion-sdk/object-detection-api.md?ask= ``` The question should be specific, self-contained, and written in natural language. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.