Google Summer Of Code 2022 Report

Overview

Over the past three months, I have had the valuable opportunity of contributing to an OpenCV open source project as a part of GSoC 2022. The main goal of this project was to create a cost-effective vision system for teams competing in the FIRST Robotics Competition by using the OpenCV DepthAI/OAK-D platform. Through this project, OAK-D can begin to be part of a solution to reduce build time by providing a generic and proven vision solution for all types of robots.

Goals

Task 1: Create a new abstraction layer in FRC for person tracking and distance.
Task 2: Create a new abstraction layer in FRC for object tracking and distance.
Task 3: Create a new abstraction layer for AprilTag and ArUco tag homography.
Task 4: Provide a way to simplify the calibration process/auto calibration.
Task 5: Create a new abstraction layer for object identification.
Task 6: Create a new process to have the training environment easily usable and consumable models from community.
Task 7: Create documentation and tutorial on using this FRC software package.
Task 8: Create documentation and tutorial on tuning a model and uploading to OAK-D.

During the working period, I started by mainly focusing my assigned work on person tracking and distance. When I had met this goal, I worked with my mentors and moved onto completing the other goals that built off of this initial step, such as simplifying the training environment and documenting the setup instructions. This way, the full start-to-end toolkit that I had created with the person detector could be applied to the other abstraction layers as well. Project Description #

Development Environment

Oak-D DepthAI

DepthAI is the Embedded, Performant, Spatial AI+CV platform, composed of an open-source hardware, firmware, software ecosystem that provides turnkey embedded Spatial AI+CV and hardware-accelerated computer vision. OAK was developed, which is a modular, open-source ecosystem composed of MIT-licensed hardware, software, and AI training. Oak-D is the stereo camera which runs the DepthAI. FRC lacks advanced vision-based solutions including the open source software and hardware that can be quickly adopted by developers. The proposal is to create a package for the OAK-D camera for FRC. Beyond that, the package can be used by any mobile robot to quickly implement vision based tasks

JavaCpp-Presets

The JavaCPP Presets modules contain Java configuration and interface classes for widely used C/C++ libraries. It provides efficient access to native C++ inside Java, not unlike the way some C/C++ compilers interact with assembly language. Since the DepthAI that was used for this project was created with C++ language, in order to create a Java version, we need to use a Java CPP glue layer which is what Java-Cpp is about.

FRC RoboRio Simulation

RoboRio Simulation is a desktop tool designed to help developers test FRC robot code. The simulator allows developers to test their robot code without the need for a physical robot or the RoboRIO component. It fully supports the FRC WPILibrary in Java.

Streamlit

Streamlit is an open source app framework in the Python language. It helps us create web apps for data science and machine learning in a short time. For this project, we used Streamlit to create a web-based user interface for teams to adjust parameters for the vision pipeline.

Implementation

The Python Implementation with test GUI

The DepthAI platform has low level functions for object detection and depth calculation but these examples are separate programs. Developers will need to understand the details and gist of how object and depth detection works in order to put them together. In the first phase of the working period, I spent time familiarizing myself with the depths of this code. Afterwards, I created two separate programs for detection based off of input, which was either a live feed from the OAK-D camera or a video from a file. In my initial iterations, I created an asynchronous detection process with the use of a long running thread. This worked in some cases but not all. Based on mentor feedback, I created a synchronous call instead so that once a person is detected, then at that point a result will be returned. The result of the pipeline includes values such as a boolean statement, bounding box coordinates, and depth. Once the initial detection networks were functioning, I created a web-based UI through streamlit to simplify the process of fine-tuning parameters to alter the result. With a web-based UI, developers can tune the pipeline regardless of their operating system. I added functionalities that included specifying the pipeline input, model file, confidence threshold, and a live display. Finally, I made it possible to return the results from the python code to the robot using the FRC WPILibrary NetworkTables function, which is a distributed message bus so that other software components can read the results, even if these components are in different processors.

Refer to this link for setup instructions.

The Java Implementation with RoboRio Simulator

Due to the large number of FRC teams that have Java as their preferred programming language, I felt that it was necessary to add Java support for this project as I was finishing up my Python code. However, the Java implementation was quite an involved process. Since DepthAI does not have a Java library, this work would have had to be done based on low-level code from the C++ library. Thankfully, there is an open-source project, JavaCPP-Presets, which provides some form of a JNI-layer wrapper for some common C++ components. I modified the JavaCPP to create a depthAI jar wrapper and then ported my person detection Python code to Java. Through unit testing, I made sure that this process was successful. Then, I utilized NetworkTables from the FRC WPILibrary in order to use the outputs from the pipeline as input to the RoboRIO processor found on FRC robots, so that teams can use the results for positioning or other judgements. Since I didn’t have access to the RoboRIO hardware, I chose to install and use a FRC RoboRIO simulator which can mimic the NetworkTables function. With the setup complete, I tested the Java code with a OAK-D and this simulator and it worked as expected.