Team:SZU-China/Application
Application
Our product(Micrancide) needs a suitable way of application. In order to solve this problem, we went to BGI's experimental field to visit. We would like to know how they applied the medicine.
At the very beginning, we planned to combine the applet with the drone. After a few times of text, we found that the function of our applet could not work when it combined with the drone. It meant we need to find another method to reach our purpose. We interviewed Mr. Zhiqiang Xie, the CEO of Cloudroom Technology Co., Ltd. He recommended that we used SDK for drone development and helped us understand the functions of the various parts of the drone. So we started looking for potential partners to help us develop drones.
Fig.1 The Photo of Mr. Zhiqiang Xie and the SZU-China Team
We learned that the RobotPilots Robot Club is the first technical club of Shenzhen University to participate in robotics events. The club has won numerous awards in the China National Robotics Competition (RoboMaster). They have much experience in the development of robots and drones with SDK. We contacted them through the website, introduced our project to them. They were interested in our project. Therefore, we worked together to develop the drone. They provided DIY herbicide spray peripherals. Taught us to plant our identification and positioning program into the micro-computer of the drone. We converted the signal of the recognition program into a spray instruction together. What is more, they guided us to conduct flight tests of the drone.
Fig.2 The Photo of RobotPilots and the SZU-China Team
the drone designed by us can fly in the woods, cities, and fields with the remote control connected by mobile phones. The light frame and efficient motor can provide up to 40 minutes of flight time.
our drone can recognize Mikania micrantha and other plants in various environments according to the trained models, and give us feedback after recognition.
after confirming that the recognition result is Mikania micrantha, the drone can use our additional spray hardware to spray Mikania micrantha.
Video 1. Principle of Operation
Fig.3 Component Diagram of Drone
Tabel 1. Hardware Source Surveillance
Fig.5 The Camera We Used in the Drone
Fig.6 Water Pump
Fig.7 Basic Drone Configuration (Without Our Additional Hardware)
Basic Parameters of Drone Click here to see more details
-
Frame: Caffe -
Platform: Ubuntu 16.04.3
-
Hardware: Intel NUC Mini PC (i5 seven generation CPU), Daying USB camera
-
Software: opencv3.4.4, C + + 11
-
Platform: Ubuntu 16.04.3
-
Software files: Upload to GitHub
-
Function realization:
1. It is observed that the difference between the leaves of weed and micro chamomile is relatively significant. Combined with the flight speed of drone, the classic, simple and effective LeNet neural network is used for training, which can be used for relatively fast identification.
2. Realize the use of the drone to collect flight status frames, manual classification and standard naming, convenient for a follow-up operation.
3. Considering the single plant posture in the recorded video frame. We wanted to enhance the robustness, and we use the python library imgaug to enhance the image data and carry out random rotation, random translation, Gaussian blur, brightness adjustment, contrast adjustment, and other processing to increase the sample.
4. Randomly select 5% of the images as the verification set.
5. Use the Caffe sample script to generate the picture list TXT file of the training set and verification set.
6. Generate the LMDB data file for Caffe training according to the list file.
7. Set the initial learning rate, learning strategy, training times, model saving times, model saving path and other parameters in prototype
8. Call the Caffe executable file, set the network file, training parameter file, and other parameters, then start the training, get the caffemodel type file, then use the DNN module of OpenCV to call.
In the Ubuntu environment, the OpenCV computer vision library is used to write c + + code. At the same time, the data communication protocol agreed with the embedded system, and then debugging can be started.
1. Use the camera and Mini PC on the drone to process the real-time image: process the image captured by the camera in grayscale, resize it into 28 * 28 image and send it to the model interface. If there is a micro chamomile in the image, the function return value is 1. If there is no, return 0.
2. The Mini PC communicates with the single-chip microcomputer. It sends the return value of the function in the first unsigned char data. That is to say, if the micro chamomile is recognized, it will send 1; if it is not recognized, it will send 0 to the single-chip microcomputer. Then let the single-chip microcomputer control the switch of the valve.
Fig.8 The Logic Diagram of Identification Process
Demand determination:
Realize the communication function with Mini PC
Realize the spraying function
Ensure the regular power supply of electronic components and primary control, provide protection and waterproof, and maintain the environment required for regular operation.
Fig.9 System Block Diagram
The program judges the difference between the current system time and the last system time by timestamp to control the cycle and execute different tasks in turn.
Fig.10 The Logic Diagram of Overall Implementation
The system can judge whether it can receive remote control data in real-time. If it cannot receive the data, it will enter the out of control protection, and automatically force to turn off the spraying device.
When the remote control data is received typically, the program usually executes and judges the received visual information. If it is Mikania micrantha, the timing will start. If the cumulative time exceeds 150ms, the spraying will start. If one frame is judged as weed, the cumulative time will be cleared. The guarantee is stable identification, not fluctuation value
Here is the data generated by the spraying process of our drone. We took a short segment to show it.
The first column is a timestamp. The second column is the result of visual recognition. The third column is the time-consuming of recognition. And the fourth column is the Boolean value of whether to make spraying operation (yes means to perform spraying operation, no means not to perform any operation).
Table.2 Program Data Capture Part
Fig.11 We Were Transforming the Drone
The liquid sprayed in the Spraying Test of Drone Automatic Identification is PURE WATER
We use our own planted Mikania micrantha and Arabidopsis to crisscross, use the drone to identify and spray liquid (pure water).
The drone sprayed when above the Mikania micrantha, not sprayed above the Arabidopsis. This result can prove that our drone can accurately identify Mikania micrantha and spray liquid. Our test video is as follows.
Video 4. Demo of Indoor Test
We wanted to test our drone in a more complex environment in the wild. Make sure it recognizes Mikania micrantha with the same accuracy and sprays the liquid.
We chose a path with complex vegetation for testing. The test results showed that our drone can still maintain a high recognition accuracy and complete the spraying of liquid.
Our test video is as follows.
Video 4. Demo of Outdoor Test
At present, we are using the small drone to transform, and the effect is excellent in small drones. Later, we can promote it to large agricultural drones. Using the model training method we explored, the agricultural drones also have the function of automatically identifying Mikania micrantha and spraying liquid. In this way, it can apply to some urban green belts, rugged mountains that are hard for human beings to reach, and so on. In these places, Mikania micrantha can be removed by applying medicine. It dramatically reduces the difficulty of removing Mikania micrantha and protects people's safety.
Fig.12 The Agricultural Drone
DJI agricultural drones have the function of intelligent route planning. Our self-developed applet continues to draw the map of Mikania micrantha invasion land We can combine the intelligent route planning function of DJI with the map of Mikania micrantha invasion areas drawn by Shenzhen citizens. In this case, the drones can automatically fly according to the set route and apply herbicide to Mikania micrantha.
Fig.13 Intelligent Route Planning of DJI Drones
This automatic and efficient herbicide application method can not only be used in the treatment of Mikania micrantha by using drones precise identification. In the future, if similar invasive plants are encountered in any country or region, this application mode can be adopted. We can use the model training method that RobotPilot and we jointly explore to train the model, to achieve high-specific accurate recognition. Significantly improved the efficiency of the application, while ensuring the safety of the applicators.
