AI-Flight
In the last decade, drones are showing a great potential as data acquisition platforms, which can support industrial systems in achieving further automatization, for instance in tasks such as mapping/surveying and inventory taking. However, especially in indoor operations, the implementation of drone-based solutions must be supervised by safety pilots within visual line of sight, and often the pilot must manually fly the drone during the task. This necessity stems from the fact that such an operation requires the detection of safety hazards and bringing the drone to a safe halt during such situations.
Goals
AI-Flight aims towards achieving the autonomous operation of drone systems for indoor industrial applications, as well as towards the intermediate step of beyond-visual-line-of-sight operation. Many commercially available off-the-shelf (COTS) drones readily provide interfaces for the deployment of machine learning models to analyze the drone’s video stream and acquired imagery and for the customization of the drone’s operation. AI-Flight will assess the applicability of image processing artificial intelligence (AI) in the context of autonomous drone operation to achieve a dependable operation of COTS drones beyond visual line of sight in industrial indoor environments. This approach enables service providers to choose the COTS drone best suited for the application and to reduce their dependency on specific drone manufacturers.
Therefore AI-Flight will explore and test the provisioning of AI models to achieve situational awareness for COTS drones, which allows a drone during operation to detect, track and react to third parties (vehicles and people) and to obstacles inside and around the working area, and guide the compilation of further methodological steps and research directions required to achieve such a system.
Approach
In AI Flight, rather than building a fail-safe drone, we have investigated, tested and implemented safety extensions by means of external monitoring that enable a safer operation of COTS drones towards the achievement of a fail-safe operation inside industrial warehouse environments. AI-Flight will concretize the goals (situation awareness and external monitoring) into metrics and perform first necessary quantitative evaluations within a real logistics environment, focusing on warehouses arranged in corridors.
In addition, the project AI-Flight has had a focus on achieving automated drone-based inventory management, therefore, AI methods to add new capabilities to COTS drones related to this industrial application have been explored. Keeping stock of warehouse inventory requires the employed drone to be able to count. Due to its camera, it has the necessary perceptive capabilities but still lacks the intelligence to count. We intend to equip the drone with counting capabilities via a trained convolutional neural network which generates density maps of desired objects. These can then be used to estimate the number of objects present during inventory taking.
Expected and Achieved Results
As shown in Figure 1, our investigations have led us to the design of a Mobile Monitoring Station (MMS) to achieve the external monitoring of the drone. The MMS is an Autonomous Ground Vehicle (AGV), which monitors the COTS drone repetitively from the same point of view. Additional equipment – computing resources, AI methods and sensor systems – are installed on the MMS to monitor the immediate surroundings of the drone and its operational area. In our system: (1) the COTS drone can achieve stable flight inside the warehouse, but has only limited capacity to avoid collisions with obstacles; therefore (2) the MMS monitors at all times the flight of the COTS drone and its surroundings and provides commands to the drone to perform the task safely and in an optimal manner. The usage of a repetitive point of view of the drone from the MMS, allows a very reliable and repetitive drone tracking behavior by the MMS, allowing the deployment of such a system in different buildings, applications, and scenarios.
The main topics that have been addressed regarding our envisioned drone system are omni-directional situational awareness – the automatic monitoring of the drone system and its surroundings –, the automatic in-place configuration of the drone system in the warehouse and the cost‑effective adaptability of such a system to different application scenarios. Regarding the omni-directional situational awareness, we have tested and evaluated AI‑based omni‑directional depth estimation approaches for fisheye multi‑camera sensors, the automatic image/point‑cloud segmentation of ground versus storage racks and the omni-directional detection of certain objects – that is, either air-borne obstacles (e.g., drones), ground-based vehicles (e.g., fork-lifts) and human co-workers. Regarding the automatic in-place configuration of the system and in view of our external monitoring concept, we have investigated AI methods ensuring a good accuracy of the anchoring transformation of the MMS with respect to the surrounding hallway.
Our initial prototype for the automatic object counting AI method already achieved competitive results. Transfer learning was done with a small dataset of >100 samples. On the unseen validation samples the counting error was 25%. Our counting method has potential for improvement and further research. This will be explored over the course of follow-up projects.
Our experiments have shown the capabilities of our system to track the drone with high reliability and positioning accuracy. In our system, the drone is tagged to ease the vision-based re-detection and tracking task and the tracking camera is placed statically inside the working environment. Our work represents a first step towards achieving the safe operation of drones in warehouse industrial environments. The experiments demonstrate the feasibility of utilizing automated external monitoring in indoor drone operations. Among other advantages, our system provides more information to the operator, such as flight statistics over time and the repetitive acquisition of metrics that reflect the performance of the drone in each particular area of the warehouse where the system is deployed.
For drone operations as a whole, our external monitoring solution signifies an additional layer of security, which can be used as part of the navigation architecture towards achieving fail-safety in drone industrial operations.
The project partners organized a workshop at Roto Frank GmbH (Kalsdorf) in February 2022 for stakeholders interested in the topic of "safe indoor drone operations". The workshop attracted, among others, several safety certifications experts, potential end users, as well as 2 manufacturers of automated AGVs for warehouse solutions. The content of the workshop influenced our planning and realization of other tasks and work packages, as well as the writing of our roadmap "towards an AI‑enabled drone operation".
