WorkIT
The vision of WorkIT is to establish cognitive capabilities in today’s conventional products (e.g. power tools or machines). To achieve this, anything surrounding the worker in an industrial context, be it product, robot or machine, needs to be self-aware and possesses self-organisation capabilities to orchestrate the recognition of workflow and tool processes with others. In addition, the self-organisation approach needs to have the worker in the loop. Overall a collective behaviour of humans and machines needs to be formed, as (i) machines actively adapt to users behaviours and needs by (ii) algorithms and systems which facilitate coordination and collaboration and (iii) exploiting worker and machines strengths (HABA/MABA). WorkIT along with sibling projects SeeIT and Guide is a component of and targets the creation of a general cognitive assistance system.
However, establishing cognitive abilities in product components, tools or machinery requires three levels of awareness: context-, activity- and self-awareness. Specifically for context awareness, we are interested in recognizing the state in the underlying workflow, composed of activities, i.e. work steps, which are performed in sequence or parallel by humans and machines. For activity awareness, we are interested in the recognition of atomic tasks executed by workers, e.g. grab/place a workpiece, which together form a work step. Finally, we are integrating machine/robot states and tool and material parameters, e.g. selected or measured torque in a dynamometric key or detection of metal sheet size, to formulate self-awareness of devices.
The main focus of WorkIT will therefore be centred on recognition of explicit human behaviour, i.e. recognition of the work tasks but also by combing that information with implicit human behaviour, e.g. cognitive load, to formulate an overall cognitive state of an industrial worker. For these recognition tasks, WorkIT will deploy both infrastructure-embedded and mobile-wearable sensors, which either solely or collectively can be used to deduce the worker state. For explicit behaviour recognition, we target to recognize activities, which are of large value in the industrial field of manufacturing, such as screwing in complex assembly tasks or assessing quantitatively quality values of metal working tasks, like welding or metal bending. For implicit behaviour, we aim to leverage existing advances in detection of cognitive state, e.g. using GSR or Pupil Dilation, and aligning them with work tasks to estimate task complexity.
WorkIT’s deduced cognitive state of the collective of worker, machines and tools will be used as input for sibling project Guide, e.g. to deduce action plans, or SeeIT, e.g. to provide what the next task is.
Goals
The goal of the project is to provide awareness of three kinds: context-, activity- and self-awareness for cognitive machines and tools or as a pluggable embeddable component, which could even be part of robots or stand-alone. While the second form of awareness, activity awareness, is targeting industrial workers, the last form, self-awareness, is targeting machines or tools, the first form, context-awareness is concerned with underlying processes, i.e. workflows, where men and machine need to cooperate to achieve a particular -- e.g. manufacturing -- goal. Vice versa, in order to understand context, i.e. recognize workflows, one needs to recognize human behaviour, both explicit and implicit, and machine behaviour on a shop floor.
With the industrial setting provided by the company partners, the project will support workflow recognition for assembly processes, metal forming and joining by leveraging information from embedded-infrastructure and wearable-mobile sensing devices. The goal is to combine various information sources making it not only feasible to do workflow recognition but also to provide a high level of reliability by providing redundant, overlapping or disjoint sensor data either of the same or different kind, such as e.g. multiple camera image sensing onto the same viewpoint, depth sensing from multiple angles, or combing data from IMUs and head-worn cameras. Additionally, we aim to recognize task complexity by also incorporating physiological data indicative for the cognitive state.
Approach
The system will be designed around a self-description approach and publish-subscribe principle of sensors and their data. High-level awareness modules, i.e. context-, activity-, and self-awareness, harness this approach by potentially combing data from various sources. We aim to receive data from tools, e.g. welding pistols, machines, e.g. bending machines, and from human-focused dedicated sensors either attached to a worker or embedded in the environment of the shop floor. Based on this self-awareness of machines and tools, and the activity-awareness of dedicated sensor software/hardware modules, we aim to implement a high-level context awareness, in the form of workflow recognition. A particular focus of the project will be to fuse together various data sources into a multi-layered recognition chain for a high level of reliability.
Expected and Achieved Results
The project aims to develop a recognition and awareness architecture, capable of combing and processing multi-sensor data streams from various information sources, such as machines, tools or human behaviour sensing devices, in near real time to correctly and reliable recognize industrial workflows. To his end the project recorded collect multi-modal sensor streams of sensors embedded into products (e.g. welding torch), machines (e.g. bending machine) or the industrial environment (e.g. top down RGB-D cameras), or worn by workers themselves (e.g. eye tracker), which recorded internal states for products or machines, implicit and explicit behaviour, biometric and physiological, but cognitive indicative, data from humans on the shop floor. We implemented various established (e.g. message queues), but also create novel, pre-processing (tree based reactive streams) mechanisms for this huge and frequent volume of data, to deal with redundant, overlapping and disjoined data by either same or different sensor sources. Based on the raw or pre-processed data, we deployed machine learning algorithms to check for the alignment with workflow specifications and report progress or deviations, by a mechanism, which supports correct task execution, provided by sibling projects Guide and SeeIT.
Additionally we still aim to provide knowledge to ensure correct execution of task order, performance and result, but also aim to distinguish between attentional and systematic errors in manual assembly processes by making use of strategic knowledge, which is created on the fly by repeated work task executions and stored for analysis in a workflow database. This database is also what we used to capture data for classification on the shop floors of our industrial partners. Additionally, we already we able to identify bottlenecks by implementing quantitative assessment methods within the production procedure, such as inefficiencies in single work step, erroneous operation of industrial tools and machines.
