SeeIT
Harsh industrial environments, where workers interact with machines, tools or other entities (such as robots or other workers) provide a difficult terrain for human computer interaction. It is the aim of SeeIT to provide workers with digital augmentation using different forms of deployment, body worn/mobile or infrastructure based different fidelity of transmitted content, e.g. manuals or video snippets and different sensory targets, such as visual, auditory or tactile channels. The aim is to create a holistic human-in-the-loop system as pluggable interaction component, which can be used by sibling projects, Guide and WorkIT.
Due to the complex needs of industrial reality and the harshness found there, e.g. noise pollution or changing lighting conditions, perception capabilities of workers are limited and need to be considered carefully, especially when critical information needs to be received by a worker. The interaction design is therefore three folded based on general suitability, information flow (unidirectional vs. bidirectional) and content perception. SeeIT considers therefore several factors when providing augmentation: (i) environment factors like noise pollution or changing lightning conditions, (ii) actuator availability such as distance, viewpoint or powered off, (iii) content availability like media available for a specific sense or device, (iv) worker preferences and (v) worker applicability such as sensorial load for a given sense. In addition to that the system will provide the user with a way to interact with the presented information, either implicitly, e.g. by behavior change, or explicitly, by providing input via devices. The major focus will however be on implicit interaction, such as capturing the response of a worker, i.e. in terms of behavior change, to a system triggered information display. We are therefore particular interested in observing overt behavior changes, such as turning the head towards the audio source or avoiding to get closer to a dangerous machine. But we will also investigate intrinsic behavior changes, such as a reduction of cognitive load measured by eye tracking devices or GSR wrist bands. Finally, we aim to use sensory channels which are not already highly engaged, as proposed by Wickens’ Multiple Resource Theory. In addition, we strive to not bind a huge amount of attention for long periods of time but aim to make use of subliminal stimuli, such as vibration or ambient lightning, and to provide brief information exchanges, e.g. through video snippets. These three categories, general suitability, information flow, and content perception, will be used to select the best suitable modality in case the system, e.g. Guide or WorkIT wants to exchange information with a worker.
Goals
The project goal is to communicate digital information on different modalities (visual, haptic, auditory), whereas the content is prepared independently of a specific device at the information source, e.g. WorkIT’ workflow recognition module, but tailored to a specific device at the information sink. E.g. an attention capturing notification, will be sent by the source and will be translated by the system into a specific modality, the best suitable at this point in time, which then at the receiving device node will be translated into e.g. the exact PWM frequency for the vibration motor to raise the attention. The translation processes considers various factors such as environmental limitations, e.g. ambient noise, or worker limitations, e.g. viewpoint towards a screen or prior engagement/load of a sensory channel but also contextual information, such as the current workstep in a workflow.
In addition, the presented information can be acted upon both explicitly by using input devices, such as eye trackers or implicitly by changing or not changing of current behaviour, be it overt -- e.g. body pose -- or intrinsic -- e.g. decrease of cognitive load. The interaction response again will be used to manipulate or stimulate further feedback but also to train the system into learning how successful feedback is given not only on a general basis (i.e. pretrained) but for the system to learn how to provide feedback for a specific user individually.
Approach
The system will be designed based on previous work and make use of a message oriented architecture. Trigged by components of other project, i.e. Guide and WorkIT, rather generic feedback, e.g. alert, will be processed in two stages. The first stage will select the best suitable modality, where several factors are considered: (i) environment factors, (ii) actuator availability, (iii) content availability, (iv) worker preferences and (v) worker applicability such as sensorial load for a given sense. In a second stage the selected modality and content will be transformed on a device node into an actuator specific format, e.g. a vibration message will be converted into the corresponding PWM sequence. The feedback can be interacted with both explicitly with dedicated devices or implicitly by observing overt and intrinsic behaviour (changes). In addition, the system will use the response of the user to learn and self-optimise how to best provide feedback in future interactions.
Expected and Achieved Results
The project should result in a multimodal hardware and corresponding software prototype, which is able to provide feedback using infrastructure and mobile/wearable devices, which are able to stimulate visually, auditory or haptic. We plan to have each output device fully connected via an IP network. A central controller acts as mediator between the information source, e.g. a workflow recognition module, and information sink, an output device node. This controller receives generic feedback mechanism, e.g. alerts, and first derives by sensor data from the network which modality: (i) visual, (ii) auditory or (iii) haptic is best suited based on the following: (i) environment factors like noise pollution or changing lightning conditions, (ii) actuator availability such as distance, viewpoint or powered off, (iii) content availability like media available for a specific sense or device (iv) worker preferences and (v) worker applicability such as sensorial load for a given sense. Secondly, based on the selected sensory modality the now selected content will be send to a dedicated output device node, which may translate or adapt the content for a specific actuator, e.g. a vibration pattern is converted into a pwm sequence.
After the feedback is provided by the system, it waits either for explicit interaction using dedicated devices, such as eye trackers, or observes both overt behaviour, like body pose, and intrinsic behaviour, like cognitive load. Hence, through this machine perception of how feedback is affecting a worker, we aim to close the loop and provide that again as an input for both selection of modality but also about when and how to interact with a user. Therefore, the system will optimise itself to best fit its user. We foresee that, in conjunction with Guide and WorkIT, we will be able to provide work step relevant information such as position in workflow, work step instructions, material and tool parameters, while at the same time decreasing information overload by ensuring that cognitive load is within expected values.
