Interaction design

Big thanks to Veronika Domova for the material on this page. Everything good on this page is thanks to her and if something seems wrong it is most likely that you should blame Patric the editor.

The user in the age of industry 4.0

Modern manufacturing industry is facing a number of challenges, mainly to do with the constant strive towards more efficient, cost-saving, reliable and safe operations. The most prominent of these challenges is arguably rooted in digitization, leading to a situation where industrial machines, devices and settings are increasingly connected and instrumented which in turn opens possibilities to monitor the equipment while being on distance and yields massive amounts of data for monitoring and control. Approaches based on autonomous agents are increasingly used to synthesise data and offload tasks, leading to complex ensembles of humans, machines, agents and sensors where validated design patterns for successful operation are generally lacking.

It is clear that these developments are potentially useful, possibly even game-changing for the industry sector. However, with this new setup comes the challenge of how to appropriately support the work-practices of industrial personnel who now need to monitor large fleets and solve complex problems while being located remotely from the industrial settings. An entire new set of tools is needed that would enhance this kind of work practice.

With the respect to this background, there is a need for a broad research aiming to explore approaches to visualisation and interaction that enable users to monitor and control large ensembles of digitally connected and instrumented machines and agents in ways that contribute to increased efficiency, reliability and safety.

Analysis of the problem

With increasing digitization of industry, the amount of operation and maintenance data available to the operator grows manifold. A modern gas turbine installation, for example, may have thousands of sensors measuring and reporting real-time data on virtually any aspect of the energy production process. Similarly, runtime information and maintenance status of every component of the partially customized turbine installation is available through digital communication.

What is more, industrial machines in general are becoming more and more reliable. With recent advances in manufacturing and robotics, error rates are dropping and maintenance intervals are growing, which in turn means that operators are able to oversee larger fleets of machines. Furthermore, the principles of modern economy require from companies to maximise the utilization of machinery with a minimal amount of people involved. As such the industrial processes are being monitored by fewer personnel who also often have other more challenging responsibilities. Having too large amounts of data to monitor hinders operators from taking the right decisions at the right time which potentially can lead to safety, reliability, privacy, and security issues.

Advances in artificial intelligence and autonomous systems further imply that the operators’ tasks move into the territory of human-automation collaboration. The scale issue comes up again: human-automation collaboration on the order of one human operator and one autonomous or semi-autonomous agent is reasonably well understood, but the human-centered design of large, complex ensembles with emergent properties is essentially unexplored.

When these developments co-occur, we end up in a situation of multiplying scale effects: much more available data from many more machines with much more local autonomy. The operators’ tasks of monitoring, control and maintenance span an increasingly wide scope from rapid overviews and situational awareness of thousands of units located throughout the world, to detailed inspection of time-series data from one component of one machine in one location.

Key terms / concepts / areas

User-centered design (focus on the user needs) vs designer-centered design (focus on ideas of the designer)
1. https://en.wikipedia.org/wiki/User-centered_design Links to an external site.
2. Important Stages
  1. User studies
    1. Empathy
    2. Interviews
    3. Observations
  2. Lo-Fi/Hi-Fi sketching of the prototypes
    1. Paper prototypes
    2. Wire frames
    3. Photoshop or similar for realistic mock ups
    4. Buxton, B. (2007) Sketching User Experiences: Getting the Design Right and the Right Design. Elsevier, San Francisco, USA. [book]
  3. User evaluations
    1. Wizard of Oz: fake the technology to collect the preliminary feedback
    2. Qualitative: 1-2 users is enough. Give them to try your technology, observe, reflect
    3. Quantitative: 10>users. Measure the actual performance in numbers. Statistics.
Physical constraints of the user
1. Memory: A user can only keep 4 items in the working memory
2. Perception: A user can only perceive 40 stimuli at the same time (hardness of surface she stands on, temperature on the skin, wind in the face, someone smoking to the left, woman with cute dog in the distance, .... and the red light blinking)
3. Mental load in order of effort required
  1. Cognitive load: thinking
  2. Visual load: looking
  3. Motor load: clicking, typing
Automation levels, human-automation interaction and collaboration.
1. Automation is not black or white. There are many levels of automation, only on the highest level of automation takes the user out of the loop (if everything works well). However, there are very few examples where such a high automation level has been achieved. Also, automation and technology tend to fail, which means there is always a need for a user to potentially take over the control.
  More information about
  1. Levels of automation
    1. Sheridan, T. B., Verplank, W. L., & Brooks, T. L. (1978, 1978/01/01/). Human/computer control of undersea teleoperators. Proceedings of the International Conference on Cybernetics and Society, Place of Publication: New York, NY, USA; Tokyo-Kyoto, Japan.
    2. Endsley, M.R. 1999. Level of automation effects on performance, situation awareness and workload in a dynamic control task. Ergonomics, 42 (3). 462-492.
  2. Adaptive and adaptable automation.
    1. Visser, E.J.d., LeGoullon, M., Freedy, A., Freedy, E., Weltman, G. and Parasuraman, R. 2008. Designing an Adaptive Automation System for Human Supervision of Unmanned Vehicles: A Bridge from Theory to Practice. Proceedings of the Human Factors and Ergonomics Society Annual Meeting, 52(4), 221-225.
    2. Kaber, D.B. and Endsley, M.R. 2004. The effects of level of automation and adaptive automation on human performance, situation awareness and workload in a dynamic control task. Theoretical Issues in Ergonomics Science, 5(2), 113-153.
  3. Problems of automation
    1. Sheridan, T.B. and Parasuraman, R. 2005. Human-Automation Interaction. Reviews of Human Factors and Ergonomics, 1(1), 89-129.
    2. Sarter, N. B., & Woods, D. D. (1995). How in the World Did We Ever Get into That Mode? Mode Error and Awareness in Supervisory Control. Human Factors, 37(1), 5-19. Norman, D.A. 1990.
    3. The 'problem' with automation: inappropriate feedback and interaction, not 'over-automation'. Philosophical transactions of the Royal Society of London. Series B: Biological sciences, 327 (1241). 585-593.
    4. Vicente, K. and Rasmussen, J. 1992. Ecological interface design: Theoretical foundations. IEEE Trans. Systems, Man and Cybernetics, 22(4), 589-606.
Interaction means: depends on a context
No single means of interaction works everywhere. For example, a speech interface which works well in a domestic setting, is far from ideal on a workshop floor surrounded by loud machines, a touch interface which we all use every day on our smartphones and tables is not great when you need both hands for something else, etc.
1. Mouse-Keyboard-Screen
2. Tangibles
  1. https://en.wikipedia.org/wiki/Tangible_user_interface Links to an external site.
3. Gesture
  1. Kinect
  2. Leap motion
  3. …
4. Gaze
  1. Tobii,…
5. Speech
  1. Alexa, Siri …
6. Virtual reality
  1. Oculus rift, …
7. Augmented reality
  1. Microsoft Hololens,…
8. Hand-held devices
  1. Tablets, smartphones
9. Wearables
  1. Smart textile
  2. Sensors
  3. Actuators
  4. E.g. smart watch
Interaction design research
1. Research through design https://www.interaction-design.org/literature/book/the-encyclopedia-of-human-computer-interaction-2nd-ed/research-through-design Links to an external site.
2. Fallman, D. The interaction design research triangle of design practice, design studies, and design exploration. Design Issues 24, 3 (2008), 4–18.
3. Some key people researching interaction design at the WASP universities: