Dr Jacob Greene, senior human factors engineer and Chris Forrest, innovation, and growth engineer at BAE Systems, discuss with Ed Hill the development of so called ‘haptic’ or tactile technology to provide improved prompts for pilots during complex and stressful combat situations and mitigate cognitive overload.

BAE Systems delivers a broad range of solutions and services enabling its customers to successfully carry out missions. The company has a long legacy of providing large-scale systems in engineering, integration, and sustainment services across air, land, sea, space, and cyber domains.

Working alongside academic and industrial leaders it develops emerging and game-changing technologies to ensure it continues to provide the world’s most advanced, technology-led defence, aerospace, and security solutions.

Q) What was the thinking behind developing systems that can help with pilot cognitive overload?

Fighter pilots operate in a complex environment; they are exposed to extreme G-forces, engine noises, changing light levels, and visual instruments, all while performing fast jet operational manoeuvres – making cognitive overload a likely outcome. This can have dire consequences, especially when compounded by stressful situations like system failures or combat operations. At BAE Systems we have been measuring cognitive loads on pilots for many years and wanted to explore ways to give pilots more room to think, with the goal of alleviating the physical demands and mental workload of a cockpit environment.

Q) Can you tell us more about the haptic vests that you are developing? Why are these better than visual interfaces with the aircraft?

We have been focusing on the best ways to draw the pilot’s attention to a particular issue when they are fully engaged with other operational tasks. We took a commercial off-the-shelf ‘haptic’ gaming vest, more commonly used to simulate the feel of action in virtual reality video games, and used it to create vibration alerts across a pilot’s chest and back. In airborne trials, the initial feedback has been incredibly positive, as vibration uses an entirely different part of the brain to the audio-visual cortex, which means it can be consciously registered more easily, especially in stressful situations.

We have done multiple tests on pilots, comparing reaction times between visual and haptic vest cues. We have increased their cognitive load by giving pilots tasks to do on a computer tablet, simulating their attention being diverted elsewhere when the cue arrives. These trials showed the pilots were much more responsive to cues from the haptic vest, as they reacted more quickly and intuitively in comparison to other cues. Following positive trials of this technology in small commercial aircraft, our next step is to progress it with military customers and sophisticated combat aircraft, as well as rolling it out in a training environment.

Q) Tell us more about the automation of aircraft management systems and evading missile attacks, etc.?

Our new aircraft management system helps pilots under attack in the air by automatically tracking missile threats, and giving the pilot a haptic signal to help avoid them, mitigating the risk of cognitive overload. In the event of an inbound missile accelerating towards the aircraft, the haptic signal would move the flight stick just at the right time to take evasive action.

We have been testing the technology at our simulator facility in Rochester to improve pilot performance and aircraft safety through threat warnings. In one scenario, we simulated very difficult flying environments for pilots, including reduced visibility and enemy attacks. We tested the impact of various inputs, including haptic feedback from the active flight stick, as well as 3D audio and visual cues through augmented reality glasses.

Our tests aimed to determine what information we can efficiently deliver to the pilot when they are under a high cognitive load, in order to ensure that they perform to their maximum potential. Like the haptic vest trials, haptic cues via the active stick were the most effective medium in these high-stress environments. In fact, it was able to give more than one cue at a time! The flight stick technology has been adapted for both military aircraft, such as the F-35 and commercial jets such as the Gulfstream G500/600.

Q) Will there be a role for AI in the development of these systems?

AI will undoubtedly play an increasingly significant role in the development of these systems as we create ever more realistic and complete synthetic environments to test and refine the capabilities that bring the most operational benefit. As an example, AI could generate the flight path and behaviour of airborne drone swarms in response to a variety of tactical situations, to test how our systems could assist a pilot interacting and managing those assets.

As AI expert systems become more capable, and most importantly more reliable, we expect to see their staged deployment into these intelligent systems. Such a capability would be able to monitor both aircraft situation and pilot awareness in order to, for example, provide priority information to the pilot. Eventually we will see such systems able to respond to priority threats, be it through launching of defensive countermeasures, or actively controlling the aircraft to avoid an immediate threat such as an incoming missile, or collision hazard during low level manoeuvres.

We are already seeing the significant benefit of employing this technology, particularly in terms of driving critical pace and efficiency across our programmes. This is allowing us to break the rising time and cost cycle of our ever more complex work.

In partnership with academic and industrial leaders, for example, we are using data from the complex systems used in fast jets or gathered at our testing facilities to create simulations and prototypes more quickly and effectively than ever before. These can be used to test things like platform aerodynamics, real-time aircraft radar object recognition and stealth capability.

Q) Can this technology be retrofitted to aircraft and how will it develop from concept to being manufactured at scale?

Our systems are constantly being refined in terms of their size, weight, and power footprint, and as such can be retrofitted to a range of aircraft, from lightweight two-seater propeller driven platforms, to future fast jet platforms.

Design for manufacture is always a primary consideration in the development of our systems so they are manufacturable at scale and can be maintained and supported through life. We have a good track record of adapting our new technology to replace and enhance existing legacy capabilities, such as the digital LiteHUD retrofit replacing cathode-ray tube-based head-up displays.

Q) Will this research and technology contribute to the Tempest/FCAS programme?

Absolutely, the capabilities required by Tempest in the future will be fulfilled through research and development projects such as this taking place today.

Q) With so much research going into automating various aspects of flying and deploying weapons, would it not be easier to develop more sophisticated UAVs?

We are very clear that there are significant benefits to unmanned systems, an area we have been working in for many years, but there should always be a human ‘in the loop’ when it comes to key decisions.

This technology instead brings mission benefits of freeing up the pilot to focus on higher order tasks, such as commanding assets and collaborating with multi-domain forces, while the lower order tasks are handled automatically and safely by the on-board systems.

www.baesystems.com