Pointing the Way

A new tool for simulator instructors

Full-motion flight simulators have become a key component of pilot training, giving students a chance to show what they have learned while safely on the ground rather than up at 36,000ft and offering instructors the opportunity to test prospective pilots under almost real conditions.

BR&T Australia human factors researcher Nathan Perry said new technology would potentially enable the student and instructor to get more out of their flight simulator session through using a gaze-tracking system that measured the pilot’s instrument scan.

“The system extracts flight parameter data from the simulator and overlays symbology representing the pilot’s scan on a real-time video feed of the simulator’s displays on a device like a wireless tablet that the instructor can use during a live simulator session,” Perry explains to reporters during a media visit to Boeing Flight Services on December 7.

That provides, says Perry, “Precise, real-time insight into the student’s instrument scan behaviour.”

And real-time insight means better and more immediate feedback to the student, ensuring any bad habits are quickly identified and worked on, rather than allowed to linger.

“What we are also doing with this technology is we’re using all the data that’s available in the simulator, and analysing that data in real time to provide the flight instructors with real-time analysis of how students are performing relative to the competencies that are being evaluated in a training exercise, so the instructors can get that objective insight into how their student is performing,” Perry said.

“That should make sim training more efficient and instructors will be able to detect performance issues earlier.

“During the live training session the flight instructor can pause the recording and playback what they were just seeing to the student to enable them to give the student better insight into their own performance and allow the instructor to communicate any corrective actions to be taken.”

Instructors get a front-row seat thanks to augmented reality

BR&T is also looking at the use of augmented reality technology to give instructors a better view of the pilot’s actions in the flight simulator.

Traditionally, instructors sit at the rear of the simulator monitoring the session, meaning an obscured view of what a pilot is doing.

But BR&T researchers hope augmented reality, deployed via Microsoft HoloLens head-mounted smartglasses, will change all that. “We got one and started exploring different applications for its use,” BR&T Australia associate tech fellow John Towers explained.

“One of the problems that instructors have is they are seated behind the student in the simulators, and one of their tasks is to assess how well the students scan the instruments.

While a tablet could do the task, Towers said it was less than ideal, describing it as a “head-down interface that they’ve got to attend to if they want to get access to any of this type of analysis”.

“So we thought using an augmented display, could we display that information over the flightdeck?” he asked.

“Effectively we’re already extracting the video from the flightdeck, we’ve got the eye-tracker information, can we display that in a way through augmented reality so that effectively the instructor is looking through the student and seeing the displays through the student.

“It’s overlaid behind them.”

While the glasses on display during the media visit on December 7 was an early concept demonstrator, Towers said this would hopefully evolve into something easy for instructors to use.

“Eventually we imagine it will be like a pair of glasses that the instructor will just flick down and they’ll see this type of information,” Towers said. “It’s very early days with this sort of technology, but we’re looking at where this might be an enabling technology, a more immersive and intuitive presentation of information.”

In another example being evaluated, a virtual reality headset was able to offer “a multi-modal learning experience” on the Boeing 737’s fuel system. In this case pilots can see information such as the different capacities of the fuel tanks, where the refuelling is done at the wing, and how that related to the fuel system on the panel on the flightdeck, projected in virtual reality.

“We’ve been looking at virtual technology for a few years now and the goal is to really increase the quality and efficiency of training at the earlier stages of training like a type conversion to a Boeing aircraft,” explains Leighton Carr, virtual reality lead for BR&T – Australia.

“At the moment they’ll do a lot of Powerpoint slides or computer-based training and we’re exploring the use of this technology to allow them to experience the flightdeck at an earlier stage of that training.”

BR&T collaborates with Boeing Flight Services

All this research and evaluation is being done inside Boeing Flight Services’ Brisbane training campus, a facility run in partnership with Virgin Australia that features five full-flight simulators (two 737NGs, a 777-300ER, a 717 and an Embraer E190 sim).

Carr said the recent move from BR&T’s offices in the city to set up its “tech incubation centre” at Boeing Flight Services was to be closer to the subject matter experts, the regulator and customers.

“This is somewhat of a unique relationship in that we’re actually embedded here working very closely with Boeing Flight Services with new technologies. That’s somewhat unique for Brisbane, that sort of close symbiotic but there is a broader range of VR and AR tech across the enterprise,” Carr said.

“That, hopefully, is already starting to shape and focus our research strategy and the type of tech that we’re looking to develop to better focus on the business unit needs.”

BR&T has taken a similar approach in Melbourne, where it has set up shop inside Boeing Aerostructures Australia’s facility at Fisherman’s Bend, where it is focused specifically on resin infusion carbon fibre and robotic technology.

Making training more efficient

As reported in Preflight, Boeing expected Oceania to need an additional 30,000 pilots and maintenance engineers over the next 20 years in order to meet the expected growth in air travel into, out of and within the region.

That level of demand means scarce training resources are often stretched to the limit, as any pilot who has had to book in for a flight simulator session at one o’clock in the morning can easily attest to.

Boeing Flight Services vice president Sherry Carbary talked up the potential benefits of virtual technology in aviation training.

However, she cautioned that the immediate goal was to try to understand the technology. “You are able to do a lot through virtual reality,” Carbary said.

“You’re actually seeing the airplane, you’re seeing the flightdeck, you’re seeing the training manuals, you’re seeing everything you need to make a decision and so I think that will if not accelerate the training it will certainly enhance the training experience for pilots and technicians.

“You think about who we are trying to attract. It is all the younger generation and that’s what they have grown up on.”

Boeing Flight Services senior manager, strategy, marketing and business development Jeff Cooper added: “It can’t just be a toy, it has got to be something that actually creates value, going out and running actual pilots or customers through something like this and making sure that we’ve tested it and proved that there is a gain, a value in using these kinds of technologies is going to be instrumental. That’s part of the next steps.”

It can’t just be a toy, it has got to be something that actually creates value. JEFF COOPER