Eyes in the sky: an Australian company is literally examining how we look at flying

The company is involved in developing computer vision technologies which enable machines to see, understand and assist people. Through artificial intelligence analysis of the heads, faces and eyes of operators, such as pilots, the vision platform can monitor a person’s attention state in real-time, including their drowsiness and distraction.

Seeing Machines’ first commercial solution was delivered to the mining industry with its Driver Safety System (DSS). The DSS was deployed across the world on mining sites where heavy vehicle operators were identified as at a high risk of experiencing fatigue, due to the nature and length of shifts. They wree also prone to distraction. This system provides real-time in-cabin intervention, aimed at preventing fatigue and distraction‑related driver events.

Due to high interest and industry take-up, Seeing Machines then worked closely with the American Fortune 100 company, Caterpillar Inc, and in 2015 licensed the Off-Road business exclusively to them on a worldwide basis. The next step saw ‘Guardian’ launched. A version of the Driver Safety System for a commercial fleet of road trucks, it is now protecting the vehicles of over 200 customers globally.

Although the Seeing Machines driver monitoring platform was for automotive applications, it is now being adapted for use across a range of industries, including aviation. Using the investment and experience gained from these other industry sectors, Seeing Machines Aviation is in a strong position to apply the technology to the complex, sophisticated, and highly regulated aviation industry.   

The emerging aviation business within Seeing Machines is under the stewardship of general manager Patrick Nolan, and already the company relates that there is strong interest from manufacturers, carriers and operators alike. From the company’s perspective, it is seeking to leverage its existing core capability and systems to build solutions that support applications in aircraft, simulators, and consoles that are typically used in air traffic control and unmanned aerial systems (UAS) operations.

How does it work?

If most communication is non-verbal it would seem that a pilot’s face can also relay information without a word being spoken. Each renewal of a medical certificate calls for testing of the eye’s ability to see, but there may be more to be learnt than merely how far away a pilot can spot an aircraft, or how well they can read the fine print on a chart.

We have all felt the slow blinking of the eyes and ultimately the head nods as fatigue begins to bite as we sit watching Netflix. Similarly, pilots have felt the speed and efficiency of their instrument scan affected by an extended absence from flying, or even a change of cockpit and flight panel design. Conversely, what happens to our scan under periods of high workload?

These issues, and others, raise the question whether we can monitor and quantify such behaviour as it develops? Furthermore, is there the ability to train these behaviours, or even intervene?

These are some of the answers that Seeing Machines is seeking. The technology being trialled to date is non-intrusive, fully automatic and camera-based and does not require the user to wear any form of hardware or sensors.

The current prototype solution being used for pilot and crew training support is a simple (non-integrated) and temporary after-market installation of a small camera and two infrared light sources. The camera is typically placed between the two key instruments, the primary flight display and navigation display, while the light sources are placed on the outer points of the instruments. The arrangement is replicated for both the captain’s and first officer’s positions. There is additional computing equipment required to run the system, capture data and to provide real-time tracking displays but that is not visible to the pilot.

The technology detects and locates a human face and then tracks in real time without any need for calibration prior to use. It gathers a variety of accurate head and eye-related data, measures, and metrics including precise gaze-tracking, microsleep detection and very accurate measures of pupil diameter.

To track the head and eyelids, the system provides precise detection and measurement of the frontal area and sides of a subject’s face and head in real time and returns a comprehensive model of the face that includes the coordinates of all facial features, their current state and their rate of change. This includes a very accurate measure of blink rate and eyelid aperture.

‘Eye gaze tracking’ is the measurement of where the pilot is looking. By directing a safe, invisible light source at a subject’s eye and then using a special camera to track the glint, it can be interpreted to precisely ascertain where or what the subject is looking at. Combining this information with an accurate understanding of the flightdeck involved, gaze tracking can be used in real time (or recorded and played back) to help assess exactly how the subject is processing their visual surrounds.

Although eye-tracking and related data on its own will not prevent the impact from these identified problems, they stand to provide a new, relevant and rich data source to supplement and optimise the current aviation recruitment, training, and assessment processes with an underlying alignment to safety.

Seeking answers

As with any new form of technology in the aviation sphere, there is a significant process of data collection, analysis and trials involved in developing the system.

Seeing Machines has been conducting this proof-of-concept data collection across a range of flight and air traffic control environments. These include full-flight simulators with multiple carriers and operators from UAE, US, and Asia Pacific, and the training and operational consoles of Australian and UK and Europe-based air navigation service providers. There have also been multiple flights during which pilot alertness data has been gathered.

Emirates, through its Human Factors Group and Emirates Training College, has shown significant interest and commitment to a collaboration with Seeing Machines.  The collaboration to date has seen multiple data collections in both part task trainers and more recently in the A380 full-flight simulator.

In addition to the partnership with Emirates, there is work underway with several other carriers, expanding to include rotary‑wing work, and more recently Seeing Machines says there has been interest in the development of military applications, specifically in the ab-initio pilot training space.

The findings from the first part of the project were primarily focused on the use of the equipment, the methods for data collection and how to use that data. This demonstrated the potential of eye-tracking for pilot training, understanding the requirements of different procedures and to analyse events that occurred from a pilot performance perspective. The second part, recently carried out, was still focused on the use of equipment and data. From this we expect data from specific flight scenarios that will contribute important information about pilot performance in these scenarios.

The data collection was conducted with a view to certain questions. Where is the pilot looking, and does it matter? What does a good scan pattern look like and can you train that? How important is monitoring, do you train how to monitor and in turn, how do you monitor that? And critically, can you detect fatigue during operations? The areas where such data and the associated technology could potentially be utilised with pilots and air traffic controllers are far-reaching, including recruitment, assessment and training, monitoring skills, and fatigue‑related impacts.

Seeing Machines has the potential to supply aviation training providers and instructors with intelligent data that can be used in the selection, training and assessment of pilot candidates, as well as in the continuing training of line pilots. The technology identifies what the pilots are looking at, be that the speed tape, altitude, approach guidance, flight mode annunciations (FMA) and the like at any given point in time during a flying sequence. This information can be further used to identify breakdowns in instrument scans, missed information and even crew interaction from a CRM perspective.  

Potential fatigue issues are another growing field of concern within aviation as pilots, air traffic controllers, employers and regulators seek to manage their resources in the safest possible manner. The nature of flight operations means that flightcrew regularly perform complex, safety-critical tasks using highly sophisticated aircraft systems. Furthermore, irregular schedules, long duty periods, early start times, and flying during the body’s ‘Window of Circadian Low’ are all part and parcel of a career in aviation.

Through using the Seeing Machines technology to better understand their pilots’ alertness and fatigue state in real time, during flight, the data can in turn be employed by companies to evaluate the effectiveness of various aspects of their operation. They can review their schedule and crew trip design, validate that their FRMS is performing as expected, and identify and mitigate against systemic operational fatigue risks.

The growth of the aviation industry over the coming decades does not solely impact airlines and pilots. Air traffic controllers are also affected by the increasing demand on their skill‑set, growth in air traffic and rostering patterns that may seem at odds with the human’s natural desire to sleep when the sun sets.

In the air traffic control sphere, Seeing Machines’ Operator Alertness System can help air navigation service providers and controllers assess alertness and attention state in live operations and consequently provide valuable and timely data to air traffic controller teams. Again, like the pilots, this can potentially support air traffic controllers with more effective resourcing scheduling and decisions.

The potential uses for the Seeing Machines technology in aviation are broad in their range, but quite precise in the data that can be collected. It is apparent that through this data there is the opportunity to optimise the performance of the pilot or air traffic controller as well as the environment in which they work.

Eyes in the sky

As all technologies continue to evolve, it is the interface with the human element that must be continually monitored and assessed and adapted. Unlike our digital creations, humans are not designed to operate at peak performance for 24 hours each day and seven days per week.

It may be some time until the Seeing Machines technology is incorporated on flightdecks around the world, but there is no doubt that this Australian company is at the cutting edge in the way that it is addressing certain human aspects of operating in aviation, not the least of which is recognising and managing fatigue.

Rather than drawing a hypothesis, this technology is going directly to the source – the human – and monitoring performance very closely under a range of conditions, timeframes and environments. The subtle movement of the eye as it busily scans across flight instruments to draw the maximum amount of data has not previously been tracked. And that same eye may hold the earliest clues to a decrease in performance as fatigue insidiously draws down upon the eyelids.

Moving forward means that new solutions must be found to the new challenges that inevitably arise. In the case of human performance in aviation, the key may actually lie in looking very, very closely.

This feature article first appeared in the July 2018 issue of Australian Aviation.

To read more stories like this, subscribe here. Digital editions of the magazine can be purchased on Zinio and Issuu, or in the Apple app store.