Artificial intelligence and computer vision to improve turnaround management at Gatwick - Exclusive interview with Christiaan Hen, Assaia's Chief Customer Officer

Blog Interview

Gatwick recently started using Assaia’s computer vision based technology to improve the performance of the turnaround process. We talk about this but also about the future of turnaround management with Assaia’s Chief Customer Officer Christiaan Hen.

Assaia’s tag line is “The Apron AI”. Can you tell us more about your vision, mission and products? How do you use AI, which can be applied to so many aspects of airport management? 

The demand for air travel has been increasing over the last couple of decades. Typically, capacity at airports has been relatively slow and costly to develop. Therefore, there is a mismatch between supply and demand which causes a very high degree of stress on the air transport industry. Aircraft have gotten bigger while infrastructure has remained pretty much the same. Therefore, both time and space are scarce as more and more aircrafts need to be handled. Many of today’s delays and safety incidents are due to this mismatch in capacity and demand. Besides the high cost and time required to develop infrastructure, there are also often physical or environmental constraints.

If we look forward, IATA again predicts a doubling in the demand of air travel in the next 15-20 years. There have been studies published by Eurocontrol that calculate the effect of these developments. Enormous amounts of time will be wasted due to delays and a fair chunk of the demand in air travel will not be fulfilled. That means that people want to fly but sometimes simply won’t be able to unless something is done. In order to address both the capacity as well as environmental concerns, the only option is to become more efficient. Therefore, at Assaia it is our mission to make the apron a more efficient, safer and more sustainable place in order to keep air travel sustainable in the decades to come.

The aviation industry is characterized by the fact that there are many different stakeholders required in order to create the product. A good synchronization and collaboration between these stakeholders is essential. However, with the growth of the industry this collaboration has become more and more challenging. Today, all the individual parts and players optimise their performance at the lower level rather than trying to optimise the entire system as a whole.

In order to achieve a fully optimised system we envision that there will be real time data about every individual part of the flight journey shared among all stakeholders. This availability of high quality real time data will also enable to optimise the system on a higher level, which will both generate sustainability as well as capacity benefits. The CDM framework has been a good first step in the right direction. However, the lack of granularity, quality and high latency of the data within the framework prevent it from reaching its ultimate goal. We envision automated real time data from every part of the aircraft journey that can be used for optimisation and automation whereby an autonomous airport could be the ultimate challenge. 

Our Apron AI product addresses the issue of data quality in the aircraft turnaround process (everything that happens between in-block and off-block time). It is an AI / Computer Vision software product that consumes video streams, preferably from existing cameras, and turns these video data into structured data. Structured data are typically timestamps for events like ‘bridge connected’, ‘fueller on stand’, ‘fueller connected’, etc. These structured data are the primary output of the software.

Assaia's Apron AI image with tags
Apron AI example - extracted from Assaia's YouTube channel

However, if we apply some business logic to these data we can create alerts in order to optimise the outcome of the turnaround. One example related to increasing on time performance is for example to alert ramp operators as soon as one of the turnaround sub-processes starts too late or takes too long. By directly being notified, the operator can take action and minimise the impact of this deviation on the overall flight. Another safety related alert could be an alert to a ground controller whenever an aircraft is on its way to an aircraft stand that is not empty because there is for example a piece of ground service equipment parked on the stand.

A third output category is predictions, both about the sub-processes (how much longer do we expect catering to take) as well as about the overall turnaround. The latter is what we call predicted off-block time or POBT. Ground handlers and airlines can use this POBT to prevent predicted delays from happening. Airports can optimise their stand allocation based on more accurate information about which aircraft is going to be ready for departure. Finally, air traffic control can use the information to optimise the use of taxi- and runways as well as airspace. 

Our tagline is the Apron AI for 2 reasons actually. First, the computer vision technology that we use to create the structured data from video, is indeed an AI technology. However, it does not stop there. The algorithms we create for the prediction of flight state (like POBT) and our algorithms for the optimisation of stand allocation, taxiway usage and/or runway usage are all machine learning algorithms that can also be regarded as AI technology. 

What is the company history, where are you coming from? Are you still a start-up?

Assaia was founded by 2 highly skilled computer science experts and a serial entrepreneur. The combination of technological skills and entrepreneurial spirits is the foundation of our company. Initially, the founders did not know where to apply the technology in order to create business value. Over the course of several workshops with people from different industries, different use cases were identified and tested. The reaction from the aviation industry on the idea of applying computer vision to gather real time data about the turnaround process was so overwhelming that they decided to drop all other ideas and fully focus on that specific concept. Since then we have grown to be a fully specialized aviation AI data and optimisation company. 

There are obviously many different opinions about how long you can call yourself a ‘start-up’. Even though we have grown a lot and have involved into a mature company with commercial contracts and funding for the years to come, I still think of us as a start-up. Especially considering all the work that we have not still taken on ourselves.  

Gatwick Airport has recently become the first airport to make use of your Artificial Intelligence (computer vision technology) based product to measure the turn-around process at the apron. Can you tell us more about the feature set you provide to Gatwick? What is the scope and coverage of your project there (i.e. which milestones do you produce and for how many stands)? What is the gain for all stakeholders compared to classical coordination by a crew or ground handler’s supervisor? How do you move from measuring to improving?

At Gatwick, we will initially be deploying the Apron AI software across all of Pier 6 and stands on Pier 2 and 5. To me it makes perfect sense that London Gatwick Airport is going to be the very first airport in the world that will have this kind of technology implemented. Gatwick is one of the busiest single runway airports in the world. The quest for operational efficiency with a given amount of infrastructure (as discussed earlier) couldn’t be higher. Besides that (or maybe because of that), Gatwick has a very mature and professional innovation function that has proven to be very good at identifying, testing and implementing innovative solutions. At Gatwick, we will be creating real time data about virtually all turnaround sub-processes. The data will both be used by Gatwick to optimise the usage of their infrastructure as well as shared with the airlines.

The beauty of airports sharing the data, is that if airlines use the data in order to improve their performance, the performance of the airport eventually also increases. The data we will generate is currently not available to any of the stakeholders. Through the implementation of the Apron AI solution, I expect that we will see a positive effect on OTP, higher airport asset utilization and an option for all stakeholders to automate certain activities and therefore save on costs.

Together with Gatwick we will be measuring the effects of the introduction of the system to the airport. The results will be collected in a case study that we will be more than happy to publish.  

Events generated by Assai's Apron AI
Apron AI example - extracted from Assaia's YouTube channel

How do the milestones your product generate integrate within an existing A-CDM context? How do you dispatch the information you generate about the progress of the ground handling process? Is it going to all partners or only to the ground coordinator? 

As mentioned before, we are very positive about the concept of A-CDM even though the implementation today has several shortcomings. First, in many cases the data is not shared in raw format in real-time. Second, the level of granularity of the data is too low. And third, the quality of the data is too low as it is often manually generated data.

We advocate and facilitate full sharing of the data between all airport stakeholders as we believe that only then the value of the data will be maximized. We are working together with IATA in the Ramp of the Future working group in order to set standards and try and expand the A-CDM thinking beyond what it is today.

Your product can be used to monitor the exact times of various operations and therefore the performance of each sub-team (i.e. fueling, baggage handling, catering, …). How do the concerned people react, is there a risk of getting a “Big Brother” feeling?

Privacy and information security are obviously always a topic during our initial discussions with customers. We appreciate the sensitivity and have therefore taken a wide array of measures in order to be compliant with the most stringent regulations. First of all, we have developed an automatic and irreversible people-blurring function that can be applied as a first step of the video processing. By blurring out all humans from the video, it is no longer regarded as personal identifiable information, which makes the handling of the data much easier. Furthermore, we can support a full on-premise deployment which means that no video data needs to leave the customer’s premises. These measures in combination with our very elaborate information security policy, have ensured that we have so far always been able to get the green light from customers’ legal and cyber security teams.

Can you see other ways AI can contribute to the implementation of Airport Collaborative Decision Making (A-CDM)? Do you have plans to move to other applications, including machine learning or big data applied to the A-CDM milestones as a whole?

Absolutely! I think AI can be used in other areas for data collection. But as mentioned before, I believe the real value will be unlocked when we will use AI algorithms that use these data in order to create predictions, optimisations and recommendations. In multiple projects we are already combining AODB and CDM data from our customers with the data we generate about the turnaround to predict the off-block time of an aircraft. We can see that even with very primitive algorithms we are already consistently outperforming the current-day alternative, the TOBT. 

At Network level, what are the implications for the ATM operational performance? What are the performance areas (Safety, Environment, Capacity, Cost-Efficiency) that can be improved?

With the introduction of the AOP concept, the CDM thinking will be extended both inn time as well as in scope. Again, we could not be a bigger fan of this way of thinking as it focuses on the optimization of the entire system rather than just a part of it. However, if the data that is being fed into the network are low in quality, we will still not succeed. Therefore, I believe these efforts go hand in hand and in order to be truly successful in optimising the ATM performance, data such as we can produce will be essential. 

Besides efficiency gains I also see major potential on the safety side. There are many safety and compliance regulations at airports. Most of these are visually auditable. However, because today it is done manually there is very limited data about safety incidents. The Apron AI can be used to continuously monitor for a very wide array of safety/compliance events. As a result, incidents can be prevented, root cause analysis can be done, damage can be claimed and safety training can be optimised. 

What is your current market development strategy? What are the types of end-users for which you see a possible market and the development of targeted applications? How do you see the future of turnaround management in the longer term?

The core of our company revolves around the AI technology for data generation, predictions and optimisation. However, we do also offer several application for different user groups. In most cases the airport is our primary customer as they own the camera infrastructure from which we require the video feed. The airport is typically interested in different things than the airline and therefore we offer different applications that target different stakeholders, users and use-cases.

To give an example, for airports we have a multi-turn view that shows all ongoing turnarounds on a map. It only shows if a stand is occupied or not, if the POBT for a flight is before / after the SOBT (is the flight predicted to leave on time?) and a countdown timer towards the predicted off-block time. This view is more high level and shows a lot of flights. Another application that we offer is our real-time turnaround dashboard. This dashboard shows a live feed of the turnaround and an overview of all subprocesses. This application has been designed with either an airline or ground handler in mind. It can be complemented with our smartwatch application that gives a turnaround coordinator / dispatcher an alert if any of the sub-processes deviate from their schedule. 

Can you please describe your role within the company in your current position of Chief Customer Officer? What are your key responsibilities? What do you see as the most challenging aspect of your job?

I joined Assaia one year ago as Chief Customer Officer. In practice this means that I am mainly involved with business development, project management and product development. Obviously, we are working hard to try and get as many airports and airlines to experience our product. We are working closely together with them in order to keep improving the product to be able to deliver more value for our customers.

One of the most challenging aspects is that we are selling a solution that is totally new to the industry. There is literally no airport in the world that is using this technology today. Therefore, there are many things that we have to figure out and prove to our prospective customers. I do believe that this challenge will become smaller now that we have signed our first customers and the first implementations are underway. We also see this as final proof of the concept and product. Therefore, we are currently investing heavily in growing the team in order to scale our efforts and realise our mission. 

About Christiaan Hen

Christiaan Hen has spent his entire career in the aviation industry. He worked for
over 8 years at Amsterdam Airport Schiphol in different roles like Airport Development, Capacity Management and Operations Management. The last three years at Schiphol Christiaan was Head of Innovation. During this time he met Swiss startup company Assaia. Discussions about a partnership between the airport and Assaia evolved into a partnership between Assaia and Christiaan personally. 

At the end of 2018 Christiaan joined Assaia to become their Chief Customer Officer. In this role he is involved in both business as well as product development.

Christiaan uses his extensive knowledge of the aviation industry to further enhance the Apron AI product to make the apron more efficient, safer and more sustainable.

About the author

Andrej Němec is an Aviation consultant with over 15 years of international experience in Air Traffic Management, Satellite-Based Navigation, Aviation Safety, and Regulatory Affairs. A Slovenian-Italian national, he holds an MSc in Aerospace Engineering from the University of Rome “La Sapienza”.

He is the CEO and founder of NEMAND Ltd, an SME providing Aviation Consulting and Engineering Services currently active with EUROCONTROL and in the recent past also with the European Commission. Since April 2019 he is a contributor to the FoxATM Market Radar.