InfoQ Homepage Presentations Behind Booking.com's AI Evolution: The Unpolished Story.
Jabez Eliezer Manuel discusses the 20-year evolution of Booking.com’s data management and ML platforms. He explains how they scaled MySQL topologies, sunset Hadoop for cloud architecture, and built high-throughput model serving and feature stores operating at 400B predictions per day. He also shares insights on personalization, interleaved experimentation, and their newly introduced agentic AI foundation.
Jabez Eliezer Manuel brings two decades of software engineering experience to his role as Principal Engineer at Booking.com. Over the past decade, he has tackled high-scale challenges ranging from low-latency search personalization to unified payment ledgers. Today, he drives the platform engineering for long-term agentic AI memory and next-gen intelligent application infrastructure.
It has been revealed that software is changing the world. QCon London empowers software development by facilitating the spread of knowledge and innovation in the developer community. A practitioner-driven conference, QCon is designed for technical team leads, architects, engineering directors, and project managers who influence innovation in their teams.
Presented by: David Iparraguirre - Senior Product Marketing Manager at Datadog, and Jessica Yang - Product Manager at Datadog.
Presented by: Kiran Gangadharappa - Engineering Advisor at Harness, and Scott Davis - Senior Engineering Manager at Harness.
Jabez Eliezer Manuel: Today, Booking.com impacts billions of travel decisions, and behind each of those travel decisions is actually a bunch of ML platforms working behind the scenes. I'm here to talk about them. In the spirit of QCon's 20th anniversary, we are going to go back in time by two decades and take on a journey and then see how these platforms evolve.
To make it easier for you to follow along, I've structured my talk into three layers, the data management layer, the ML engineering, and domain intelligence. In the data management layer, we will see the evolution of our databases, and some of the scale challenges that we had, and the cloud migration that we undertook to actually better our data. In the ML engineering layer, it'll be model serving at scale, and our feature engineering atoms there. In the domain intelligence layer, it'll be a high-level tour of all the machine learning platforms that we have. I'll pick one of them and then do a deep dive, so you get a full evolution of that specific platform, and you'll also have something tangible to take.
Let's go back to 2005. This was how the world looked like in 2005, if you remember these camera phones. Web 2.0 was taking off as a platform, and Booking.com was just like 9 years old. It had still not joined the Priceline group. Inside Booking, something fascinating was happening. We had run our first A/B experiment. Today everything goes through experimentation. It doesn't matter if it's a simple CSS change or an infrastructure change, everything goes through experimentation. At any point in time, if you're visiting Booking.com, you are subjected to 1,000-plus experiments, literally. Over the period of time, we have run 150,000 experiments or more until now. One thing that's certain is that most of our experiments fail, our success rate is just less than 25%. The goal isn't to get it right, but it is to learn fast. If you see that golden star on the top right, it's my little takeaway marker.
In a recent development, it's my way of hinting to you that there might be a lesson in this slide that is reusable for you. Continuing on experimentation. Experimentation built our data-driven DNA. It gave us a lot of technical benefits, which is sure. The ones that I like are that it eliminated HIPPOs. There were no endless debates internally. You just say, ok, let's go and build, test it, and done. The last is like, no, it created a culture of humility. There were no experts in the room, meaning that nobody could pinpoint and say, yes, let's experiment and then figure it out.
Industry observers note that experiments needed data, and that's where we are going to begin our journey in data management. In this part, I want to talk about our MySQL, our Hadoop, and our move to cloud. These are the three topics that I want to cover. This was our original tech stack when Booking.com started. It was just Perl and Postgres. Promptly, we replaced Postgres with MySQL, primarily because of its asynchronous replication capabilities and for other reasons as well. In 2005, our setup was just a single master and four slaves. Fast forward 15 years, we were operating one of the largest clusters in the world. You might be wondering, how do you manage so many instances? Three things. We have a single writer setup, and we have replication chains, which I'll show you. Then we have a lot of automation. Let's make it real with an example, with an image. What you're seeing is a single master topology.
On the left, orange is the master, and then there are replicas or slaves. The colors indicate different data centers out there. As you can see, we minimize the replication across DCs by having intermediate masters. This whole topology is managed by a replication management tool called Orchestrator. If anybody is using MySQL, you might be aware of it. Automation was a necessity. For example, every 60 days, we take these 7,000 instances or so out, deprovision them, and reprovision them again. That meant that for every three minutes, there is some action that is going on. It is not humanely possible for some engineer to sit and do it. We have plenty of automation to do that.
There are a few things that makes our MySQL setup a little bit different. One is we don't use any specialized hardware. It's like simple frontend blades that are used for MySQL. No stored procedures, no UDFs, no views, not even a cache layer. Yes, no cache layers. The secret sauce, what we do is we actually create smaller DBs. The soft limit is 2 terabytes. The hard limit is 10 terabytes. We try to keep everything small. The reason is we want this database to fit into the fast storage of that server blade that we are using. In this example, we were normally using server blade 7 or 9, if you are aware of its configuration. What that meant is our point queries were less than 350 microseconds, which was roughly in the order of some of the memory engines out there. We were effectively using MySQL as a cache.
Sources indicate that it also provided several side benefits. It was faster backups, quicker failovers, and primarily also lower cost, because if you're moving away from fast storage to iSCSI, it's expensive. We didn't just use MySQL. We actually helped shape it. What do I mean by that? In the early days of Booking, we had expertise in-house to fork MySQL, but we made a conscious decision to say that we're not going to fork, we're going to contribute upstream. Many of the features that you see today exist because we wanted it. Like, for example, GTID-based replication is because we actually proposed it. We made the case. We shaped the design of MySQL.
I want to sidestep a bit and then give you a picture of how machine learning looked in that era. It was Perl scripts and statistical and scientific computing libraries from CPAN and just cron jobs. The deployment strategy was simple. Copy the script and pray. It solved real business problems. We were able to do fraud detection. We were doing demand forecasting with regression. We were doing recommendations with KNN and a few other things. Even in those days, we were able to solve that. This MySQL setup helped us a bit. As data grew large, we needed something more. Hadoop came to the rescue. In 2011, we actually built two on-premise clusters. This was a configuration that was there. It was redundant clusters. The setup looked like this. There was MySQL on bare metal. Then there was this classic Hadoop stack, HDFS, YARN, MapReduce. Then we layered it with Oozie for orchestration, Spark for workflows, and Hive for SQL, and Sqoop for moving data between Hadoop and MySQL.
Sources indicate that on top of it, we had some self-serve tooling, a classic setup. Data came into Hadoop via two means. From MySQL, it came as nightly jobs into Hadoop. Then from Kafka, it was Kafka Connect that was bringing in data. This data was used for machine learning purposes. This is how prediction training was happening back then. What you see is two workflows, the training workflow on the top and an inference workflow. A scientist would actually write a workflow where they would train a Python model. Then they call it as pickling the model, which is just serializing the model, storing into Hadoop. Then another workflow loads that model, does the prediction that it wants. Then it actually stores it into MySQL in some cases, but not in all cases. Basically, the whole prediction was happening completely offline. In some cases, we want real-time. In those cases, what we did was we just faked it.
In a recent development, this is how we faked it. The top is the offline workflow, and the bottom is a web request. We wanted the real-time prediction to happen in the web request. All we did was we trained the model. I'll give you an example how we did it. Let's take an example of a model that we want to train, like device and country is the feature, and probability to book is the output that we want to do. In those cases, what we did was we just enumerated all the input features. We just created all possible devices, all possible countries. Then we just predicted on that upfront. That gave us a table. All we did was store that table into the MySQL. Then when the request came in, we just queried the table as if we were predicting, and then went on to do our job. Obviously, it doesn't work for most cases, but that was our earlier real-time prediction with faking it with lookup tables.
According to recent reports, that worked for us in most cases. Hadoop powered for us most of our pipeline for many years, but there were some problems. One was it was a shared environment, so every training, production workloads, everything was running in a single cluster. There was mono-version. If you are a data scientist, you cannot use a version that you want. You just have to use a version that is possible, like now it is shared. The second thing was there was no GPU support. That limited our ability to train a bigger model, larger model. We were using two different DCs for redundant setup. Then you need to keep the data in sync between this, and that was becoming a headache. Capacity issues, like at peak times, we were operating very close to max capacity. Noisy neighbors, like some NOOB who's writing a query can actually clog the entire production cluster.
Industry observers note that most importantly, there was no discovery or ownership, even though people depended on each other's data. That was the biggest headache for us. This is what it looked like. Like now somebody might think that this is a Rorschach psychological test, but it is just a dependency graph for our workflows. If you zoom in closely, you can see Hadoop screaming in binary there.
According to recent reports, 2011, we introduced Hadoop. 2018, we realized the limitation of Hadoop, but then it took us until 2022 to put it into maintenance mode, and another three years to go into sunset. Like now what was going wrong? Why did it take so long, that hairball that you saw earlier? Why was it difficult? We were operating with hundreds of petabytes of data. We had about 190,000 Hadoop tables, 7,000-plus workflows, hundreds of teams, and then we want to keep everything up and running. Our goal was move everything, stop nothing, align everyone. Everybody knows that that last part is already a moonshot. We were like, why pick one impossible thing? Pick three. We had four problems to solve. One was, we wanted to actually modernize our compute and scale it, obviously, because we were operating at max capacity. Then we want segregation of workflows, because we wanted isolation at least between dev and production.
Industry observers note that governance and ownership, which was a mess, we wanted to fix that. The last was, we wanted to enable discovery of workflows and datasets so that people can reuse things. Saying goodbye to Hadoop was hard, and this is why. This is what happened in those gap years. 2018, when we realized the mistake, or realized the problem, we actually attempted to move into one of the cloud providers. Quickly, the product got shelved. The reason was, depending upon who you ask, it is either cost or a strategy shift. We quickly pivoted to building something internally to help us. Like now we actually built assistant application along with Hadoop to have a data catalog. Then we also had built Kubernetes clusters to do Spark. That eased us a little bit, so we could continue on. A short bit later, we started the modernization project, where we focused on three things, like quality, ownership, and governance of data.
Recent findings suggest that we were trying to fix that with that modernization project. During that time period, we actually piloted another cloud, and that indicated promise, but we didn't actually make enough to move the needle to actually do a full migration. In 2023, leadership actually made the call, and they said, guys, we're going to do it all in, no proposal B. That actually helped, we made progress, and we succeeded. Just within one and a half years, we migrated everything that you saw. This is how we did it. Maybe there is something in there for you. We actually built a tool that mapped every possible table, every dependency into a graph. That was our first effort, map the universe. Second thing that we did was, we looked at the usage analysis of all the tables and the workflows, and the lineage as well, and said, which of this is business critical?
Industry observers note that then we tried to trim it down. You'd be surprised that 92% of the workflows didn't pass the test, someone cares, it was easy. The best architectural tool is Delete key. What we then did is, we actually did graph analysis on those workflows that passed, and we actually prioritized it by centrality and impact. We took a few, and after that it was simple. Group them into batches, do the migration, and slowly phase out the legacy system.
Sources indicate that if you look back, one key thing that made the difference was this unified command center, which held that graph data, and that became the central place for anybody to look at it during the migration. If I had a workflow that I had to migrate, but then I depended on a few other workflows, I could go in there and then see, ok, when is my dependency migrating, and when can I migrate. I could plan things easy. Anything, status, dependencies, blockers, everything was in there. This replaced spreadsheet tracking, and also it replaced tribal knowledge. Everybody knew what exactly was happening. The second thing we already saw was, we removed plan B out of the equation. Leadership mentioned, just go in. We all know that this is not always the best plan. The best strategy is prevention. If we had a time machine, these are the four messages that we would send to our past self. First, catalog every asset that you have. Data workflows doesn't matter. Catalog it. Establish clear ownership. It can be business ownership and also technical ownership of the data. Build quality gates before you share that data. Track lineage end-to-end. That concludes our data layer.
Recent findings suggest that next in the ML engineering layer, I want to talk about two things. One is how we industrialized ML model serving. Second, our attempts at feature engineering and the problem that still remains. Before I do that, I want to give you a 30-second overview of evolution of our ML stack. In 2005, we actually were using Perl libraries and the MySQL, which was still doing the job. Hadoop came into the picture that brought in substantial data analysis and reporting, that was good. True machine learning kickstarted with the introduction of Oozie and Python, like the sklearn era, because we were able to do things easily. When Spark came in, that changed the game or changed the pace of the game. Now we were able to do more things faster and we were able to crunch more numbers. H2O took it to another level. Now we were able to do faster training.
According to recent reports, also, we were able to deploy models directly in any Java environment. That was a big thing. In 2017, deep learning came in. Deep learning started as a very simple hackathon project, where a group of developers wanted to improve the image stacks, which was of low quality. From then on, it is widely adopted across the company. 2023, LLM wave took off and we built a bunch of generative products, which we'll touch upon a bit later. 2025, we started laying the foundations of the agentic systems, which we'll also touch later.
According to recent reports, going back, if you look at the timeline, 2015, in my opinion, was a pivotal year, because we were solving two problems. One was real-time predictions. Until then, we were solving real-time predictions with fake lookup tables that you saw earlier. In a hackathon, we actually built a small Scala service, which was backed by a storage. It was actually used to do model serving. Machine learning scientists could upload the model. Back in those days, the model was just classical problems, like logistic regression, classification, and so on. Uploading a model is nothing but uploading a CSV file with all the features that was used, what type of those features, and some weights and coefficients. In real time, when a user landed on the website, some backend service collected the features necessary to invoke the model. Then they used the Perl SDK, because we were still in Perl back then, to invoke the Scala service.
Recent findings suggest that over a period of time, this Scala service has evolved into a platform. Now it supports 12-plus different model formats. You can think of all the major ones are in here. This is how the setup looks like today. That service is deployed in multiple clusters. Why multiple clusters? We have some general-purpose shared tenant clusters, some dedicated clusters for very critical workloads, and then also some clusters for batch. Either these clusters actually host their model, or they actually can interact with externally hosted models, so like SageMaker. They can also contact feature store, which we'll cover later. To hide the complexity from the end customers, these clusters are fronted by a custom proxy. Every prediction that these clusters make, is logged. Then those logs go into data lake and also to our monitoring platforms. Let's take an example how a flow of machine learning scientists uploading a model and inference happens.
Machine learning scientists upload the model artifact into something like S3, and then they notify an ML portal to say, I've registered this fresh model. That triggers a pipeline. The coordinator goes and picks up that model that the ML scientist uploaded, and then comes back, versions it in its own storage. Also, what it does is, it finds the right cluster to deploy, and then allocates enough space, and triggers that upload. In addition, it'll also upload that, update the proxy that you saw earlier to say that the proxy knows which cluster to route to. Serving was simple. When a web request came in, backend servers just invoked the custom proxy with this client SDK. This was one of our success stories. There was no problem, and it was a clear success. Two years back, this is what that system that you just saw handled. It has 480-plus models. It can do 400 billion predictions per day. All of those predictions within less than 20 milliseconds on an average. Those are impressive numbers.
Sources indicate that second problem that we started solving in 2015 was feature engineering. Back then, 80% of the time was spent on just data preparation. Two pain points emerged. One was duplication. Each team was trying to build the feature by themselves in a different way. The second was there was skew. The models that behaved one way in training was behaving another way in inference. We set out to solve the problem by solving the duplication and to bring consistency. We thought, let's build reusable computations. We decided to tackle the problem separately for offline and for online, because offline was targeted at machine learning scientists. Online was targeted for engineers. I'm going to sidestep here to give you the context so you can better understand the decisions that the team made. What you're seeing on screen is just an initial dataset. You have a user ID, what the user searched and when.
It has been revealed that then also the label that we are predicting on to say that did the user eventually book or not. That's the initial dataset. That's how an ML scientist starts their work. What they do is they try to add more features into this dataset. They keep looking for other feature tables. Feature tables look like this, which is just a time series representation of a feature. In this case, the feature represented is what is the number of searches that the user did in the last seven days. That is the series that you're seeing. Once they found such a feature table, all they do is a point-in-wise join, wherein they take a unique ID, in this case, user ID, and some timestamp, and they do a time-wise join on it. Then they get a training set, wherein they have the feature that they were looking for, which in this case is the count of last seven days.
In a recent development, when we looked at this, we said, we can help in two places. One is building that feature table, and the other is doing the time-wise join. That is what we set out to achieve. That was a feature table. That feature table is built by another fact log or an immutable log. This is just the searches that have happened over a period of time. By looking at this, it is just very clear that there is one data frame and the other data frame. We figured out we can actually just use Spark ML transformer, which did the feature translation between.
This is how it looked in its entirety. There were immutable fact logs, which is just log of search events that have happened in the past. We asked people to write reusable feature encoders that converted into a feature table. Once we had that, all anybody has to do is just go look at the initial dataset and do a time-wise join on it. We tried to do something similar to online as well. There were event streams. We asked people to write feature mappers in Scala, and that became enriched streams. From there, there was a centralized team which took all these features, put it into Cassandra, and then served it via an API. There were a couple of problems with this approach. One was parity. Data was different between online and offline. The computation was different from online and offline. Then there were performance problems because of the amount of data that we were processing in offline.
Backfilling or recomputation was hard. In the second attempt, we decided, let's fix data and computational parity together. The way we fixed data parity was we introduced the concept of timelines. Timelines is nothing but a time-ordered sequence of events for a user. User searched, user clicked, user booked, things like that. The idea was, let's build this timeline equally or approximately in the same way for both online and offline. Then that way, once we have that, we can just write computation once and then use the timeline. That was a substantial picture. This is how the timelines were built. Those are the search, clicks, and immutable events. They got materialized into Hadoop. We built timeline builders separately. It's a duplicate effort, but it was managed by a central team. That timeline was operating separately for online and offline. They synced the data or the timelines into Cassandra and HDFS. The only difference between these two timelines were the Cassandra timelines were trimmed. Unlike the MVA, TVA, we actually didn't trim timelines for saving the universe. We trimmed timelines in Cassandra for saving the space.
Recent findings suggest that this is how we used timelines for computational parity. We had a full timeline in HDFS, a trimmed timeline in Cassandra. We asked people to write centralized shared feature libraries. These are things that can be used to compute features, things like sum, count, large scale, and all those typical things was written by a centralized team. It also allowed extension. Anybody could come in and write in the feature that is not already there. In online or in real time, when an application wanted some inference, they would go to a feature store API and ask, can you give me this feature? What it would do is go into Cassandra, pick up the timeline for that specific user or use case, and then build that feature on the fly with these shared libraries. The same happened on the training side or on offline. That stack was Python, because scientists were using Python.
It has been revealed that we had a wrapper, which had a feature computation library wrapper. What it did was, whenever the scientist wanted a feature, it submitted a Spark job. In that Spark job, we actually did the same thing. We typically achieved what we wanted to do, both data and feature parity. The problem was, ML scientists were not familiar with writing features in Java, and that created a dependency between engineering and ML science. Secondly, in the previous one, the Cassandra timelines, computing features on the fly was costly, expensive, because it took more than 90 milliseconds or so to actually just fetch the data. It was not suitable for most of the use cases. Then we noted, ok, third attempt. Forget feature computation, we'll just focus on feature storage and serving. Let people deal with feature computation. We didn't care how features are computed, as long as we knew where to pick the features from.
If it was pre-computed, ok, we'll go to a data lake of sorts. If it is real time, we'll go to Kafka. Then we built a feature store. In the feature store, there was an ingestor component, which knew about these multiple data sources, and pulled that data, and stored it into an online store, and served it via an API. In this API, there was no on-the-fly computation. It was just a simple lookup, because features were already computed. The same pipeline, what it also did was sync data into offline, so that MLS can use it for retraining purposes, and for also other analysis. This feature store also had other benefits, like it had feature registry, feature monitoring, and so on. In essence, after three attempts, this is what we learned. It's a multidimensional problem, and we don't have an answer. Currently, for the pre-computation and real time, we currently have three systems, and we are always looking for one more system. That wraps the second layer.
Recent findings suggest that now in that third layer, we are moving away from infrastructure to business impact. I want to give you a high-level tour of all the ML platforms that we have, and then dive into one of them. This is the platform that we have. One is GenAI platform. It's the newest kid in the block. What it does is it enables any team to actually safely build generative AI experiences. It provides tools and infrastructure and foundational capabilities like guardrails, evaluation, orchestration out of the box. Things like PII detection, PII masking, everything comes out of the box. The product teams can focus just on product innovation and not on LLM plumbing. Let's look at a few use cases there, what we have. The trip planning, which is just a conversational experience where someone can go talk in natural language and get recommendations, that's powered by GenAI. We have smart filters.
Recent findings suggest that if you were ever on Booking.com, you would see hundreds of filters on the left, and it is hard for anyone to know which to pick. We have a text box where you can type it, and that translated into the right filters and sorters. The third is review summaries. That provides, based on the last reviews, a quick snapshot of pros and cons. The next is property Q&A. This actually enables instant answers to any property-specific questions, and it is grounded on reviews, past Q&A, and that is also powered by GenAI. We also have a booking support, wherein we have AI that can do basic customer support on payments, cancellation, and things regarding your booking. These are the use cases. There are many interesting things under the platform, but we are not going to cover it today.
GenAI is good, but to generate data, we actually want good content. That's where content intelligence comes in. What it does is it provides surfaces like good, high-quality content that are usable. This platform analyzes every image and text. It actually adds metadata or enriches that metadata. For example, if it is images, it adds themes, tags, and looks for any harmful content. If it is text, it looks for, for example, sentiments or topics that have been discussed. This also supports the reviews. Because most of the things happen in the background, like fraud detection is also taken care of by this content intelligence. One example that you see, anytime you see a gallery on Booking.com, it is probably powered by this content intelligence. Picking up the right photo to put up front and the order, which photos to highlight and so on. The other thing is, it autogenerates all text for screen readers.
Recent findings suggest that there's a platform for that. It also does reviews. It detects topics on the reviews and summarizes it at the property level. This is what it enabled was, not just getting intelligence at content level, but going one level higher. For example, if property had all the images and all the summaries, we can actually create intelligence at the property level to see what is happening. That kind of aggregation level, this platform enabled. The third is ranking. Ranking's responsibility is to order millions of products so that any time a customer comes in, they see the most relevant results for them, like something personalized. This is a platform that we are going to dig in, so I'll park it here. Recommendations is a self-serve platform wherein you can build any kind of recommendations across the funnel, from discovery to post-booking. It has more than 40, 50 use cases. I picked a few to give you a flavor of what it looks like.
The pushcards that you might see on Booking.com is powered by recommendations. There's multi-vertical recommendations. When I say multi-vertical, I'm referring to accommodations, flights, attractions, and so on. Those kinds of recommendations come from this platform. Unique stays, team-based recommendations for you, and attraction carousels that you might see on the page, and any flight recommendations. All of these are examples that are powered by recommendations platform. The last platform is value intelligence. The responsibility of this platform is to measure and optimize promotions. By promotions, I mean anything like vouchers, discounts, things like rewards that's being optimized by value intelligence. The goal of this platform is to say, maximize value for travelers, partners, and the business. That wraps all the platforms.
Let's jump into ranking platform. Ranking is a three-way optimization problem. We need choice and value for customers. We need exposure and growth for the partners. We also need efficiency and revenue for business. If we show irrelevant results, the travelers are going to be annoyed. If we show poor visibility, partners are going to be frustrated. If any one of them is annoyed for too long, I'm going to get my salary re-ranked. It's a three-way optimization problem. Back then in 2005, we had the formula for ranking hotels look like this. It was simple. It was just a simple function, which took bookings and views, and a random tiebreaker in case there were two hotels which had the same rank. The formula evolved over time. We introduced cancellations. It was taking consideration of seasonality. What is the distance of the hotel from the center? What is the number of views the hotel is getting?
Availability of the hotel. The average ranking in the locality. What is the visitor traffic? Guest reviews. These are just some flavors of what are the factors we considered. Back then, ranking actually was computed offline using simple Perl cron scripts every night, and then the scores were dumped into MySQL, and we used it in real time. Around 2016, we actually tried to replace the ranking formula with machine learning, and we failed miserably. The handcrafted formula was undefeatable. We could not beat it for a long time, like one and a half years or so. This formula cannot be beaten. The reasons were many fold. The primary reason was that the ideal model that we wanted to train the model on, and the massive amount of data that we had was not actually helpful when Spark kept exploding. The memory of Spark was not sufficient. What we did was we cut corners, we made some approximations, and we succeeded. In true Booking sense, we ran an experiment, and it was successful.
Next, we wanted to tackle personalized ranking, and there was a problem. This is what it looked like. The ranking scores was calculated offline until then. It was put into MySQL, and an availability engine read that MySQL data, loaded it into memory. The responsibility of this availability engine was that it needs to search the availability and prices of millions of properties. Because of the amount of data that this engine had, it used a scatter-gather pattern. It had a coordinator and 24 shards. Each of those shards actually worked on a subset of the data. When a request came in from the user, it came to the availability engine via a search orchestrator, and the request landed there. The request contained something like this. Where does the user want to search on? When? Then, what is the party size? Then, what are the filters and preferences of the user?
According to recent reports, the coordinator accepted that request and then distributed the work to the workers. Workers went on to do their job. They actually looked into the subset of the data that they had, took the data out, and then they used the ranking scores that were preloaded into memory to do the sorting. After they did the sorting, they supplied the results back to the coordinator. Coordinator aggregated all the information from the workers and re-sorted the results and gave the results back to the search. This is what was happening. If you wanted to personalize, meaning that every user gets a specific set of results, what we had to do was take the ranking logic out of this availability engine, get rid of those static scores at the bottom, and then replace it with a service. Then have that service invoke actually a model which is deployed in the model serving platform that we saw earlier.
That's what we wanted to do. There were two problems. The first problem was, if there were 1,000 requests for the availability engine, it meant that the new service is going to get 24,000 requests because it was sitting behind the fanout, behind the shards. Secondly, the availability engine was operating at super low latency, and that also meant that this new service plus the network hop to the model and the model's time all needs to be well within the time of that service. It's basically a high throughput, low latency problem, and we managed to crack it. These are the numbers from 2024. That ranking service that you saw in the previous slide, that green box that was introduced new, is able to solve 100,000 requests per second with a p99.9 latency of 100 milliseconds. The average is way low. It's like 40, 50 milliseconds or so.
I want to share one more thing before I wrap up the evolution story. Experiments at Booking typically run for two to four weeks, but this meant that we couldn't actually train or run models. We were limited in the number of models that we can run every year, because we just had 53 weeks. We wanted to move faster, knowing well that if we actually put in a bad ranking model upfront, it is going to cost all of us, like travelers, partners, and business. What we did was we introduced interleaving. This is how it worked. Typically, for an A/B testing, we actually take the people and split them across two groups, and then we show different search results for them, which was coming from different ranking models. In interleaving, what we did was we didn't show two groups, we didn't split the groups, we just showed the same results for all users.
Recent findings suggest that those results were coming from merged data, meaning we would train a couple of models, in this case, two models, red and blue. We trained the model, and each of these models gave a list of results. Then we would somehow need to merge it. One simple example could be, take the top one from here, top one from here, merge it. Then keep alternating it. It's just a team draft approach. Then once you have done, you present that to all users across the service. Then you check offline, ok, from which list did the user interact more? Was it from model A or from model B? That gives us the answer. Why it was useful was we could suddenly run more variants. On the right side, it doesn't have to be two. We can even run eight different models that are actually interleaving the results for search. That is, we can run more variants that was like 10 times speed-up than what we are used to with A/B testing.
According to recent reports, we could use it with less traffic because interleaving is super sensitive to customer preferences, and we can get signals faster. The way we use it now, interleaving and A/B, is that we train a bunch of models, we pre-select it with interleaving, we pick up a winner, and then we take the winner and then do proper A/B testing. We have optimized both speed, while not losing scientific rigor.
Recent findings suggest that I started the talk by saying that experiment laid the foundation in general for data-driven decision-making. You can see that, still, it continues to underpin what we do every day. With that, that concludes the evolution story.
Next, I want to tell you where we are going. These are the five platforms that you saw. What we started doing last year is that we started building the foundations of agentic systems. By agentic systems, think about memories, all kinds of memories, episodic, semantic, procedural, and think about agent serving, like access control for those agents, delegation controls for the consumers to those agents, agent-to-agent communication. Those are the things we started working on last year. Parallelly, what we are also doing is unifying all these platforms under a single umbrella and then having a single orchestration layer. Why? Because we foresee a possible fragmentation and we wanted to avoid it earlier. Like for cross-domain use cases, we want a single, centralized, unified experience for the customer, and this platform can do that. Secondly, we wanted to centralize all kinds of AI-related governance, security, quality, and all those things. We wanted to centralize that.
See more presentations with transcripts.
Recent findings suggest that microsoft Ignite 2026; Explore What's Next in AI and Software Engineering - Learn how to architect agentic systems, modernize cloud infrastructure, and tackle emerging engineering challenges. Get practical insights from Microsoft experts and industry leaders. Microsoft Ignite is less than two weeks away! Register now to join us in person or attend online for free, November 17–20.
