Auto Trader Engineering Blog
-
Cultural Learnings of Data Academy for Make Benefit Glorious Company of Auto Trader: or how we're promoting data literacy.
At Auto Trader, we are attempting to democratise our data.
We put together a team of data engineers, signed up to AWS, built a Data Platform, replaced all the components at least once because they weren’t up to scratch and, having done all that, finally realised we needed some people to use it.
Luckily, as we were progressing, we engaged with our potential users, who provided plenty of interesting problems for us to work on. Unfortunately, we will never have enough time to generate the code to answer all of the questions those developers, analysts, scientists, business and sales folk have, so we’ll need them to self-serve.
-
How buyers are affected by an AI-driven product: Price Indicator
Artificial intelligence (AI), is the ability of machines to learn from data. Often, this data is derived from human interactions. In this blog, we will explore how Auto Trader has used AI and Machine Learning (ML) to directly influence the buyer’s and seller’s experience.
-
Keeping the engine running efficiently event—March 7th
Building on our previous Open Evening of tech talks, we want to showcase what is keeping the Auto Trader engine running efficiently. Our speakers come from various operational disciplines and are all keen to give us an insight into what they are working on.
-
All The Small Things
Operations Engineers at Auto Trader play a critical role responding to service impacting issues and quickly restoring service. Our web applications run on Linux and Windows servers and are hosted on VMware, private and public clouds. We are responsible for monitoring these applications and responding to system generated alerts and incidents whilst performing routine trend analysis. In previous years the Operations Engineer discipline followed a daily task timetable which included lots of manual jobs. This is very inefficient and can be impacted by human error. Over the past year, the Operations Engineers have been learning about various programming languages and leveraging the benefits of third party APIs to help with reducing the amount of repetitive, tedious tasks. This has been with a goal of making our processes more efficient, which has freed up time to allow us to carry out more proactive work and increase our knowledge of new technologies.
-
Data Universe event videos
It’s been a few weeks now since our last public tech talks event—a big thank you to all who came. Thanks also for all the feedback we received on the evening. We plan to use this to help shape our next event. For anyone who missed the event, please find the videos of all the talks below. Watch this space for details of our next event in the new year.
-
Lessons from the data lake, part 1: Architectural decisions
It’s eighteen months since the inception of the Auto Trader Data Platform, a cloud-based storage lake and computing solution designed to unlock the power of Auto Trader’s huge and growing data set of vehicle transactions in the UK. It’s time to ask, what have we learnt, what have we changed, and did we deliver what we intended?
-
Delve into our data universe event—November 21st
Building on our previous Open Evening of tech talks, we want to delve further into our data universe. What good is having all this data if our people can’t easily access and glean insight from it? Our speakers come from the disciplines of Data Analysis, Data Science and Data Engineering. What brings us all together is our Data Platform. Find out about the rewards and challenges of delivering on Auto Trader’s self-serve data mission.
-
Projection Search: Review of Persistence Technologies
Having previously established what we mean by a projection search, it’s time to have a think about how we might build it. Let’s start by reviewing the current landscape of Java persistence technologies, comparing and contrasting these, with an eye on suitability for our search. We’re not going to cover all of these, but we are going to cover a representative sample.
-
Greater Manchester College Hack 2018
The Greater Manchester College Hack is open to anyone at college in the area who wishes to develop their digital skills by participating in challenges set by local tech companies. Auto Trader sponsored the Hack in June this year by providing a challenge, prizes and mentors. This is our experience.
-
How we used Databricks notebooks, MLeap and Kubernetes to productionize Spark ML faster
Machine learning (ML) models are nothing new to us. We’ve used them to power products such as our used car valuations and Price Indicators. The processes involved in training and serving these models are complex, involving code written by both data scientists and developers in a variety of languages including Python, Java and R. Serving predictions from these models in real time typically involves a custom application to extract the coefficients from the saved model, and apply them to an input request.
-
Projection Search: Introduction
What do we mean by Projection Search? Almost all web applications need to provide their user with a search, providing a summarised view of their data. An email inbox is a good example of this. Within AutoTrader, we need the ability to list a user’s vehicles and adverts.
Usually, this type of search returns a matching list of resource summaries, each summary being a subset of the actual resource, with some mechanism - such as a link - for finding the resource.
Our recurring design headache is to decide which fields are included in the summary object.
-
Sending Spark logs to ELK using logstash-gelf
An important part of any application is its underlying log system. Logs are fundamental for debugging and traceability, but can also be useful for further analysis and even in areas such as business intelligence and key performance indicators (KPIs). At Auto Trader, we emphasise the importance of building a robust application logging system that can be integrated into our ELK stack that serves as a centralised log store.
-
Let's Encrypt at Scale
Enabling HTTPS on 3,000+ websites is a bit of a pain. But as we are now in the age of increasing online privacy, we had to knuckle down and find a way to do it. We provide a platform for trade dealers to upload and advertise their stock online. As an extension of this, we offer a product that allows customers to host a private website using this stock, under their own domain. We wanted to provide HTTPS support to all of these websites.
-
Customisable Logging on Kubernetes
Centralised logging has been a part of Auto Trader for the best part of eight years—the last three of which we’ve adopted Elasticsearch as our choice of software.
From application logs to system logs, monitoring logs to security logs; we try our best to provide a central location for users to log what they care about and we try to make it as easy as possible to do so. This is true for our current platform and we aim to make it true for our next generation platform, running Kubernetes on Google Cloud.
This post will cover the solution we’ve implemented to achieve good logging on Kubernetes.
-
Speaking about Continuous Deployment at Pink18
Last year Dave Whyte and I were lucky enough to be invited to speak at Pink18 in Orlando, Florida. The Pink Elephant conference is the worlds largest and most prestigious IT Service Management conference, having run for over two decades.
This years theme was Integrated Service Management, which is about aligning traditional Service Management processes with DevOps and Lean/Agile.
This post is about the talk I gave, the public speaking experience and the preparation required to deliver a cohesive talk.
-
A Boot Camp Career Reboot
Software is eating the world say the pundits, and consequently companies like Auto Trader are hungry for developers. The excess of demand over supply has led to initiatives such as TechReturners and our own recent Returners Discovery Day, which aim to lure back those who have left the industry. But there is another source of coding talent which has emerged in recent years: the coding boot camp. Can a few months of coding practice be enough to prepare someone with little or no experience of coding for a career as a developer? Even someone like me?
-
Experiments and the 3 Lies we Pin on Them
It is said that there are three types of lies: lies, damn lies and statistics. However, you don’t need a degree in statistics to be able to recognize these lies, and redesign your experiment to tell the truth.
Before Christmas, we launched a new advert creation journey, and after significant development effort, I’ve been very invested in making it a success. Lean principles state that we should try to get a product or service out to the user as soon as possible, and observe the effects. However, the work we do to launch a new feature often motivates us to root for it too much, and ignore problematic ways in which we gather the data. This often results in a phenomenon known as the sunk cost fallacy, where people convince themselves to throw good money after bad, to avoid the pain of losing. Here are a few tips on how to spot the mistakes we often introduce into the data when experimenting on users.
-
Use templates for better Git commit messages
Commit messages are important. They are a means of communication with yourself and your team throughout the life of your codebase (remember that team members are likely to come and go over time.) In fact, given that they live alongside the code, they’ll probably be the best source of documentation that you have about the evolution of your codebase. Commit messages are important!
-
Using AWS Lambdas for data lake monitoring
Here at Auto Trader, a core part of our strategy is to help improve the process of buying and selling vehicles through the provision of data driven intelligence. This data driven approach isn’t just something we suggest to our customers; we also use data internally to drive our product development. For example, our product teams use KPI dashboards when adding new features or developing new products to ensure that they’re making informed decisions. The data that drives these dashboards needs to be correct, and we need to know if anything goes wrong.
-
How we (almost) completed a natural language recipe recommendation engine using Twitter in a 24-hour hackathon
On 28th October we four Java developers attended HAC100’s Hack Manchester event. A 24-hour hackathon of coding, coffee and chaos (25 hours really as the clocks went back). We were one of three Auto Trader teams out of about 50 in total. Our team took on the dunnhumby challenge, “to use technology to enhance the retail experience for a customer in the home or in store.” By the end, we had built a personalised recipe recommendation engine with a natural language based Twitter interface.
