TRANSCRIPT
EPISODE 17: JURE LESKOVEC
Jim Freeze Hi! And welcome. I’m Jim Freeze, and this is The ConversAItion, a podcast airing viewpoints on the impact of artificial intelligence on business and society.
[UPBEAT MUSIC]
On today’s episode, we’ll be speaking with Jure Leskovec, Chief Scientist at Pinterest and a professor at Stanford University.
For years, Pinterest has been a popular social media platform for discovering and sharing images and information. Amid the pandemic, it’s not surprising that the site’s usage has grown – Pinterest recently reported over 400 million users.
In this episode, we’re sitting down with Jure to learn more about the technology behind the recipes and travel guides we see on our Pinterest feeds. We’ll talk about the powerful recommendation engine he and his team have built that serves up relevant, personalized suggestions for hundreds of millions of users, and hear his vision for the future of AI and social platforms.
Jure, welcome to The ConversAItion.
Jure Leskovec Hi, thank you for the invitation.
Jim Freeze Yeah, we’re very happy to have you with us today. So you’re the chief scientist at Pinterest, and you’re also a professor at Stanford University. Could you talk a little bit about your academic area of focus and kind of what drew you to that specialty?
Jure Leskovec Yeah, so I’m a professor in the computer science department at Stanford University, been there now for over 10 years. And my area of research is data analytics, machine learning, specially applied to complex data types in particular like graph or network data, social networks, and interactions between different types of objects or entities. And I’ve been basically developing machine learning algorithms to analyze and predict all kinds of human behaviors, recommender systems. And this is kind of then also the connection, why Pinterest is in some sense, such a great place where we can take our latest kind of theoretical ideas and also have a test bed where we can really put them to work.
Jim Freeze Yeah absolutely. And we’re going to explore that a little later. I’m very interested in understanding a little more about image rich datasets. But we’ll talk about that in a bit, but I’d kind of like to explore how you ended up at Pinterest. You co-founded a company called Kosei machine learning ad startup that was acquired, I think in 2015 by Pinterest. Could you tell us about Kosei and what inspired you to start the company?
Jure Leskovec Yes. I mean, at Stanford, we are kind of ingrained with entrepreneurship and we are kind of always talking to various industries to understand what are the big problems, what is the state of technology and so on. And I was working on recommender systems trying to understand human behavior and build capabilities that would allow us to better anticipate, predict human behavior and how it’s going to evolve. So it quickly became clear that this technology can be applied in recommender systems. And at that point in time, you know, there was kind of, in advertising there were a lot of times when you would kind of go to some website and then the ads from that website, I know be it about the couch or skiing boots would kind of follow you wherever you went for weeks.
So it really seemed like there is an opportunity to develop technology that would allow for much better, more personalized and more timely recommendations. And then you can also look at advertising as giving recommendations to the user or giving the right, showing the right ads to the right person at the right time. And this is kind of how we then started the company with two other colleagues.
Jim Freeze Yeah it’s interesting, so it sounds like it’s a much more sophisticated method of retargeting. If you go to look at a particular site and then, you know, ads pop up and quite often they’re actually not that relevant. It might’ve just been once in a kind of a blue moon, you went to a particular website and all of a sudden you’re getting ads served up for that website. So this sounds like it’s a lot more sophisticated.
Jure Leskovec Yeah. We did not want to do retargeting exactly. It’s all kinds of problems with the retargeting including privacy and so on. And the current recommender systems, right, they can either retarget, but going a bit to the left or to the right from that exact thing, they kind of totally deteriorate. And we started working on a system that would actually allow us to make proper recommendations rather than just show the same thing over and over again.
Jim Freeze God bless you. I get these retargeting ads all the time. So I’m very happy about that. So I know there’s a really good story behind the Kosei Pinterest connection. I believe it started with a former student of yours, is that correct?
Jure Leskovec Exactly. It’s a, you know, Silicon Valley is very well connected and a very small place and the information travels very fast. So basically I was just at, you know, at some, at some party. Met an ex student of mine, kind of casually mentioned to her that, you know, we are working on an exciting technology focused startup. This is kind of how the connection, and she was working with Pinterest at that point in time. And this is kind of how I think the connection happened. And then of course it took some time. We really did not want to be acquired. We wanted to kind of focus on our vision and execute along that, but, you know, after quite a lot of discussion and kind of getting to know each other and building trust, it kind of became clear that our vision can be perhaps amplified and we can develop it faster by basically taking the technology we have, joining Pinterest—who at that time was still a small company was around, you know, 200 people—and scale it up. And that’s kind of how it went.
Jim Freeze Yeah it’s a small world, isn’t it? It’s amazing, especially an area as big as the Bay Area you wouldn’t think, you know, everybody would know everybody, but they do so. So at Pinterest now you’re the Chief Scientist and you have been since the acquisition of Kosei. Can you talk a little bit more about, you know, in layman’s terms, how the technology impacts the experience of a typical Pinterest user?
Jure Leskovec Yeah, so kind of my role at Pinterest is to look at technology and kind of create road roadmaps and systems that we can then keep improving on. But every about two years we have to come up with something completely new and completely from scratch that kind of sets us to the next level. So technology, I mean, or if I say data analytics, machine learning, AI is kind of, is the core of Pinterest. I think perhaps an interesting problem is that because Pinterest is not kind of a following based social network. What this means is that any user can see any piece of content, any pin out of, you know, sets of billions of these pins.
So this means we need a very fast and very responsive and very scalable recommender systems that to any user at the given time will select the right, you know, 100 pins out of this set of several billion. And this is really kind of the core of what makes Pinterest, and then out of these recommendations, we compose the home feed, which is the home screen, the newsfeed of Pinterest, we compute what are the related pins. And we also use this to serve search results. And then of course advertising is essentially the same problem. We just don’t serve organic content. We serve the paid content, right?
Furthermore, if you look at Pinterest, it’s a visual discovery engine. So we also include a lot of computer vision technology that allows you, for example, to select a subset of an image and we quickly search billions of Pins to find your related Pins so you can take a picture of the object in the real world and we would find it on Pinterest. And this is very interesting because it kind of requires you to really understand what these Pins are, what these images are, what’s the content of the image, and what’s kind of the semantics of that image. So you can find things that are similar. And when I say similarity, especially this similarity can be quite nuanced.
If you for example, think of fashion or furniture where, you know, there is a given style and perhaps, the color is very different, but the style is the same. So if we need the quite sophisticated technology to basically be able to compare things by style, and then, I mean, kind of the last place to mention where let’s say machine learning is super useful is in keeping Pinterest safe which means identifying a spammy, unsafe Pins, malicious users, spammers, things like that. So this is another place where machine learning really helps us keep kind of Pinterest safe and trustworthy.
Jim Freeze Yeah what’s really interesting, and I want to kind of drill down on something you just talked about there, and you also talked about it in some of your opening comments, is obviously AI needs data and lots of data. And it’s something we’re very familiar with at Interactions. And we very much believe that AI needs humans and humans need AI. And in particular, we use humans in real time with voice and text to build really sophisticated data models that support customer service operations. I’m interested in the notion of human curation of the image rich datasets, I think it’s for many folks, the notion of being able to build datasets and apply AI to an image as opposed to something like voice or text, where there are words that have meaning, it’s kind of an abstract concept. Can you drill down on that a little more?
Jure Leskovec Yeah. So I mean, working with the images is kind of much more complex than in some sense language or sequence type data and really kind of the deep learning revolution, the deep neural networks and representation learning ability of this model which basically can take in the raw data and can come up with the presentation of the raw data that captures the relationships and content of those objects is very important because in the in the old days, right, you would have to take an image and then you’d say how do I describe the image? And you would come up with a lot of different heuristics to identify, address and various kinds of features of the image to kind of say, Oh, now I understand what’s in the image. Today we don’t do that anymore, but we kind of use deep neural networks that can kind of learn this featurization of images and that is very important to us.
What is also very important is this curation loop that Pinterest has invented where basically people look at these Pins, these objects, these are images with a bookmark, with a hyperlink to the target web page. And these are not just images. They also have a title, they have a description and so on, and what Pinterest users do they take these Pins and they save them or organize them into collections that they create. And those collections are called boards right? But what this means is that the essentially Pinterest users are doing two things that are super useful to Pinterest. First is that they are categorizing and almost like semantically labeling these images because if two images belong to the same collection, they must have something in common. Right?
So for example, if I create a board of fireplaces and I take an image into it, it’s likely an image of a fireplace, but somebody else may take that same image and put it into, I know another collection called vintage kitchens. And then it means that this is both the image of a fireplace and of a vintage kitchen. Right. So that’s one important part is that these things get categorized and labeled by users.
But then the other thing is that these workers, you know, who are doing this work for Pinterest, if you want to think of it this way, these are our users. So it’s not only that they label images, but each person labels images in their own unique way. So this means we understand how that person, in some sense, perceives the world, what things they are interested in and how they put things together into collections. So it means, in some sense, we get the data on both ends. We get data about what are the relationships between pins between the images. But we also get data about the uniqueness of every user. And it is kind of the combination of these two that really allows us to make the technology, the recommender systems that truly personalize to the user, because you need to know the content and you need to know the human to be able to recommend.
Jim Freeze Yeah it’s a really interesting concept because, you know, one of the things, certainly on social media sometimes ads or content is served up to individuals. And sometimes I feel like the platforms I personally use sometimes know me better than I know myself. What’s your perspective on, is there a line where you’re kind of stepping over it and almost getting almost creepy because you know so much about that individual. How do you guys think about balancing what you’re able to do in terms of personalization versus kind of that notion of privacy and not stepping over the line, how do you balance that?
Jure Leskovec Yeah. So the way we think of Pinterest, right? Like Pinterest is not a social network where people expose themselves right, or feel exposed. What this means is that Pinterest is a safe place where people really ask themselves, what are my interests? What would I like to do? What do I want to do? You know, with my free time, what topics am I interested in? And then that’s kind of, it’s an internal motivation to use Pinterest rather than kind of trying to post things to others. So this means that Pinterest is a safe place, and it is very important to us to keep that safety, and make sure that in some sense, our users feel safe, which means that we put a lot of effort to make sure that our technology does not amplify biases, does not discriminate and that it is safe and inclusive.
So if we don’t only say, Oh, you know, does this model increase some metrics, but we actually put a lot of effort to understand the implications of the, let’s say, technologies, machine learning AI systems that we are developing. So we are mindful of that. And similarly, when, you know, when developing these systems, we are careful what kind of signals are we using? So that users are not surprised when they see our recommendation.
Jim Freeze That’s fantastic to hear. I think the sensitivity of that is obviously very important to users. Kind of a last question for you. You know, I think it’s inevitable that all aspects of AI will continue to shape online platforms, e-commerce experiences. Do you have a vision for what’s next, you know, how will large scale, deep recommendation engines transform the way we seek out and engage with content or with each other?
Jure Leskovec Yeah. Yeah. I would say that today, let’s say on the, we have kind of the social platforms, we have the e-commerce and users have to jump between a lot of different sites, applications in order to accomplish what they want to do. And here what we think is that connecting these applications together. So in some sense, get the user from the beginning where they are kind of just thinking, tinkering about perhaps doing something, buying something, showing them a set of possibilities so that they can identify the few they like, and then, you know, walking with them, or kind of helping them to go all the way from the inspiration all the way at the beginning to the execution and actually doing the thing at the end and kind of Pinterest is really trying to connect, you know, it started very much on the inspirational the side, but it’s kind of moving more and more towards actually doing so that we support not only the user getting inspired, but actually say, but if I’m inspired by this and I want to do it, here are the steps I need to take.
And I think the hard part here is how do you take the user all the way from the inspiration down to actually taking an action. And I think here, for example, what would be interesting is that there is a lot of additional kind of tools and features that can be built in terms of, let’s say augmented reality, virtual reality that allows the user to make decisions in an easier way. So you can imagine how AI can help you figure out what kind of furniture you would want to put in your living room or how lipstick would look on you. So, for example in the makeup space or in the hairstyle space fashion and so on.
So that really, you would kind of have an assistant that helps you from this inspiration side of things where, you know, you say, Oh, I like this, I don’t like that, I would want it more like this, but less like that and through examples through kind of this visual discovery identify what is it you want, what is your style? And then kind of carry you all the way to actually making an action, doing the thing, perhaps buying something, going somewhere or taking some action on it.
Jim Freeze Fantastic I learned a lot today. I really appreciate you joining us today. It’s been so interesting to hear about how these personalized recommendations show up in our Pinterest feeds and how AI is at the core of it. We really appreciate you taking the time to join us today and share a bit more about the technology. Thanks so much Jure.
Jure Leskovec Yeah. Thank you very much.
Jim Freeze On our next episode of The ConversAItion, we’ll be joined by Alex Capecelatro, co-founder and CEO of Josh.ai, an AI platform that provides smart home automation services. We’ll cover how AI-powered voice control can create a more seamless at-home experience, and discuss the critical importance of privacy for smart home technology.
This episode of The ConversAItion podcast was produced by Interactions, a Boston-area conversational AI company. I’m Jim Freeze, signing off, and we’ll see you next time.
[UPBEAT MUSIC]
+++
