How Benedict Evans Separates Tech Hype From Reality

There’s no hotter topic in tech right now than generative AI.

Since tools like the image generator DALL-E started capturing mainstream attention and ChatGPT arrived promising to upend online search, entrepreneurs have been in a hurry to launch new companies and venture capitalists to fund them. Start-ups are offering generative-AI solutions for tasks from clothing design to marketing copy.

The frenzy, however, feels eerily reminiscent of the crypto gold rush just a few years ago. It’s enough to make bystanders looking on from outside the industry wonder if generative AI is just the next obsession in the hype cycle.

Benedict Evans has spent more than 20 years analysing technology, several of them as a partner at venture-capital firms including Andreessen Horowitz. For him, separating hype from reality is as much a part of the job as trying to predict the ways technology will change the world and the businesses in it.

The independent analyst — and former VOICES speaker — shared his thoughts on how he approaches new products, the potential of generative AI, the likelihood of the metaverse ever becoming reality and how Shein is like Netflix. (The interview has been edited and condensed for clarity.)

Marc Bain: When you think about how to separate hype from reality in the tech world, what are you actually looking for?

Benedict Evans: There’s not a general answer. What is this product? How useful is it? How does it work? How close is it to deployment?

MB: Generative AI is on everyone’s mind right now. On the one hand, it seems like the next obsession in the tech hype cycle as money floods in. But on the other hand, it does seem like a tool that businesses are beginning to use in various ways, and lots of regular consumers are at least playing around with. How are you thinking about this? Is it a potential game changer, or is that yet to be determined?

BE: Generative machine learning is a pretty profound technical breakthrough in solving a broad class of problem. What we’re trying to do now is work out, ‘Ok, where do you apply that?’

If you go back and think about the last wave of machine learning back in 2013, 2014, you had this shift. Stuff that had sort of worked but not very well suddenly started working really well. It seems to be able to do image recognition perfectly. What does that mean? Well, it generalises and it’s not really image recognition. It’s pattern recognition. Where do we have patterns? Where can we apply that? We quickly work out it’s not just images. It’s also translation. It’s natural language. It’s audio processing. But then go beyond that, it’s credit card processing or it’s network planning, or all sorts of things. It’s a whole class of thing that you couldn’t automate before that now you can automate, or maybe we hadn’t realised were things we could automate.

We are going through a similar process now with generative machine learning, which in very crude terms, takes the same models and runs them backwards. You can make anything if you’ve got a sufficient number of examples to provide a pattern. We are trying to work out what that would mean. There’s an absolute explosion now of people very quickly creating companies and creating actual products that you can use, trying to apply those to solving problems for real companies and real people.

MB: Judging by the way Microsoft and Google are going about things, there’s some belief that this could fundamentally change search on the internet. Do you think that’s overblown at this point?

BE: Nothing about this is overblown. This is a big fucking deal. This isn’t metaverse. This is not NFTs. This is like once-every-10-or-20-years structural change in what you can do in software.

Trying to apply this in general search I think is very seductive because in principle you can apply it generally to ‘all the text on the internet’ and therefore it can answer anything where there’s text on the internet. The challenge is that, because of how this works, it’s not actually producing an answer. It’s producing something that looks like what an answer might be. It’s just doing pattern prediction. There’s an error rate inherent in these systems, and the question is, does the error rate matter and can you tell? If you ask ‘What are the symptoms of appendicitis?’ it would be roughly right, probably. But it might not be, and you can’t tell. If, on the other hand, you are saying, ‘Here’s a press release. Write a one-paragraph summary of it.’ Then you can tell what the mistakes are and you can fix it.

That’s the problem with using it for general search: it’s going to be wrong and you’re not going to be able to tell that it was wrong. Now, this is all still very early and the models are getting better very quickly. Say the error rate is 90 percent. Say it will go to 1 percent. There is always that question of at what point is it good enough.

The other side is to what extent is this a product question rather than a science question. Because, after all, Google doesn’t just give you one answer. It gives you 10 answers and says, ‘I don’t know. It’s probably one of these.’ Whereas ChatGPT is saying, ‘This is the answer.’ So it could be that there are ways of presenting this from a product side to communicate the uncertainty.

MB: You suggested generative AI is a much bigger deal than NFTs and crypto. While I definitely wouldn’t call you a crypto booster, I also don’t get the impression you think it’s all a scam. What are some of the useful and valuable features that might have a viable future, assuming you think there are any?

BE: Crypto is a very low-level technology that would enable a whole range of different applications in about five years’ time, once an awful lot of intermediate infrastructure has been built. But at the moment it’s like looking at the internet without web browsers. There are a lot of intermediate layers between what we have now and what an actual useful application might look like. At the point that you are actually able to build and scale applications, well if you were to build ‘Instagram on a blockchain,’ then it would work differently in a bunch of interesting and important and potentially useful ways. We’re not really able to do that yet.

MB: One talked-about use of NFTs would be to enable users to prove ownership of a digital asset and be able to bring it with them across different virtual spaces. You could buy a digital item and use it in different gaming environments, which could be important for digital fashion. Do you think this sort of interoperability is achievable?

BE: I don’t think this is really a technical problem. I think this is a product problem. To put it very simply, if I go into a flight simulator and I buy an F14, and then I close that game and I open Fortnite, what am I supposed to do with an F14 in Fortnite? If I buy a costume in Fortnite and then I close Fortnite and I open FIFA, what happens with that? The degree to which assets have meaning between different games is not necessarily very strong. So I’m sort of hesitant about this idea that somehow all the assets will move between all the different games. Technically it’s not very difficult. It’s just from a business point of view and a product point of view, I’m slightly perplexed as to what that would mean and in what context that would actually make sense.

MB: You also implied the idea of the metaverse is overblown. Do you think we’ll ever have a metaverse that looks the way people like Meta founder Mark Zuckerberg imagine it?

BE: My conceptual problem here really is with the term ‘the.’ The idea that there’s sort of one thing that all works in one centralised, unified way. To give context to this, if you go back to the early ‘90s, there’s a moment when people realise that these PCs are a big deal and lots of people are going to have a PC and maybe they’re going to be connected to networks. What would that mean? And so you get a whiteboard and you write all sorts of stuff on the whiteboard, ideas like multimedia and interactivity and video and graphical user interfaces and convergence. You draw a box around this on the whiteboard and you call it the information superhighway. Who’s going to build this? Well, Disney and The New York Times Company and AT&T and Bertelsmann and Viacom. Here we are 25 years later and we are doing all of that stuff, but it’s not the information superhighway and it’s not those companies and it’s not one unified system.

People have these conversations, ‘Well, in the metaverse it’ll work like this.’ Number one, you can’t possibly know the structure of the output of thousands of companies trying to work out what to build and consumers working out what to use in 10 years’ time. It’s like sitting down in the year 2000 and describing how the mobile internet was going to work.

MB: I noticed you keep an eye on Shein, which is unusual for a tech analyst. How much of its success do you think is a result of data prowess versus having this fairly unique supply chain set up that no company outside China can really copy? How much of a technology company is Shein really from your point of view?

BE: I tend to draw a line from Shein to Netflix and say, ‘What are the questions that matter for Shein?’ They’re really all apparel questions. What are the questions that matter for Netflix? They’re basically all TV questions. There are no technology questions here.

I look at it because I think it’s interesting to see this company using these models to shift fast fashion, using the internet as a new channel and a new route to market in not very different ways to the way that Netflix does.

MB: As somebody who watches a broad swath of the tech industry, are there any other emerging technologies that you’re excited about that it’s worth fashion and retail keeping an eye on?

BE: I think part of what’s going on at the intersection of tech and everything else is that most of what’s being deployed is ideas from 10 and 20 years ago. The tech industry is obsessed by what’s going to happen in 10 years’ time or five years’ time. But meanwhile, most of what’s actually getting built is ideas from 10, 20 years ago — ideas like maybe people will buy stuff on the internet. We are simply working out how to deploy ideas of basically 10 and 20 years ago to new sectors in new ways.