Table of Contents
- Introduction
- I. What Is AI Engineering?
- II. Keeping a Book Current in a Field That Moves by the Week
- III. From Data → Model → Product, to Product → Data → Model
- IV. First Steps: Prompting, Context, and RAG
- V. The BM25 Test — Why Boring Retrieval Still Wins
- VI. Fine-Tuning: The Last Resort, Not the First
- VII. Rolling Out AI Thoughtfully — The Customer Support Case
- VIII. Staying Focused on the Problem, Not the News
- IX. Why Evaluation Gets Harder as Models Get Smarter
- X. Functional Correctness, LLM-as-Judge, and Comparative Evaluation
- XI. Measuring What Users Actually Want
- XII. Common Mistakes Teams Make With GenAI
- XIII. Systematic Thinking Never Goes Out of Style
- XIV. How to Learn AI Engineering
- XV. Why AI Is Not the End of Software Engineering
- XVI. Education, Entertainment, and the Enterprise
- XVII. Rapid Fire Round
Introduction
Chip Huyen is a computer scientist and writer, author of the book AI Engineering — currently the most-read title on the O'Reilly platform. She previously worked as a researcher at Netflix, was a core developer of NeMo (NVIDIA's GenAI framework), worked as an ML engineer at Snorkel AI, and founded and sold an AI startup called Claypot AI. She has taught machine learning systems design at Stanford, and her current book is her second on ML and AI engineering. In this conversation, host Gergely Orosz and Chip cover what AI engineering is and why it feels so much more full-stack than machine learning engineering did, what the typical steps are for building an AI application — from choosing a model, through using RAG, all the way to fine-tuning — and what practical ways exist for software engineers to get started building AI applications.
GERGELY OROSZ: Chip, welcome to the podcast.
CHIP HUYEN: Hey, hi, I'm Chip. I'm very excited to be here.
GERGELY: I've been following your newsletter for a while, so I was really looking forward to this chat. First of all, I really want to congratulate you on this book. I've started reading it — I haven't read the whole thing, I've gone through some chapters and gone deeper into others. What I found is, when I looked at the table of contents, I thought, well, this looks good in terms of breadth — it goes from how do you understand foundation models, how do you evaluate them, what is prompt engineering, and into things like fine-tuning, RAG, dataset engineering. But then within each of those sections it starts to go deep — there's an introduction, but then it goes deeper. For example, for the evaluation methodology chapter — just looking at the table of contents — I started reading it thinking, well, we know it's important to evaluate AI models, we know it's harder to do, but then you go into things like LLM-as-judge, or ranking models with comparative evaluation, and the challenges of it. That's where, in some parts, I had to slow down and look things up. So it does go really deep into a lot of these sections, which I found pretty refreshing — it's got a mix of breadth but also depth. This is definitely not a fast read for me, but it's one of those things I'll keep coming back to.
CHIP: Thank you. It was not a fast write either — it took quite a while, and a lot of references. I think I cited over a thousand references in the book, so I probably should have read even more papers. I went through a lot of GitHub repos — I have a tracking sheet of thousands of repos, at least 800 stars now — so I went through a lot of those codebases, and a lot of blog posts, and other books, some from the seventies, eighties, nineties. I also published about a hundred reference links I found really useful while writing the book, so if you just want to look at those references on their own, they're on my GitHub.
GERGELY: And I was surprised by some of the really original research — it's not just that you wrote about these papers, or here's what I read, but — as you mentioned, the thousand repos — you actually have a whole section about how the number of GitHub repositories changed over time: the ones about infrastructure, the ones at the application level, and so on. You have more than 900 repos mapped out. I've never seen anything like that, and clearly that took your own slicing and dicing and your own research.
CHIP: Yeah, I feel like I do a lot of manual labor. I think I get a lot of value out of doing things the non-optimal way. I feel like a lot of people focus on "what's the quickest way to do it, what's the fastest way to do it" — but if you're willing to put in the effort into things that a lot of people aren't willing to, I think you can get some kind of insight that other people don't get.
I. What Is AI Engineering?
GERGELY: One thing that's a little interesting about this book is that it's about AI engineering, and this field moves so quickly — just in a week we've had a new model come out, for example DeepSeek, that people have been talking about for a few weeks now. How did you write this book? How were you able to write a book about such a fast-moving industry, so that by the time it's released — which was clearly a few months after you finished it — it would still be relevant?
CHIP: That's a great question. When I started writing the book, I was thinking the same thing — I thought it wasn't the right time to write it, because there are so many things still changing. But then, when ChatGPT really came out, like a lot of people, I had this existential crisis. I was in a group chat and everyone was like, "oh no, what does this mean for us as engineers." I feel like there are two things I usually identify with — being an engineer, and being a writer. And guess what use cases AI is really good at — writing code, and writing. So I was like, oh shoot, what does this mean for me?
So I started interviewing a lot of people, I started reading a lot, I talked to a ton of people, and I made a lot of notes. And in that process, what I realized is that a lot of things that seem new — a lot of the fundamentals have actually been around for a while. Language modeling isn't a new task — Claude Shannon introduced that back in the 1950s. We talk about RAG all the time, but RAG isn't new either — it's based on retrieval-augmented generation, and retrieval is a very old technology, already powering a lot of use cases across the internet, like search or recommendation systems. Vector databases have been around for a while, and vector search already has so many cool algorithms.
So I thought, okay — first, a lot of things aren't actually new. And second, I tried to focus on asking: is this due to a fundamental limitation of AI, or is it just due to the current, temporary capabilities of AI? And if it's due to current capabilities, how fast is that capability changing? In the early days, for example, a lot of people shared prompt tips — like, try bribing the model, "hey, if you answer this correctly I'll give you $200" — or people talked about prompt robustness, how sensitive a model is to small changes in a prompt. I was reading about this and I felt like models were actually getting more and more robust to prompts — from GPT-3 to GPT-3.5, it was already so much more robust, meaning small changes to a prompt produced a lot less variation in performance. So I felt this kind of thing wasn't going to stick around — prompt tips weren't going to matter for very long. That was already happening at the height of people saying there might be a job called "prompt engineer." You could already see it trending down.
So I think writing is a bit like making a bet — when you write about a topic, you're betting on whether it's going to stay relevant. I tried to look at the trajectory of progress and see what would still be true in a year or two. Another example is context length — people wanted longer context, but then I kept seeing it move really fast, from something like 8,000 tokens to 128,000 in a few months. Super fast. So I thought, maybe the question is less about context length and more about context efficiency — can a model actually use that context well? Those were the kinds of bets I made, and certain things happened during the writing process that made me more confident in them. Multimodality is another one — when I wrote about multimodal models back in 2023, people told me I was too early, that everyone was still working on language, we're not there yet. But it just felt inevitable to me — we've learned how to work with language, but I wanted to do a lot more than just language. And now multimodality is everywhere — almost every model is multimodal now.
II. Keeping a Book Current in a Field That Moves by the Week
GERGELY: So the title of the book is AI Engineering, and we now have this term "AI engineer" spreading like wildfire. How would you define AI engineer, or AI engineering? Because I feel like it's become a bit of a loaded term these days.
CHIP: It is. I feel like a lot of terms nowadays are loaded — you're not allowed to say certain things anymore, you're not allowed to use certain words anymore. When I was agonizing over the title for the book, I knew we needed a different term from "machine learning engineering." The reason is: with foundation models, a lot of the fundamentals, the systematic approaches, are still the same as in machine learning engineering, but there are also a lot of new things.
One thing is: before, if you wanted to build a machine learning application, you had to build your own model — that meant you needed your own data, and you needed the expertise to train and babysit a model. Nowadays, if you want to build an application leveraging machine learning or AI, you can just send a direct API call and get access to this wonderful capability. That really lowers the barrier to entry — you don't need data anymore, you don't need a fancy AI degree anymore.
A second thing is: before, you needed distribution, because you deployed a machine learning application as part of an existing application. If you built a recommender system, you needed an e-commerce website to attach it to. Fraud detection was deployed as part of, say, a banking or payments app. But now you can put it out as a standalone application — you don't need an existing distribution channel, though having one is still really useful.
Another very big thing is that it's a shift from less machine learning and more engineering and product. Before, if you were a good machine learning engineer, you started from data — you had to gather data, maybe get human annotations, then you trained a model, and once the model was good, you deployed it into your product. Nowadays, you actually start with a demo — you have a cool idea, you say, "let's just try it out and see if it works." You start with a product, and once it's working pretty well, you say, "I want to make it better," so then you start gathering more data — maybe more prompt examples, or, in very rare cases (I don't recommend most people do this early on), fine-tuning. Then maybe you're sending a lot of API calls to OpenAI, Anthropic, Google, and you decide it's getting too expensive, so you start hosting your own model, using an open-source alternative, or fine-tuning one.
So before, machine learning engineering went from data, to model, to product. Now, with AI engineering, it goes from product, to data, to model. That places a lot more focus on product and data, which is where the comparative advantage lies once everyone shares similar underlying AI capabilities. I did think we needed a different term to separate it from machine learning engineering, but I didn't know which term to use — so I thought, let's just ask people. I surveyed a bunch of people I thought were doing this — building applications on top of foundation models — and almost everyone said "AI engineering." So I thought, okay, if that's what people are already calling it, let's go with that.
GERGELY: So do I understand correctly that the biggest difference is: machine learning engineers did a lot more groundwork — getting the data and building the model — whereas with AI engineering you have a lot of that available, at least initially, through APIs, so there's more of an engineering flavor, you hack things together, and then over time, as things get more serious or your product gets bigger, you do more of that — you might build your own model, host it, maybe even build your own one day, but it comes a lot later. A lot of the machine learning engineering work only comes down that path if the product is big enough and it works, whereas with AI engineering it's the other way around — you put in the effort up front and then see if it even works.
CHIP: First, I feel like every company has a different definition of the role — even within the same company, people with the same title can do very different things, so it's never really a clear-cut definition. Second, I don't think the real question is "machine [learning] or engineering." In the vast majority of GenAI systems I've seen, there's still a very strong traditional machine learning or classifier component. Imagine you're building a customer support chatbot — whenever I ask about this at a conference, I see a lot of people raise their hands, it's such a classic GenAI application. You get a request from a customer, and maybe you have several potential solutions for it — if it's an easy query, you send it to a cheap model; if it's harder, you send it to a more expensive one. But something very sensitive, like "hey, why did you charge me twice for the bill last month," you probably want to send to a human operator. So you'd have a router, or an intent classifier, to decide what goes where — that's a traditional, classical machine learning model you can build yourself. Or, after you get a response from an AI model, you might check whether it contains PII, because you don't want to send users a response with private information — PII detection can be a classifier, toxicity detection can be a classifier. Retrieval in RAG systems is also, I think, in the realm of classical machine learning you can build yourself.
III. From Data → Model → Product, to Product → Data → Model
GERGELY: What are the most common techniques used when building AI applications — things a software engineer going into AI engineering should know about? The assumption here is you've tried a lot of other solutions, and now you think GenAI is a solution for you — what should you do first?
CHIP: I think those techniques are useful, and a common pattern I've seen is a certain developmental path. Initially, the first thing I'd say is to try to understand what a good response is and what a bad response is — you want a mental model of that, and it's not always intuitive. LinkedIn has a great example here — they built a candidate job-fit assessment, and they found that the majority of their time went into just understanding what candidates actually needed from the model. Initially they focused on correctness, but they realized candidates didn't find that helpful — if a candidate asks, "am I a good fit for this job," and the AI responds, "you're a terrible fit," the candidate is left thinking, "okay, what am I supposed to do with this information?" What candidates actually wanted was more understanding of the gaps, and how they could fill those gaps, or suggestions for other roles that might be a better fit right now. So once you have that picture, you build a guideline — given this kind of response, answer like this, be helpful, show them the gaps — a very clear guideline for the model, in the prompt. You try those prompts, look at the output, add more examples, and keep going until you get really good responses. Then you evaluate — maybe create a set of queries and expected responses, and use both automated metrics, like LLM-as-judge, and human evaluation, to measure progress.
IV. First Steps: Prompting, Context, and RAG
Once you've done prompting and added more examples, you might want to make the system more complex — give the model more context so it can answer better. So when a user asks a question, you might have the model pull up related documents, or job listings, or information about the company, or a candidate's résumé. You build a system that augments the context with documents — that's the RAG pattern. I do think RAG is a very powerful pattern, and there's nothing really fancy about it.
What's interesting is that a lot of people equate RAG with vector search. When people say "I want to use RAG," the very next thing they ask is "which vector database should I use" — people jump straight to vector search.
GERGELY: Yeah, because you have the chunks, and the embeddings can be stored as vectors, so as engineers we're like, "I need a vector search database."
CHIP: People love databases. But I don't think the first solution should be jumping straight to embedding-based retrieval, because you need to build an embedding model, and the quality is highly dependent on the quality of the embeddings — bad embeddings, bad retrieval. Vector databases can also be quite expensive to run, and there's latency to think about. And embeddings can obscure certain keywords — if I'm searching for a specific error code, an embedding search doesn't necessarily surface the exact code. So there are real challenges with vector databases and vector search. The usual, common approach is to start with something as simple as keyword retrieval — extract the keywords from the user's query and find documents that match. Then maybe the documents are too long to fit into context, and that's when you start chunking — figuring out how to break documents up so they fit. And chunking creates its own problems — maybe a document is about "Company X," but says early on, "from now on, Company X is referred to as the Company," so the rest of the document never says "X" again, and if you search for X, you miss the chunks below that point. So you might want to extract keywords, add metadata to every chunk, add the document title, or add a summary — some people do that. Anthropic has a very good article called "contextual retrieval," where they have Claude generate key contextual metadata for each chunk and prepend it, so you retrieve the right chunk. Data preparation gives a really huge performance boost — I've seen it give a much bigger boost than agonizing over which vector database to use. It's not that vector databases aren't useful, it's that in the beginning you want to try the simplest thing with the biggest performance gain, and move up in complexity from there.
V. The BM25 Test — Why Boring Retrieval Still Wins
CHIP: For a lot of retrieval, someone told me something a little hot-takey but very interesting — he said, "I'm not going to take any retrieval system seriously if they don't benchmark against BM25." BM25 is a pretty old-school retrieval method, over twenty years old now — term-based retrieval, not embedding-based — and it's really, really hard to beat. A lot of the time, when you want more complexity, you combine term-based retrieval with a vector-database solution, so you get the semantic side from the embeddings, and the exact keyword match from the term-based side. Hybrid search is very, very common.
VI. Fine-Tuning: The Last Resort, Not the First
GERGELY: So we've talked about prompt engineering, adding more examples, RAG — I guess after you've maxed out on a lot of those things, which usually takes a while, people might consider fine-tuning.
CHIP: I think there's usually a lot of reservations against fine-tuning, because it brings a whole host of new problems you need to deal with. First, once you've fine-tuned a model, you need to think about how to host it — a lot of these models are big, with a lot of parameters.
GERGELY: I actually read this part in the book — you go into detail on the problems with memory size, and you cover alternatives that need less memory but bring their own trade-offs. So it's trade-offs within trade-offs within trade-offs — you solve one problem and get a bunch of others, and you have to decide whether it's worth your time, effort, and resources.
CHIP: Exactly. Also, once you fine-tune a model, you own that fine-tuned model, and the question becomes how do you maintain it. We have this whole world of very smart people releasing new models constantly, with capabilities increasing rapidly, so the question is: how long can your fine-tuned model keep outperforming the new models coming out? You might spend a lot of energy fine-tuning a model, and then some random company you've never heard of releases something extremely fast and extremely good. So it's quite challenging. Fine-tuning is the last resort, not the first line of defense.
GERGELY: What I've heard, if I've got it right, is: take a structured approach — start with prompting, start simple, get to responses that make sense, then add more data. You can do this with RAG, with chunking, keyword extraction, data preparation, which makes a bigger difference than a lot of people think — and then move on to more advanced things. There's a whole host of things you could do, but my understanding is you'll get there over time. Initially the basics will keep you busy, and you'll probably be able to build a pretty good system just with the basics and a bit of engineering — and, most importantly, understanding the problem you're trying to solve, rather than building whatever the shiniest new technology happens to be.
CHIP: The approach can be a bit different for an individual developer versus an enterprise. One thing I've seen, especially early in a technology's life, is that enabling new use cases actually brings more return than incrementally improving existing ones. So instead of pouring energy into squeezing out a little more performance with something fancier and more complex, maybe use the same stack you already have and open up new applications with it. That's why I think a lot of companies will take a while to get to the fine-tuning phase.
VII. Rolling Out AI Thoughtfully — The Customer Support Case
GERGELY: Let's say at my company we decide to build an AI solution — take the example of customer service automation. What are typical approaches I should know about? You cover some of this in the book too.
CHIP: For customer support, I'd say the first thing to look into is what the bottlenecks in your current solution actually are. For example, I worked with a setup that had a lot of customer support requests and didn't know how to answer them all. Their solution was interesting — they tried to drive a lot of the questions into a common channel, like a public Discord, so other users could help answer them, and in the future, if someone had a similar question, you could just point them to the previous discussion. Another solution that was popular around 2018–2019 was routing requests to the right department. The bottleneck there was in triaging — you get a request and don't know which department to send it to. A bunch of startups tried to build systems to predict, is this a finance query, is this a technical-support query — smart routing alone already reduced a lot of the friction.
If you decide you do need GenAI, I really recommend the framework Microsoft introduced, called "crawl, walk, run" — going from a lower-stakes to a higher-stakes deployment. It doesn't have to be a support jackpot from day one — initially you might have a human in the loop. For every request, instead of a human agent writing the response from scratch, the AI suggests a few options, and the human picks one or uses it as a starting point and makes a quick edit before sending it. Once you see the acceptance rate getting really high — say 90 percent for a category of queries — you feel more confident rolling it out further, maybe to more users, or maybe internally first. You give it more autonomy but keep the scope of deployment small, and once you're really happy with it, you roll it out to more users.
GERGELY: That's nice, because what I was expecting you to say is, "oh, you just build this AI framework, deploy it, and see" — which is what I feel a lot of companies are doing. A lot of teams say, "oh, GenAI, let's grab a model from ChatGPT or Anthropic, put it in, and put it out there." But I really like that what you're describing isn't really specific to GenAI — you look at the business problem, look at the options, which include traditional machine learning like classifiers, not just GenAI, and then you check whether these tools actually solve your problem before you roll it out, instead of blindly rolling it out. All of this sounds like it's not really new — you could have said the same thing two or three years ago, before GenAI, except we wouldn't have had these GenAI tools to play with.
CHIP: Yeah — actually, before our chat, I looked back at one of my old talks, about things that haven't changed, things that feel very similar across eras of engineering. Dealing with a new technology is one of the things that never changes. Every time a new technology comes out, I can hear the collective sigh of senior engineers everywhere saying, "not everything is a nail" — people just try to make the new technology work for everything.
I do think a very common mistake is jumping straight into GenAI when you don't need GenAI. There are two different headlines here: one is "I use AI," and the other is "I solved the problem." If you're focused on the first headline, sure, you use AI — but if you want to solve the problem, you need to understand what the problem actually is, what the roadblocks are, and remove them using the simplest solution, not the fanciest one.
GERGELY: I feel like there's a bit of a really strong fear of missing out across most tech companies — everyone knows this is such a transformative technology, it gives so many new capabilities that it's going to be important, everyone knows their company will be using it, but now there's a fear of missing out: what if my team doesn't build it, what if someone else gets ahead of me? So a lot of teams are building it, using a hammer looking for nails, even if they don't need it yet. I'm not sure that's necessarily a bad thing, because people at least get experience with it — but it's a very interesting time, because usually when a new backend framework comes out, or something domain-limited, some people jump on it, but this is the first time I've seen the whole industry jump on something at once, everyone trying to use it whether it works or not.
VIII. Staying Focused on the Problem, Not the News
CHIP: I definitely agree with you on the FOMO. I do think everyone jumping on it is actually a pretty good thing — the energy is incredible, I've never seen so many smart people focused on the same problem before, and the progress is amazing. But I do think there's an irony to it — the more we try not to miss out on things, the more things we actually miss, because if we try to keep up with the news, jumping from one piece of news to the next, we always stay at the surface level and never go deep into anything. I actually don't read the news much — I find it a bit distracting. My approach is: pick a problem you care about, and only pay attention to things that help you solve that problem. If some news comes out, I ask, does this help me solve this problem? If it doesn't, it can wait — because if something is actually important, it'll still be important two weeks from now, a month from now. I don't drop everything to go understand it right away. I try to stay a bit calmer about it.
IX. Why Evaluation Gets Harder as Models Get Smarter
GERGELY: When you're building an AI system, one of the things you'll come across is that you need to evaluate the output — how well does it work, does it solve your problem. Why is it difficult to evaluate AI systems, and what are common ways to do that?
CHIP: Evaluation, I think, is a billion-dollar question — a trillion-dollar one, given how much is being invested now. You need to go big, and if you go big, you'd better go really big. I think it's challenging because the smarter AI becomes, the harder it is for humans to evaluate it. Before, if an AI was incoherent, you could easily tell a response was bad — it just didn't sound good. But nowadays it's pretty coherent — if you ask ChatGPT to summarize a book, and the summary sounds convincing, you actually don't know whether it's a good summary — you might have to read the whole book yourself just to evaluate it. It's the same with math. I personally use AI to ask a lot of questions I don't know the answer to — and because I don't know the answer, I don't know if the answer it gives is correct. A lot of people can tell whether a first-grade math solution is correct, but very few people can tell whether a proof of some fancy equation is correct. I remember when o1 came out, Terence Tao — one of the best mathematicians of our time — took the time to evaluate it, and he said the experience of using o1 was similar to advising a mediocre, not-completely-incompetent graduate student. That makes me think — if we really need the brightest minds today to evaluate AI, we're soon going to run out of really smart people to evaluate it with.
So what would be the next step forward? For a long time, we used humans as the gold standard for AI performance — humans would write out how you should respond, and the AI would try to copy the human. But now, for a lot of tasks, AI performs way better than humans. I thought about several approaches to deal with this, which is actually why I split this into two chapters in the book — I originally had one chapter on evaluation, but the more I wrote, the more I realized there was just so much, so it became two fairly long chapters: the first on general methodology, the second on the specific techniques for evaluating an AI system.
X. Functional Correctness, LLM-as-Judge, and Comparative Evaluation
CHIP: One methodology is functional correctness — you evaluate an application's output based on how well it performs the task. If you say, "use AI to save energy," you can see how much energy it actually saves. If you say, "use AI to play this video game," you can see the score. A very common use case for this is coding — I don't think it's a coincidence that coding is one of the most popular use cases, because we actually know how to evaluate generated code. We might not know how to evaluate a generated essay, but we know how to evaluate code, because we've been testing code for a long time. With code you can use functional correctness — first, does it compile, does it run; second, does it produce the expected output.
The second approach is using AI to evaluate other AI. We've been using AI to automate a lot of applications, so can we also use it to automate evaluation? It's actually doing pretty well. Even back in 2023, the LangChain "State of AI" report found that the majority of applications they surveyed already had some sort of LLM-as-judge, and I think that's only grown. It's getting pretty cost-effective and useful, though of course there are a lot of challenges around it too.
Another approach I found very interesting is comparative evaluation. As humans, it can be hard to give an absolute score to something, but if you give us two versions of something, we can usually tell which is better. We've seen studies showing that even for tasks where AI is performing at a level human experts can't reach, humans can still detect the differences between two outputs. This has been guiding not just evaluation, but also model development.
GERGELY: Sounds like there's no simple answer — you need to go through all these options and figure out, for your case, what makes sense for cost, for what you can do, whether you can put a human in the loop. There's no real silver bullet.
CHIP: I don't think there's a simple solution. That's one reason I'm a little skeptical about evaluation tooling — a lot of the challenge with evaluation isn't that we don't know how to evaluate, it's that it requires discipline and hard work, and a lot of that just can't be automated by tools.
XI. Measuring What Users Actually Want
CHIP: One thing about evaluation is that we need to evaluate an application based on what users actually want — and we usually don't. That means going and talking to users, looking at their interactions, because a lot of what we think we should measure isn't what actually matters. We have to measure what matters. I have several examples of how counterintuitive this can be — you think you're measuring one thing, but users actually care about something else.
I have a friend building a pretty big application, basically a meeting-summarization tool. Initially they tried to measure correctness — does the summary cover the content of the meeting — and they agonized over format, whether users wanted a three-sentence summary or a five-sentence one, because they assumed users wanted shorter summaries. But eventually they found out that users don't really care about the whole content of the meeting — people only want to know, what's the action item for me, what do I have to do after this. So that changed things: correctness still matters, they still don't want the summary to make things up, but the real focus became not missing action items specific to the person asking.
Another example — a pretty big tax firm built a chatbot to help people with tax preparation, and the response was very lukewarm. They were measuring usage, and people just weren't using it much, and they wondered why — was it hallucinating, was there some other issue — they tried to measure their own kind of metrics. But in the end, they found people simply weren't using it because they hate typing. And also, if you're facing a domain you don't know much about, you don't even know what questions to ask — people use tax software precisely because they don't know a lot about tax. So they didn't know what to type, and they didn't understand the domain — they'd gone to the product because they wanted someone else to take care of their taxes. What the team started doing was trying to understand what kind of questions people would ask, and suggesting those up front — guiding users with, "here's a question you should ask," and then the answer, and it keeps going from there.
A lot of this comes down to understanding your problem domain — go talk to users, look at the data. I still think looking at data is very, very important. Greg Brockman has a great quote about this: manual data inspection is one of the activities with the highest ratio of value to prestige. People don't think highly of manually inspecting data — "let's give it to an intern," "let me think of something fancier, some algorithm" — but it's actually extremely high value, because by looking at the data you detect patterns and understand how users actually use the product. I usually highly recommend to teams: don't forget human evaluation. You use LLM-as-judge, but the judge has real challenges — its quality depends on the underlying model and the prompt, and it's fairly non-deterministic, so things can change over time. But if you also have human evaluation — very consistent, very clear guidelines, every day going and looking at, say, 50 samples of actual interactions, or more if you have the resources, 500 or a thousand — you get a real picture of how your users are using your product, any changes in behavior tied to current events, or you can correlate it with your automated metrics. If the AI judge score starts diverging from the human judge score, that's something you need to investigate.
XII. Common Mistakes Teams Make With GenAI
GERGELY: So, as you said, you can't really skip the hard work if you want good results, and you can't fully pull humans out of the loop, at least initially. What are some common mistakes you've seen when teams are building AI applications?
CHIP: I don't want to say it in a way that sounds like everyone's an idiot, but — we've touched on several already. One very common mistake is using GenAI when you don't need GenAI. A startup once pitched me on using GenAI to help people optimize their electricity usage — you'd tell the chatbot about your household's energy-intensive activities, charging a car, doing laundry, and it would tell you the best time to do each one to minimize your bill. They said their research showed you could save 30 percent on average — "it's free money, why wouldn't anyone want that." I asked what the savings would be compared to just manually scheduling the most energy-intensive activity during off-peak hours — just charge the car at 10pm. They said, "we haven't done that comparison yet, but we'll try it and let you know" — and they never got back to me, and later abandoned the idea. A lot of these optimization problems can be solved greedily, without GenAI at all.
On the other end of the spectrum, I see a lot of companies giving up on GenAI because they tried it and it didn't work, when in fact — I'm often surprised to find — another company with a very similar use case had it working really well. When you look into it, it's usually a bad product: they didn't prompt well, didn't understand their users, or didn't even know how to evaluate it properly. For example, I worked with a company doing résumé information extraction — someone submits a résumé (a PDF, not raw text), and the system extracts the raw text, then extracts the organizations the person worked for. I asked how it was working, and they said terribly — they got the organization wrong about 50 percent of the time. I asked where in the pipeline it was failing: the PDF-to-text step, or the text-to-organization step? They said, "we don't know, we didn't check that." If you can't localize where something fails, you can't fix it. It sounds like common sense, but this keeps coming up.
Another mistake is overcomplicating things — jumping straight to fancier vector databases, or fine-tuning. A common one nowadays is: you see a fancy agent framework and think, "let's use this, let's try it." Frameworks are really cool, and I'm genuinely grateful for a lot of them — they make my life easier — but a framework should encode best practices and be heavily tested, and I think we're still in the phase of learning what the best practices even are. A lot of frameworks can introduce unnecessary, painful bugs. When I was going through the codebases of several popular frameworks, I noticed something interesting — a lot of them ship default prompts to help you get started quickly. But every single one of those default prompts had typos, and they kept changing — someone would submit a quick PR to fix a typo that never made it into a proper release. So if you're using one of those default prompts and your application's performance suddenly changes, you don't know why, because the prompt changed underneath you.
GERGELY: It's interesting, because what you're describing — using this technology when you don't need it, giving up on it for reasons a bit of common sense would have fixed, using a new framework that isn't really high quality or doesn't really encode best practices — all of this sounds like stuff you could swap "GenAI" out for any new technology or stack and hear similar stories. It's not really new, is it — whoever tells you they're the expert still has, at most, a year or two of experience with it, it's still changing constantly, and there's no established best practice yet.
XIII. Systematic Thinking Never Goes Out of Style
CHIP: I definitely agree. Even though technologies change over time, systematic thinking — systematic approaches to problems — usually doesn't. If you want to solve a problem, you start by breaking it down, seeing where the challenges actually are, and going through different solutions. It sounds obvious, but a lot of the time FOMO gets in the way — you know it's the right thing to do, but you feel like you need to check this other thing out first, and you keep doing that three times a day, and the day is gone, and you never really get the time to sit down and think deeply about what you're actually trying to do.
GERGELY: So I guess we're going to see a lot of the same mistakes that happen with any new technology, and if some listeners have adopted new technologies before, they can probably reuse those approaches, localized for GenAI, and avoid some of these.
XIV. How to Learn AI Engineering
GERGELY: Speaking of picking up new technology — as a software engineer who wants to get into AI engineering, what would your recommendation be for how to learn? Things change so fast — you mentioned the importance of fundamentals — what would you focus on?
CHIP: I have a lot of thoughts on learning, because I like learning a lot, and over time I've tried to observe patterns — though the way I learn might not be the way you learn, people have different learning styles. In general, I think of learning as having two approaches: project-based and structured. Project-based learning is: you choose a project, work on it, try to solve every problem in it, and finish it. Structured learning is more like taking a course or reading a book — someone else has laid out what you should learn and in what order.
There's quite a bit of debate about this. Somebody told me recently that a good friend of theirs thinks the problem nowadays with people who want to become engineers is that they spend too much time learning and not enough time doing — forget the courses, forget the books, just pick a project and work on it. I do think project-based learning is very valuable. But if you think of it as, here's a set of skills and knowledge I want to build, project-based learning can hit a lot of those points, but not always all of them, and it can sometimes leave you confused.
Another thing about project-based learning is that a lot of people follow tutorials — someone else has already walked through how to do this. Tutorials are really cool, and I do them a lot myself, but I've noticed it's very easy to just mindlessly click from one cell to the next, run cell after cell, and never stop to ask why it's being done this way, why this library matters, why the code is written this way, why the batch size is 16 instead of 64. There's no built-in mechanism to force you to stop — you just want to run to the end and see the output, then make changes by best guess.
Something funny happened when I was doing market research on an open-source project I'd worked on, called Ibis — I knew that if you wanted to use it, you had to write "import ibis." So I searched GitHub for repos with "import ibis" in them, and found a lot of repos where that import was present but Ibis wasn't actually used anywhere else in the codebase. I realized a lot of those repos had copied from a tutorial that used "import ibis" — probably the original developer had that import, deleted the code that used it, but forgot to delete the import line, and then everyone else copied the same thing. That's something a little dangerous about tutorial-based learning — it's great, but it's important to be able to stop and ask questions, and sometimes structured learning helps you know which questions to ask.
So before starting, I'd recommend a mixture: choose a project — it doesn't have to be big or fancy, just pick one — and complement it with structured learning: a book, a course with friends, reading papers. Reading papers is its own skill, it can be time-consuming, and you need to know what you want to get out of it. But start a project, and complement it with structure.
There's also an exercise I found very useful, at least for me, when I was starting out: for a week, observe and note down everything you do, and think about what percentage of it could be automated by AI. Then try to use AI to do those things. It gave me a lot of ideas for use cases — you just think about what matters to you, and if an application can solve even one problem for you, that's already great.
GERGELY: That's an unconventional way to look at it, but a good one — it can help you get ahead of the dread of "what would AI do to my job," because you actually see what happens when you automate things. Which leads to my next question — there's a lot of fearmongering around AI meaning the end of software engineering, since it's very good at coding, better than in a lot of other areas. What's your take — as AI gets better, will it actually end software engineering, change it, or not change it that much?
XV. Why AI Is Not the End of Software Engineering
CHIP: I think it goes back to the question of what software engineering actually is. An analogy might help — writing. We tend to confuse the most salient activity of something with the thing itself. For writing, in the past, writing meant the physical act of putting words onto paper — people took pride in calligraphy, "oh, you have beautiful handwriting, you must be smart." Then we had computers, and now writing doesn't refer to that physical act anymore — it refers to the process of arranging ideas into a readable format. I think the same is true for coding. People think of engineering as the physical act of putting code into an editor, but that's not what software engineering actually is. Software engineering is about solving problems — here's a problem, how do I come up with an executable program to solve it. Coding itself is just the physical act. So yes, AI may automate coding, but I don't think it's going to fully automate problem-solving, because you still need to know what the problem is, and only you can understand what problem you're actually facing.
Software engineering is also about doing things very precisely. The reason the job of software engineer exists is that it's very hard to be precise, to speak the computer's language — move an if-statement, change a variable, and suddenly the program crashes with a stack overflow, which of course you understand if you're a software engineer. But if you're a business user who just says, "I want the button to move over when you resize the window," it's easy to say, but as the engineer you know the edge cases, the environment, the system events you need to account for, and you write code for all of it. I'm sure we'll get to a point where AI can generate a lot of that, but there will still, at some point, be a need for someone who understands the code and can find the gap — because English isn't as precise a language as a programming language. Programming languages were invented to be precise and unambiguous — there's essentially a one-to-one mapping from assembly to a higher-level language, but going from English to a programming language is very fuzzy.
I think that profession isn't going to go away. For casual users, it might work fine for hobbyist use cases — you say something, you get something roughly right, you try a few times and you're happy. But for a business or professional use case, you'll need people who can guarantee you get exactly what you want.
I'm actually really excited about AI automating part of coding, because it frees software engineers to build much more complex software. Going back to the writing analogy — when writing meant manually copying words onto paper, a book of 5,000 or 10,000 words was considered big, because it took so long to copy by hand. Now we have books hundreds of thousands of words long, and that shift made things a lot easier. It's the same with software engineering — if you don't have to hand-write every line, you can turn ideas into executable programs much faster, which enables us to write much more complex software. Maybe one engineer will be able to command, debug, or maintain a much more complex system on their own — right now, a million-line codebase usually needs several engineers, it's rare to have just one person own that, not even counting dependencies. That would be an interesting shift.
XVI. Education, Entertainment, and the Enterprise
GERGELY: What other use cases are you excited about, outside of coding?
CHIP: I'm excited about education. I think AI can help people learn a lot faster. One thing I've realized is that nowadays, if you already know the question, finding the answer is actually pretty easy — you can ask AI and get a good answer, or at least a lot of references to dig into. What's still hard is coming up with the right questions in the first place. So I think education needs to focus on building the habit of asking questions and genuinely understanding, and I think AI can make learning a lot more effective — people can learn more, faster. And if we can learn better and faster, we can actually do more things.
Another use case I'm excited about is entertainment. We tend to think of entertainment and education as separate things, but I don't see why we couldn't have games that help us learn — strategy games teaching negotiation, for example, could be really fun. Or just more intellectually stimulating content generally — movies or shows don't have to be pure escapism; I like content that makes me think a little, that helps me understand more about different fields. AI can help create content that's both entertaining and intellectually stimulating. As a simple example, we already have a lot of medium adaptations — a book becomes a movie, a movie becomes a game, a paper becomes a podcast — and if AI can help adapt content across mediums like that, it could be really exciting.
There are a lot of smaller problems I'm interested in too. I haven't touched much on the enterprise side, but that's where most of the money still is. I do think enterprise and company organizational structures are going to change. Think about what the job of middle management actually is — aggregating information from reports and transmitting it up to executives, and transmitting direction back down to lower layers. Information aggregation is something AI can do really well. I think companies can become a lot more efficient because of it.
XVII. Rapid Fire Round
GERGELY: Let's close with some rapid questions — I'll shoot some questions and you tell me what pops to mind. What programming language did you use most building AI applications, or doing ML engineering, and why?
CHIP: Python and JavaScript.
GERGELY: JavaScript as well? How come?
CHIP: Oh, definitely — a huge part of building products is building demos quickly, and JavaScript is really handy for that. I'm not very good at it — I've always been a little scared of JavaScript — but I'm grateful AI has made it a lot easier to get started with nowadays.
GERGELY: Which is your favorite LLM right now, and why?
CHIP: I don't really have a favorite — I use different ones for different things. I used to use ChatGPT out of habit, because I already had a bunch of prompts set up. I use Claude sometimes for creative writing, because I think it's a bit less clichéd. I've been reading up on R1 — who isn't, these days — just trying it out. I've also used some of the Llama models, like the vision version, for things like screenshot-to-code, just testing it out, having fun with it. But I'm not emotionally attached to any of them.
GERGELY: And what's a neat AI tool you've used and liked?
CHIP: I built something myself that's really helped with research. When I come across a link to a paper, I usually go through the same process — read the abstract, look up the authors, do a bit of digging, ask questions, check the citations — so I built a small tool that just goes and scans all of that for me and hands me the information.
GERGELY: A tool to scratch your own itch.
CHIP: Exactly — I think that's the beauty of AI now, you can build something like that in a very small amount of time. Before, it would have taken me weeks, but now I can just build one. That's something to be excited about.
GERGELY: I agree. And what are one or two books you've read and would recommend?
CHIP: I recommend a lot of books, but recommending books can feel a bit like forcing people to enjoy what you enjoy. I like books that give me a new perspective, or insight into topics I don't know much about. I really like Complex Adaptive Systems — a very interesting book on systems thinking, on how to design social dynamics so people work toward the goals you want. It forces you to think about systems. I like The Selfish Gene, because it makes you think about free will — the idea that you can live on either through your genes or through your ideas: genes live on through offspring and reproduction, but ideas can also replicate, like memes. And I like Antifragile — I think the author is a fascinating character, and I've genuinely enjoyed his books.
GERGELY: Thank you for the recommendations, and thank you for being on the podcast. AI engineering is such a new field, and it was great to hear from someone who's gone so broad and so deep, and who's been in this field even before it was called AI engineering. Thank you for this.
CHIP: Thank you so much for letting me ramble on the show. I really appreciate it. One thing I really enjoy about writing or talking publicly is that I get feedback — sometimes it's "I'm less convinced," sometimes it's "I agree with you," which is great to hear but isn't always great for the ego. But I really like pushback — "you didn't think about this," "you forgot this" — so if there's anything you feel I missed, please let me know. I'd really appreciate it.
Transcript source: "The Pragmatic Engineer" podcast, hosted by Gergely Orosz, published February 5, 2025. Cleaned up from the raw episode transcript, categorized and formatted for readability.
No comments:
Post a Comment