Source URL: https://simonwillison.net/2024/Sep/24/xkcd-1425-turns-ten-years-old-today/
Source: Simon Willison’s Weblog
Title: XKCD 1425 (Tasks) turns ten years old today
Feedly Summary: XKCD 1425 (Tasks) turns ten years old today
One of the all-time great XKCDs. It’s amazing that “check whether the photo is of a bird" has gone from PhD-level to trivially easy to solve (with a vision LLM, or CLIP, or ResNet+ImageNet among others).
The key idea still very much stands though. Understanding the difference between easy and hard challenges in software continues to require an enormous depth of experience.
I’d argue that LLMs have made this even worse.
Understanding what kind of tasks LLMs can and cannot reliably solve remains incredibly difficult and unintuitive. They’re computer systems that are terrible at maths and that can’t reliably lookup facts!
On top of that, the rise of AI-assisted programming tools means more people than ever are beginning to create their own custom software, which I love.
These brand new AI-assisted proto-programmers are having a crash course in this easy-v.s.-hard problem.
I saw someone recently complaining that they couldn’t build a Claude Artifact that could analyze images, even though they knew Claude itself could do that. Understanding why that’s not possible involves understanding how the CSP headers that are used to serve Artifacts prevent the generated code from making its own API calls out to an LLM!
Via @chrisalbon
Tags: clip, ai-assisted-programming, xkcd, ai, llms
AI Summary and Description: Yes
Summary: The text discusses the evolution of AI technologies, particularly in relation to LLMs, and reflects on the nuances of tasks that these systems can or cannot perform. It highlights the challenges faced by new programmers using AI tools and the implications of understanding task difficulty in software development.
Detailed Description: The content analyzes the current landscape of AI-assisted software development and the inherent complexities involved with LLMs. It mentions the following key points:
– **Evolution of Task Complexity**: The text refers to XKCD 1425 (Tasks) as a hallmark representation of the evolution of machine learning capabilities, noting how tasks that were once complex, such as image recognition, have become relatively straightforward due to advancements in AI models like LLMs, CLIP, and ResNet+ImageNet.
– **Misconceptions Around LLMs**: While LLMs have improved task automation, they have also created confusion regarding their capabilities. The author points out that many LLMs struggle with fundamental tasks, particularly mathematics and fact-checking, which illustrates the limitations of these models.
– **AI-Assisted Programming Tools**: There is a growing trend of individuals using AI programming tools to create custom software. However, this emergence leads to new challenges as these “AI-assisted proto-programmers” grapple with the complexities of what tasks their AI tools can execute effectively.
– **Technical Challenges**: The text highlights a specific challenge regarding the integration of AI models into applications. A user’s inability to create a Claude Artifact that effectively analyzes images illustrates the importance of understanding systems limitations, particularly how CSP (Content Security Policy) headers affect API calls made by generated code.
– **Insight on Developer Experience**: The text emphasizes that a profound depth of experience is necessary to navigate the “easy versus hard” problems within the realm of software development, underscoring the ongoing need for education in this rapidly evolving field.
The reflections in this text have implications for security and compliance professionals, especially in understanding the limitations and operational boundaries of AI systems in software development environments. The integration of secure coding practices with the rise of AI tools will be crucial to mitigate risks and ensure compliance with industry standards.