III - Understand the capabilities of AI

Back to the previous chapter II - Organize well with the right team

1. Get realistic about AI’s limits

You have a better chance of shipping great AI feature by underestimating its capacity than the opposite.

</aside>

While tools like ChatGPT may create this illusion, generative AI is far from magical. To unlock its full potential in real-world applications, you must understand its capabilities thoroughly.

Don’t be misled by impressive academic benchmarks. Claims that AI is "passing the bar" or "solving advanced math problems" can create a misleading impression that AI has human-like abilities and that creating an AI feature almost wouldn’t require engineering.

These benchmarks are valuable for tracking progress and comparing models in controlled environments, but they don't reflect AI's practical limitations.

2. Treat AI like an average college intern

<aside> <img src="https://prod-files-secure.s3.us-west-2.amazonaws.com/d8baae53-7dc0-4bad-a65f-26898d6a633d/13f7d500-73a1-4dfb-a2af-168ccccebd57/Silex_Brand_Symbol.png" alt="https://prod-files-secure.s3.us-west-2.amazonaws.com/d8baae53-7dc0-4bad-a65f-26898d6a633d/13f7d500-73a1-4dfb-a2af-168ccccebd57/Silex_Brand_Symbol.png" width="40px" /> Ask yourself: “would the average college level student be able to perform this task consistently?”

</aside>

Imagine AI as an average college student intern: capable of reading and writing adequately, with limited reasoning skills, and no experience or context about your specific tasks.

To get meaningful results from your "AI intern," you'll need to provide detailed, precise instructions and all the necessary context. Expect to repeat explanations and refine your guidance before the AI becomes proficient at the task.

While it's true that AI opens up exciting possibilities for automating tasks once reserved for humans, don’t expect it to be any easier or more reliable than working with a freshly arrived intern. The promise is real, but it comes with challenges that require careful management and realistic expectations.

3. Focus on the tasks that AI can do well

<aside> <img src="https://prod-files-secure.s3.us-west-2.amazonaws.com/d8baae53-7dc0-4bad-a65f-26898d6a633d/60e16bfc-42ed-47f3-b4c2-e0955098dd83/Silex_Brand_Symbol.png" alt="https://prod-files-secure.s3.us-west-2.amazonaws.com/d8baae53-7dc0-4bad-a65f-26898d6a633d/60e16bfc-42ed-47f3-b4c2-e0955098dd83/Silex_Brand_Symbol.png" width="40px" /> Understand the generic tasks that LLMs do well. Combine these building blocks to create your features.

</aside>