AISA Finds Users Overestimate Their AI Skills by 18 Points

An August analysis by AI skills assessment firm AISA evaluated 274 participants who predicted they would score an average of 63.4 out of 100 on an AI literacy test. Instead, they averaged just 44.9, overestimating their abilities by 18.5 points. Even engineering professionals overshot their scores by 15.4 points. In a broader dataset of 1,172 assessments, workflow application was the highest-scoring category at 49.6, while safety and technical understanding lagged behind at 41.5 and 41.3, respectively.

This trend of overconfidence is mirrored in academic and corporate research. In a study titled 'AI makes you smarter but none the wiser', 246 participants used AI to answer 20 LSAT logical reasoning questions. While their performance beat the norm by three points, they overestimated their scores by four points. A follow-up study with 452 participants replicated these findings, showing that higher AI literacy actually correlated with less accurate self-assessment. Meanwhile, Anthropic's AI Fluency Index analyzed 9,830 multi-turn Claude conversations from January 2026. It found that when Claude generated structured artifacts, users performed 3.7 percentage points less visible fact-checking and questioned the model's reasoning 3.1 percentage points less often.

For AI practitioners, these findings show that polished outputs often mask underlying errors, leading to unearned trust. To counter this blind spot, professionals must shift from merely prompting models to actively verifying their work. This means establishing objective criteria for what makes a result usable before starting a task, rather than relying on whether an answer simply looks good. Practitioners should also explain key AI-generated conclusions in their own words and conduct independent verification, such as running code or checking original sources, rather than asking the chatbot to review itself.

For organizations, the data suggests that training must evolve beyond teaching basic prompting. Employers need to design assessments that test whether workers can identify subtle, AI-generated errors. Cultivating critical evaluation skills alongside generation skills will ensure that teams do not mistake fast, well-formatted drafts for accurate, production-ready work.