ChatGPT is an language model that uses probability to create a text-based answer to your question. But math and accounting hinge on accuracy, not probability. Here’s why that matters.
Over the past year or so, large language model (LLM) ChatGPT has demonstrated an uncanny ability to best humans at some of the things that are the cornerstone of our young professional lives.
It has passed all three notoriously difficult exams for medical school, got through the law school bar exam, and passed an MBA exam from the Wharton school of business at the University of Pennsylvania.
The scores posted by the LLM were modest passing grades. But its later avatar — GPT-4 — is supposedly an even better student than its parent, having sailed through the bar exam with a 90th percentile score and getting near-perfect marks on the GRE Verbal test.
So, it must come as an immense source of both satisfaction and relief for us humans that there is at least one thing that LLMs like ChatGPT are not good at — or in fact terrible at: accounting.
Many users of ChatGPT have commented publicly on how the simplest math functions have foxed it. However, there’s a sizeable and rigorously executed study into ChatGPT’s accountaing capabilities that Brigham Young University (BYU) professor of accounting David Wood undertook several months ago.
Wood decided to harness the power of the global accounting fraternity via a pitch on social media that solicited help to put ChatGPT through the paces of a global accounting exam of sorts.
There was a deluge of takers: 327 co-authors from 186 educational institutions located in 14 countries participated in the study. They collectively pooled together 25,181 classroom accounting exam questions — as well as 2,000-plus questions from his own department at BYU — to pose to ChatGPT.
Typical of a comprehensive accounting examination, questions ranged across all major topics. such as financial accounting, auditing, managerial accounting, tax, and others, and were of different types (multiple choice, short answers, true/false) and difficulty levels.
