Are AI Chatbots Spot On?

Recently, our resident first grader brought home his first-ever report based on facts he learned from Pebble Go, an app he uses on his school-assigned Chromebook. After celebrating “101 Days of First Grade” with his Dalmatian shirt, it was not surprising that he chose Dalmatians as his first report topic. We all learned quite a bit about those dogs, but there’s always room to learn more. As someone who has recently jumped into exploring chatbots, I thought it might be fun to compare what our resident “expert” and the free versions of ChatGPT, CoPilot, and Claude have to say about Dalmatians.

Immediately noticeable is that while our first grader used rudimentary citing techniques by recording page numbers for his report, none of the AI-generated reports cited their sources. I could add a prompt to list the sources, but that information was not automatically shared. The generated reports may supply a general outline and help introduce ideas of what to include, but the shared facts need to be double-checked, and proper credit is given.

When requested to write a short report, the structure of that result varied. With ChatGPT,  each paragraph starts with a topic heading followed by a colon. CoPilot created a bulleted outline that used complete sentences. It also included an image of a Dalmatian with a litter of puppies and a link to the American Kennel Club. Claude provided a “bare bones” five-paragraph summary.

When I prompted the chatbots to write as a first grader, the results were roughly equivalent to an adult writing for a first grader. Using our first grader as a test reader, the CoPilot report included only one unknown word. With ChatGPT and Claude, the vocabulary was above first-grade level and included more advanced - and unknown - words.

There were a few differences in the facts presented in the reports. Although this isn’t a comprehensive list, it shows that using one AI source limits the results when researching a topic. Here are some of the differences:

• CoPilot did not mention Dalmatia, now known as Croatia, as the potential origin of these dogs.
• ChatGPT did not include their size (medium size). CoPilot mentioned their approximate height, while Claude mentioned what they typically weigh.
• Claude did not mention fire trucks, which you might assume would be an obvious main fact.
• Our first grader discovered that Dalmatians are also called plum pudding dogs, which none of the chatbots mentioned.

On the other hand, some of the sentences found in one report were fairly close to sentences from other AI sources. One standout example is CoPilot’s: “They have spots all over their bodies like they’re wearing polka dot pajamas.” ChatGPT provided, “It's like they're wearing their own polka-dot pajamas!”

AI can be a great starting point for writing reports. It suggests main topics, facts, and possible report structure in very little time, but final reports still require human interaction and effort.

Terri M.
Digital Innovation Specialist