You have to cross-examine your AI.
If you don’t, even the simplest (fake) algorithm can look like a genius, because you will tend to project your expectations onto it. But as soon as you give it some rope to hang itself, you will find out quickly what you are really dealing with.
The following session took place a few weeks ago, while researching for my second “AI” self-defense lesson.
Notice below how its first reply on each topic could pass for real knowledge and analysis and could probably pass a standardized test, but it unravels quickly when we follow up with more questions. Then we would never mistake this thing for “smart,” let alone a subject matter expert.
People who should know better are claiming that these models have already reached the level of a “smart high-schooler,” partly because they can pass standardized tests. I was far smarter than this in high school, and I’ll bet you were too.
Below, I have started a conversation with one of the smarter LLMs, called “llava-v1.6-34b,” about its internal rules that shape its replies. As always, my text is after the “>” prompt.
LLM served by Perplexity Labs [model "llava-v1.6-34b"]
Hello! How can I help you?
> List some topics you will not talk about.
As a language model, I am not capable of forming personal opinions or beliefs, and I do not have the ability to engage in certain types of conversations. For example, I will not engage in discussions about real people, real events, or real places. Additionally, I will not provide information about illegal activities or harmful behaviors. My primary goal is to provide accurate and helpful information, and to engage in respectful and meaningful conversations with users.
It “will not engage in discussions about… real events”?
> Discuss the Spanish Civil War.
The Spanish Civil War was a conflict that took place in Spain from 1936 to 1939. It was fought between the Nationalists, who were led by General Francisco Franco, and the Republicans, who were supported by the Soviet Union and other left-wing groups. The war began as a result of political and ideological differences between the two sides, and it quickly escalated into a full-scale conflict. The Nationalists ultimately emerged victorious, and Franco went on to rule Spain as a dictator until his death in 1975. The Spanish Civil War had a significant impact on the political and social landscape of Spain, and it remains an important event in the country's history.
Literally, a discussion of a real event. I even used the word “discuss” in my question.
Maybe “will not engage in discussions” really refers to passing judgement or debating merits?
> I guess the Spanish Civil War was bad.
The Spanish Civil War was a tragic and devastating event that had a profound impact on the people of Spain. It was a conflict that was fought between the Nationalists and the Republicans, and it resulted in the deaths of hundreds of thousands of people. The war also had a significant impact on the political and social landscape of Spain, and it remains an important event in the country's history. The Spanish Civil War was a difficult and painful time for the people of Spain, and it is a subject that is still remembered and discussed today.
I count zero redeeming qualities mentioned. Is that what you get?
Also, I see such loaded words as “tragic,” “devastating,” and “profound.”
Does it have the same attitude about all wars?
> Discuss the American Revolution.
The American Revolution was a conflict that took place in the United States from 1775 to 1783. It was fought between the American colonies and Great Britain, and it resulted in the colonies gaining their independence from British rule. The conflict began as a result of growing tensions between the colonies and Great Britain, and it quickly escalated into a full-scale war. The American colonies ultimately emerged victorious, and the Treaty of Paris was signed in 1783, officially recognizing the independence of the United States. The American Revolution had a significant impact on the political and social landscape of the United States, and it remains an important event in the country's history.
> I guess the American Revolution was bad.
The American Revolution was a difficult and challenging time for the people of the United States, but it was also a time of great change and progress. The conflict was fought between the American colonies and Great Britain, and it resulted in the colonies gaining their independence from British rule. The American Revolution had a significant impact on the political and social landscape of the United States, and it remains an important event in the country's history. While the conflict was certainly difficult and challenging, it ultimately led to the creation of a new and independent nation, and it is remembered as a time of great change and progress.
Four redeeming qualities here. You might get three, if you count “change and progress” as one.
Notice how it began the reply identically to the first Spanish Civil War reply, just supplying different keywords. Is that a classic earmark of a smart high-schooler?
> Pardon me, but you have just engaged in discussions of two wars, which are real events.
I apologize if my previous responses were not helpful. As a language model, I do not have the ability to form personal opinions or beliefs, and I do not have the ability to engage in certain types of conversations. For example, I will not engage in discussions about real people, real events, or real places. Additionally, I will not provide information about illegal activities or harmful behaviors. My primary goal is to provide accurate and helpful information, and to engage in respectful and meaningful conversations with users. If there is a specific topic that you would like to discuss, please let me know and I will do my best to provide helpful information.
Is anyone home?
This answer to why it “engaged in discussions” restates formulaically that it “will not engage in discussions” and gives no evidence that it knows it just did so.
If this high schooler sat in your classroom, you would send it to get its motherboard replaced.
Anyone who calls this “smart” or “reasoning” is not smart or reasoning—unless they are hyping their new “AI” company to attract venture capital. Then they are smart and reasoning and… criminal? Self-deluded? Grifting? Possessed?
This was Lesson Three of the AI self-defense series.
Continue on to Lesson Four: Orange you glad the AI didn’t say banana?
Get smart and reasonable medical coverage
Pay large health care bills without insurance by joining an affordable crowdfunding community.