Close only counts in horseshoes and hand grenades…and AI. Hear me out.
Read a paper written by AI, and you are likely to think, “This sounds really stiff and something is just a bit off,” like a cut-and-paste job from a student who was watching Netflix at 2 a.m. while crashing on an assignment. Look at AI-generated images or videos and you’re likely to see very attractive people with six fingers. Listen to an AI-generated rock song and you’ll hear a ripping guitar solo that you’ll never want to hear again.
That’s AI slop. “Nice try!” you might tell your 12-year-old as they write their first composition. Or Google as it tosses an AI Overview at you about a topic you genuinely understand. Isn’t that charming! Now, if you plan to entrust your life to artificial intelligence — say, through a self-driving car, or an air-traffic control system, or even as a personal financial advisor — close enough isn’t going to be good enough.
On the other hand, if you are a criminal, good enough is just great. If you make 100,000 phone calls claiming you have a grandchild hostage and demanding a ransom, your AI-generated voice clone doesn’t need to be perfect. Just good enough to create a few moments of panic. So while we are all living in a beta test of slop, criminals don’t mind these clumsy missteps- they’ve been handed a superpower straight out of a comic book.
Not-So-Super Intelligence
AI still makes mistakes. Lots of them. It often reminds me of that screw-up friend who is really great at saying “I’m sorry,” and somehow keeps getting away with it. (“You’re right. Those two planes are going to hit each other.” ) Just follow the links Google offers up as source material for its AI findings and you’ll often discover the bits of wisdom it has confidently shared are nowhere to be found in the source material.
Of course, there are perfectly predictable uses for these consumer-grade AI tools. This week I read a New York Times story by a reporter who let Muse run his life for a while. He was most impressed (“blown away!”) by Muse’s ability to stay on hold with his insurance company. I smell the latest reality TV show coming soon. Forget mano y mano. This is Bot vs Bot. In this fight to the death, will there be a clear winner before the patient dies awaiting a prior authorization?
Hear me out: What if we used those billions in AI investments to hire a couple of competent customer service agents to answer the phone? Crazy, I know. You see where this is going. Credit bureaus like Equifax have long battled automated consumer complaints. Trust me, corporations will win any stare-off-game-of-chicken you ever try to play by wielding your cute AI agent against their revenue-enhancement bot army.
The nightmare is here
But I digress. There’s no time to fret about this futuristic nightmare because the AI Nightmare is already here. Criminals don’t need their agents to be perfect when launching AI grenades at us. Slop is just fine when you don’t care about who you are hurting.
Here’s an example of AI-crime in action I read recently.
Microsoft detected an email invoice scam in August that was highly personal and went to 1 million potential victims. It was a traditional business email compromise on AI steroids. Recipients — mainly accounting departments — got notes that appeared to come from an executive asking that an invoice be paid immediately. That part is familiar. But the attention to detail in these one million requests was astonishing and could only have been accomplished by bots from the Dark Side.
The attack combined “executive impersonation, vendor branding, fabricated invoices, and supporting email conversations into a unified narrative,” the report says. The detailed ServiceNow invoices attached (ServiceNow was not involved; it was impersonated, too) also included personalized details.
Recipients even saw a forwarded email thread with believable conversations attached. From the report:
The threat actor impersonated executive team members (such as a CEO, CFO, President) of multiple targeted companies, attempting to convince accounts payable departments of the same companies to process an ACH payment of nearly $50,000. More specifically, the CEOs were impersonated in multiple places in the email such as in the sender display name, reply-to display name, and in the email signature. Email bodies contained a simple and direct “approval” of the “invoice below” as well as urged users to request a PDF version if they need it. Additionally, as mentioned earlier, the email signature contained certain details about the spoofed CEO such as name and email address.
Look at this sample email
And sample invoice:
These are high-value spear phishing attacks. At incredible scale. Remember, everything in these emails was personalized. One million times. Microsoft provided evidence that AI was used to do so, given the spacing and em dash use in the software’s comments — you can read that in the report.
Back to the horseshoes and hand grenade part of this story. The personalization in these one million emails does not need to be perfect. Thousands of those emails could have an obvious error in them — say, failure to use a nickname that a finance employee would recognize. Who cares? The hit rate could literally be one in a million and it might be a cost-effective success.
Contrast that with the AI fantasy we’re being sold right now, the so-called “paradise of machines” that Pope Leo recently warned about. If a self-driving car made a million turns and thousands of mistakes, a self-driving car program would (hopefully?) be shut down immediately. AI makes so much more sense as an agent of crime than as an agent of intelligence.
Naturally, Microsoft’s report explains why their technology is smart enough to recognize the small mistakes in this AI attack; its clients were safe, it says. In this bot vs bot war, the criminals lost, their better bots won, it claims. But what about regular human beings who can’t afford the bigger bots? AI-enhanced scams are already penetrating our world, making realistic cloned voices for digital kidnapping, allowing male call-center criminals to appear as sexy women during video calls. In all these cases, close enough is an effective hand grenade.
I am worried about the AI doomsday scenarios we keep hearing about; I’m worried about “skill offloading” for young people who only know how to regurgitate bot answers in school; I’m very worried AI will continue to be shoved at consumers and used as a tool for cost cutting;
But right now I’m far more worried about the individual doomsdays that will happen as cheap, sloppy AI is continually unleashed on unsuspecting consumers by criminals. This is a beta test none of us have signed up for. Our incentives are misaligned; companies that make these tools need to face more liability for the damage they cause. That’s the only way we can slow down what feels like an inevitable slide into the very sloppy future.
PS: I read a very wise, even-handed look at the risks and benefits of AI in this New York Times opinion piece today. In it, an experienced doctor compares his AI tool to a committee of experienced friends he can run diagnoses by; but he frets that young doctors rely too much on software and won’t develop the medical intuition that also plays a large in a good medical practice. Give it a read.
Be the first to comment