How AI Supercharges Email Fraud
I got an email last week that showed the ways in which AI can perpetuate fraud and trick even the savviest of human recipients. We need to stay alert. Here is what to look for...
Sitting at a pool in Utah last week (a topic for a later post about how AI can be a wonder in organizing and managing a vacation), I got an email from (what appeared to be) a publisher.
Now, I’m naturally quite cynical and immediately side-eyed the prospect of a senior SVP at a publishing house wanting to just “catch up” about my work. But, on the other hand, the email seemed very real. In fact, it wasn’t like any spam email I’ve ever received…
Email from *MacMillan*
Here is what dropped into my email last week…
See how “human like” the note is. The information about my book is accurate and there is even a nod to the second one I’m currently writing. It’s clear and articulate with a professional tone that makes no promises—just an expression of “interest” in my work.
On first glance, the details about Allison also seem very believable. The structure of the email is “clean” with a nice signature, a photo, and details about the publishing company that are(mostly) correct.
The Role AI Plays
What made the email feel authentic wasn’t just that it praised my work—it was that it reflected my actual aspirations. In “studying” humans AI systems understand what messages might be most emotionally compelling to us. This is one of the most important things to understand about AI…in aggregating and analyzing human output and behavior, a system in the wrong hands can coerce and manipulate quite easily.
An AI system can take publicly available information—in this case, my website, LinkedIn profile, Amazon book description, interviews, or media appearances—and weave those details into a highly personalized message. The result feels authentic because much of it is true.
It’s also important to note that AI is able here to easily employ tactics that humans use to coerce others. It’s not that a system is somehow more clever or manipulative, it’s that we can be and AI has learned from us. It can collate those things that work. For instance:
It referenced a real person with a prestigious title.
It contained accurate, specific compliments about my work.
It expressed interest rather than urgency (making it feel less like a scam).
It didn’t ask for money immediately, lowering suspicion.
It invited an ongoing professional relationship rather than demanding action.
The Fraud Markers
Identifying an email like this as fraud isn’t too difficult if you know the markers. But first, no matter how compelling, believable, or personal an email is, if it’s not from someone you’ve heard from before, or would expect to hear from, you should take a moment to consider its veracity anyway. We are prone to optimism, but need to look at any outreach that may be “unrealistic” with suspicion from the get-go.
These include email addresses in the header that do not come from a company’s domain. That’s a big reason to stop, and if you have any questions, forward it to someone at the company via an official domain.
Then there are also little flaws such as an additional email copied that wasn’t mine (but it was my name); a few capitalization inconsistencies; and other details that are too familiar or sound “strange.”
Your instincts can be the guide. But the goal should be to first, just stop and do nothing.
Tips to Protect Your Family
The Email Address is the Giveaway
If the email is not coming from an actual company domain, don’t respond or click. Ever. And be 100% sure. So for instance, you might get an email from facebookcompany.com or another that on first glance “seems” legit. But don’t do a thing before checking first.
From there you can ignore, of course. But if you want to understand if there is even a chance that the note is real (or want to help root out fraud), you can Google to find the actual email of the person or a commonly-available email like legal@company.com. In this case the email would have come from macmillan.com.
Watch for the Follow Up
If you were shaky in your resolve after getting the first email, then a follow-up might wear you down. You might ask, “why would a fraudster send such a thoughtful follow-up?” Well, that’s the cleverness of this crime… and AI can make it easy.
It Takes a Village
The Publishing Fraud Slayers
It’s all of us who can work together to ensure that AI is used for good and not to benefit criminals—and it starts with spreading the word. In this case, it didn’t take long for me to find an intrepid group documenting and fighting this very fraud. The Science Fiction and Fantasy Writers of America have been collecting examples of this very issue and publishing the results.
The emails are incredible when looked at en masse. The Writer Beware blog breaks down individual cases like mine in detail, while their main site tracks scams across the industry more broadly. Between the two, you’ll get both the specifics and the bigger picture.
Speak Up
Yes, it takes extra work to do your part to help root out this type of fraud. But this is the way we educate our communities and ensure a future where we are all much savvier and in control.
If you get an email like this, consider reporting it to any interested party, from the person targeted to lawyers and industry organizations as I have above. It’s exactly the kind of documentation that helps the rest of us spot the pattern faster. And, of course, it’s critical to help our kids understand what is happening as well.
Have questions? Let me know.