AI Written Poetry is SHOCKINGLY Better Than Human Writers

Doug Downs (00:11):

Can machines write poems we hold so dear? Or are they just pretending to draw near today? We'll look at how verses play and why some people like them more this way. Do humans lose their hold on art's True soul? When AI starts to take a leading role, the answers might surprise you as we see what's next for poetry and artistry, all that written by AI today on stories and strategies can AI poetry, outwit Whitman? Out, charm Chaucer? Outshine Shelley?

(00:57):

My name is Doug Downs. I've got a thank you off the top. Thank you to JGW who left the five star rating on Apple in Canada. And this review about our episode on Jaguar. I'm a journalist in agriculture with a new podcast, the Extensionist, so I got to plug in there for you JGW, and I'd like to hear Doug's tips on how not to boost your brand. After listening to the Jaguar episode, I will definitely scratch Zoolander video from my to-do list. Really appreciate the review, kind of an inside joke, but you have to listen to the Jaguar episode, which is a couple back. Now to get that joke.

(01:50):

My guest this week is Brian Porter joining today from Pittsburgh. Hey Brian. Hey, Doug. How are you? I'm good. How are things in steel town? I have been, by the way, love the city and I actually really loved Pennsylvania overall. How are things in steel town today? It's great. A little

Brian Porter (02:06):

Rainy, but it's above freezing in December, so as someone from the south, that's fantastic weather for me.

Doug Downs (02:12):

Yeah, absolutely. Brian, you're a researcher and data analyst at the University of Pittsburgh. You have a background in logic and cognitive science. So first off, this is an academic study. This isn't like some survey of 12 people and we don't hold it scientifically accountable. This is scientific process. You co-authored with Ed Edward Maie and sought to find if people can tell the difference between a poem generated by AI as opposed to one that was written by a human type brain and which did they prefer? What did you find?

Brian Porter (02:50):

Well, we found that our participants, which we ran two studies, the first had a little over 1600 people in it, the second about 800 people in it. And we found that people could not tell the difference between AI generated poetry and poetry written by human poets ranging from Chaucer, Emily Dickinson, Sylvia Plath, Shakespeare. We had 10 different poets in the study and people could not tell the difference. And in fact, they were a little bit worse than random guessing. People were a little bit more likely to guess that the AI generated poems had been written by a human than to guess that the actual poems written by humans were written by a human.

Doug Downs (03:33):

And which ones did they like in general? Was there a leaning? Did they prefer the AI or the people poems?

Brian Porter (03:42):

They preferred the AI poems. In a study where we asked assessments, we asked people to rate poems, rate the overall quality on a scale from extremely good to extremely bad rate, how meaningful, profound, original, witty, how beautiful it was, to what extent it conveyed a mood or emotion. How was the rhythm, how was the sound? Did it rhyme? And across all of our rating dimensions, not only were the human written poems not preferred, but the AI poems were rated higher, and that's pretty consistent. Over 78% of our participants gave higher average ratings to AI poems than to human written poems. And all of the AI poems had a higher average rating than all of the human written poems.

Doug Downs (04:30):

Okay, we got to break this apart, right? The machines have taken over. It's like a Stephen King novel. Suddenly we got to understand the why. And I'm tempted to ask chat GPT why this happened, but I'll go to your abstract. You theorize that the human preference for AI poetry could be because the AI stuff, it's just more straightforward. It's not this Shakespearean gobbledygook, I don't know what he's saying. They can actually follow the flow or the plot of the poem itself. Is that what you heard?

Brian Porter (04:58):

Yes. So we found that in our initial study when we asked people to judge whether the poems were written by human or generated by ai, we gave them the option to explain their answers. And most people didn't. But those who did when judging the human written poem was generated by ai, they often said things like, this doesn't make sense or this seems like nonsense. One person reading a TS Elliot poem wrote in, this doesn't seem like it was written by a human or someone with feelings, but it was written by a human being with feelings. But there's something sort of difficult or that requires time and maybe historical context in order to appreciate about a lot of the human poems. Whereas the AI generated poems are very straightforward. They look like you expect a poem to look, and it's pretty easy to see what the poem is about on your first read of it. And so just to pick two poems that we used in our study, one poem the people did not have very high ratings of is a TS Elliot poem, which is sort of an opaque satire, how it go opaque now.

Doug Downs (06:22):

Yeah.

Brian Porter (06:24):

One of the TS Elliot poems in our study is a satire of the readers of the Boston Evening Transcript, which is a newspaper that does not exist anymore,

(06:32):

And without historical context, it's not that crazy that people prefer the AI generated Walt Whitman style poem about how awesome nature is. Maybe the rhythm's a little clunky here or there, but that's a much more pleasant reading experience, especially if you're sitting with no, has no historical context, no historical background. And I think part of what we maybe found is that how you interpret a choice that a poet makes depends a lot on giving them some credit for having made it on purpose and knowing what they're doing is just of hypothesis here. But when Chaucer in one of the Chaucer poems repeats a line multiple times, if you think it might be AI generated, it might just be lazy or an error. But when you know that Chaucer wrote it, when you know someone who was good at this did that on purpose, that sort of forces you to reinterpret it a little bit. And we found that if you tell people that a poem was written by a human or tell them that that same poem was generated by ai, that has a pretty substantial effect on the qualitative ratings. People like the same poem much better and rate it better overall quality as more meaningful, as more original as Whittier. They like the rhythm better when you tell 'em it was written by human as opposed to telling them that it was generated by ai.

Doug Downs (08:06):

That's the cognitive bias there than I'm buying in more

Brian Porter (08:09):

Readily. Yeah.

Doug Downs (08:11):

So what I'm pulling from that is the feedback you got in your study is that people get it more readily, more quickly, more widespread when it's AI generated. And that's probably a key lesson for us as communicators and marketers to pull from here that if you're dealing with masses, they need to get it very, very quickly and you can't guide them to an inside joke or an inside story or tap it. But if there is a niche audience that you're going for, maybe it is more valuable to tap into that inside story. So let me ask you when you, because I know you went to some poetry experts in some capacity and said, what do you think? It's not in your abstract, but I know you went. So the experts must have easily figured out, oh, no, no, because they're in it, right? They know Chaucer and they know Whitman. That poetry experts must have said, no, no, I prefer the human generated poetry and easily deducted and distinguished the difference between the two.

Brian Porter (09:19):

Well, in a small pilot, we did look at poetry experts, people in MFA programs for poetry, people with English PhDs studying poetry. The smart

Doug Downs (09:35):

People, the smart people who know the difference.

Brian Porter (09:37):

Yes, some of them had published poetry, they were published poets, and they did do pretty well at the discrimination task, but a hundred percent of them reported having seen several of the poems before that were

Doug Downs (09:53):

AI generated.

Brian Porter (09:53):

No, they said they'd seen the human written poems before they just recognized that they could easily, they could go, oh, that's Shakespeare's sonnet one 15 or whatever.

Doug Downs (10:05):

I get it.

Brian Porter (10:06):

So then it's a memory task,

Doug Downs (10:08):

Right?

Brian Porter (10:09):

Less than a can you tell what sounds human or what sounds AI generated? So we are working on getting a more reliable test of poetry experts. I'm working on a project now with a researcher in Edinburgh to have poets contribute unpublished poems, and we're hoping to create an online interface and release it and let people in the public, you can go online and take the test yourself and see how you do. And hopefully since the poems aren't published, there won't be a risk that the experts already know the right answers for half the questions.

Doug Downs (10:52):

So if I as a public relations pro, a marketing pro, if my sort of thought is chat PT or cloud are great, but they can't tap the emotion the way I a professional can tap into emotion because they don't have any human experience, am I coming to the right conclusion or am I missing something rather gaping here?

Brian Porter (11:15):

Well, it's important to remember that the large language models that we're looking at, whether it's Chat g PT or Claude or Gemini or Llama three, these have never had a human emotion, but they have read much more than you have about human emotions. They've read everything that's ever been written about every emotion, and they're really good at imitating that. So if one of the things we found in our study was that part of the tendency for people to think AI generated poems were written by humans at fairly high rates. Part of that is that they underestimated how good AI would be at communicating human emotions, and it's actually very good, at least imitating the communication of human emotions and evoking human emotions. A couple of our participants wrongly guessed that AI generated poems were written by humans because the poems made them laugh and they didn't think AI could do that.

(12:19):

But AI can be funny sometimes, and if you want to evoke an emotion or communicate an emotion, AI may be a useful tool. It can help you do that. If you want to say something new and original about human experience, if you want to say something no one's ever said before or articulate something in a genuinely new way the way that I think a lot of poets aim to do, well then AI is at a pretty substantial disadvantage because it doesn't have the human experience that you're trying to communicate something about. It only has what's been said before. So real originality is kind of a weakness, and it is a weakness that's sort of baked into the core concept and sort of fundamental to the enterprise that it's only trained on predicting what the most likely next token is. It's trained on figuring out what sounds like everything that came before. So if you want something with mass appeal, AI is probably going to do a very good job. If you want to make something unique that captures something about someone's experience in a way they've never heard before, ai AI's probably not going to help you do that.

Doug Downs (13:28):

I've watched sitcoms and Hollywood movies and I just have to think the Hollywood Screenwriter's Guild is not going to like this study at all. But any notable difference between genders or age groups? Did anything stand out that way or No?

Brian Porter (13:44):

So we asked participants a lot of demographics questions, and we didn't find any impact of age or gender or even experience with poetry. Now, there's a bit of a ceiling on the level of expertise that participants reported having, but people who read Poetry weekly didn't do better than people who read poetry less than once a year. People who'd taken poetry courses before didn't do better than people who hadn't. There is one minor caveat there, which is we did find that even when controlling for how much you read poetry and how much you like poetry and things like that, people who in our demographics questionnaire identified as non-binary did perform better at the discrimination task. But approximately 2% of our participants reported being non-binary. So the sample size is a little small to draw conclusions from reliably, but that is a statistically significant effect according to our models.

Doug Downs (14:58):

Let's pull some pontifications and potential conclusions here. Is this suggesting that AI should be embraced as a legit artistic collaborator or put more darkly? Is it a tool that risks overshadowing human creativity?

Brian Porter (15:17):

I think my answer is different for the two ways of putting it, the bright way of putting it, I think absolutely. I think AI as an artistic collaborator can be very useful for brainstorming. It's great. It can come up with 15 different ways to say something in a matter of seconds. Even if you don't use any of the ideas it spits out, it might help you figure out what you don't want. Sort of like the Thomas Edison finding 2000 ways not to make a light bulb chat, g PT will get you to 2000 ways, not to phrase something much faster than you could on your own.

(15:51):

So as an artistic collaborator, it can be useful in a lot of ways. There's an artist, Sasha St. Styles, who is a sort of AI poet who writes poetry in collaboration with an AI model and is very optimistic about AI as a tool for art and poetry and thinks of language as a technology for getting an idea from my head to someone else's head. So if language is a technology for getting ideas from one head to another, then AI as a tool for producing language is just another form of technology. It's another tool in your tool belt to get an idea out of your head and into someone else's, which is an important part of almost every form of communication. It's maybe what pretty good definition of communication.

Doug Downs (16:41):

So

Brian Porter (16:41):

As a collaborator, AI is potentially very useful depending on how you use it as a replacement. I don't think it's going to replace human creativity anytime soon. It's not going to make us obsolete. It might make us more efficient and might mean that you need fewer writers to get the same output each year. That's sort of what the software industry has discovered in the last couple of years, is that one senior software developer with a copilot and Claude Sonnet can produce all of the code that a team of junior developers used to produce. So there may be some changes to how human creativity and how many people there are for the same output. There may be changes coming there to a variety of industries, but we're not going to be completely replaced at any point because to have this sort of new and interesting thing to say or to have an idea of what it is that you're trying to communicate, the idea still has to come from a human head. The thing you're trying to communicate still has to come from a person.

Doug Downs (18:01):

The dark side of me would say there's nothing actually new in this world and everything that is to be said or created has been said or created in a previous iteration. So you and I could philosophically go back and forth on that. Your study looked at the written word and the spoken word, what about art? Could AI generate paintings? Is that a potential study in the future? Because I could not tell the difference. And my wife's an artist, she's an oil on canvas artist, and she would probably scold me for saying that, and she does listen. Could this be replicated with art?

Brian Porter (18:42):

I think there is some evidence that AI generated images are already to a general audience indistinguishable from human made paintings. I think you are probably right. You and I probably could not tell the difference whether your wife, the artist could that remains to be seen. I haven't seen a study that tested professional artists and painters, and that would be interesting to see. But I think paintings and poetry are currently at the same point where the general public cannot tell the difference, at least not without seeing examples ahead of time. You need either to be trained to distinguish or you need domain expertise.

Doug Downs (19:33):

Appreciate this. I really appreciate Tim time on this today, Brian.

Brian Porter (19:37):

Yeah, thanks for having me.

Doug Downs (19:39):

Well, in our previous episode, our guest, Jess Jensen of copilot communications, she left a question for you, have a listen to this.

Jess Jensen (19:46):

So I know that you did a fun short podcast around the idea of a bamboo and a bamboo plant. Yes, I'm sure you remember, and I loved the symbolism of how planting a bamboo is similar to growing a podcast audience and it takes quite a while for the roots to sort of take hold and then eventually, and then you see rapid growth with the bamboo. And so my question would be for your next guest, which analogy in nature might you use to symbolize your business and why?

Brian Porter (20:21):

So it's a good question. I think I would say that, well, my business is sort of data analysis and using data to try to find answers to questions. And I think of it kind of like bees in a beehive where each individual bee produces almost no honey over the lifetime of a bee. You get, I think something like one third of one milliliter of honey out of it. But if take a couple thousand of them, now you've got a useful product, now you've got something good and it's sort of trying to take data and make an insight out of it is a lot like that one is almost worth nothing but a couple thousand. Now you can get a picture of things and get something useful that you can use to find some answers and solve some problems.

Doug Downs (21:09):

That's exceptional. I absolutely love that answer. Thanks for that. Thank you. So okay, your turn. I'm expecting a killer question now because you nailed the answer. What would you like to leave behind for the next guest?

Brian Porter (21:22):

Well, I'm going to be a little bit selfish if I can and ask a question that I have been thinking about and would love to hear an answer from your next guest and hear what they have to say. Because having looked at AI generated texts and how people perceive them, there is, most of us in most businesses and industries are, if we haven't already, we're going to have a situation where we have AI generated content that we know is good and we know people will like, but given biases against AI generated content, they're not going to like it if you tell them it's AI generated. So as a business and as an audience, what should we do when we are producing content and outputting text? Do we have an obligation to say that it was AI generated? And as an audience, are we okay with things being generated by AI and maybe not always being disclosed as ai? Do we care and should we care

Doug Downs (22:30):

Ethically? For me, it comes down to are we somewhat diversified in who's creating the software that has created the ai? Because if it's all 30, 40 something white dudes in Silicon Valley, that's a problem and that will help shape the narrative of society. And I think society wants, Hey, let's add some more ingredients to the mix because that is the more natural way.

Brian Porter (22:57):

Yeah, I think that's right. And some linguistic evidence. So AI and chat GT especially has measurable sort of linguistic quirks. It uses some vocabulary at higher rates, it uses some sentence constructions at higher rates than humans in general do. And as it becomes more prominent, it's going to have an impact on how we talk when we communicate because it will be we based how we talk on what we've read and heard, and that's already happening. And there's some evidence that it started happening earlier than we might think, especially among young people. So who creates these models and whether or not we accept their prominence and accept not knowing and not necessarily being able to recognize when they're being used. I don't know what the right answer is, and I don't know what answer we're going to settle on, and I'm not sure they're the same thing.

Doug Downs (23:56):

Well, I'm glad it begins with a question instead of, it feels like in society we often begin with an answer and try to shape the questions. After that, I think it should be the question and then the answer. So really do appreciate your time today, Brian.

Brian Porter (24:10):

Oh, thank you for having me. I really enjoyed

Doug Downs (24:11):

It. So here are the top three things I got from Brian Porter in this episode. Number one, simplicity and accessibility in communication. AI generated poetry was preferred by participants because it's straightforward, it's easily understood, clarity and accessibility are crucial, complex or abstract messages might alienate those lacking the context or the expertise to interpret them. Number two, cognitive bias in perception. The study revealed that people rated the same poem higher when they were told it was written by a human rather than by ai. Another way of saying this is branding counts, reputations count. And number three, AI is a communication tool, not a replacement. AI excels in generating widely appealing content by mimicking existing patterns and evoking emotions, making it a powerful collaborator for creating content. If you'd like to send a message to my guest, Brian Porter, we've got his information at the University of Pittsburgh in the show notes, stories and strategies as a co-production of JGR Communications and Stories and strategies, podcasts. Hey, check out our YouTube channel. We just eclipsed a hundred thousand views on YouTube. We're kind of pumped about that. Full episodes on YouTube. Thanks to Producer Emily Page. I hope you have a fantastic holiday season. Whatever your personal or family tradition is, I hope it makes you happy this year. Happy is good. One last note, please forward this episode to one friend. Thanks for listening.