of scientific studies had to toss out weeks of data because of a 56-second TikTok video by a teenager.
The July 23rd video is short and simple. It opens with recent Florida high school graduate and self-described “teen author” Sarah Frank sitting in her bedroom and smiling at the camera.
“Welcome to side hustles I recommend trying — part one,” she says in the video, pointing users to the website Prolific.co. “Basically, it’s a bunch of surveys for different amounts of money and different amounts of time.”
That video got 4.1 million views in the month after it was posted and sent tens of thousands of new users flooding to the Prolific platform. Prolific, a tool for scientists conducting behavioral research, had no free screening tools in place to make sure that it delivered representative population samples to each study. Suddenly, scientists used to getting a wide mix of subjects for their Prolific studies saw their surveys flooded with responses from young women around Frank’s age.
For researchers who rely on representative samples of the US population, that demographic shift was a major problem with no obvious cause and no immediately clear way to fix.
DOING SCIENCE ON THE INTERNET
Though not particularly well known, Prolific is part of a small collection of online tools that have transformed the way corporations and scientists study the way people think and act. The first and largest of these research platforms is Amazon-owned Mechanical Turk, which was released in 2005 as a general-purpose platform for crowdsourcing work on repetitive tasks. Soon after it was released, behavioral scientists realized its potential value for their research, and it quickly revolutionized several research fields.
“Before Mechanical Turk existed, all social science research had to happen in the laboratory. You’d need to bring in college sophomores and put them through questionnaires and surveys and whatnot,” said Nicholas Hall, director of the Behavioral Lab at the Stanford School of Business.
“That’s an enormous time- and labor-intensive endeavor. Online research makes it so much easier. You program a survey... you put it online, and within a day, you have 1,000 responses,” Hall told The Verge. “That changed the face of social science.”
The Behavioral Lab at Stanford mainly uses the newer, smaller Prolific platform for online studies these days, Hall said. While many Mechanical Turk customers are big businesses conducting corporate research, Prolific gears its product to scientists.
The smaller platform offers more transparency, promises to treat survey participants more ethically, and promises higher-quality research subjects than alternative platforms like Mechanical Turk.
Scientists doing this sort of research in the United States generally want a pool of subjects who speak English as a first language, are not too practiced at taking psychological surveys, and together make up a reasonably representative demographic sample of the American population.
Prolific, most agreed, did a good job providing high-quality subjects. The sudden change in the platform’s demographics threatened to upend that reputation.
TIKTOK AFTERMATH
In the days and weeks after Frank posted her video, researchers scrambled to figure out what was happening to their studies.
A member of the Stanford Behavioral Laboratory posted on a Prolific forum, “We have noticed a huge leap in the number of participants on the platform in the US Pool, from 40k to 80k. Which is great, however, now a lot of our studies have a gender skew where maybe 85% of participants are women. Plus the age has been averaging around 21.”
Wayne State psychologist Hannah Schechter seems to have been the first person to crack the case.
“This may be far-fetched,” she tweeted, linking to Frank’s video, “but given the timing, virality of the video, and the user’s follower demographics....”
Long-standing Prolific survey-takers complained on Reddit that Frank had made it difficult to find paid surveys to take on the overrun platform.
“Now it’s just another bullshit site to spend hours and make pennies on,” wrote one user, who said they had previously made $30 a week on the platform.
Frank, who “guesstimated” she had made a total of about $80 taking surveys on Prolific before her video, told The Verge she also noticed a difference on the platform.
“Less studies have been available for me and everyone else,” she told The Verge. “I’ve received some really mean comments accusing me of single-handedly ruining the site and being selfish — even though I received no compensation for that video.”
She added that she hoped Prolific would be able to set up a system to deal with its changed demographics.
“I also predict that a lot of people who signed up after seeing my video will forget about it, and the surge will die down,” she said.
Prolific co-founder and CTO Phelim Bradley told The Verge that many of the new users do seem to be dropping off.
“Prior to Tiktok, about 50% of the responses on our platform came from women,” he wrote in an email. “The surge knocked this up as high as 75% for a few days, but since then, this number has been trending down, and we’re currently back to ~60% of responses being from women.”
:no_upscale()/cdn.vox-cdn.com/uploads/chorus_asset/file/22873438/2CC6A4E2_1910_45C9_9FF7_310DB9DA5B9C.jpg)
According to Bradley, about 4,600 studies were disrupted by Frank’s TikTok, around a third of the total that were active on the platform during the surge. Of those, he said, the vast majority should be salvageable.
Prolific has refunded researchers whose studies were significantly impacted by the surge in women survey takers and introduced a new suite of demographic screening tools. The company announced these steps a month after Frank posted her video. The company has now re-organized, putting a team in charge of demographic balancing in order to more quickly recognize and respond to this sort of problem in the future.
“Honestly, we were somewhat caught by surprise, and we didn’t predict how large the impact was going to be,” Bradley said.
The surge isn’t all bad. Refreshing the pool of survey takers probably has long-term benefits, says Vlad Chituk, a Yale graduate student in psychology who was running several pilot studies on Prolific when the surge hit. When subjects take lots of psychological surveys, they learn the tricks scientists use to gather data, and that can impact the way they respond to future survey questions. Fresh subjects provide higher-quality data.
“Young women who enjoy TikTok are people too,” he said.
As for Frank, she says that her side hustle video is now the most popular TikTok she’s ever posted.
“It definitely didn’t occur to me that the video would blow up. I just posted it for my friends and followers, not for the reach it ended up getting,” she said. “I think it blew up because the site is genuinely so cool, and people love efficient ways to make money.”
For the time being, Frank has paused most of her own side hustles as she settles into her freshman year at Brown.
Correction 9/27: A source at Prolific originally said that the company didn’t have demographic balancing in place in July. He informed us after publication that they did have a gender screening tool available for a fee at the time.
