The test seems flawed. I got the first one (after the attention check) wrong - I said it was fake, but it was real. I thought it was fake because Trump's hair was clearly very smeared out, not at all realistic. But it seems that this must have been an artifact of heavy video compression. I don't see how this is a meaningful test of anything.
Yeah, but how is the test taker supposed to know whether to take the smearing as indicating it's fake or not?
Better would be to let them see 10 videos all at once, half of which are fake, and ask them to divide into a fake set of 5 and a real set of 5, after looking at all of them as many times as they like. Asking "fake or real" when there is no basis to tell whether flaws should be taken as indicating "fake" or just attributed to compression seems meaningless.
Or tell people what aspects of fakeness they're trying to assess - eg, forget about video artifacts, just pay attention to the audio.
Using clips of Trump and Biden is also a bad idea. They ask you to say if you've seen one before, but aren't many people going to have seen one, but not clearly remember that, and then be influenced to think it's real by sub-conscious recognition?
Why not present pairs of videos of the same non-famous person, one fake one real, both presented with the same amount of compression, and ask one to chose which is the real one? Using many different people, of course - why would you introduce doubt about the generality of your results by using only two people?
Of course, in practice people may be less able to recognize fakes when video quality is poor, which would be useful to know, but I think one would need to investigate that issue separately, not in combination with other reasons that fakes might or might not be recognizable.
