Jakana Thomas' IO article on "Wolves in Sheep's Clothing" and replication data

7 months - 232 posts - Latest - RSS
OP: 10 Goods vs 2 No Goods
Thread: 705 Goods vs 393 No Goods

Political Science Rumors » Political Science » Research / Journals

Jakana Thomas' IO article on "Wolves in Sheep's Clothing" and replication data

« Previous 123…12 Next »

Merla

There are also differences in the expanded and non-expanded datasets. If you use the non-expanded dataset, you can replicate the results perfectly to model 2 but you can't estimate table 1. If you use the expanded dataset, you can replicate table 1 but the number for table 2 are slightly difference as outlined above. Didn't read the article or appendix so I don't know what the differences are (lol).
Either way, seems like the biggest issue here is reporting 1 tailed test and someone replicating and seeing 2 tailed test and freaking out.

I can assure you the article uses the term, de"tail", but never 1 "tail"ed test.

I guess we can publish our work with p-values likes these. What journals?

p-value for the first replicated coefficient: 0.682
The second: 0.131
The third: 0.027
The fourth: 0.142

7 months ago # QUOTE 2 YEA 2 NAY !
Bettie

I don't care whether the paper replicates. It was bad to begin with.

7 months ago # QUOTE 7 YEA 1 NAY !
Merla

"Authors who report that results are “statistically significant” should use the 0.05 level or lower. Statistical tests using the 0.10 level are not acceptable for work submitted to the AJPS."

There are also differences in the expanded and non-expanded datasets. If you use the non-expanded dataset, you can replicate the results perfectly to model 2 but you can't estimate table 1. If you use the expanded dataset, you can replicate table 1 but the number for table 2 are slightly difference as outlined above. Didn't read the article or appendix so I don't know what the differences are (lol).
Either way, seems like the biggest issue here is reporting 1 tailed test and someone replicating and seeing 2 tailed test and freaking out.

I can assure you the article uses the term, de"tail", but never 1 "tail"ed test.
I guess we can publish our work with p-values likes these. What journals?
p-value for the first replicated coefficient: 0.682
The second: 0.131
The third: 0.027
The fourth: 0.142

7 months ago # QUOTE 2 YEA 0 NAY !
Poppie

"Authors who report that results are “statistically significant” should use the 0.05 level or lower. Statistical tests using the 0.10 level are not acceptable for work submitted to the AJPS."
There are also differences in the expanded and non-expanded datasets. If you use the non-expanded dataset, you can replicate the results perfectly to model 2 but you can't estimate table 1. If you use the expanded dataset, you can replicate table 1 but the number for table 2 are slightly difference as outlined above. Didn't read the article or appendix so I don't know what the differences are (lol).
Either way, seems like the biggest issue here is reporting 1 tailed test and someone replicating and seeing 2 tailed test and freaking out.
I can assure you the article uses the term, de"tail", but never 1 "tail"ed test.
I guess we can publish our work with p-values likes these. What journals?
p-value for the first replicated coefficient: 0.682
The second: 0.131
The third: 0.027
The fourth: 0.142

OP received a B in her class.

7 months ago # QUOTE 1 YEA 4 NAY !
Michelle

Can you please use PubPeer to publish critiques against that paper?

https://pubpeer.com/ => look for DOI => add critique

7 months ago # QUOTE 0 YEA 0 NAY !
Sheila

Think IO replicated it before publishing it? I doubt it, which is bad enough. But if they did replicate it and didn't ask JT to specify one-tailed tests - or did so and didn't care - that's far worse.

7 months ago # QUOTE 0 YEA 2 NAY !
Gerrard

And you shouldn't really report one-tailed tests. If you do, you have to be very explicit.

"Authors who report that results are “statistically significant” should use the 0.05 level or lower. Statistical tests using the 0.10 level are not acceptable for work submitted to the AJPS."
There are also differences in the expanded and non-expanded datasets. If you use the non-expanded dataset, you can replicate the results perfectly to model 2 but you can't estimate table 1. If you use the expanded dataset, you can replicate table 1 but the number for table 2 are slightly difference as outlined above. Didn't read the article or appendix so I don't know what the differences are (lol).
Either way, seems like the biggest issue here is reporting 1 tailed test and someone replicating and seeing 2 tailed test and freaking out.
I can assure you the article uses the term, de"tail", but never 1 "tail"ed test.
I guess we can publish our work with p-values likes these. What journals?
p-value for the first replicated coefficient: 0.682
The second: 0.131
The third: 0.027
The fourth: 0.142

7 months ago # QUOTE 6 YEA 0 NAY !
Gerrard

What difference does that make?

The results either replicate or not. It has nothing to do with OP's stat level.

"Authors who report that results are “statistically significant” should use the 0.05 level or lower. Statistical tests using the 0.10 level are not acceptable for work submitted to the AJPS."
There are also differences in the expanded and non-expanded datasets. If you use the non-expanded dataset, you can replicate the results perfectly to model 2 but you can't estimate table 1. If you use the expanded dataset, you can replicate table 1 but the number for table 2 are slightly difference as outlined above. Didn't read the article or appendix so I don't know what the differences are (lol).
Either way, seems like the biggest issue here is reporting 1 tailed test and someone replicating and seeing 2 tailed test and freaking out.
I can assure you the article uses the term, de"tail", but never 1 "tail"ed test.
I guess we can publish our work with p-values likes these. What journals?
p-value for the first replicated coefficient: 0.682
The second: 0.131
The third: 0.027
The fourth: 0.142

OP received a B in her class.

7 months ago # QUOTE 5 YEA 0 NAY !
Poppie

What difference does that make?
The results either replicate or not. It has nothing to do with OP's stat level.
"Authors who report that results are “statistically significant” should use the 0.05 level or lower. Statistical tests using the 0.10 level are not acceptable for work submitted to the AJPS."
There are also differences in the expanded and non-expanded datasets. If you use the non-expanded dataset, you can replicate the results perfectly to model 2 but you can't estimate table 1. If you use the expanded dataset, you can replicate table 1 but the number for table 2 are slightly difference as outlined above. Didn't read the article or appendix so I don't know what the differences are (lol).
Either way, seems like the biggest issue here is reporting 1 tailed test and someone replicating and seeing 2 tailed test and freaking out.
I can assure you the article uses the term, de"tail", but never 1 "tail"ed test.
I guess we can publish our work with p-values likes these. What journals?
p-value for the first replicated coefficient: 0.682
The second: 0.131
The third: 0.027
The fourth: 0.142
OP received a B in her class.

It matters about as much as AJPS' publishing guidelines matter for a paper published in IO.

7 months ago # QUOTE 3 YEA 0 NAY !
Geneva

Think IO replicated it before publishing it? I doubt it, which is bad enough. But if they did replicate it and didn't ask JT to specify one-tailed tests - or did so and didn't care - that's far worse.

IO requires all results to replicate before they proceed with publication. Including the appendix. They'll kick the replication files back to you for correction if something doesn't hold up. Not sure what happened here...

7 months ago # QUOTE 4 YEA 0 NAY !
Candice

Can I ask one thing...assuming the study wasn't poorly done...what's significant about the findings? How does this advance our understanding of politics, suicide terrorism, or extremism at all?

7 months ago # QUOTE 4 YEA 0 NAY !
Cindy

Can I ask one thing...assuming the study wasn't poorly done...what's significant about the findings? How does this advance our understanding of politics, suicide terrorism, or extremism at all?

It doesn't.

7 months ago # QUOTE 1 YEA 0 NAY !
Merla

Can I ask one thing...assuming the study wasn't poorly done...what's significant about the findings? How does this advance our understanding of politics, suicide terrorism, or extremism at all?

Can I ask one thing...assuming the study wasn't poorly done...what's significant about the findings? How does this advance our understanding of politics, suicide terrorism, or extremism at all?

Nothing. The findings can't be verifiable, so the article is inseminating erroneous findings about female suicide terror.

The article argues that females are more lethal weapons than males; however, the replicated results show no gender differences whatsoever.

7 months ago # QUOTE 1 YEA 0 NAY !
Madonna

With all due respect to JT, this just smells like p-hacking.

The lethality of suicide terrorism and women's civil society participation? I mean, come on

7 months ago # QUOTE 9 YEA 0 NAY !
Poppie

Why would somebody p-hack a flimsy result? And if the results don't replicate, there is something else going on here, something potentially much worse than p-hacking. It could also be benign error.

As for the whole "smells like p-hacking" trend we're seeing in how people discuss results, all this means is that the result isn't immediately obvious to the person. If the theory isn't logically consistent, assumes things it should not, or is missing things that should be considered, that's a problem and may indicate p-hacking. Seeing a variable you never thought to relate to some other variable does not indicate somebody p-hacked the result. In this case, it surprised me that it had taken so long for somebody to run this study.

7 months ago # QUOTE 1 YEA 6 NAY !
Roderick

Default in every common statistical function in stata or r is to do a two tailed test, JT using a one tailed test to pass the, admittedly, arbitrary bar of significance is clear fraud.

7 months ago # QUOTE 0 YEA 1 NAY !
Herman

Clearly you don’t know what p-hacking means

Why would somebody p-hack a flimsy result? And if the results don't replicate, there is something else going on here, something potentially much worse than p-hacking. It could also be benign error.
As for the whole "smells like p-hacking" trend we're seeing in how people discuss results, all this means is that the result isn't immediately obvious to the person. If the theory isn't logically consistent, assumes things it should not, or is missing things that should be considered, that's a problem and may indicate p-hacking. Seeing a variable you never thought to relate to some other variable does not indicate somebody p-hacked the result. In this case, it surprised me that it had taken so long for somebody to run this study.

7 months ago # QUOTE 4 YEA 0 NAY !
Brice

The table caption on table 2 does report a one tailed test. I'm not sure if it's reported elsewhere in the main text. And unfortunately this isn't something that reviewers are likely to notice (although they will notice that typo on p. 5 bc they are very smart).

7 months ago # QUOTE 0 YEA 1 NAY !
Roderick

The table caption on table 2 does report a one tailed test. I'm not sure if it's reported elsewhere in the main text. And unfortunately this isn't something that reviewers are likely to notice (although they will notice that typo on p. 5 bc they are very smart).

Then who cares this is a non-issue. Another example of IR work that is poor on the empirical side.

7 months ago # QUOTE 2 YEA 0 NAY !
Jodie

I'm sure OP is one of the racist faculty members of the MSU political science department. They are just but hurt.

7 months ago # QUOTE 0 YEA 4 NAY !

« Previous 123…12 Next »

Jakana Thomas' IO article on "Wolves in Sheep's Clothing" and replication data

Reply »