New article - how did this get published? « Political Science Rumors

1 year - 92 posts - Latest - RSS
OP: 32 Goods vs 0 No Goods
Thread: 301 Goods vs 108 No Goods

Political Science Rumors » Political Science » Research / Journals

New article - how did this get published?

12…5 Next »

Lysette

New article just published online in Social Science Quarterly on attitudes about gun violence.

Authors take eight separate polls on gun violence from 1999 to 2018. They predict whether or not respondents endorse this phrase: “Government and society can take action that will be effective in preventing shootings" (i.e., a dichotomous dependent variable) using respondent age as their main independent variable.

For the earliest four polls (1999, 2000, 2001, 2011) they use OLS (?) and they find no significant effect of age. For the latest four polls (2012, 2015, 2017, 2018) they use logistic regression and they do find a significant effect of age.

From this they conclude that young people make up a "massacre generation" that reacts differently to shootings than older people.

"We believe there is no age effect prior to 2012 because mass and active shootings had not yet become a major issue in public discourse and therefore did not yet trigger issue saliency in younger generations." They actually say that.

There are just so many things wrong here. Granted, this is not a top journal, but it's still a moderately well respected peer-reviewed journal with lots of big names on the editorial board. How did this get published?

1 year ago # QUOTE 32 YEA 0 NAY !
Florence

SSQ is for new ideas that might be right but are hard to nail down. Especially when they’re kind of provocative. It gives a lifeline in the literature until people can figure out how to show it.

1 year ago # QUOTE 5 YEA 8 NAY !
Trent

For the earliest four polls (1999, 2000, 2001, 2011) they use OLS (?) and they find no significant effect of age. For the latest four polls (2012, 2015, 2017, 2018) they use logistic regression and they do find a significant effect of age.

r u for real

1 year ago # QUOTE 10 YEA 0 NAY !
Pauleen

SSQ has a very strict replication policy:

“It is the policy of SSQ to publish papers only if the empirical data used in the analysis are clearly and precisely documented and are readily available to other researchers for purpose of replication. Details of the computations sufficient to permit replication must be provided to other scholars on request. The Editor should be informed upon submission if, for some reason, this requirement cannot be met. Our policy does not mean that the data could be reconstructed from some source (e.g., the Census or the ICPSR) but rather that the author agrees to make the data set available to other interested scholars.“

So if it’s a big deal to you, you can just email the authors and ask for the data.

1 year ago # QUOTE 8 YEA 1 NAY !
Read

For the earliest four polls (1999, 2000, 2001, 2011) they use OLS (?) and they find no significant effect of age. For the latest four polls (2012, 2015, 2017, 2018) they use logistic regression and they do find a significant effect of age.
r u for real

Different models using different samples produce different results. Science!

1 year ago # QUOTE 13 YEA 0 NAY !
Jewell

I didn't spend too much time reading it closely, but I didn't see any justification for using OLS v. logistic regression. Maybe I missed it. Doubt it affected results. But still. And it is probably a typo, but the 2015 sample size varies in the two tables.

1 year ago # QUOTE 3 YEA 1 NAY !
Miriam

what does "government and society" mean? what is "society"? How can it take action?

1 year ago # QUOTE 2 YEA 3 NAY !
Pauleen

Could be a chance for Karl to redeem himself after his recent flops.

SSQ has a very strict replication policy:
“It is the policy of SSQ to publish papers only if the empirical data used in the analysis are clearly and precisely documented and are readily available to other researchers for purpose of replication. Details of the computations sufficient to permit replication must be provided to other scholars on request. The Editor should be informed upon submission if, for some reason, this requirement cannot be met. Our policy does not mean that the data could be reconstructed from some source (e.g., the Census or the ICPSR) but rather that the author agrees to make the data set available to other interested scholars.“
So if it’s a big deal to you, you can just email the authors and ask for the data.

1 year ago # QUOTE 5 YEA 0 NAY !
Yasmin

I didn't spend too much time reading it closely, but I didn't see any justification for using OLS v. logistic regression. Maybe I missed it. Doubt it affected results. But still. And it is probably a typo, but the 2015 sample size varies in the two tables.

You didn’t miss it. It’s not there.

There is no good reason to use different methods to run the same models with same IVs and same DV, unless this specification gave them the result they were looking for.

1 year ago # QUOTE 6 YEA 1 NAY !
Grier

I'm guess the logit model choice has more to do with making the substantive effects. You can still find the size of the effects by looking at the odds, but it is less intuitive.

I know these researchers. They seem like nice people, but one of them said they got their position (with no publication at the time) because they had a "nice teaching statement."

1 year ago # QUOTE 3 YEA 1 NAY !
Galadriel

Here's a nice teaching statement
https://twitter.com/torrespolisci/status/1525473610273538048

1 year ago # QUOTE 1 YEA 2 NAY !
Pierrick

None of this has any relationship to reality -- as none of quant does.

1 year ago # QUOTE 0 YEA 6 NAY !
Weston

2011: age no sig, not massacre generation
2012: age sig, massacre generation

I am a scientist.

1 year ago # QUOTE 13 YEA 0 NAY !
Weston

In the first table, the age coefficient is negative, extremely small, and non significant.

In the second table, the age coefficient is negative, extremely small, and significant.

What’s the difference between the two tables?

Along with the shift in method between the two tables, it should be noted (but isn’t in the article) that the sample sizes for the polls in the second table are way bigger—sometimes 2-3 times bigger, than the sample sizes for the polls in the first table.

What a mess of a paper.

1 year ago # QUOTE 14 YEA 0 NAY !
Dax

Those who can't do, criticize

1 year ago # QUOTE 0 YEA 17 NAY !
Weston

Those who can't do, criticize

This, of course, is the opposite of the truth. It is those who have the most experience, and the strongest skills, who have the greatest ability to see through shoddy research. They make the best critics. This is why we have the system of peer review. That’s the whole point of it.

Which brings us back to the title of the original post. How is it possible that this paper made it through peer review?

1 year ago # QUOTE 19 YEA 0 NAY !
Lloyd

I didn't spend too much time reading it closely, but I didn't see any justification for using OLS v. logistic regression. Maybe I missed it. Doubt it affected results. But still. And it is probably a typo, but the 2015 sample size varies in the two tables.

Yikes.

Also, they don't report SEs, but do report t statistics for each coefficient, and at least one of the variables reported as significant at p<.001 in Table 1 is actually NS, according to their own reported t statistic.

This article is a perfect candidate for replication. The regressions are so simple that if you got the data you could replicate it the whole thing in less than an hour.

At the very least, a detailed correction is in order. Editors should seriously consider retraction.

1 year ago # QUOTE 12 YEA 0 NAY !
Quianna

They report sample sizes for all eight polls; in nearly all cases, the sample sizes differ from the number of cases reported in the tables, sometimes by more than 700 cases. No explanation is given for why data were missing or how missing data were handled. There are no descriptive statistics in the article to help the reader understand the characteristics of the various samples.

1 year ago # QUOTE 7 YEA 0 NAY !
Renard

I've been rejected from journals roughly on this level (can't remember if I ever sent to SSQ itself, probably not)for papers that were way better than this. Not necessarily fancy methods, but not this kind of sleight of hand.

1 year ago # QUOTE 5 YEA 0 NAY !
Elle

Not the first poli Sci fraud on gun violence effects. The other is more sophisticated but also much better published.

1 year ago # QUOTE 0 YEA 1 NAY !

12…5 Next »

New article - how did this get published?

Reply »