How could a paper that makes every rookie mistake in intro quant analysis get published?
JOP article on prohibition and representation
-
What are the mistakes?
For starters, OLS assumptions don't hold as their DV is a proportion, they match on a post-treatment variable that is influenced by their treatment variable (lots of endogeneity going on), they lose 60% of observations when matching, they also use 0/1 votes to adjourn that have nothing to do with the bill content, they interpret interaction effects incorrectly, they aggregate data over several years ...
-
This is pretty common in the "new" HPE subfield. They take some pretty bad data from archives and beat it to death.
No, this is a new low. Their conclusion is basically: on 40% of observations that we could match because they were close in Mahalanobis distance on a post-treatment variable, we find that there are no differences in representation between treatment and control group. Laughable.
Also, set.seed(53074) for the matching looks a lot like cherrypicking.
-
This is the best bit:
The Author:
https://polisci.wustl.edu/people/michael-olson
Bentley from the Jeffersons:
-
What are the mistakes?
For starters, OLS assumptions don't hold as their DV is a proportionYou lose. Next.
You're correct on this point, but the other points are valid, especially the point about matching on a post-treatment variable. It's a publishable paper, but probably not one that JoP should publish.
-
What would be a better way to do this paper?
What are the mistakes?
For starters, OLS assumptions don't hold as their DV is a proportion
You lose. Next.You're correct on this point, but the other points are valid, especially the point about matching on a post-treatment variable. It's a publishable paper, but probably not one that JoP should publish.
-
So do you think their claims are false?
Could you have done it in a better way and shown that?
If so, you could maybe get a publication.
If you are just going down a methods checklist and stressing things that won't matter in practice, not so much.The authors claim they "push back" against previous literature on parties. And they base that wild claim on very crude methods which they apply in the most clueless possible way.
-
What would be a better way to do this paper?
What are the mistakes?
For starters, OLS assumptions don't hold as their DV is a proportion
You lose. Next.
You're correct on this point, but the other points are valid, especially the point about matching on a post-treatment variable. It's a publishable paper, but probably not one that JoP should publish.You appear to assume that if there isn't a better way to do this paper, JoP should publish it. That's not a reasonable assumption to make. As for a better way to do this paper, perhaps you'll get a chance to review it in the future.
-
Yes, I am problem-driven, not methods-driven, unlike some people.
What would be a better way to do this paper?
What are the mistakes?
For starters, OLS assumptions don't hold as their DV is a proportion
You lose. Next.
You're correct on this point, but the other points are valid, especially the point about matching on a post-treatment variable. It's a publishable paper, but probably not one that JoP should publish.You appear to assume that if there isn't a better way to do this paper, JoP should publish it. That's not a reasonable assumption to make. As for a better way to do this paper, perhaps you'll get a chance to review it in the future.
-
What are the mistakes?
For starters, OLS assumptions don't hold as their DV is a proportion
You lose. Next.Dependent variable is a scaled binomial with range 0-100. OLS linearity assumption is violated. Predicted values can fall below 0 or go above 100.
We are literally going to have the same basic fight every generation until we die, aren’t we?
OLS parameters are still unbiased in this scenario. That’s not something to give up lightly.