SEs of Predicted Probabilities in Logit « Political Science Rumors

8 years - 30 posts - Latest - RSS
OP: 0 Goods vs 3 No Goods
Thread: 106 Goods vs 32 No Goods

Political Science Rumors » Off Topic » Off Topic

SEs of Predicted Probabilities in Logit

« Previous 12

Nell

Ok, I’m back. Not a lot of lines of code to go through, actually.

This is what he says he does in Fig.1:

“Left panel estimated from probit model with offer size, treatment condition, interaction between offer and treatment, and dichotomous variable for the repetition of the experiment … Standard errors clustered at individual level and confidence intervals calculated using a parametric bootstrap running for 1000 iterations.”

Sample size is very small (only 72 students), so I can see why a reviewer asked him to bootstrap his data. Also, data is clustered, so simple bootstrap won’t do.

Replication data is here: http://hdl.handle.net/1902.1/14820

From his readme file:

“The .R and .do files provide all of the statistical analyses.”

There are only one of each. Stata file only does tests of equality of proportion (prtest).
R file has the Zelig code for the probit models:

ProbReject<-zelig(OfferReject~Delta*SubjAOffer+ repetition , model="probit.gee", robust=TRUE, id="ID", corstr = "exchangeable", data=ExpData)

Several such models, none using the bootstrap option.

Zelig documentation says that if bootstrap argument is not specified, the default is bootstrap=FALSE. That is, no bootstrap.

http://docs.zeligproject.org/articles/bootstraps.html

“The zelig() bootstrap argument has a default of FALSE, and can be set to TRUE or a numeric value giving the number of bootstrapped datasets to run. If set to TRUE the default is 100 bootstraps. The bootstrap works in combination with other Zelig arguments as follows:
…

Examples:

z.out <- zelig(unem ~ gdp + capmob + trade, model = "ls", data = macro, bootstrap = 500)“

Oops, I just couldn’t find any evidence of bootstrap in the code. Check for yourselves and report here if you find any.

3 years ago # QUOTE 5 YEA 2 NAY !
Garrick

Ok, I’m back. Not a lot of lines of code to go through, actually.
This is what he says he does in Fig.1:
“Left panel estimated from probit model with offer size, treatment condition, interaction between offer and treatment, and dichotomous variable for the repetition of the experiment … Standard errors clustered at individual level and confidence intervals calculated using a parametric bootstrap running for 1000 iterations.”
Sample size is very small (only 72 students), so I can see why a reviewer asked him to bootstrap his data. Also, data is clustered, so simple bootstrap won’t do.
Replication data is here: http://hdl.handle.net/1902.1/14820
From his readme file:
“The .R and .do files provide all of the statistical analyses.”
There are only one of each. Stata file only does tests of equality of proportion (prtest).
R file has the Zelig code for the probit models:
ProbReject<-zelig(OfferReject~Delta*SubjAOffer+ repetition , model="probit.gee", robust=TRUE, id="ID", corstr = "exchangeable", data=ExpData)
Several such models, none using the bootstrap option.
Zelig documentation says that if bootstrap argument is not specified, the default is bootstrap=FALSE. That is, no bootstrap.
http://docs.zeligproject.org/articles/bootstraps.html
“The zelig() bootstrap argument has a default of FALSE, and can be set to TRUE or a numeric value giving the number of bootstrapped datasets to run. If set to TRUE the default is 100 bootstraps. The bootstrap works in combination with other Zelig arguments as follows:
…
Examples:
z.out <- zelig(unem ~ gdp + capmob + trade, model = "ls", data = macro, bootstrap = 500)“
Oops, I just couldn’t find any evidence of bootstrap in the code. Check for yourselves and report here if you find any.

For those of us too stupid/not trained in quant, this means that the author lied about the bootstrap right? If so, what repercussions does it have for the results?

3 years ago # QUOTE 5 YEA 2 NAY !
Damien

You can simulate coefficients from a normal distribution and then do the calculations matrix algebra + inverse logit.

Look at Gelman & Hill, they have a chapter on simulation (R code)

Or look at the Hanmer and Kalkan AJPS observed value approach (STATA code)

3 years ago # QUOTE 3 YEA 1 NAY !
Nell

For those of us too stupid/not trained in quant, this means that the author lied about the bootstrap right? If so, what repercussions does it have for the results?

Could be negligence, not necessarily lying... though six models out of six is a lot of negligence. You'd have to re-run his code with the bootstrap option to see how results would change.

3 years ago # QUOTE 4 YEA 1 NAY !
Edgar

LOL, what would you expect from Hatemi's pal? He should have taken lessons from RE on how to make it hard to get busted. Split the code across lots of files and make it so messy no one would ever have the patience to go through it.

3 years ago # QUOTE 7 YEA 0 NAY !
Sylviane

Formal theorist here. I can't speak to the bootstrap question or to whether this guy is p-hacking on not. I'll leave that to you empirical quants. But he misunderstood and butchered Fearon's model in that paper.

3 years ago # QUOTE 6 YEA 2 NAY !
Finnegan

You can't just re-run his probit.gee code with the option to bootstrap within clusters because that's still not currently supported in Zelig. So it can't be a matter of negligence (say he forgot to add the bootstrap argument to the zelig function in all six models). If the reviewer asked him to bootstrap because of his small sample size he would have needed to write his own bootstrapping code, matching data within clusters. The Zelig documentation confirms this (see link). There is no code for the bootstrap in the replications files, and the figures are obviously done with the simple (non-bootstrapped estimates). It would have been very difficult for someone not to realize that they are making a false statement in the method description, and that they had not, in fact, written any such code. At AJPS, this would have likely not cleared the Odum institute, but the ISQ team didn't verify his analyses.

http://docs.zeligproject.org/articles/zelig_probitgee.html

The method of bootstrapping generally should not be used in GEE models. If you must bootstrap, bootstrapping should be done within clusters, which is not currently supported in Zelig. For conditional prediction models, data should be matched within clusters.

For those of us too stupid/not trained in quant, this means that the author lied about the bootstrap right? If so, what repercussions does it have for the results?

Could be negligence, not necessarily lying... though six models out of six is a lot of negligence. You'd have to re-run his code with the bootstrap option to see how results would change.

3 years ago # QUOTE 5 YEA 0 NAY !
Damien

Sample size is very small (only 72 students)

I don't anything about this paper, but there are a lot of debates about the lowest number of observations and there is not an official number on it.

3 years ago # QUOTE 1 YEA 1 NAY !
Finnegan

It's certainly possible to disagree with a reviewer who says your sample size is too small. But then why not be honest and just say so to the reviewers and the editors? Why mislead them with the claim that significant effects and low p-values are still there after you wrote the bootstrap code, which you know you never did? Would they still have accepted the paper if you told them the truth? And if you lied about something this in the paper, why would I trust anything else in your research?

Sample size is very small (only 72 students)

I don't anything about this paper, but there are a lot of debates about the lowest number of observations and there is not an official number on it.

3 years ago # QUOTE 4 YEA 0 NAY !
Leann

This is also part of his dissertation and likely his job market paper (I can't recall exactly). Isn't this just another Lacour who wasn't caught early enough? How many more people like this are there in the discipline?

3 years ago # QUOTE 5 YEA 1 NAY !

« Previous 12

SEs of Predicted Probabilities in Logit

Reply