Political Science Rumors Topic: SEs of Predicted Probabilities in Logit
https://www.poliscirumors.com/topic/ses-of-predicted-probabilities-in-logit
Political Science Rumors Topic: SEs of Predicted Probabilities in Logiten-USWed, 28 Jul 2021 23:26:13 +0000http://bbpress.org/?v=1.0.2<![CDATA[Search]]>q
https://www.poliscirumors.com/search.php
PoliSci on "SEs of Predicted Probabilities in Logit"
https://www.poliscirumors.com/topic/ses-of-predicted-probabilities-in-logit/page/2#post-1048640
Fri, 04 May 2018 20:44:39 +0000PoliSci1048640@https://www.poliscirumors.com/<p>This is also part of his dissertation and likely his job market paper (I can't recall exactly). Isn't this just another Lacour who wasn't caught early enough? How many more people like this are there in the discipline?
</p>PoliSci on "SEs of Predicted Probabilities in Logit"
https://www.poliscirumors.com/topic/ses-of-predicted-probabilities-in-logit/page/2#post-1048008
Fri, 04 May 2018 04:59:23 +0000PoliSci1048008@https://www.poliscirumors.com/<p>It's certainly possible to disagree with a reviewer who says your sample size is too small. But then why not be honest and just say so to the reviewers and the editors? Why mislead them with the claim that significant effects and low p-values are still there after you wrote the bootstrap code, which you know you never did? Would they still have accepted the paper if you told them the truth? And if you lied about something this in the paper, why would I trust anything else in your research?</p>
<blockquote><p>
Sample size is very small (only 72 students)</p>
<p>I don't anything about this paper, but there are a lot of debates about the lowest number of observations and there is not an official number on it.
</p></blockquote>PoliSci on "SEs of Predicted Probabilities in Logit"
https://www.poliscirumors.com/topic/ses-of-predicted-probabilities-in-logit/page/2#post-1048004
Fri, 04 May 2018 04:45:42 +0000PoliSci1048004@https://www.poliscirumors.com/<blockquote><p>
Sample size is very small (only 72 students)
</p></blockquote>
<p>I don't anything about this paper, but there are a lot of debates about the lowest number of observations and there is not an official number on it.
</p>PoliSci on "SEs of Predicted Probabilities in Logit"
https://www.poliscirumors.com/topic/ses-of-predicted-probabilities-in-logit/page/2#post-1047990
Fri, 04 May 2018 04:29:08 +0000PoliSci1047990@https://www.poliscirumors.com/<p>You can't just re-run his probit.gee code with the option to bootstrap within clusters because that's still not currently supported in Zelig. So it can't be a matter of negligence (say he forgot to add the bootstrap argument to the zelig function in all six models). If the reviewer asked him to bootstrap because of his small sample size he would have needed to write his own bootstrapping code, matching data within clusters. The Zelig documentation confirms this (see link). There is no code for the bootstrap in the replications files, and the figures are obviously done with the simple (non-bootstrapped estimates). It would have been very difficult for someone not to realize that they are making a false statement in the method description, and that they had not, in fact, written any such code. At AJPS, this would have likely not cleared the Odum institute, but the ISQ team didn't verify his analyses. </p>
<p><a href="http://docs.zeligproject.org/articles/zelig_probitgee.html" rel="nofollow">http://docs.zeligproject.org/articles/zelig_probitgee.html</a></p>
<blockquote><p>
The method of bootstrapping generally should not be used in GEE models. If you must bootstrap, bootstrapping should be done within clusters, which is not currently supported in Zelig. For conditional prediction models, data should be matched within clusters.
</p></blockquote>
<blockquote><p>
For those of us too stupid/not trained in quant, this means that the author lied about the bootstrap right? If so, what repercussions does it have for the results?</p>
<p>Could be negligence, not necessarily lying... though six models out of six is a lot of negligence. You'd have to re-run his code with the bootstrap option to see how results would change.
</p></blockquote>PoliSci on "SEs of Predicted Probabilities in Logit"
https://www.poliscirumors.com/topic/ses-of-predicted-probabilities-in-logit/page/2#post-1047915
Fri, 04 May 2018 02:23:24 +0000PoliSci1047915@https://www.poliscirumors.com/<p>Formal theorist here. I can't speak to the bootstrap question or to whether this guy is p-hacking on not. I'll leave that to you empirical quants. But he misunderstood and butchered Fearon's model in that paper.
</p>PoliSci on "SEs of Predicted Probabilities in Logit"
https://www.poliscirumors.com/topic/ses-of-predicted-probabilities-in-logit/page/2#post-1047874
Fri, 04 May 2018 01:53:59 +0000PoliSci1047874@https://www.poliscirumors.com/<p>LOL, what would you expect from Hatemi's pal? He should have taken lessons from RE on how to make it hard to get busted. Split the code across lots of files and make it so messy no one would ever have the patience to go through it.
</p>PoliSci on "SEs of Predicted Probabilities in Logit"
https://www.poliscirumors.com/topic/ses-of-predicted-probabilities-in-logit/page/2#post-1047866
Fri, 04 May 2018 01:43:25 +0000PoliSci1047866@https://www.poliscirumors.com/<blockquote><p>
For those of us too stupid/not trained in quant, this means that the author lied about the bootstrap right? If so, what repercussions does it have for the results?
</p></blockquote>
<p>Could be negligence, not necessarily lying... though six models out of six is a lot of negligence. You'd have to re-run his code with the bootstrap option to see how results would change.
</p>PoliSci on "SEs of Predicted Probabilities in Logit"
https://www.poliscirumors.com/topic/ses-of-predicted-probabilities-in-logit/page/2#post-1047863
Fri, 04 May 2018 01:40:10 +0000PoliSci1047863@https://www.poliscirumors.com/<p>You can simulate coefficients from a normal distribution and then do the calculations matrix algebra + inverse logit.</p>
<p>Look at Gelman & Hill, they have a chapter on simulation (R code)</p>
<p>Or look at the Hanmer and Kalkan AJPS observed value approach (STATA code)
</p>PoliSci on "SEs of Predicted Probabilities in Logit"
https://www.poliscirumors.com/topic/ses-of-predicted-probabilities-in-logit/page/2#post-1047850
Fri, 04 May 2018 01:26:23 +0000PoliSci1047850@https://www.poliscirumors.com/<blockquote><p>Ok, I’m back. Not a lot of lines of code to go through, actually.<br />
This is what he says he does in Fig.1:<br />
“Left panel estimated from probit model with offer size, treatment condition, interaction between offer and treatment, and dichotomous variable for the repetition of the experiment … Standard errors clustered at individual level and confidence intervals calculated using a parametric bootstrap running for 1000 iterations.”<br />
Sample size is very small (only 72 students), so I can see why a reviewer asked him to bootstrap his data. Also, data is clustered, so simple bootstrap won’t do.<br />
Replication data is here: <a href="http://hdl.handle.net/1902.1/14820" rel="nofollow">http://hdl.handle.net/1902.1/14820</a><br />
From his readme file:<br />
“The .R and .do files provide all of the statistical analyses.”<br />
There are only one of each. Stata file only does tests of equality of proportion (prtest).<br />
R file has the Zelig code for the probit models:<br />
ProbReject<-zelig(OfferReject~Delta*SubjAOffer+ repetition , model="probit.gee", robust=TRUE, id="ID", corstr = "exchangeable", data=ExpData)<br />
Several such models, none using the bootstrap option.<br />
Zelig documentation says that if bootstrap argument is not specified, the default is bootstrap=FALSE. That is, no bootstrap.<br />
<a href="http://docs.zeligproject.org/articles/bootstraps.html" rel="nofollow">http://docs.zeligproject.org/articles/bootstraps.html</a><br />
“The zelig() bootstrap argument has a default of FALSE, and can be set to TRUE or a numeric value giving the number of bootstrapped datasets to run. If set to TRUE the default is 100 bootstraps. The bootstrap works in combination with other Zelig arguments as follows:<br />
…<br />
Examples:<br />
z.out <- zelig(unem ~ gdp + capmob + trade, model = "ls", data = macro, bootstrap = 500)“<br />
Oops, I just couldn’t find any evidence of bootstrap in the code. Check for yourselves and report here if you find any.
</p></blockquote>
<p>For those of us too stupid/not trained in quant, this means that the author lied about the bootstrap right? If so, what repercussions does it have for the results?
</p>PoliSci on "SEs of Predicted Probabilities in Logit"
https://www.poliscirumors.com/topic/ses-of-predicted-probabilities-in-logit/page/2#post-1047849
Fri, 04 May 2018 01:23:59 +0000PoliSci1047849@https://www.poliscirumors.com/<p>Ok, I’m back. Not a lot of lines of code to go through, actually.</p>
<p>This is what he says he does in Fig.1:</p>
<p>“Left panel estimated from probit model with offer size, treatment condition, interaction between offer and treatment, and dichotomous variable for the repetition of the experiment … Standard errors clustered at individual level and confidence intervals calculated using a parametric bootstrap running for 1000 iterations.” </p>
<p>Sample size is very small (only 72 students), so I can see why a reviewer asked him to bootstrap his data. Also, data is clustered, so simple bootstrap won’t do. </p>
<p>Replication data is here: <a href="http://hdl.handle.net/1902.1/14820" rel="nofollow">http://hdl.handle.net/1902.1/14820</a></p>
<p>From his readme file:</p>
<p>“The .R and .do files provide all of the statistical analyses.”</p>
<p>There are only one of each. Stata file only does tests of equality of proportion (prtest).<br />
R file has the Zelig code for the probit models:</p>
<p>ProbReject<-zelig(OfferReject~Delta*SubjAOffer+ repetition , model="probit.gee", robust=TRUE, id="ID", corstr = "exchangeable", data=ExpData)</p>
<p>Several such models, none using the bootstrap option.</p>
<p>Zelig documentation says that if bootstrap argument is not specified, the default is bootstrap=FALSE. That is, no bootstrap.</p>
<p><a href="http://docs.zeligproject.org/articles/bootstraps.html" rel="nofollow">http://docs.zeligproject.org/articles/bootstraps.html</a></p>
<p>“The zelig() bootstrap argument has a default of FALSE, and can be set to TRUE or a numeric value giving the number of bootstrapped datasets to run. If set to TRUE the default is 100 bootstraps. The bootstrap works in combination with other Zelig arguments as follows:<br />
…</p>
<p>Examples:</p>
<p>z.out <- zelig(unem ~ gdp + capmob + trade, model = "ls", data = macro, bootstrap = 500)“</p>
<p>Oops, I just couldn’t find any evidence of bootstrap in the code. Check for yourselves and report here if you find any.
</p>