<p>I think the underlying premise of the question -- e.g., &quot;Why was this sample bias not thoroughly addressed before?&quot; -- is somewhat incorrect. Previous papers, including papers by those making the &quot;Planet 9&quot; claim, <em>have</em> attempted to address sample biases; a secondary issue is that the new paper uses data unavailable to previous studies (and also ignores some of the previous data).</p>
<p>To begin with, the original paper by <a href="http://adsabs.harvard.edu/abs/2016AJ....151...22B" rel="nofollow noreferrer">Batygin &amp; Brown (2016)</a> does include a brief discussion of the possible effects of selection biases (in Section 2). This was, admittedly, pretty cursory, but they have followed it up with more extensive analyses (<a href="https://adsabs.harvard.edu/abs/2017AJ....154...65B" rel="nofollow noreferrer">Brown 2017</a>; <a href="https://adsabs.harvard.edu/abs/2019AJ....157...62B" rel="nofollow noreferrer">Brown &amp; Batygin 2019</a>). From the abstract of the latter paper: &quot;To determine if observational bias can be the cause of these apparent clusterings, we develop a rigorous method of quantifying the observational biases in the observations of longitude of perihelion and orbital pole position. From this now more complete understanding of the biases, we calculate that the probability that these distant KBOs would be clustered as strongly as observed in both longitude of perihelion and in orbital pole position is only 0.2%.&quot;</p>
<p>So Napier et al. are not, as you suppose, the first to consider selection biases (as they indeed acknowledge). What, then, is the difference? Part of it is that Napier et al. argue for performing an analysis using detailed simulation taking full account of individual survey characteristics <em>if</em> they are known. In order to do this, they focus on objects found in three recent surveys. This means that they deliberately <em>ignore</em> the 6 objects used for the original claim of Batygin &amp; Brown: &quot;The six ETNOs considered in the Batygin &amp; Brown (2016) (BB16) analysis were discovered in an assortment of surveys with unknown or unpublished selection functions, making it difficult to establish that the observed angular clustering was indeed of physical origin.&quot; Instead, they use 14 objects &quot;detected by three independent surveys with characterized selection functions, all published since BB16.&quot;</p>
<p>(Note that the Batygin &amp; Brown 2019 bias analysis also used 14 objects, <em>but not the same 14</em> -- they included the original 6 objects that Napier et al. exclude, but did not have access to objects detected after their study, which Napier et al. do. We are in a situation where there are very few data points, and there is the potential for divergent results based on the small number statistics of different studies.)</p>
<p>Eric Jensen has already posted links to some Twitter discussions by Batygin on this topic; I can point you to a <a href="https://twitter.com/plutokiller/status/1361754940121640962" rel="nofollow noreferrer">Twitter thread by the other original author (Mike Brown)</a>, as well as a recent <a href="http://findplanetnine.blogspot.com/2021/02/is-planet-nine-finally-dead.html" rel="nofollow noreferrer">blog post</a> by him which tries to understand why they and Napier et al. get such different results.</p>

CC BY-SA 4.0
2021-02-19T22:50:03.573Z
Thank you for your thorough, considered and well-sourced answer! With this roadmap I may now be able to read in more depth, and though I normally don't, I'll try to follow the exchange in twitter.
uhoh
CC BY-SA 4.0
2021-02-21T07:10:20.110Z
There's another issue, Batygin & Brown use a heuristic model for biases that are completely untested on known surveys with known biases instead of using the actual published biases for the surveys and instead of a Bayesian (and almost certainly more correct) approach for understanding those biases that Napier et al. did use. Brown also makes some shockingly ignorant statements in his blog about uncertainties (you can't put uncertainties on things like "this is true to 95% confidence" – an uncertainty is meaningless). FWIW, I listened to a 30-min talk by a statistician about this 2 days ago.
Stuart Robbins
CC BY-SA 4.0
2021-02-21T09:19:00.840Z
@Stuart Robbins -- To be fair, Brown is not complaining about a missing "uncertainty on an uncertainty"; he's complaining about a missing uncertainty on a clustering measurement, which is obviously not the same thing.
Peter Erwin
CC BY-SA 4.0
2021-02-21T12:18:31.560Z
The slightly concerning thing for me about the Napier et al. analysis is that in their comparison with Brown & Batygin (2019), they note that if they exclude the five most recently discovered objects (which were not available to B&B19), the $P$-value from their analysis goes from "0.17 to 0.94" down to "$P < 0.005$". Which suggests that: a) B&B19's approach was not *wholly* erroneous; and b) Napier et al.'s results are *really* sensitive to the inclusion/exclusion of small numbers of objects, so their quoted $P$-values are, well, kind of uncertain.
Peter Erwin
CC BY-SA 4.0
2021-02-19T16:43:33.780Z
2021-02-20T13:21:32.553Z
2021-02-20T13:21:32.553Z
Peter Erwin
<p>Konstantin Batygin, one of the authors of the original Planet 9 paper, has an interesting Twitter thread <a href="https://twitter.com/kbatygin/status/1360381088216936451?s=21" rel="nofollow noreferrer">here</a> where he discusses this paper.</p>
<p>I encourage you to read that, but briefly, his argument is that the Dark Energy Survey (DES) is already biased (not a judgment term here, just describing the sky footprint) to discover objects that are in the cluster of KBOs that constitute the evidence for Planet 9. So when looking at that subset of the data only, you can fit the data either with a uniform distribution or with a clustered distribution.</p>
<p>Note that that’s what the paper concludes - finding that this subset of the data by itself doesn’t <em>require</em> clustering is different from saying that the clustering isn’t there.</p>
<p>Edit:  <a href="https://twitter.com/kbatygin/status/1360390787653980161?s=21" rel="nofollow noreferrer">Here</a> is another thread that explains this with an analogy.</p>
<p>The key question - which I don’t know the answer to - is how heavily the discovery of the clustering in the first place relies on the subset of the data analyzed in the Napier et al. paper.</p>

CC BY-SA 4.0
2021-02-19T15:10:11.837Z
The answer to your “key question” is “not at all”, because the six objects that were the basis for the original claim are (deliberately) left out of the Napier et al. analysis, which relies instead on 14 objects discovered *after* the original claim.
Peter Erwin
CC BY-SA 4.0
2021-02-21T07:13:16.547Z
Put another way: Batygin & Brown (2016) used 6 objects and tested for clustering and found what they think is clustering.  Napier et al. (2021) used a larger set of surveys (a meta-analysis) and specifically asked the question: Can the null hypothesis that the data are consistent with NO clustering be rejected?  They could not reject the null hypothesis.  These are two quite different ways of studying the problem and posing the question, and I would personally argue the latter is a more statistically honest way of doing it.
Stuart Robbins
CC BY-SA 4.0
2021-02-19T13:16:33.377Z
2021-02-19T13:36:42.000Z
2021-02-19T13:36:42.000Z
Eric Jensen
<ul>
<li>Science <a href="https://www.sciencemag.org/news/2021/02/claim-giant-planet-nine-solar-systems-edge-takes-hit" rel="nofollow noreferrer">Claim for giant 'Planet Nine' at Solar System's edge takes a hit</a></li>
<li>arXiv <a href="https://arxiv.org/abs/2102.05601" rel="nofollow noreferrer">No Evidence for Orbital Clustering in the Extreme Trans-Neptunian Objects</a></li>
</ul>
<p>The brightness of distant solar system objects varies as <span class="math-container">$1/r^4$</span> and so for elliptical orbits objects are more likely to be discovered near periapsis.</p>
<p>If discovery telescopes are pointed in a biased direction, resulting discoveries will show a bias in orbital elements due to this pointing bias.</p>
<p>That's a rough summary of the recent paper.</p>
<p><strong>Question:</strong> Why was this sample bias not thoroughly addressed before? Was there perhaps insufficient statistics until now for such an analysis, or were there other complicating factors?</p>

CC BY-SA 4.0
2021-02-16T10:58:57.050Z
If you're thinking that corrupt, perhaps deliberately negligent editors let the previous papers through, what makes you think this won't be true of this paper? (Granted that it seems this paper hasn't been peer-reviewed yet...)
Peter Erwin
CC BY-SA 4.0
2021-02-16T11:08:19.553Z
@PeterErwin I think you are trying to read more into what I wrote than what's actually written, for a sensational effect; not that that bears any resemblance to anything discussed above :-)
uhoh
CC BY-SA 4.0
2021-02-16T17:47:41.900Z
The (purported) clustering is in the orientation of the orbits around the Sun when looking down from above (see Fig 2 and 6). So in addition to distance biases, there are also complicated biases based on where ETNO distribution (concentrated along the ecliptic) and stellar density distributions (concentrated towards the MW plane and Galactic Center) interact which makes debiasing the (multiple) survey sensitivities to ETNOs very hard.
astrosnapper
CC BY-SA 4.0
2021-02-17T00:29:52.773Z
@astrosnapper thanks for pointing out the complexities; I've adjusted the question's wording, its goal should be to simply provide a space for answers.
uhoh
CC BY-SA 4.0
2021-02-19T13:48:25.553Z
I find questions in the line of "why wasn't it done before" problematic and answerable mostly subjectively. Science is funded. You prove what you do by writing papers. You can do so much in your time. And even when you did not consider every possible explanation, and/or dismiss a certain view or analysis, doesn't mean it's bad science. Science lives from the evolution of judgement and re-assessment of old evidence by new people and also by the same people in the light of newer evidence (or so far neglected, overlooked or for whatever reason not-considered).
planetmaker
CC BY-SA 4.0
2021-02-19T14:00:10.203Z
@planetmaker I think this one isn't, and that's why I've asked it. Astronomers have been avid and skillful users of population studies for a century or more. This isn't some esoteric third-order effect. If you look only left and say "hey, all the stars are on the left" before looking to the right to see if there are stars that way as well, then it's not a case of not "considering every possible explanation", it's a case of not asking the obvious question before shouting about the peculiar left-clustering of stars.
uhoh
CC BY-SA 4.0
2021-02-19T16:07:06.417Z
Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/119945/discussion-on-question-by-uhoh-sample-bias-contribution-to-planet-9-hypothesi).
called2voyage
<observational-astronomy><solar-system><data-analysis>
Sample bias' contribution to "Planet 9" hypothesis, why was this not thoroughly addressed before?
uhoh
# Sample bias' contribution to "Planet 9" hypothesis, why was this not thoroughly addressed before?

(Asked by: username_0 on 2021-02-16)

* Science [Claim for giant 'Planet Nine' at Solar System's edge takes a hit](https://www.sciencemag.org/news/2021/02/claim-giant-planet-nine-solar-systems-edge-takes-hit)
* arXiv [No Evidence for Orbital Clustering in the Extreme Trans-Neptunian Objects](https://arxiv.org/abs/2102.05601)

The brightness of distant solar system objects varies as $1/r^4$ and so for elliptical orbits objects are more likely to be discovered near periapsis.

If discovery telescopes are pointed in a biased direction, resulting discoveries will show a bias in orbital elements due to this pointing bias.

That's a rough summary of the recent paper.

**Question:** Why was this sample bias not thoroughly addressed before? Was there perhaps insufficient statistics until now for such an analysis, or were there other complicating factors?

**Tags:** observational-astronomy, solar-system, data-analysis

> **Comments:**
> **username_1**: If you're thinking that corrupt, perhaps deliberately negligent editors let the previous papers through, what makes you think this won't be true of this paper? (Granted that it seems this paper hasn't been peer-reviewed yet...)  
> **username_0**: @username_1 I think you are trying to read more into what I wrote than what's actually written, for a sensational effect; not that that bears any resemblance to anything discussed above :-)  
> **username_2**: The (purported) clustering is in the orientation of the orbits around the Sun when looking down from above (see Fig 2 and 6). So in addition to distance biases, there are also complicated biases based on where ETNO distribution (concentrated along the ecliptic) and stellar density distributions (concentrated towards the MW plane and Galactic Center) interact which makes debiasing the (multiple) survey sensitivities to ETNOs very hard.  
> **username_0**: @username_2 thanks for pointing out the complexities; I've adjusted the question's wording, its goal should be to simply provide a space for answers.  
> **username_3**: I find questions in the line of "why wasn't it done before" problematic and answerable mostly subjectively. Science is funded. You prove what you do by writing papers. You can do so much in your time. And even when you did not consider every possible explanation, and/or dismiss a certain view or analysis, doesn't mean it's bad science. Science lives from the evolution of judgement and re-assessment of old evidence by new people and also by the same people in the light of newer evidence (or so far neglected, overlooked or for whatever reason not-considered).  
> **username_0**: @username_3 I think this one isn't, and that's why I've asked it. Astronomers have been avid and skillful users of population studies for a century or more. This isn't some esoteric third-order effect. If you look only left and say "hey, all the stars are on the left" before looking to the right to see if there are stars that way as well, then it's not a case of not "considering every possible explanation", it's a case of not asking the obvious question before shouting about the peculiar left-clustering of stars.  
> **username_4**: Comments are not for extended discussion; this conversation has been [moved to chat](https://chat.stackexchange.com/rooms/119945/discussion-on-question-by-username_0-sample-bias-contribution-to-planet-9-hypothesi).  

---

## 2 answers

### Answer by username_1 on 2021-02-19. Score: 7

I think the underlying premise of the question -- e.g., "Why was this sample bias not thoroughly addressed before?" -- is somewhat incorrect. Previous papers, including papers by those making the "Planet 9" claim, *have* attempted to address sample biases; a secondary issue is that the new paper uses data unavailable to previous studies (and also ignores some of the previous data).

To begin with, the original paper by [Batygin & Brown (2016)](http://adsabs.harvard.edu/abs/2016AJ....151...22B) does include a brief discussion of the possible effects of selection biases (in Section 2). This was, admittedly, pretty cursory, but they have followed it up with more extensive analyses ([Brown 2017](https://adsabs.harvard.edu/abs/2017AJ....154...65B); [Brown & Batygin 2019](https://adsabs.harvard.edu/abs/2019AJ....157...62B)). From the abstract of the latter paper: "To determine if observational bias can be the cause of these apparent clusterings, we develop a rigorous method of quantifying the observational biases in the observations of longitude of perihelion and orbital pole position. From this now more complete understanding of the biases, we calculate that the probability that these distant KBOs would be clustered as strongly as observed in both longitude of perihelion and in orbital pole position is only 0.2%."

So Napier et al. are not, as you suppose, the first to consider selection biases (as they indeed acknowledge). What, then, is the difference? Part of it is that Napier et al. argue for performing an analysis using detailed simulation taking full account of individual survey characteristics *if* they are known. In order to do this, they focus on objects found in three recent surveys. This means that they deliberately *ignore* the 6 objects used for the original claim of Batygin & Brown: "The six ETNOs considered in the Batygin & Brown (2016) (BB16) analysis were discovered in an assortment of surveys with unknown or unpublished selection functions, making it difficult to establish that the observed angular clustering was indeed of physical origin." Instead, they use 14 objects "detected by three independent surveys with characterized selection functions, all published since BB16."

(Note that the Batygin & Brown 2019 bias analysis also used 14 objects, *but not the same 14* -- they included the original 6 objects that Napier et al. exclude, but did not have access to objects detected after their study, which Napier et al. do. We are in a situation where there are very few data points, and there is the potential for divergent results based on the small number statistics of different studies.)

username_6 has already posted links to some Twitter discussions by Batygin on this topic; I can point you to a [Twitter thread by the other original author (Mike Brown)](https://twitter.com/plutokiller/status/1361754940121640962), as well as a recent [blog post](http://findplanetnine.blogspot.com/2021/02/is-planet-nine-finally-dead.html) by him which tries to understand why they and Napier et al. get such different results.

> **Comments:**
> **username_0**: Thank you for your thorough, considered and well-sourced answer! With this roadmap I may now be able to read in more depth, and though I normally don't, I'll try to follow the exchange in twitter.  
> **username_5**: There's another issue, Batygin & Brown use a heuristic model for biases that are completely untested on known surveys with known biases instead of using the actual published biases for the surveys and instead of a Bayesian (and almost certainly more correct) approach for understanding those biases that Napier et al. did use. Brown also makes some shockingly ignorant statements in his blog about uncertainties (you can't put uncertainties on things like "this is true to 95% confidence" – an uncertainty is meaningless). FWIW, I listened to a 30-min talk by a statistician about this 2 days ago.  
> **username_1**: @username_5 -- To be fair, Brown is not complaining about a missing "uncertainty on an uncertainty"; he's complaining about a missing uncertainty on a clustering measurement, which is obviously not the same thing.  
> **username_1**: The slightly concerning thing for me about the Napier et al. analysis is that in their comparison with Brown & Batygin (2019), they note that if they exclude the five most recently discovered objects (which were not available to B&B19), the $P$-value from their analysis goes from "0.17 to 0.94" down to "$P < 0.005$". Which suggests that: a) B&B19's approach was not *wholly* erroneous; and b) Napier et al.'s results are *really* sensitive to the inclusion/exclusion of small numbers of objects, so their quoted $P$-values are, well, kind of uncertain.  

---

### Answer by username_6 on 2021-02-19. Score: 4

Konstantin Batygin, one of the authors of the original Planet 9 paper, has an interesting Twitter thread [here](https://twitter.com/kbatygin/status/1360381088216936451?s=21) where he discusses this paper.

I encourage you to read that, but briefly, his argument is that the Dark Energy Survey (DES) is already biased (not a judgment term here, just describing the sky footprint) to discover objects that are in the cluster of KBOs that constitute the evidence for Planet 9. So when looking at that subset of the data only, you can fit the data either with a uniform distribution or with a clustered distribution.

Note that that’s what the paper concludes - finding that this subset of the data by itself doesn’t *require* clustering is different from saying that the clustering isn’t there.

Edit: [Here](https://twitter.com/kbatygin/status/1360390787653980161?s=21) is another thread that explains this with an analogy.

The key question - which I don’t know the answer to - is how heavily the discovery of the clustering in the first place relies on the subset of the data analyzed in the Napier et al. paper.

> **Comments:**
> **username_1**: The answer to your “key question” is “not at all”, because the six objects that were the basis for the original claim are (deliberately) left out of the Napier et al. analysis, which relies instead on 14 objects discovered *after* the original claim.  
> **username_5**: Put another way: Batygin & Brown (2016) used 6 objects and tested for clustering and found what they think is clustering.  Napier et al. (2021) used a larger set of surveys (a meta-analysis) and specifically asked the question: Can the null hypothesis that the data are consistent with NO clustering be rejected?  They could not reject the null hypothesis.  These are two quite different ways of studying the problem and posing the question, and I would personally argue the latter is a more statistically honest way of doing it.  

---

