  Much evidence from biological theory and empirical data indicates that, gene
tree, phylogenetic trees reconstructed from different genes (loci), do not have
to have exactly the same tree topologies. Such incongruence between gene trees
might be caused by some ``unusual'' evolutionary events, such as meiotic sexual
recombination in eukaryotes or horizontal transfers of genetic material in
prokaryotes. However, most of gene trees are constrained by the tree topology
of its species tree, that is, the phylogenetic tree of a given species
following their evolutionary history. In order to discover ``outlying'' gene
trees which do not follow the ``main distribution(s)'' of trees, we propose to
apply the ``tropical metric'' with the max-plus algebra from tropical geometry
to a non-parametric estimation of gene trees over the space of phylogenetic
trees. In this research we apply the ``tropical metric,'' a well-defined metric
over the space of phylogenetic trees under the max-plus algebra, to
non-parametric estimation of gene trees distribution over the tree space.
Kernel density estimator (KDE) is one of the most popular non-parametric
estimation of a distribution from a given sample, and we propose an analogue of
the classical KDE in the setting of tropical geometry with the tropical metric
which measures the length of an intrinsic geodesic between trees over the tree
space. We estimate the probability of an observed tree by empirical frequencies
of nearby trees, with the level of influence determined by the tropical metric.
Then, with simulated data generated from the multispecies coalescent model, we
show that the non-parametric estimation of gene tree distribution using the
tropical metric performs better than one using the Billera-Holmes-Vogtmann
(BHV) metric developed by Weyenberg et al. in terms of computational times and
accuracy. We then apply it to Apicomplexa data.
