Inferring the history of speciation in house mice from autosomal, X‐linked, Y‐linked and mitochondrial genes

<jats:title>Abstract</jats:title><jats:p>Patterns of genetic differentiation among taxa at early stages of divergence provide an opportunity to make inferences about the history of speciation. Here, we conduct a survey of DNA‐sequence polymorphism and divergence at loci on the autosomes, X chromosome, Y chromosome and mitochondrial DNA in samples of <jats:italic>Mus domesticus, M. musculus</jats:italic> and <jats:italic>M. castaneus</jats:italic>. We analyzed our data under a divergence with gene flow model and estimate that the effective population size of <jats:italic>M. castaneus</jats:italic> is 200 000–400 000, of <jats:italic>M. domesticus</jats:italic> is 100 000–200 000 and of <jats:italic>M. musculus</jats:italic> is 60 000–120 000. These data also suggest that these species started to diverge approximately 500 000 years ago. Consistent with this recent divergence, we observed considerable variation in the genealogical patterns among loci. For some loci, all alleles within each species formed a monophyletic group, while at other loci, species were intermingled on the phylogeny of alleles. This intermingling probably reflects both incomplete lineage sorting and gene flow after divergence. Likelihood ratio tests rejected a strict allopatric model with no gene flow in comparisons between each pair of species. Gene flow was asymmetric: no gene flow was detected into <jats:italic>M. domesticus</jats:italic>, while significant gene flow was detected into both <jats:italic>M. castaneus</jats:italic> and <jats:italic>M. musculus</jats:italic>. Finally, most of the gene flow occurred at autosomal loci, resulting in a significantly higher ratio of fixed differences to polymorphisms at the X and Y chromosomes relative to autosomes in some comparisons, or just the X chromosome in others, emphasizing the important role of the sex chromosomes in general and the X chromosome in particular in speciation.</jats:p>