With rapid urbanization in China, a large number of rural students have enrolled in urban schools. We investigate the impact of the proportion of rural students on urban students’ cognitive ability in urban schools. Using the sample of students who are randomly assigned to classes, we find that in urban schools, the proportion of rural students in the class has a negative effect on the cognitive ability of urban students. This negative effect is more pronounced for boys and students whose parents have a high level of education. We also find that the effect is less salient in schools located closer to the center of the city or in more developed cities. Evidence suggests that urban students in classes with a higher proportion of rural students are more likely to be late for class, have poorer quality friends, and receive more criticism from their head teachers, which consequently hurts their cognitive performance. The poor academic performance of rural students is a plausible explanation instead of the bad behaviors of rural students or social integration. Improving the level of teaching in classes with heterogeneous populations will be the focus of policymakers in the future