Universally across modern cultures children acquire the meaning of the words one, two, and three in order. While much research has focused on how children acquire this knowledge and what this knowledge represents, the question of why children learn numbers in order has been comparatively neglected. To address this question, a non-verbal anticipatory looking task was implemented. In this task, 35 14- to 23-month-old infants were assessed on their ability to form implicit category structures for the numbers one, two, and three. We hypothesized that children would be able to form the implicit category structure for the number one but not for two or three because sets of two and three objects would exceed the working memory capacities of infants. We found results consistent with this hypothesis; infants (regardless of age) were able form a category for sets with one object, as evidenced by their looking behavior while the looking behavior for the numbers two and three did not demonstrate a statistically significant pattern. We interpret our results as consistent with our hypothesis and discuss implications for parallel individuation, number acquisition theories, and the development of working memory resources.