Introduction Aspirin has been associated with a reduction in mortality in patients diagnosed with colorectal cancer (CRC). A possible mechanism for this is related to the programmed cell death 1 (PD-1) immune checkpoint pathway. Aspirin may have a synergistic effect with PD-1 inhibitors via inhibition of prostaglandin E2 (PGE2) production, which can reverse the ability of tumor cells to evade the immune system. This appears to be strongest in cancers that express PI3 kinase (PI3K) signaling activity, which aspirin downregulates. However, the benefit of pre-diagnosis aspirin use on CRC overall survival (OS) and cancer-specific survival is still controversial, and most studies have been performed in racially homogenous populations. Our study examines the effect of pre-diagnosis aspirin therapy on OS in a racially diverse group of patients with CRC. Methods This is a retrospective chart review of 782 patients diagnosed with CRC from January 2007 to December 2020. Kaplan-Meier curve was created to study the association of aspirin exposure compared to no exposure on OS. In addition, univariate and multivariate binary logistic regression analyses were done to investigate potential predictors of survival. Results Of the 782 patients with CRC, 55.1% were males, 22.2% whites, 58.5% Asians, and 17.7% Pacific-Islanders. Moreover, 38.4% of the patients had a history of aspirin use, 79% of them used it for more than one year. There were more patients with hypertension (HTN), hyperlipidemia (HLD), diabetes mellitus (DM), and chronic kidney disease (CKD) among those with a history of aspirin use. There was no difference in one, three, and five-year OS among aspirin users compared to non-users, p-value = 0.63. Age, grade, and stage were potential predictors of worsened OS. However, treatment with chemotherapy and CKD were potential predictors of worsened OS on univariate analysis only. No significant association was noticed with gender, tumor location, or other associated comorbidities. Conclusion The effect of pre-diagnosis aspirin use on CRC survival is not clear. In this retrospective analysis of a racially diverse population of CRC patients, we found that aspirin use was not associated with improved OS. Therefore, physicians should be careful about using aspirin as adjuvant therapy in CRC patients until high-quality prospective data are available, given the potential associated complications.