The COVID-19 pandemic caused by SARS-CoV-2 infection poses a serious threat to public health. An explicit investigation of COVID-19 immune responses, particularly the host immunity in recovered subjects, will lay a foundation for the rational design of therapeutics and/or vaccines against future coronaviral outbreaks. Here, we examined virus-specific T cell responses and identified T cell epitopes using peptides spanning SARS-CoV-2 structural proteins. These peptides were used to stimulate peripheral blood mononuclear cells (PBMCs) derived from COVID-19-recovered subjects, followed by an analysis of IFN-γ-secreting T cells by enzyme-linked immunosorbent spot (ELISpot). We also evaluated virus-specific CD4 or CD8 T cell activation by flow cytometry assay. By screening 52 matrix pools (comprised of 315 peptides) of the spike (S) glycoprotein and 21 matrix pools (comprised of 102 peptides) spanning the nucleocapsid (N) protein, we identified 28 peptides from S protein and 5 peptides from N protein as immunodominant epitopes. The immunogenicity of these epitopes was confirmed by a second ELISpot using single peptide stimulation in memory T cells, and they were mapped by HLA restrictions. Notably, SARS-CoV-2 specific T cell responses positively correlated with B cell IgG and neutralizing antibody responses to the receptor-binding domain (RBD) of the S protein. Our results demonstrate that defined levels of SARS-CoV-2 specific T cell responses are generated in some, but not all, COVID-19-recovered subjects, fostering hope for the protection of a proportion of COVID-19-exposed individuals against reinfection. These results also suggest that these virus-specific T cell responses may induce protective immunity in unexposed individuals upon vaccination, using vaccines generated based on the immune epitopes identified in this study. However, SARS-CoV-2 S and N peptides are not potently immunogenic, and none of the single peptides could universally induce robust T cell responses, suggesting the necessity of using a multi-epitope strategy for COVID-19 vaccine design.