Globally, the use of ERP systems has increased significantly. Organizations in developed and developing countries are pursuing ERP systems to stand out globally for facilitating their growth beyond their previous inhouse systems. Even if public organizations were not the main target of ERP systems, as ERP systems were first developed for manufacturing organizations, due to the countless benefits gained from the utilization of the systems they became part of the worldwide trend of implementing ERP systems long ago. These organizations have different culture and encounter numerous challenges due to several reasons. The purpose of this study was to explore ERP system implementation challenges by developing countries’ public sector organization’s and identify which critical factors has high impact in the success or failure of the implementations. The participants were 11 experienced IT experts with adequate knowledge on the ERP currently or previously involved in ERP implementation in Ethiopia, chosen case country in public sector organizations. The results of this study are comprised of systematic literature review and qualitative analysis, and data were collected using documentary search and semi-structured online interviews, respectively. Through deductive thematic analysis, themes were identified which are presented in 5 categories: critical success factors, ERP systems pre-implementation challenges, during implementation challenges, post implementation challenges, success of ERP system implementation and its measurements. The results are discussed based on comparison of the literature findings and qualitative analysis.