The purpose of this paper is to investigate if semi-compatibilism can neutralize the skeptical arguments from hard determinism and prove that justified moral responsibility is compatible with determinism. I aim to prove that this cannot be done without an account of free will. I argue that hard determinism and its skeptical arguments pose a relevant and significant threat to semi-compatibilism by highlighting and defending the justice aspect of Galen Strawson's and Bruce Waller's skeptical arguments. Through a skeptical justice argument, I argue that a key element of semi-compatibilism generates unjust systems of moral responsibility. This is put to the test by carefully examining the semi-compatibilist theories of Harry Frankfurt and John Martin Fischer and analyzing how their theories holp up to the justice argument. The result indicates that their theories of moral responsibility result in seriously unfair judgements, showing that their accounts of moral responsibility cannot be justified, due to unfairness. I argue that the unfairness of semi-compatibilism is a result of the semi-compatibilist need to remain coherent with determinism.