Past studies (primarily on steady state problems) that have compared the penalty and the velocity-pressure finite element formulations on a variety of problems have concluded that both methods yield solutions of comparable accuracy, and that the choice of one method over the other is dictated by which of the two is more efficient. In this work, we show that the penalty finite element method yields inaccurate solutions at large times on a class of transient problems, while the velocity-pressure formulation yields solutions that are in good agreement with the analytical solution. Numerical studies are conducted on various problems to compare these two formulations on the basis of rates of convergence, total number of equations to be solved and accuracy of results. We found that both formulations give almost the same rates of convergence in all problems, however the penalty formulation involves lesser number of equations than the velocity-pressure formulation due to implicit treatment of pressure field, and hence is more efficient. In some of the problems we have also compared a finite volume method with the penalty and velocity-pressure formulations on the basis of accuracy and computational cost.