In this thesis, for the under-determined system Fx = b with the matrix F ∈m¡Ñn (m ≤ n), new error bounds involving the traditional condition number and the effective condition number are established. Such error bounds are simple than those of over-determined system. The errors results implies that for stability, the condition number and the effective condition numbers are important if the perturbation of matrix F and vector b are dominant, respectively. This thesis is also devoted to the application of Neumann problems, where the consistent condition holds to guarantee the existence of multiple solutions. For the traditional Neumann conditions, the discrete consistent condition has to be satisfied to guarantee the existence of numerical solutions. Such a discrete consistent condition can be removed, to greatly simplify the numerical algorithms, and to retain the same convergence rates. For Neumann Problems, we may solve its ordinal discrete linear equations, or the underdetermined systems by ignoring some dependent equations, or the fixed variables methods. Moreover, we may choose different equations to be ignored, and different variables to be fixed. The comparisons of these different methods and choices are important in applications. In this thesis, the new comparisons and relations of stability and accuracy are first explored, and some interesting results and new discoveries are found. Numerical examples of Neumann problem in 1D are carried out, to support the analysis made. However, the algorithms and stability analysis can be applied to the complicated Nuemann problems in 2D and 3D, such as the traction problems in linear elastic problems.