In this thesis, we prove the existence and uniqueness of the solution to a Neumann boundary problem for an elliptic differential operator with singular coefficients, and reveal the relationship between the solution to the partial differential equation (PDE in abbreviation) and the solution to a kind of backward stochastic differential equations (BSDE in abbreviation).This study is motivated by the research on the Dirichlet problem for an elliptic operator (\cite{Z}). But it turns out that different methods are needed to deal with the reflecting diffusion on a bounded domain. For example, the integral with respect to the boundary local time, which is a nondecreasing process associated with the reflecting diffusion, needs to be estimated. This leads us to a detailed study of the reflecting diffusion. As a result, two-sided estimates on the heat kernels are established. We introduce a new type of backward differential equations with infinity horizon and prove the existence and uniqueness of both L2 and L1 solutions of the BSDEs. In this thesis, we use the BSDE to solve the semilinear Neumann boundary problem. However, this research on the BSDEs has its independent interest. Under certain conditions on both the "singular" coefficient of the elliptic operator and the "semilinear coefficient" in the deterministic differential equation, we find an explicit probabilistic solution to the Neumann problem, which supplies a L2 solution of a BSDE with infinite horizon. We also show that, less restrictive conditions on the coefficients are needed if the solution to the Neumann boundary problem only provides a L1 solution to the BSDE.