After the first plausible fully homomorphic encryption (FHE) scheme designed by Gentry, interests of a building a practical scheme in FHE has kept increasing. This paper presents an engineering study of accelerating the FHE with BGV scheme and proves the feasibility of implement certain parts of HElib on GPU. The BGV scheme is a RLWE-based FHE scheme, which introduces a set of algorithms in polynomial arithmetic. The encryption scheme is implemented in finite field. Therefore, acceleration of the large polynomial arithmetic with efficient modular reduction is the most crucial part of our research efforts. Note that our implementation does not include the noise management yet. Hence all the work is still in the stage of somewhat homomorphic encryption, namely SWHE. Finally, our implementation of the encryption procedure, when comparing with HElib compiled by 9.3.0 version NTL library on Xeon CPU, has achieved 3.4x speedup on the platform with GTX 780ti GPU.