The first ocean general circulation models developed in the late sixties were based on finite differences schemes on structured grids. Many improvements in the fields of engineering have been achieved since three decades with the developments of new numerical methods based on unstructured meshes. Some components of the first models may now seem out of date and new second generation models are therefore under study, with the aim of taking advantage of the potential of modern numerical techniques such as finite elements. In particular, unstructured meshes are believed to be more efficient to resolve the large range of time and space scales present in the ocean. Besides the classical continuous finite element or finite volume methods, another popular new trend in engineering applications is the Discontinuous Galerkin (DG) method, i.e. discontinuous finite elements presenting many interesting numerical properties in terms of dispersion and dissipation, errors convergence rates, advection schemes, mesh adaptation, etc. The method is especially efficient at high polynomial orders. The motivation for this PhD research is therefore to investigate the use of the high-order DG method for geophysical flow modeling. A first part of the thesis is devoted to the mesh adaptation using the DG method. The inter-element jumps of the fields are used as error estimators. New mesh size fields or polynomial orders are then derived and local h- or p-adaptation is performed. The technique is applied to standard benchmarks and computations in more realistic domains as the Gulf of Mexico. A second part deals with the use of the high order DG method with high-order representation of geometrical features. On one hand, a method is proposed to deal with complex representations of the coastlines. Computations are performed using high-order mappings around the Rattray island, located in the Great Barier Reef. Numerical results are then compared to in-situ measurements. On the other hand, a new method is proposed to deal with curved manifolds in order to represents oceanic or atmospheric flows on the sphere. The approach is based on the use of a local high-order non-orthogonal basis, and is equivalent to the use of vectorial shape and test functions to represent the vectorial conservation laws on the manifold's surface. A method is finally proposed to analyze the dispersion and dissipation properties of any numerical scheme on any kind of grid, possibly unstructured. The DG method is then compared to other techniques as the mixed non-conforming linear elements, and the impact of unstructured meshes is studied.