Cette thèse présente des travaux effectués dans le cadre de l'optimisation de forme de pièces mécaniques, sous critère dynamique, par approche isogéométrique. Pour réaliser une telle optimisation nous mettons en place dans un premier temps les éléments coque au travers des formulations Kirchhoff-Love puis Reissner-Minlin. Nous présentons une méthode permettant d'atteindre les vecteurs normaux aux fibres dans ces formulations au travers de l'utilisation d'une grille mixte de fonctions de base interpolantes, traditionnellement utilisées en éléments finis, et de fonction non interpolantes issues de la description isogéométrique des coques. Par la suite, nous détaillons une méthode pour le couplage de patch puis nous mettons en place la méthode de synthèse modale classique dans le cadre de structures en dynamique décrites par des éléments isogéometriques. Ce travail établit une base pour l'optimisation de forme sous critères dynamique de telles structures. Enfin, nous développons une méthode d'optimisation de forme basée sur le calcul du gradient de la fonction objectif envisagée. La sensibilité de conception est extraite de l'analyse de sensibilité au niveau même du maillage du modèle, qui est obtenue par l'analyse discrète de sensibilité. Des exemples d'application permettent de montrer la pertinence et l'exactitude des approches proposées. / Isogeometric method is a promising method in bridging the gap between the computer aided design and computer aided analysis. No information is lost when transferring the design model to the analysis model. It is a great advantage over the traditional finite element method, where the analysis model is only an approximation of the design model. It is advantageous for structural optimization, the optimal structure obtained will be a design model. In this thesis, the research is focused on the fast three dimensional free shape optimization with isogeometric shell elements. The related research, the development of isogeometric shell elements, the patch coupling in isogeometric analysis, the modal synthesis with isogeometric elements are also studied. We proposed a series of mixed grid Reissner-Minlin shell formulations. It adopts both the interpolatory basis functions, which are from the traditional FEM, and the non-interpolatory basis functions, which are from IGA, to approximate the unknown elds. It gives a natural way to define the fiber vectors in IGA Reissner-Mindlin shell formulations, where the non-interpolatory nature of IGA basis functions causes complexity. It is also advantageous for applying the rotational boundary conditions. A modified reduce quadrature scheme was also proposed to improve the quadrature eficiency, at the same time, relieve the locking in the shell formulations. We gave a method for patch coupling in isogeometric analysis. It is used to connect the adjacent patches. The classical modal synthesis method, the fixed interface Craig-Bampton method, is also used as well as the isogeometric Kirchhoff-Love shell elements. The key problem is also the connection between adjacent patches. The modal synthesis method can largely reduce the time costs in analysis concerning structural dynamics. This part of work lays a foundation for the fast shape optimization of built-up structures, where the design variables are only relevant to certain substructures. We developed a fast shape optimization framework for three dimensional thin wall structure design. The thin wall structure is modelled with isogeometric Kirchhoff-Love shell elements. The analytical sensitivity analysis is the key focus, since the gradient base optimization is normally more fast. There are two models in most optimization problem, the design model and the analysis model. The design variables are defined in the design model, however the analytical sensitivity is normally obtained from the analysis model. Although it is possible to use the same model in analysis and design under isogeomeric framework, it might give either a highly distorted optimum structure or a unreliable structural response. We developed a sensitivity mapping scheme to resolve this problem. The design sensitivity is extracted from the analysis model mesh level sensitivity, which is obtained by the discrete analytical sensitivity analysis. It provides exibility for the design variable definition. The correctness of structure response is also ensured. The modal synthesis method is also used to further improve the optimization eficiency for the built-up structure optimization concerning structural dynamics criteria.