This master's thesis deals with the issue of the security of GSM and UMTS mobile communication systems. In the thesis the principles of the authentication and encryption of both the mobile systems are described. Further, the constituent algorithms used for identity verification, key generation and encryption are mentioned. The commonly used algorithms are described along with their weaknesses. In the following part of this thesis, well-known attacks on GSM system are mentioned. In the part where UMTS system is dealt with one can find the algorithms used in this system to increase the security of transmitted data and authentication in comparison with GSM system. Therefore the whole process of authentication and encryption is described separately. In the last part the creation of programming models of authentication and encryption in GSM and encryption in UMTS is described. These models are created in the environment of Matlab version 2009b.