This thesis focuses on description of the current state of research in the detection of network attacks and subsequently on the improvement of detection capabilities of specific attacks by establishing a formal definition of network metrics. These metrics approximate the progress of network connection and create a signature, based on behavioral characteristics of the analyzed connection. The aim of this work is not the prevention of ongoing attacks, or the response to these attacks. The emphasis is on the analysis of connections to maximize information obtained and definition of the basis of detection system that can minimize the size of data collected from the network, leaving the most important information for subsequent analysis. The main goal of this work is to create the concept of the detection system by using defined metrics for reduction of the network traffic to signatures with an emphasis on the behavioral aspects of the communication. Another goal is to increase the autonomy of the detection system by developing an expert knowledge of honeypot system, with the condition of independence to the technological aspects of analyzed data (e.g. encryption, protocols used, technology and environment). Defining the concept of honeypot system's expert knowledge in the role of the teacher of classification algorithms creates autonomy of the~system for the detection of unknown attacks. This concept also provides the possibility of independent learning (with no human intervention) based on the knowledge collected from attacks on these systems. The thesis describes the process of creating laboratory environment and experiments with the defined network connection signature using collected data and downloaded test database. The results are compared with the state of the art of the network detection systems and the benefits of the proposed approximation methods are highlighted.