In our thesis, we investigate the detection of malicious behavior in cooperative networks. Our model contain one source, one destination and two relays where relays adopt Orthogonal Space Time Block Code (OSTBC) to achieve spatial diversity. Cooperative communication takes two phases to forward signal to the destination. During first phase, source broadcasts data symbols to relay and destination, and a few tracing symbols are inserted randomly in data symbols. The values and positions of tracing symbols are known at source and destination. The random placement of tracing symbols is to prevent relays evade the detection of malicious behavior. In second phase, relays adopt orthogonal space time block code to forward the received signals after decoding source message successfully. We consider two scenarios based on the decoding capability at relays. The first scenario assumes perfect source¡Vrelay links. So, relays can always decode symbols correctly. Second scenario considers decoding failures at relays. In both scenarios, relays have a certain probability to perform maliciously. After receiving symbols at destination, the destination extracts and detect the tracing symbols. The malicious behavior of relay is detected depending on the value of the correlation between detected and exact tracing symbols. Moreover, depending on the average received energy, we can distinguish whether relays behaves as in outage. Through computer simulation, we can verify that our proposed tracing algorithm and decoding strategy reduce bit error rate.

To combat channel fading, cooperative communication achieves spatial diversity for the transmission between source and destination through the help of relay. However, if the relay behaves abnormally or maliciously and the destination is not aware, the diversity gain of the cooperative system will be significantly reduced, which degrades system performance. In our thesis, we consider an one-relay decode and forward cooperative network, and we assume that the relay may misbehave with a certain probability. If the relay is malicious, it will garble transmission signal, resulting in severe damage to cooperative system. In this work, we discuss three kinds of malicious behavior detection. More specifically, we adopt sequential detection to detect the behavior of relay. If tracing symbols are inserted among the source message, the destination detects malicious after extracting the received tracing symbols. We adopt log-likelihood ratio test to examine these tracing symbols, and then determine the behavior of relay. If the source does not transmit tracing symbols, the destination detects misbehavior according to the received data signal. Furthermore, we employ sequential detection to reduce detection time for a given probabilities of false alarm and miss detection. Through simulation results, for a certain target on probability of errors, our proposed methods can effectively reduce numbers of observations. On the other works, the destination can effectively detect misbehavior of relay, and eliminating the damage causes by malicious relay without requiring large numbers of observations.