Cooperative communications is an emerging technique that has spatial diversity inherent in wireless multiuser communication systems without multiple antennas at each node. Most studies in the literature assume that users acting as the relays are normally operated and trustworthy, which, however, may not always be true in practice. This thesis considers the design of robust cooperative communication in physical layer for combating relay misbehaviors. This thesis considers both models in which the cooperative communications is with direct path (WDP) and without direct path (WODP). Two signal-correlation-detection rules for both WDP and WODP are proposed, respectively. Utilizing the proposed signal-correlation-detection mechanism, the destination identifies the misbehaving relays within the cooperative communication network and then excludes their transmitting messages when performing the diversity combining to infer the symbols of interest sent by the source. The proposed signal-correlation-detection rules are optimally designed in accordance with either the criterion of the minimization of the probability of misbehavior misidentification or the criterion of the maximum generalized likelihood detector. In addition, this thesis also provides the BER analysis of the cooperative communications employing the proposed misbehaving relay detectors. The simulation result demonstrates that the proposed schemes have excellent robust performance when the relay misbehavior is present in the cooperative communication networks.

In the cooperative communications, the users relay each other's signal and thus form multiple transmission paths to the destination and therefore the system can achieve spatial diversity gain. Most studies in the literature assumed that cooperative users acting as the relays are normally operated and trustworthy. However, this may not always be true in practice. When the relay misbehaviors are present in the cooperative communications, the communication performance may degrade dramatically and the users may be even better off without cooperation. Therefore, it is necessary for the destination to determine the misbehaving relays and to take appropriate actions to ensure that cooperative advantages are preserved. This thesis considers both models in which the cooperative communications are with direct path (WDP) and without direct path (WODP). Utilizing the proposed Kolmogorov-Smirnov test mechanism, the destination identifies the misbehaving relays within the cooperative communications and then excludes their transmitting messages when performing the diversity combining to infer the symbols of interest sent by the source. In addition, this thesis provides the bit error rate (BER) analysis of the cooperative communications employing the proposed misbehaving relay detectors. The simulation results demonstrate that the proposed methods have robust performance when the relay misbehaviors are present in the cooperative communications.