Cooperative communications by pooling available resources—for example, power and bandwidth—across the network, is a distributed solution for providing robust wireless transmission. Motivated by contemporary applications in multi-hop transmission and ad hoc networks, the classical three-node relay channel (RC) consisting of a source–destination pair and a relay node has received a renewed attention. One of the crucial aspects of the communication over relay networks (RNs) is the design of proper relaying protocols; that is, how the relay should take part in the transmission to meet a certain quality of service. In this dissertation, we address the design of reliable transmission strategies and quantification of the associated transmission rates over RNs. We consider three canonical examples of RNs: the classical RC, the multiple-access RC (MARC) and the two-way RC.We also investigate the three-node RC and MARC with state. The capacity of the aforementioned RNs is an open problem in general except for some special cases. In the thesis, we derive various capacity bounds, through which we also identify the capacity of some new classes of RNs. In particular, we introduce the class of state-decoupled RNs and prove that the noisy network coding is capacity achieving under certain conditions. In the thesis, we also study the effect of the memory length on the capacity of RNs. The investigated relaying protocols in the thesis can be categorized into two groups: protocols with a finite relay memory and those with infinite relay memory requirement. In particular, we consider the design of instantaneous relaying (also referred to as memoryless relaying) in which the output of the relay depends solely on the presently received signal at the relay. For optimizing the relay function, we present several algorithms constructed based on grid search and variational methods. Among other things, we surprisingly identify some classes of semi-deterministic RNs for which a properly constructed instantaneous relaying strategy achieves the capacity. We also show that the capacity of RNs can be increased by allowing the output of the relay to depend on the past received signals as well the current received signal at the relay. As an example, we propose a hybrid digital–analog scheme that outperforms the cutset upper bound for strictly causal relaying. / <p>QC 20110909</p>