Worm attacks have always been considered dangerous threats to the Internet since they can infect a large number of computers and consequently cause large-scale service disruptions and damage. Thus, research on modeling worm attacks, and defenses against them, have become vital to the field of computer and network security. This dissertation intends to systematically study two classes of countermeasures against worm attacks, known as traffic-based countermeasure and non-traffic based countermeasure. Traffic-based countermeasures are those whose means are limited to monitoring, collecting, and analyzing the traffic generated by worm attacks. Non-traffic based countermeasures do not have such limitations. For the traffic-based countermeasures, we first consider the worm attack that adopts feedback loop-control mechanisms which make its overall propagation traffic behavior similar to background non-worm traffic and circumvent the detection. We also develop a novel spectrumbased scheme to achieve highly effective detection performance against such attacks. We then consider worm attacks that perform probing traffic in a stealthy manner to obtain the location infrastructure of a defense system and introduce an information-theoretic based framework to obtain the limitations of such attacks and develop corresponding countermeasures. For the non-traffic based countermeasures, we first consider new unseen worm attacks and develop the countermeasure based on mining the dynamic signature of worm programs’ run-time execution. We then consider a generic worm attack that dynamically changes its propagation patterns and develops integrated countermeasures based on the attacker’s contradicted objectives. Lastly, we consider the real-world system setting with multiple incoming worm attacks that collaborate by sharing the history of their interactions with the defender and develop a generic countermeasure based on establishing the defender’s reputation of toughness in its repeated interactions with multiple incoming attackers to optimize the long-term defense performance. This dissertation research has broad impacts on Internet worm research since this work is fundamental, practical and extensible. Our developed framework can be used by researchers to understand key features of other forms of new worm attacks and develop countermeasures against them.