Simulation is an important tool for prediction and assessment of the behavior of complex systems and situations. The importance of simulation has increased tremendously during the last few decades, mainly because the rapid pace of development in the field of electronics has turned the computer from a costly and obscure piece of equipment to a cheap ubiquitous tool which is now an integral part of our daily lives. While such technological improvements make it easier to analyze well-understood deterministic systems, increase in speed and storage capacity alone are not enough when simulating situations where human beings and their behavior are an integral part of the system being studied. The problem with simulation of intelligent entities is that intelligence is still not well understood and it seems that the field of Artificial Intelligence (AI) has a long way to go before we get computers to think like humans. Behavior-based agent modeling has been proposed in mid-80's as one of the alternatives to the classical AI approach. While used mainly for the control of specialized robotic vehicles with very specific sensory capabilities and limited intelligence, we believe that a behavior-based approach to modeling generic autonomous agents in complex environments can provide promising results. To this end, we are investigating a behavior-based model for controlling groups of collaborating and competing agents in a geographic terrain. In this thesis, we are focusing on scenarios of military nature, where agents can move within the environment and adversaries can eliminate each other through use of weapons. Different aspects of agent behavior like navigation to a goal or staying in group formation, are implemented by distinct behavior modules and the final observed behavior for each agent is an emergent property of the combination of simple behaviors and their interaction with the environment. Our experiments show that while such an approach is quite efficient in terms of computational power, it has some major drawbacks. One of the problems is that reactive behavior-based navigation algorithms are not well suited for environments with complex mobility constraints where they tend to perform much worse than proper path planning. This problem represents an important research question, especially when it is considered that most of the modern military conflicts and operations occur in urban environments. One of the contributions of this thesis is a novel approach to reactive navigation where goals and terrain information are fused based on the idea of transforming a terrain with obstacles into a virtual obstacle-free terrain. Experimental results show that our approach can successfully combine the low run-time computational complexity of reactive methods with the high success rates of classical path planning. Another interesting research problem is how to deal with the unpredictable nature of emergent behavior. It is not uncommon to have situations where an outcome diverges significantly from the intended behavior of the agents due to highly complex nonlinear interactions with other agents or the environment itself. Chances of devising a formal way to predict and avoid such abnormalities are slim at best, mostly because such complex systems tend to be be chaotic in nature. Instead, we focus on detection of deviations through tracking group behavior which is a key component of the total situation awareness capability required by modern technology-oriented and network-centric warfare. We have designed a simple and efficient clustering algorithm for tracking of groups of agent suitable for both spatial and behavioral domain. We also show how to detect certain events of interest based on a temporal analysis of the evolution of discovered clusters.