Tracking control algorithms often target the convergence of a tracking error. However, this can be at the expense of other important system characteristics, such as the control effort used to annihilate the tracking error, transient response, or steady-state characteristics, for example. Furthermore, most tracking control methods assume prior knowledge of the system dynamics, which is not always a realistic assumption, especially in the case of highly complex systems. In this thesis, a model-free optimized tracking control architectural heuristic is proposed. The suggested feedback system is composed of two control loops. The first is the tracking loop. It focuses on the convergence of the tracking error. It is implemented using two different model-free control algorithms for comparison purpose: Reinforcement Learning (RL) and the Nonlinear Threshold Accepting (NLTA) technique. The RL scheme reformulates the tracking error combinations into a form of Markov-Decision-Process (MDP) and applies Q-Learning to build the best tracking control policy for the dynamic system under consideration. On the other hand, the NLTA algorithm is applied to tune the gains of a PID controller. The second control loop is in the form of a nonlinear state feedback loop. It is implemented using a feedforward artificial neural network (ANN) to optimize a system-wide cost function which can be flexible enough to encompass a set of desired design requirements pertaining to the targeted system behavior. This may include, for instance, the target overshoot, settling time, rise time, etc. The proposed architectural heuristic provides a model-free framework to tackle such control problems, in the sense that the plant's dynamic model is not required to be known in advance. Yet, at least a subset of the stability region of the optimized gains has to be known in advance so that it can provide a search space for the optimization algorithms. Simulation results on two dynamic systems demonstrate the superiority of the proposed control scheme.