With the recent advances in the field of unmanned aerial vehicles, many applications have been identified. In tasks that require high-payload-to-weight ratios, flight times in the order of days, reduced noise and/or hovering capabilities, lighter-than-air vehicles present themselves as a competitive platform compared to fixed-wing and rotor based vehicles. The limiting factor in their widespread use in autonomous applications comes from the complexity of the control task. The so-called airships are highly-susceptible to aerodynamic forces and pose complex nonlinear system dynamics that complicate their modeling and control. Model-free control lends itself well as a solution to this type of problem, as it derives its control policies using input-output data, and can therefore learn complex dynamics and handle uncertain or unknown parameters and disturbances. In this work, two multi-input multi-output algorithms are presented on the basis of optimal control theory. Leveraging results from reinforcement learning, a single layer, partially connected neural network is formulated as a value function appropriator in accordance with Weierstrass higher-order approximation theorem. The so-called critic-network is updated using gradient descent methods on the mean-squared error of the temporal difference equation. In the single-network controller, the control policy is formulated as a closed form equation that is parameterized on the weights of the critic-network. A second controller is proposed that uses a second single-layer partially connected neural network, the actor-network, to calculate the control action. The actor-network is also updated using gradient descent on the squared error of the temporal difference equation. The controllers are employed in a highly realistic simulation airship model in nominal conditions and in the presence of external disturbances in the form of turbulent wind. To verify the validity and test the sensitivity of the algorithms to design parameters (the initialization of certain terms), ablation studies are carried out with multiple initial parameters. Both of the proposed algorithms are able to track the desired waypoints in both the nominal and disturbed flight tests. Furthermore, the performance of the controllers is compared to a modern, state-of-the-art multi-input multi-output controller. The two proposed controllers outperform the comparison controller in all but one flight test, with up to four fold reduction in the integral absolute error and integral time absolute error metrics. On top of the quantitative improvements seen in the proposed controllers, both controllers demonstrate a reduction in system oscillation and actuator chattering with respect to the comparison algorithm.