Lateral semiconductor quantum dot structures have been proposed as an effective quantum bit (qubit) for quantum computation. A single excess electron with the freedom to move between two capacitively coupled quantum dots creates a `pseudo'-spin system with the same qubit behavior as the more natural two level system of a single electron spin. The excess electron in the double dot system is restricted to one of the two dots, thereby creating two separate and distinct states (usually referred to as |L> and |R>). The benefit of these charge qubits lie in the relative ease with which they can be manipulated and created. Experiments have been performed in this area and have shown controllable coherent oscillations and thus efficient single-qubit operations. However, the decoherence rates observed in the experiments is still quite high, making double dot charge qubits not very appealing for large-scale implementations. The following work describes the effect of the electromagnetic (EM) environment of the double quantum dot system on the decoherence of the charge state. Sources of decoherence in similar systems have been extensively investigated before and this paper follows a close theoretical framework to previous work done in the area. The effect of the EM environment can been seen in the calculations discussed below, although it is clear that the decoherence seen in experiments cannot be fully explained by the voltage fluctuations as they are investigated here. The limitations of the calculations are discussed and improvements are suggested.