Pulsars are laboratories for extreme physics unachievable on Earth. As individual sources and possible orbital companions can be used to study magnetospheric, emission, and superfluid physics, general relativistic effects, and stellar and binary evolution. As populations they exhibit a wide range of sub-types, with parameters varying by many orders of magnitude signifying fundamental differences in their evolutionary history and potential uses. There are currently around 2200 known pulsars in the Milky Way, the Magellanic clouds, and globular clusters, most of which have been discovered with radio survey observations. These observations, as well as being suitable for detecting the repeating signals from pulsars, are well suited for identifying other transient astronomical radio bursts that last just a few milliseconds that either singular in nature, or rarely repeating. Prior to the work of this thesis non-repeating radio transients at extragalactic distances had possibly been discovered, however with just one example status a real astronomical sources was in doubt. Finding more of these sources was a vital to proving they were real and to open up the universe for millisecond-duration radio astronomy. The High Time Resolution Universe survey uses the multibeam receiver on the 64-m Parkes radio telescope to search the whole visible sky for pulsars and transients. The temporal and spectral resolution of the receiver and the digital back-end enable the detection of relatively faint, and distant radio sources. From the Parkes telescope a large portion of the Galactic plane can be seen, a rich hunting ground for radio pulsars of all types, while previously poorly surveyed regions away from the Galactic plane are also covered. I have made a number of pulsar discoveries in the survey, including some rare systems. These include PSR J1226-6208, a possible double neutron star system in a remarkably circular orbit, PSR J1431-471 which is being eclipsed by its companion with each orbit, PSR J1729-2117 which is an unusual isolated recycled pulsar, and PSR J2322-2650 which has a companion of very low mass - just 7 x 10⁻⁴M_⊙, amongst others. I begin this thesis with the study of these pulsars and discuss their histories. In addition, I demonstrate that optical observations of the companions to some of the newly discovered pulsars in the High Time Resolution Universe survey may result in a measurement of their age and that of the pulsar. I have discovered five new extragalactic single radio bursts, confirming them as an astronomical population. These appear to occur frequently, with a rate of 1.0^+0.6_-0.5 x 10⁴ sky⁻¹ day⁻¹. The sources are likely at cosmological distances - with redshifts between 0.45 and 1.45, making them more than half way to the Big Bang in the most distant case. This implies their luminosities must be enormous, 10³¹ to 10³³ J emitted in just a few milliseconds. Their source is unknown but I present an analysis of the options. I also perform a population simulation of the bursts which demonstrates how their intrinsic spectrum could be measured, even for unlocalised FRBs: early indications are that the spectral index of FRBs < 0.