Ultracold atomic systems have provided an ideal platform to study the physics of strongly interacting many body systems in an unprecedentedly controlled and clean environment. And, since fermions are the building blocks of visible matter, being naturally motivated we focus on the physics of ultracold fermionic systems in this thesis. There have been many recent experimental developments in these systems such as the creation of synthetic gauge fields, realization of dimensional crossover and realization of systems with synthetic dimensions. These developments pose many open theoretical questions, some of which we address in this thesis. We start the discussion by studying the spectral function of an ideal spin-12 Fermi gas in a harmonic trap in any dimensions. We discuss the performance of the local density approximation (LDA) in calculating the spectral function of the system by comparing it to exact numerical results. We show that the LDA gives better results for larger number of particles and in higher dimensions. Fermionic systems with quasi two dimensional geometry are of great importance because of their connections to the high-Tc superconducting cuprate materials. Keeping this in mind, we consider a spin-12 fermionic system in three dimensions interacting with a contact interaction and confined by a one dimensional optical potential in one direction. Using the Bogoliubov-de Gennes formalism, we show that with increasing the depth of the optical potential the three dimensional superfluid evolves into a two dimensional one by looking at the shifts in the radio-frequency spectrum of the system and the change in the binding energy of the pairs that are formed. The next topic of interest is studying the effect of synthetic gauge fields on the ultracold fermionic systems. We show that a synthetic non-Abelian Rashba type gauge field has experimentally observable signatures on the size and shape of a cloud of a system of non-interacting spin-12 Fermi system in a harmonic trap. Also, the synthetic gauge field in conjunction with the harmonic potential gives rise to ample possibilities of generating novel quantum Hamiltonians like the spherical geometry quantum Hall, magnetic monopoles etc. We then address the physics of fermions in “synthetic dimensions”. The hyperfine states of atoms loaded in a one dimensional optical lattice can be used as an extra dimension, called the synthetic dimension (SD), by using Raman coupling. This way a finite strip Hofstadter model is realized with a tunable flux per plaquette. The experimental realization of the SD system is most naturally possible in systems which also have SU(M) symmetric interactions between the fermions. The SU(M) symmetric interactions manifest as long-ranged along the synthetic dimension and is the root cause of all the novel physics in these systems. This rich physics is revealed by a mapping of the Hamiltonian of the system to a system of particles interacting via an SU(M) symmetric interaction under the influence of an SU(M) Zeeman field and a non-Abelian SU(M) gauge field. For example, this equivalence brings out the possibility of generating a non-local interaction between the particles at different sites; while the gauge filed mitigates the baryon (SU(M) singlet M-body bound states) breaking effect of the Zeeman field. As a result, the site localized SU(M) singlet baryon gets deformed and forms a “squished baryon”. Also, finite momentum dimers and resonance like states are formed in the system. Many body physics in the SD system is then studied using both analytical and numerical (Density Matrix Renormalization Group) techniques. This study reveals fascinating possibilities such as the formation of Fulde-Ferrell-Larkin-Ovchinnikov states even without any “imbalance” and the possibility to evolve a “ferromagnet” to a “superfluid” by the application of a magnetic field. Other novel fermionic phases with quasi-condensates of squished baryons are also demonstrated. In summary, the topics addressed in this thesis demonstrate the possibilities and versatilities of the ultracold fermionic systems used in conjunction with synthetic gauge fields and dimensions