The thesis deals with reverse techniques in software engineering. It presents practical application of software reverse engineering, used tools and approaches. The topic of instruction decoding is discussed in detail. Two basic methods are presented-linear sweep and recursive descent. Their strengths and weaknesses are highlighted. Subsequently a decompiler developed by AVG Technologies is introduced. The decompiler is retargetable. This feature allows to decompile applications of multiple platforms into various target languages. The aim of  the thesis is to design and implement algorithm for  decoding binary files into high-level representation. The designed algorithm is based on modified recursive descent algorithm, which uses control flow information. In order to achieve more accurate decoding results, symbol table records and other additional information are used. The proposed algorithm was implemented for the AVG Technologies retargetable decompiler. The tests showed that the implemented algorithm improved the function detection in decoded programs. Furthermore, the implemented solution allows to decode files that could not be analysed using the previous version of the decompiler.

The architecture of virtual machine (VM) interpreters has long been a subject of researchand discussion. The initial trend of stack-based interpreters was shortly thereafterchallenged by research showing the performance advantages of virtual register machines. Despite this, many VM interpreters are still stack-based, with some notable exceptions, like Lua, Android Runtime, and its predecessor Dalvik. A register architecture isusually associated with greater overhead from instruction dispatch, and to some extent, instruction decoding. By designing, and implementing a novel technique that replaces the conventional way of decoding instructions, this thesis attempts to reduce that overhead. More specifically, a VM interpreter is developed as an artifact of design-science research. The novel technique is then evaluated through benchmarking in various configurations. As the results indicate, however, using this technique showed no performance advantage, as the resulting machine instructions are exactly the same after compiler optimization. This suggests that there is no apparent decoding overhead to begin with. As a result, register-based VMs seem to not suffer from any dispatch related overhead, other than the fact that there are more operands per instruction to access. Source code is available on GitHub, at https://github.com/kaspr61/RackVM.