In a single chip multiprocessor, the ratio of off-chip communication time and on-chip processing time become larger and larger along with the advancement of VLSI technology. Hence, the number of off-chip memory accesses will become a dominant factor of system performance. We have developed a hardware/software design of an on-chip allocatable cache. In this design, we take into account the pre-measured cache size requirement of the executed program. The operating system can then allocate proper cache size to the corresponding processor by cache submodule re-allocation. Hence, programs with different cache size requirements can then be adjusted their cache size dynamically for proper cache allocation in order to increase the overall hit ratio of on-chip caches as well as the system performance. To validate the achievable the performance improvement, we designed simulators of the allocatable cache design, the dedicated cache design, and the fully shared cache design together with the multiprocessor simulation environment. We extracted execution traces from a set of real programs and measured their cache hit ratios on different sizes of cache capacities. We performed the single-chip multiprocessor simulation with these data. We randomize the time periods of cache characteristics changes to replace the executed programs in each processor during the multiprocessor simulation. The performance experiments reveal that the allocatable cache design obtains the best overall cache hit ratio and total program execution time. Although the fully shared cache design can have performance near that of the allocatable cache design, it has a draw back of much larger interconnection cost.