||Traditional Dolby™ 5.1 surround sound systems need six speakers (center, left, right, left rear, right rear and subwoofer) to deliver the surround sound effects. A Virtual Headphone-based Surround Sound (VHSS) System processes the raw signals of Dolby™ 5.1 standard to deliver simulated surround sound effects to listeners through a pair of headphones. The key components of a VHSS system are non-individualized Head-related Transfer Function (HRTF) filters that contain important information necessary to simulate directional sound cues. The purpose of this research is to optimize the complexity and performance of a VHSS system through studying the effects of simplifying and manipulating the spectra of HRTFs. Three experiments were conducted to achieve the goal. Four levels of complexities of the HRTF filters (128, 64, 32 and 18 coefficients) were studied for their effects on sound localization errors. Results indicated that reducing the number of coefficients in HRTFs significantly affected the localization errors for all directions (p<0.05) except directions 0°, 90° and 270° (p>0.1). An optimized set of filters with different complexities was obtained and implemented as a prototype VHSS system. A double-blinded usability experiment verified that the prototype VHSS system produced significantly better surround sound effects than the corresponding Dolby™ stereo channels (p=0.028). Nevertheless, front-back confusions were still the major cause for the degraded performance. In the last experiment, spectra of HRTFs, divided into six frequency bands, were enhanced at three different levels to obtain higher localization accuracy. Results showed that 12dB and 18dB enhancements of five of the six frequency bands have significantly reduced localization errors for directions 0° 45°, 135°, 180°, 225° than conditions without any enhancement (p<0.05). Actual or potential applications include the design of a cheaper and better VHSS system.