||Future wireless communication networks will be required to provide a wide range of high-quality and high-data-rate traffic, including video, data and voice transmission. However, the hostile mobile links impose a critical limit on the system performance. Moreover, the scarcity of frequency resources indicates a shortage in wireless communication capacity. One of the most recent and promising techniques for realizing extraordinary spectral efficiencies over wireless links is the MIMO (Multiple-Input Multiple-Output) antenna system. In particular, space-time trellis codes have drawn considerable attention due to their ability to achieve high frequency-efficient transmission with high diversity and coding gain. In this thesis, we investigate a layered space-time trellis coded MIMO architecture (LST-MIMO), which is used to achieve high-speed data transmission over wireless links. Multiple antennas are employed to transmit parallel groups of individually space-time trellis coded information data streams, while the receiver is equipped with multiple antennas to decode the received signals. In this thesis, a Minimum Mean Square Error based Successive Interference Cancellation (MMSE-SIC) algorithm is proposed to mitigate the co-channel interference (CCI) between different streams and achieve a higher diversity gain reception. An ordering scheme is derived to determine the optimum detection sequence of multiple data streams, so as to suppress error propagation. The LST-MIMO architecture is further investigated in conjunction with the Orthogonal Frequency Division Multiplexing (OFDM) in order to combat frequency-selective fading. In addition, the methodology of channel estimation is investigated for such system, especially in a non-sample-spaced multi-path propagation channel where power leakage results in estimation error floor. Two novel channel estimators are proposed to solve these problems and improve the estimation accuracy. Extensive simulation results show that the proposed Ordered MMSE-SIC algorithm outperforms the conventional decoding method significantly. In quasi-static fading channel, the proposed algorithm brings approximately 3.5 dB SNR reduction at Symbol Error Rate (SER) of 1%, which comes within only 4.8 dB to the theoretical channel capacity. In frequency-selective fading channel, the performance improvement is as high as 3 dB at 1% SER level. Hence, our proposed method can effectively improve the system performance and help to achieve high-speed wireless data transmission with the LST-MIMO architecture.