||Benchmark generation plays an important role in all kinds of research areas especially in optimization related problems. For most of the hard problems such as the NP-hard family, the performance of the solutions or solvers is evaluated based on experimental results. However, the current approaches on benchmark generation have two draw-backs, 1), inefficiency in tuning the parameters in the generator, and 2), bias towards some special solvers. One of the ill effects is that many researchers do not trust the experimental results, hence many heuristic algorithms are not accepted. In this work, a new framework is proposed for test case generation based on the Design and Analysis of Engineering Experiment methods. In this framework, the effect of parameters can be analyzed by conducting a small number of experiments, therefore the test cases of desired hardness can be generated efficiently and accurately. The new framework is examined in three phases. In the first phase, a special test case generator for the Multiple Unit Combinatorial Auction problem is studied. In the second phase, the framework is applied to study the general test cases for the Combinatorial Auction problem. In the third phase, it is further applied in a real-life application utilizing this methodology to evaluate the performance of RFID middle-wares.