In this paper, we have developed a new fault detection method based on vibration signal for rotor machinery. Generally, many methods related to detection of rotor fault exist and more advanced methods are continuously developing past several years. However, there are some problems with existing methods. Oftentimes, the accuracy of fault detection is affected by vibration signal change due to change of operating environment since the diagnostic model for rotor machinery is built by the data obtained from the system. To settle a this problems, we build a rotor diagnostic model by using feature residual based on vibration signal. To prove the algorithm’s performance, a comparison between proposed method and the most used method on the rotor machinery was conducted. The experimental results demonstrate that the new approach can enhance and keeps the accuracy of fault detection exactly although the algorithm was applied to various systems.