This study proposes a new diagnosis method for identifying the vibration data of the multi-level fault of gear. The diagnosis method is based on the feature representation of the vibration signal and optimized machine learning model. Firstly, a deep Auto-Encoder (AE) bottleneck network is constructed with the two hidden layers to extract the fault features of gear vibration data, named AE-BtF. In which, the AE-BtF use the basic unsupervised learning algorithm to reveal the significant characteristics in the complex data with the nonlinear, non-station properties. The obtained features can provide good discriminability for fault diagnosis task. Secondly, an optimal classifier model is formed to perform supervised fine-tuning and classification. This model is based on the least square support vector machine (LSSVM) classifier and chemical reaction optimization algorithm (CRO), named CRO-LSSVM. The meta-heuristics CRO algorithm is used to exploit the appropriate parameters for the LSSVM. Based on the vibration data of gear fault status, the proposed AE-BtF-CRO-LSSVM technique shows a good ability for identifying the gear fault accuracy. The diagnosis results have demonstrated that the AE-BtF based feature extraction in conjunction with the CRO-LSSVM classifier model can achieve higher accuracies than the other popular classifier models to 60%