This paper suggests that a simple modification to the initial search direction can also substantially improve the training efficiency of almost all major optimization methods. It was discovered that if the initial search direction is locally modified by a gain value used in the activation function of the corresponding node, significant improvements in the

convergence rates can be achieved irrespective of the optimization algorithm used. Furthermore the proposed method is robust, easy to compute, and easy to implement into well known nonlinear conjugate gradient algorithms