A Survey of Automatic Modulation Classification Techniques: Classical Approaches and New Trends.This is also a very good papaer like that of azzouz and nandi

Automatic modulation recognition of mpsk signals using constellation rotation and its 4th order cumulant