There are two main contributions in this paper. The first is to propose a joint random field (JRF) model that extends CRF by introducing auxiliary latent variables to characterize visual scene over time and enhance moving object detection in vide