This paper advances prior work by proposing a joint learning

framework to simultaneously identify the spatial and temporal

extents of the action of interest in training videos. To get pixel-level

localization results, our method uses dense trajectories extracted

videos as local features to represent actions. We first

present a trajectory split-and-merge algorithm to segment a video

into the background and several separated foreground moving

objects. In this algorithm, the inherent temporal smoothness

of human actions is exploited to facilitate segmentation. Then,

with the latent SVM framework on segmentation results, spatial

and temporal extents of the action of interest are treated as

latent variables that are inferred simultaneously with action

recognition. Experiments on two challenging datasets show that

action detection with our learned spatial and temporal extents is

superior than state-of-the-art methods.