We introduce Dynamic Deep Neural Networks (D2NN),
a new type of feed-forward deep neural network that allows
selective execution. Given an input, only a subset of D2NN
neurons are executed, and the particular subset is determined
by the D2NN itself. By pruning unnecessary computation
depending on input, D2NNs provide a way to improve
computational efficiency. To achieve dynamic selective
execution, a D2NN augments a feed-forward deep neural
network (directed acyclic graph of differentiable modules)
with controller modules. Each controller module is
a sub-network whose output is a decision that controls
whether other modules can execute.