信息熵，是随机变量或整个系统的不确定性。熵越大，随机变量或系统的不确定性就越大。相对熵，用来衡量两个取值为正的函数或概率分布之间的差异。交叉熵，用来衡量在给定的真实分布下，使用非真实分布所指定的策略消除系统的不确定性所需要付出的努力的大小。(Information entropy is the uncertainty of random variables or the whole system. The greater the entropy, the greater the uncertainty of the random variable or system.
Relative entropy, used to measure the difference between two positive functions or probability distributions.
Cross entropy is used to measure the size of the effort required to eliminate the uncertainty of the system by using the strategy specified by the non real distribution in a given real distribution.
Relative entropy = cross entropy - Information entropy:)