Séminaire Maths-Physique

A modified Ziv-Merhav cross entropy estimator

by Raphaël Grondin (McGill University)

Pellos (207) (1R2)

Pellos (207)



Specific relative entropy between two probability measures plays a fundamental role in many tasks in information theory , statistics and physics. In many practical cases the measures are not known and it is therefore natural to seek a method to estimate the relative entropy given realisations of the two underlying measures. Because the  specific relative entropy differs from the specific cross entropy  by the specific entropy  which can be universally estimated following the Lempel--Ziv parsing algorithm, we will focus on estimation of the specific cross entropy . We propose a slight modification of a well-known cross entropy estimator called the Ziv-Merhav estimator and prove that it is strongly consistent for a large class of decoupled measures.