Information changes (increases/decreases) upon adding duplicate features

[zuranski]

Hello authors,

I was analyzing the performance of the mifs feature selection and stumbled on a strange property. My features are continuous, my target is categorical. In my design matrix I accidentally used one pair of duplicate features, but to my surprise they were often both selected by the algorithm (I used JMIM). I managed to trace it to a property of the function _mi_dc(x, y, k). For features 1, 2, 3, with 1, 2 being exactly the same I obtained the following results of the _mi_dc (data attached):

• [1] 0.302
• [2] 0.302 - same, as expected
• [1, 2] 0.400 - strange, 1 and 2 are duplicates there is no extra information, but information increased
• [3] 0.389
• [1, 3] 0.447 - increase, as expected
• [2, 3] 0.447 - same increase, as expected
• [1, 2, 3] 0.423 - very strange, information decreased when adding a redundant feature

Unless I don't understand some aspect of the calculation, it looks like adding a strictly redundant feature confuses the MI calculation. Could this be rectified? Or maybe there are some alternative options of calculating the joint MI? Of course one can detect that and skip exact duplicates, but it makes me a little hesitant about using the algorithm with many closely related features.

All my best,
Andrzej

mifs_data.txt