Transpositions on fetch_olivetti_faces

[dcfq]

Hello guys, I'm a CS and AI teacher using scikit-learn for educational purposes atm.
While building class resources for a brief introduction to unsupervised learning I decided to do a small face recog project using a small dataset to motivate students to take interest in the subject. After finishing the project and deciding I'd want them to build their own small system with the faces of all participant students, i decided to dive into the code of fetch_olivetti_faces to check what if any preprocessing was done to it before returning the data.

While perusing the code there I noticed a matrix is transposed twice (faces right after remote download at line 116 with .T operator and then at line 127 a few lines before returning using .transpose()), why the double transpose?

[NicolasHug]

I think it's just a matter of getting the data into the right shape at the end. The downloaded data probably doesn't come in the expected shape or alignment (storing columns instead of rows, that kind of stuff).

Unless you think there's an issue here, it's probably not worth worrying about it too much. These transposes are quite cheap

[dcfq]

I think it's just a matter of getting the data into the right shape at the end. The downloaded data probably doesn't come in the expected shape or alignment (storing columns instead of rows, that kind of stuff).

Unless you think there's an issue here, it's probably not worth worrying about it too much. These transposes are quite cheap

Thanks for your participation @NicolasHug, i'm not that concerned with things not being right because I believe things are 100% right and i don't believe this discussion to be an issue in disguise (hence why I made this a discussion), I just want to know what is happening there and why because the students are supposed to repeat the process with their own photos.

The original dataset consists of 400 64x64 images and the returning data is 400 4096-length vectors, so each image is stored as a vector (either column or line) instead of a matrix.

However for teaching purposes a bunch of numbers is not as interesting as a a reconstructed eigenface (and using their faces is is way better to make them take notes of backgroud, lighting, etc., so i'm trying to avoid them making errors that would take the focus away from what they are supposed to be focused on.

[jnothman]

The first transpose seems to be transposing samples vs features. It puts samples on the first (0) axis. The second is done in three dimensions and transposes each image within the dataset.

[dcfq]

Yeah @jnothman I think you're right, also, maybe the first transpose helps with copying/hardware operations (to take advantage of locality or something like that?)

However, the second one i don't know, it's not that apparent to me the reason for the second one.

[jnothman]

Neither of the operations will change the order of data in memory. Transpose only changes the striding of the array, not its data. So it is definitely not intended for computational efficiency.

[dcfq]

@NicolasHug and @jnothman thanks a lot for the help and taking the time to discuss with me, i think I solved it, had to hack a bit of scikit's _base.py functions, was half-expecting it to take me much more time (that's why I asked first).

1. Images are stored at the original repository as: 4096 x 400 (features vs samples) (I'm still wondering why it was stored this way, but won't have any effect on the assignment, so only curious)
2. The first transpose gets 400 x 4096 (samples vs features), but also causes images to be flipped across the main diagonal axis (as usual for the transpose operator over images)
3. The second transpose is done for each image once they are 64x64 matrices again, re-flipping them.

End results: Students won't have to do anything since they won't store this anywhere, so... yay!? =P