You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
In the process of generating embeddings from a trained deep learning model, we perform inference on an input image and save the resulting embedding, which captures the semantic reconstructions of salient features as determined by the model. To enhance the utility of these embeddings, I propose saving the reconstruction loss alongside the embedding vector.
The reconstruction loss, calculated as the difference between the input image and the model's reconstructed output, provides valuable insights into the semantic content and anomalies present in the input image:
Images with expected semantics that align well with the model's training data will exhibit a smaller reconstruction loss, indicating that the model can effectively capture and reconstruct the salient features.
Images containing rare, unexpected, or anomalous semantics will result in a larger reconstruction loss, as the model may struggle to accurately reconstruct the input due to the presence of features outside its learned representation.
Real-world applications:
Monitoring changes in satellite imagery: By comparing embeddings and reconstruction losses of a region (e.g., Kiev) before and after significant events (war), we can detect and quantify the extent of semantic changes. Pre-event images will likely have smaller losses, while post-event images containing destruction, damage, and other anomalies will have higher losses.
Anomaly detection in various domains: The reconstruction loss can serve as a valuable metric for detecting anomalies, such as rare events (city floods, locust plagues), unusual semantics (algae blooms, green pools), or "noise" (fog, smog, ships in the ocean, image artifacts). By setting appropriate thresholds on the reconstruction loss, we can flag images containing such anomalies for further analysis.
Implementation: Modify the embedding generation pipeline to calculate and save the reconstruction loss alongside the embedding vector. This can be achieved by comparing the input image with the model's reconstructed output using e.g. the same loss, or others (MSE, ...).
Action Items:
@yellowcap to add saving the reconstruction losses in the embedding generation pipeline.
@stephen-downs to assess how to surface this in the app.
I think this change no only makes our outputs much more useful, but also of measurable relative confidence, AND highlight of operational bias (making the loss a feature, not a thing to get rid of).
In the process of generating embeddings from a trained deep learning model, we perform inference on an input image and save the resulting embedding, which captures the semantic reconstructions of salient features as determined by the model. To enhance the utility of these embeddings, I propose saving the reconstruction loss alongside the embedding vector.
The reconstruction loss, calculated as the difference between the input image and the model's reconstructed output, provides valuable insights into the semantic content and anomalies present in the input image:
Real-world applications:
Implementation: Modify the embedding generation pipeline to calculate and save the reconstruction loss alongside the embedding vector. This can be achieved by comparing the input image with the model's reconstructed output using e.g. the same loss, or others (MSE, ...).
Action Items:
I think this change no only makes our outputs much more useful, but also of measurable relative confidence, AND highlight of operational bias (making the loss a feature, not a thing to get rid of).
cc @MaceGrim @danhammer for the utility feedback.
The text was updated successfully, but these errors were encountered: