Fixing error metrics as discussed in #27 #50
Conversation
update from upstream
Modify keypoints error to return nan values for non-visible keypoints. Modify logger to mask out euclidean error for non-visible keypoints, and to correct confidence scores (to 1 minus value) for these keypoints
should store new copy of prediction and errors for each epoch
prevents warning that future h5py will change default file mode to read only
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
|
This push is to fix the issues discussed in open issue #27 |
|
I'd get rid of all the code for detecting nans and just use |
deepposekit/callbacks.py
Outdated
| confidence_mean = confidence.mean() | ||
| # keypoint_percentile = np.percentile( | ||
| # [euclidean, confidence], [0, 5, 25, 50, 75, 95, 100], axis=1 | ||
| # ).T |
There was a problem hiding this comment.
Just delete this commented code. git keeps track of changes. There's no need to keep old code.
|
We do have to detect nans still, at least to do what I've implemented here,
because the confidence values for non-visible target keypoints should be 0.
But I can make the other changes you mention, and sorry about the commented
code...that's deleted in my version of the code I must have pushed the
wrong version.
The code is a bit complicated because I didn't want to remove the option to
do confidence thresholding that you had already built in. If it's ok to get
rid of that, the code is much simpler.
…On Sun, Apr 26, 2020 at 4:45 AM Jake Graving ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In deepposekit/callbacks.py:
> - euclidean_perc, confidence_perc = keypoint_percentile
-
- euclidean_mean, confidence_mean = np.mean([euclidean, confidence], axis=1)
+ y_visible = y_visible[mask]
+
+ if ~y_visible.all():
+ euclidean = euclidean[y_visible]
+ confidence[~y_visible] = 1-confidence[~y_visible]
+
+ euclidean_perc = np.percentile(euclidean, [0, 5, 25, 50, 75, 95, 100])
+ euclidean_mean = euclidean.mean()
+ confidence_perc = np.percentile(confidence, [0, 5, 25, 50, 75, 95, 100])
+ confidence_mean = confidence.mean()
+ # keypoint_percentile = np.percentile(
+ # [euclidean, confidence], [0, 5, 25, 50, 75, 95, 100], axis=1
+ # ).T
Just delete this commented code. git keeps track of changes. There's no
need to keep old code.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
<#50 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABJNQGS56CNHGNTZGUL747TROPYAVANCNFSM4MLRFRPA>
.
|
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. |
|
This issue has been automatically closed because it has not had recent activity. |
I implemented these changes, plus two minor additions.
First, for keypoints which are supposed to be not-visible, it seemed reasonable to replace their confience score with 1 minus the confidence score, for the purposes of reporting online during training, since the target confidence for these points is 0. Euclidean errors however should be masked out, since they are not defined for non-visible keypoints. These changes only affect values printed to command line during training, but not logged values to disk. This is fine, because the keypoint errors logged to disk includes NaN values for non-visible keypoints
Second, in debugging these changes, I noticed that the logger callback appeared to be broken. Instead of adding a new entry for each epoch, it appears to have a first entry with all zeros, and a second entry with only the most recent values. This looked like a single line got misplaced outside a conditional statement, and I tested that logger works correctly after making this change.