Add loss value metric based on optimal performance definition #66
Conversation
|
Failing tests seem unrelated, but do you have an idea? |
|
Linter fixed in main branch. |
|
No, but I figured it wasn't on my branch indeed. Ill take another look! |
|
The other tests are still failing. I think tf-keras needs to be a req for ASReview. But more importantly, why is it importing transformers when running our tests... |
jteijema
force-pushed
the
loss-metric
branch
from
ccaafb7
to
11589a6
Compare
|
Tests are failing because of something fixed in #67 . Consider merging 67 and merging main into this branch to get the tests functional again. |
|
Oke, remove the draft status of that pr. |
| def _loss_value(labels): | ||
| positive_doc_ratio = sum(labels) / len(labels) | ||
| triangle_before_perfect_recall = positive_doc_ratio * 0.5 | ||
| aera_under_recall_curve = metrics.auc(*_recall_values(labels)) |
There was a problem hiding this comment.
Isn't this not something like y*np.diff(x)?
In that case, we don't need the dependency (which is installed via asreview, but yeah).
J535D165
changed the title
| def _loss(labels): | ||
| return _loss_value(labels) | ||
|
|
|
This PR needs a test and documentation. I'm not convinced this loss metric computes what's described in the PR description. The following tests both fail: def test_metric_loss_best():
labels_best = [1, 1, 1, 0]
loss = _loss(labels_best)
assert_almost_equal(loss, 0)
def test_metric_loss_worst():
labels_worst = [0, 0, 1, 1]
loss = _loss(labels_worst)
assert_almost_equal(loss, 1) |
A loss metric value based on the distance between the perfect recall curve and the actual recall curve.