Ryfujk

Antler Helmet: Can it work?

Why is there no army of Iron-Mans in the MCU?

Who can trigger ship-wide alerts in Star Trek?

Two different pronunciation of "понял"

Statistical model of ligand substitution

Windows 10: How to Lock (not sleep) laptop on lid close?

How are presidential pardons supposed to be used?

I'm having difficulty getting my players to do stuff in a sandbox campaign

I'm thinking of a number

Are my PIs rude or am I just being too sensitive?

Why is "Captain Marvel" translated as male in Portugal?

Estimate capacitor parameters

Why use gamma over alpha radiation?

How do you clear the ApexPages.getMessages() collection in a test?

New Order #5: where Fibonacci and Beatty meet at Wythoff

What would be Julian Assange's expected punishment, on the current English criminal law?

Is there folklore associating late breastfeeding with low intelligence and/or gullibility?

The following signatures were invalid: EXPKEYSIG 1397BC53640DB551

Estimated State payment too big --> money back; + 2018 Tax Reform

Do working physicists consider Newtonian mechanics to be "falsified"?

Complexity of many constant time steps with occasional logarithmic steps

What is the largest species of polychaete?

Can a non-EU citizen traveling with me come with me through the EU passport line?

Problem when applying foreach loop

What loss function to use when labels are probabilities?

.everyoneloves__top-leaderboard:empty,.everyoneloves__mid-leaderboard:empty,.everyoneloves__bot-mid-leaderboard:empty{ margin-bottom:0;
}

1

$begingroup$

What loss function is most appropriate when training a model with target values that are probabilities? For example, I have a 3-output model with x=[some features] and y=[0.2, 0.3, 0.5].

It seems like something like cross-entropy doesn't make sense here since it assumes that a single target is the correct label.

Would something like MSE (after applying softmax) make sense, or is there a better loss function?

neural-networks loss-functions probability-distribution

share|improve this question

asked 4 hours ago

Thomas JohnsonThomas Johnson

1083

New contributor

Thomas Johnson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

1

$begingroup$

What loss function is most appropriate when training a model with target values that are probabilities? For example, I have a 3-output model with x=[some features] and y=[0.2, 0.3, 0.5].

It seems like something like cross-entropy doesn't make sense here since it assumes that a single target is the correct label.

Would something like MSE (after applying softmax) make sense, or is there a better loss function?

neural-networks loss-functions probability-distribution

share|improve this question

asked 4 hours ago

Thomas JohnsonThomas Johnson

1083

New contributor

Thomas Johnson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

add a comment | 

$begingroup$

What loss function is most appropriate when training a model with target values that are probabilities? For example, I have a 3-output model with x=[some features] and y=[0.2, 0.3, 0.5].

It seems like something like cross-entropy doesn't make sense here since it assumes that a single target is the correct label.

Would something like MSE (after applying softmax) make sense, or is there a better loss function?

neural-networks loss-functions probability-distribution

share|improve this question

asked 4 hours ago

Thomas JohnsonThomas Johnson

1083

New contributor

Thomas Johnson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

$endgroup$

share|improve this question

asked 4 hours ago

Thomas JohnsonThomas Johnson

1083

New contributor

Thomas Johnson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

share|improve this question

asked 4 hours ago

Thomas JohnsonThomas Johnson

1083

New contributor

Thomas Johnson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 4 hours ago

Thomas JohnsonThomas Johnson

1083

New contributor

Thomas Johnson is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.

asked 4 hours ago

Thomas JohnsonThomas Johnson

1083

1 Answer
1

active

oldest

votes

1

$begingroup$

Actually, the cross-entropy loss function would be appropriate here, since it measures the "distance" between a distribution $q$ and the "true" distribution $p$.

You are right, though, that using a loss function called "cross_entropy" in many APIs would be a mistake. This is because these functions, as you said, assume a one-hot label. You would need to use the general cross-entropy function,

$$H(p,q)=-sum_{xin X} p(x) log q(x).$$
$ $

Note that one-hot labels would mean that
$$
p(x) =
begin{cases}
1 & text{if }x text{ is the true label}\
0 & text{otherwise}
end{cases}$$

which causes the cross-entropy $H(p,q)$ to reduce to the form you're familiar with:

$$H(p,q) = -log q(x_{label})$$

share|improve this answer

answered 4 hours ago

Philip RaeisghasemPhilip Raeisghasem

963119

$endgroup$

add a comment | 

Your Answer

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "658"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
noCode: true, onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});

Thomas Johnson is a new contributor. Be nice, and check out our Code of Conduct.

draft saved

draft discarded

Sign up or log in

StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});

Sign up using Google

Sign up using Facebook

Sign up using Email and Password

Post as a guest

Name

Email

Required, but never shown

StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fai.stackexchange.com%2fquestions%2f11816%2fwhat-loss-function-to-use-when-labels-are-probabilities%23new-answer', 'question_page');
}
);

Post as a guest

Name

Email

Required, but never shown

1

$begingroup$

Actually, the cross-entropy loss function would be appropriate here, since it measures the "distance" between a distribution $q$ and the "true" distribution $p$.

You are right, though, that using a loss function called "cross_entropy" in many APIs would be a mistake. This is because these functions, as you said, assume a one-hot label. You would need to use the general cross-entropy function,

$$H(p,q)=-sum_{xin X} p(x) log q(x).$$
$ $

Note that one-hot labels would mean that
$$
p(x) =
begin{cases}
1 & text{if }x text{ is the true label}\
0 & text{otherwise}
end{cases}$$

which causes the cross-entropy $H(p,q)$ to reduce to the form you're familiar with:

$$H(p,q) = -log q(x_{label})$$

share|improve this answer

answered 4 hours ago

Philip RaeisghasemPhilip Raeisghasem

963119

$endgroup$

add a comment | 

1

$begingroup$

Actually, the cross-entropy loss function would be appropriate here, since it measures the "distance" between a distribution $q$ and the "true" distribution $p$.

You are right, though, that using a loss function called "cross_entropy" in many APIs would be a mistake. This is because these functions, as you said, assume a one-hot label. You would need to use the general cross-entropy function,

$$H(p,q)=-sum_{xin X} p(x) log q(x).$$
$ $

Note that one-hot labels would mean that
$$
p(x) =
begin{cases}
1 & text{if }x text{ is the true label}\
0 & text{otherwise}
end{cases}$$

which causes the cross-entropy $H(p,q)$ to reduce to the form you're familiar with:

$$H(p,q) = -log q(x_{label})$$

share|improve this answer

answered 4 hours ago

Philip RaeisghasemPhilip Raeisghasem

963119

$endgroup$

add a comment | 

$begingroup$

Actually, the cross-entropy loss function would be appropriate here, since it measures the "distance" between a distribution $q$ and the "true" distribution $p$.

You are right, though, that using a loss function called "cross_entropy" in many APIs would be a mistake. This is because these functions, as you said, assume a one-hot label. You would need to use the general cross-entropy function,

$$H(p,q)=-sum_{xin X} p(x) log q(x).$$
$ $

Note that one-hot labels would mean that
$$
p(x) =
begin{cases}
1 & text{if }x text{ is the true label}\
0 & text{otherwise}
end{cases}$$

which causes the cross-entropy $H(p,q)$ to reduce to the form you're familiar with:

$$H(p,q) = -log q(x_{label})$$

share|improve this answer

answered 4 hours ago

Philip RaeisghasemPhilip Raeisghasem

963119

$endgroup$

share|improve this answer

answered 4 hours ago

Philip RaeisghasemPhilip Raeisghasem

963119

answered 4 hours ago

Philip RaeisghasemPhilip Raeisghasem

963119

answered 4 hours ago

Philip RaeisghasemPhilip Raeisghasem

963119