https://stats.stackexchange.com/questions/179026/objective-function-cost-function-loss-function-are-they-the-same-thing
These are not very strict terms and they are highly related. However:
Loss function is usually a function defined on a data point, prediction and label, and measures the penalty. For example:
square loss
l
(
f
(
x
i
∣
θ
)
,
y
i
)
=
(
f
(
x
i
∣
θ
)
−
y
i
)
2
l(f(x_i|\theta),y_i) = \left (f(x_i|\theta)-y_i \right )^2
l(f(xi∣θ),yi)=(f(xi∣θ)−yi)2, used in linear regression
hinge loss
l
(
f
(
x
i
∣
θ
)
,
y
i
)
=
max
(
0
,
1
−
f
(
x
i
∣
θ
)
y
i
)
l(f(x_i|\theta), y_i) = \max(0, 1-f(x_i|\theta)y_i)
l(f(xi∣θ),yi)=max(0,1−f(xi∣θ)yi), used in SVM
0/1 loss
l
(
f
(
x
i
∣
θ
)
,
y
i
)
=
1
⟺
f
(
x
i
∣
θ
)
≠
y
i
l(f(x_i|\theta), y_i) = 1 \iff f(x_i|\theta) \neq y_i
l(f(xi∣θ),yi)=1⟺f(xi∣θ)=yi, used in theoretical analysis and definition of accuracy
Cost function is usually more general. It might be a sum of loss functions over your training set plus some model complexity penalty (regularization). For example:
Mean Squared Error
M
S
E
(
θ
)
=
1
N
∑
i
=
1
N
(
f
(
x
i
∣
θ
)
−
y
i
)
2
MSE(\theta) = \frac{1}{N} \sum_{i=1}^N \left (f(x_i|\theta)-y_i \right )^2
MSE(θ)=N1∑i=1N(f(xi∣θ)−yi)2
SVM cost function
S
V
M
(
θ
)
=
∥
θ
∥
2
+
C
∑
i
=
1
N
ξ
i
SVM(\theta) = \|\theta\|^2 + C \sum_{i=1}^N \xi_i
SVM(θ)=∥θ∥2+C∑i=1Nξi (there are additional constraints connecting
ξ
i
\xi_i
ξi with
C
C
C and with training set)
Objective function is the most general term for any function that you optimize during training. For example, a probability of generating training set in maximum likelihood approach is a well defined objective function, but it is not a loss function nor cost function (however you could define an equivalent cost function). For example:
MLE is a type of objective function (which you maximize)
Divergence between classes can be an objective function but it is barely a cost function, unless you define something artificial, like 1-Divergence, and name it a cost
Long story short, I would say that:
A loss function is a part of a cost function which is a type of an objective function.