Keras:加权二元交叉熵

s8vozzvw  于 2023-03-12  发布在  其他
关注(0)|答案(7)|浏览(224)

我试着用Keras实现加权二进制交叉熵,但我不确定代码是否正确。训练输出似乎有点混乱。几个时期后,我只得到了~0.15的精度。我认为这太少了(即使是随机猜测)。
通常在输出中有大约11%的1和89%的0,因此权重是w_zero=0.89和w_one=0.11。
我的代码:

def create_weighted_binary_crossentropy(zero_weight, one_weight):

    def weighted_binary_crossentropy(y_true, y_pred):

        # Original binary crossentropy (see losses.py):
        # K.mean(K.binary_crossentropy(y_true, y_pred), axis=-1)

        # Calculate the binary crossentropy
        b_ce = K.binary_crossentropy(y_true, y_pred)

        # Apply the weights
        weight_vector = y_true * one_weight + (1. - y_true) * zero_weight
        weighted_b_ce = weight_vector * b_ce

        # Return the mean error
        return K.mean(weighted_b_ce)

    return weighted_binary_crossentropy

也许有人看出了什么问题?
谢谢

lh80um4z

lh80um4z1#

通常情况下,少数类的类权重较高,最好使用one_weight=0.89, zero_weight=0.11(顺便说一句,你可以使用class_weight={0: 0.11, 1: 0.89},就像评论中建议的那样)。
在类不平衡的情况下,模型看到的0比1多得多。它还将学习预测更多的0,因为这样做可以最大限度地减少训练损失。这也是为什么你看到的准确率接近比例0.11。如果你对模型预测取平均值,它应该非常接近零。
使用类权重的目的是改变损失函数,使训练损失不能通过“简单解决方案”(即预测0)最小化,这就是为什么对1使用更高的权重会更好。
请注意,最佳权重不一定是0.89和0.11。有时,您可能需要尝试一些方法,如取对数或平方根(或满足one_weight > zero_weight的任何权重)才能使其工作。

rqcrx0a6

rqcrx0a62#

您可以使用sklearn module自动计算每个类的权重,如下所示:

# Import
import numpy as np
from sklearn.utils import class_weight

# Example model
model = Sequential()
model.add(Dense(32, activation='relu', input_dim=100))
model.add(Dense(1, activation='sigmoid'))

# Use binary crossentropy loss
model.compile(optimizer='rmsprop',
              loss='binary_crossentropy',
              metrics=['accuracy'])

# Calculate the weights for each class so that we can balance the data
weights = class_weight.compute_class_weight('balanced',
                                            np.unique(y_train),
                                            y_train)

# Add the class weights to the training                                         
model.fit(x_train, y_train, epochs=10, batch_size=32, class_weight=weights)

注意,class_weight.compute_class_weight()的输出是一个numpy数组,如下所示:[2.57569845 0.68250928] .

yvt65v4c

yvt65v4c3#

我认为在www.example.com中使用类权重model.fit是不正确的。{0:0.11,1:0.89},这里的0是索引,而不是0类。Keras文档:https://keras.io/models/sequential/类_权重:将类索引(整数)Map到权重(浮点)值的可选字典,用于加权损失函数(仅在训练期间)。这对于告知模型“更多关注”来自表示不足的类的样本非常有用。

flseospp

flseospp4#

model.fit中使用class_weights略有不同:它实际上更新样本而不是计算加权损失。
我还发现当x作为TFDataset或生成器发送到www.example.com时,class_weightssample_weights在TF 2.0.0中被model.fit忽略。
这是我的加权二进制交叉熵函数的多热点编码标签。

import tensorflow as tf
import tensorflow.keras.backend as K
import numpy as np
# weighted loss functions

def weighted_binary_cross_entropy(weights: dict, from_logits: bool = False):
    '''
    Return a function for calculating weighted binary cross entropy
    It should be used for multi-hot encoded labels

    # Example
    y_true = tf.convert_to_tensor([1, 0, 0, 0, 0, 0], dtype=tf.int64)
    y_pred = tf.convert_to_tensor([0.6, 0.1, 0.1, 0.9, 0.1, 0.], dtype=tf.float32)
    weights = {
        0: 1.,
        1: 2.
    }
    # with weights
    loss_fn = get_loss_for_multilabels(weights=weights, from_logits=False)
    loss = loss_fn(y_true, y_pred)
    print(loss)
    # tf.Tensor(0.6067193, shape=(), dtype=float32)

    # without weights
    loss_fn = get_loss_for_multilabels()
    loss = loss_fn(y_true, y_pred)
    print(loss)
    # tf.Tensor(0.52158177, shape=(), dtype=float32)

    # Another example
    y_true = tf.convert_to_tensor([[0., 1.], [0., 0.]], dtype=tf.float32)
    y_pred = tf.convert_to_tensor([[0.6, 0.4], [0.4, 0.6]], dtype=tf.float32)
    weights = {
        0: 1.,
        1: 2.
    }
    # with weights
    loss_fn = get_loss_for_multilabels(weights=weights, from_logits=False)
    loss = loss_fn(y_true, y_pred)
    print(loss)
    # tf.Tensor(1.0439969, shape=(), dtype=float32)

    # without weights
    loss_fn = get_loss_for_multilabels()
    loss = loss_fn(y_true, y_pred)
    print(loss)
    # tf.Tensor(0.81492424, shape=(), dtype=float32)

    @param weights A dict setting weights for 0 and 1 label. e.g.
        {
            0: 1.
            1: 8.
        }
        For this case, we want to emphasise those true (1) label, 
        because we have many false (0) label. e.g. 
            [
                [0 1 0 0 0 0 0 0 0 1]
                [0 0 0 0 1 0 0 0 0 0]
                [0 0 0 0 1 0 0 0 0 0]
            ]

        

    @param from_logits If False, we apply sigmoid to each logit
    @return A function to calcualte (weighted) binary cross entropy
    '''
    assert 0 in weights
    assert 1 in weights

    def weighted_cross_entropy_fn(y_true, y_pred):
        tf_y_true = tf.cast(y_true, dtype=y_pred.dtype)
        tf_y_pred = tf.cast(y_pred, dtype=y_pred.dtype)

        weights_v = tf.where(tf.equal(tf_y_true, 1), weights[1], weights[0])
        weights_v = tf.cast(weights_v, dtype=y_pred.dtype)
        ce = K.binary_crossentropy(tf_y_true, tf_y_pred, from_logits=from_logits)
        loss = K.mean(tf.multiply(ce, weights_v))
        return loss

    return weighted_cross_entropy_fn
vawmfj5a

vawmfj5a5#

你可以像这样计算权重,并得到像这样的二进制交叉熵,它将编程地把one_weight设置为0.11,把one_weight设置为0.89:

one_weight = (1-num_of_ones)/(num_of_ones + num_of_zeros)
zero_weight = (1-num_of_zeros)/(num_of_ones + num_of_zeros)

def weighted_binary_crossentropy(zero_weight, one_weight):

    def weighted_binary_crossentropy(y_true, y_pred):

        b_ce = K.binary_crossentropy(y_true, y_pred)

        # weighted calc
        weight_vector = y_true * one_weight + (1 - y_true) * zero_weight
        weighted_b_ce = weight_vector * b_ce

        return K.mean(weighted_b_ce)

    return weighted_binary_crossentropy
bprjcwpo

bprjcwpo6#

对我来说,最好的方法是这样做的:

def custom_weighted_binary_crossentropy(zero_weight, one_weight):

    def weighted_binary_crossentropy(y_true, y_pred):
        y_true = K.cast(y_true, dtype=tf.float32)

        epsilon = tf.keras.backend.epsilon()
        y_pred = tf.clip_by_value(y_pred, epsilon, 1. - epsilon)

        # Compute cross entropy from probabilities.
        bce = y_true * tf.math.log(y_pred + epsilon)
        bce += (1 - y_true) * tf.math.log(1 - y_pred + epsilon)
        bce = -bce

        # Apply the weights to each class individually
        weight_vector = y_true * one_weight + (1. - y_true) * zero_weight
        weighted_bce = weight_vector * bce

        # Return the mean error
        return tf.reduce_mean(weighted_bce)

    return weighted_binary_crossentropy
zqry0prt

zqry0prt7#

如果您需要一个权重与训练损失不同的加权验证损失,您可以使用www.example.com()的参数validation_datatensorflow.keras.model.fit,方法是将验证数据集作为Numpy数组的元组,其中包含验证数据、标签和每个样本的权重。
请注意,您必须使用此技术将每个样本Map到其权重(此处按类)。
点击此处链接:https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit
tensorflow documentation

相关问题