keras 如何标准缩放3D矩阵?

vuktfyat  于 2023-02-16  发布在  其他
关注(0)|答案(8)|浏览(213)

我正在处理一个信号分类问题,希望首先缩放数据集矩阵,但我的数据是3D格式(批次、长度、通道)。
我尝试使用Scikit-learn标准定标器:

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)

但是我收到了这个错误消息:
找到dim为3的数组。应为StandardScaler〈= 2
我认为一个解决方案是将矩阵按每个通道拆分为多个2D矩阵,分别缩放,然后恢复为3D格式,但我想知道是否有更好的解决方案。
非常感谢。

ma8fv8wu

ma8fv8wu1#

只有3行代码...

scaler = StandardScaler()
X_train = scaler.fit_transform(X_train.reshape(-1, X_train.shape[-1])).reshape(X_train.shape)
X_test = scaler.transform(X_test.reshape(-1, X_test.shape[-1])).reshape(X_test.shape)
chhqkbe1

chhqkbe12#

您必须为每个通道安装并存储一个定标器

from sklearn.preprocessing import StandardScaler

scalers = {}
for i in range(X_train.shape[1]):
    scalers[i] = StandardScaler()
    X_train[:, i, :] = scalers[i].fit_transform(X_train[:, i, :]) 

for i in range(X_test.shape[1]):
    X_test[:, i, :] = scalers[i].transform(X_test[:, i, :])
lrpiutwd

lrpiutwd3#

如果您希望以不同的方式缩放每个特征(如StandardScaler),可以使用以下命令:

import numpy as np
from sklearn.base import TransformerMixin
from sklearn.preprocessing import StandardScaler

class NDStandardScaler(TransformerMixin):
    def __init__(self, **kwargs):
        self._scaler = StandardScaler(copy=True, **kwargs)
        self._orig_shape = None

    def fit(self, X, **kwargs):
        X = np.array(X)
        # Save the original shape to reshape the flattened X later
        # back to its original shape
        if len(X.shape) > 1:
            self._orig_shape = X.shape[1:]
        X = self._flatten(X)
        self._scaler.fit(X, **kwargs)
        return self

    def transform(self, X, **kwargs):
        X = np.array(X)
        X = self._flatten(X)
        X = self._scaler.transform(X, **kwargs)
        X = self._reshape(X)
        return X

    def _flatten(self, X):
        # Reshape X to <= 2 dimensions
        if len(X.shape) > 2:
            n_dims = np.prod(self._orig_shape)
            X = X.reshape(-1, n_dims)
        return X

    def _reshape(self, X):
        # Reshape X back to it's original shape
        if len(X.shape) >= 2:
            X = X.reshape(-1, *self._orig_shape)
        return X

它只是在输入到sklearn的StandardScaler之前将输入的特征展平,然后再重新整形,用法与StandardScaler相同:

data = [[[0, 1], [2, 3]], [[1, 5], [2, 9]]]
scaler = NDStandardScaler()
print(scaler.fit_transform(data))

印刷品

[[[-1. -1.]
  [ 0. -1.]]

 [[ 1.  1.]
  [ 0.  1.]]]

参数with_meanwith_std直接传递给StandardScaler,因此按预期工作。copy=False不起作用,因为整形不是就地发生的。对于二维输入,NDStandardScaler的工作方式与StandardScaler类似:

data = [[0, 0], [0, 0], [1, 1], [1, 1]]
scaler = NDStandardScaler()
scaler.fit(data)
print(scaler.transform(data))
print(scaler.transform([[2, 2]]))

印刷品

[[-1. -1.]
 [-1. -1.]
 [ 1.  1.]
 [ 1.  1.]]
[[3. 3.]]

就像在StandardScaler的sklearn示例中一样。

6ju8rftf

6ju8rftf4#

一种优雅的方法是使用类继承,如下所示:

from sklearn.preprocessing import MinMaxScaler
import numpy as np

class MinMaxScaler3D(MinMaxScaler):

    def fit_transform(self, X, y=None):
        x = np.reshape(X, newshape=(X.shape[0]*X.shape[1], X.shape[2]))
        return np.reshape(super().fit_transform(x, y=y), newshape=X.shape)

用法:

scaler = MinMaxScaler3D()
X = scaler.fit_transform(X)
tkqqtvp1

tkqqtvp15#

我使用Normalization方案来处理形状为(2500,512,642)--〉(采样、时间步长、特征/空间位置)的时空数据。以下代码也可用于Normalization及其逆过程。

def Normalize_data(data):
    scaled_data = []
    max_values  = []
    min_values  = []
    for N in range(data.shape[0]):
        temp = []
        t1   = []
        t2   = []
        for i in range(data.shape[1]):
            max_val = np.max(data[N,i])
            min_val = np.min(data[N,i])
            norm = (data[N,i] - min_val)/(max_val - min_val)
            temp.append(norm)
            t1.append(max_val)
            t2.append(min_val)

        scaled_data.append(temp)
        max_values.append(t1)
        min_values.append(t2)
    return (np.array(scaled_data), np.array(max_values), np.array(min_values))

def InverseNormalize_data(scaled_data, max_values, min_values):
    res_data = []
    for N in range(scaled_data.shape[0]):
        temp = []
        for i in range(scaled_data.shape[1]):
            max_val = max_values[N,i]
            min_val = min_values[N,i]
            #print(max_val)
            #print(min_val)
            orig = (scaled_data[N,i] * (max_val - min_val)) + min_val
            temp.append(orig)
        res_data.append(temp)
    return np.array(res_data)
t2a7ltrp

t2a7ltrp6#

s0, s1, s2 = y_train.shape[0], y_train.shape[1], y_train.shape[2]
y_train = y_train.reshape(s0 * s1, s2)
y_train = minMaxScaler.fit_transform(y_train)
y_train = y_train.reshape(s0, s1, s2)

s0, s1, s2 = y_test.shape[0], y_test.shape[1], y_test.shape[2]
y_test = y_test.reshape(s0 * s1, s2)
y_test = minMaxScaler.transform(y_test)
y_test = y_test.reshape(s0, s1, s2)

只是像这样改变数据的形状。对于零填充,使用类似的方法:

s0, s1, s2 = x_train.shape[0], x_train.shape[1], x_train.shape[2]
x_train = x_train.reshape(s0 * s1, s2)
minMaxScaler.fit(x_train[0::s1])
x_train = minMaxScaler.transform(x_train)
x_train = x_train.reshape(s0, s1, s2)

s0, s1, s2 = x_test.shape[0], x_test.shape[1], x_test.shape[2]
x_test = x_test.reshape(s0 * s1, s2)
x_test = minMaxScaler.transform(x_test)
x_test = x_test.reshape(s0, s1, s2)
at0kjp5o

at0kjp5o7#

如果要处理管道,可以使用此类

from sklearn.base import TransformerMixin,BaseEstimator
from sklearn.preprocessing import StandardScaler

class Scaler(BaseEstimator,TransformerMixin):

    def __init__(self):
        self.scaler = StandardScaler()

    def fit(self,X,y=None):
        self.scaler.fit(X.reshape(X.shape[0], -1))
        return self

    def transform(self,X):
        return self.scaler.transform(X.reshape(X.shape[0], -1)).reshape(X.shape)
xam8gpfp

xam8gpfp8#

还有一个简单的方法,不使用for循环,也不生成函数或方法,我只是用“array.reshape(-1,1)”将给定的输入数组扁平化,然后放入minmaxscaler中,再经过变换,将其转换成原来的形状。
例如,我已经实现了来自tensorflow的cifar 10数据集:

from tensorflow.keras.datasets import cifar10
from sklearn.preprocessing import MinMaxScaler

(x3d_train, y3d_train), (x3d_test, y3d_test) =cifar10.load_data()

s0_train,s1_train,s2_train,s3_train=x3d_train.shape

s0_test,s1_test,s2_test,s3_test=x3d_test.shape

scaler = MinMaxScaler()
scaler1 = MinMaxScaler()

scaler.fit(x3d_train.reshape(-1,1))
scaler1.fit(x3d_test.reshape(-1,1))

x3d_train_norm=scaler.transform(x3d_train.reshape(-1,1)).reshape(s0_train,s1_train,s2_train,s3_train)
x3d_test_norm=scaler1.transform(x3d_test.reshape(-1,1)).reshape(s0_test,s1_test,s2_test,s3_test)

相关问题