将numpy/scipy函数Map到tensorflow.data.Dataset

uz75evzq  于 9个月前  发布在  其他
关注(0)|答案(1)|浏览(119)

我试图提取使用tf.keras.preprocessing.timeseries_dataset_from_array窗口化的1D信号的某些特征,从中我获得了一个tf.data.Dataset对象ds(参见下面的代码)。理想情况下,我希望使用数据集的内置map方法将特征函数(使用numpy和scipy)Map到数据集上。
然而,当我天真地尝试这样做时:

import numpy as np
import scipy as sc
import tensorflow as tf

def feat1_func(x, sf, axis):
    x = np.asarray(x)
    feat1_value = np.apply_along_axis(
        lambda vals: sc.integrate.trapezoid(abs(vals), dx=1 / sf), axis=axis, arr=x
    )
    return feat1_value

features = ['feat1']
feature_map = {'feat1': feat1_func}

x = np.random.rand(100, 5)
y = np.random.randint(low=0, high=2, size=100)

sequence_length = 10
sequence_stride = 3

ds = tf.keras.preprocessing.timeseries_dataset_from_array(
        data=x,
        targets=y,
        sequence_length=sequence_length,
        sequence_stride=sequence_stride,
        batch_size=None,
        shuffle=False,
    )

feat_lambda = lambda x, y: (np.array([feature_map[ft](x, sf=1000, axis=0) for ft in features]), y)
ds = ds.map(feat_lambda)

字符串
我收到以下错误消息:

NotImplementedError: Cannot convert a symbolic tf.Tensor (args_0:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported.


这个问题最简单的解决方法是什么?当Map发生时,是否可以将符号Tensor转换为渴望Tensor?

mbskvtky

mbskvtky1#

解决的办法是改变路线

feat_lambda = lambda x, y: (np.array([feature_map[ft](x, sf=1000, axis=0) for ft in features]), y)

字符串

feat_lambda = lambda x, y: ([tf.numpy_function(feature_map[ft], [x, 1000, 0], tf.float64) for ft in features], y)


tf.numpy_function接受一个处理numpy数组的函数,并以渴望模式处理Dataset的Tensor(也就是具有真实的值的Tensor,而不是符号Tensor)。
ds = ds.map(feat_lambda)tf.py_function上也没有错误,但是当我试图循环数据集时,我得到了错误:

# This didn't work
feat_lambda = lambda x, y: ([tf.py_function(feature_map[ft], [x, 1000, 0], tf.float64) for ft in features], y)
ds2 = ds.map(feat_lambda)
for elem in ds2:
    print(elem)  # here I got an error with tf.py_function, not with tf.numpy_function

相关问题