scipy 在不使用scikit learn的情况下训练测试拆分

ncgqoxb0  于 2022-11-10  发布在  其他
关注(0)|答案(5)|浏览(140)

我有一个房价预测数据集,我需要将数据集拆分为traintest
我想知道使用numpyscipy是否可以做到这一点?
我现在无法使用scikit学习。

lndjwyie

lndjwyie1#

我知道您的问题是只对numpyscipy执行train_test_split,但实际上有一个非常简单的方法可以对Pandas执行此操作:

import pandas as pd 

# Shuffle your dataset

shuffle_df = df.sample(frac=1)

# Define a size for your train set

train_size = int(0.7 * len(df))

# Split your dataset

train_set = shuffle_df[:train_size]
test_set = shuffle_df[train_size:]

对于那些谁想要一个快速和容易的解决方案。

pu3pd22g

pu3pd22g2#

虽然这是个老问题,但这个答案可能会有所帮助。
这就是sklearn实现train_test_split的方式,下面给出的这个方法采用与sklearn类似的参数。

import numpy as np
from itertools import chain

def _indexing(x, indices):
    """
    :param x: array from which indices has to be fetched
    :param indices: indices to be fetched
    :return: sub-array from given array and indices
    """
    # np array indexing
    if hasattr(x, 'shape'):
        return x[indices]

    # list indexing
    return [x[idx] for idx in indices]

def train_test_split(*arrays, test_size=0.25, shufffle=True, random_seed=1):
    """
    splits array into train and test data.
    :param arrays: arrays to split in train and test
    :param test_size: size of test set in range (0,1)
    :param shufffle: whether to shuffle arrays or not
    :param random_seed: random seed value
    :return: return 2*len(arrays) divided into train ans test
    """
    # checks
    assert 0 < test_size < 1
    assert len(arrays) > 0
    length = len(arrays[0])
    for i in arrays:
        assert len(i) == length

    n_test = int(np.ceil(length*test_size))
    n_train = length - n_test

    if shufffle:
        perm = np.random.RandomState(random_seed).permutation(length)
        test_indices = perm[:n_test]
        train_indices = perm[n_test:]
    else:
        train_indices = np.arange(n_train)
        test_indices = np.arange(n_train, length)

    return list(chain.from_iterable((_indexing(x, train_indices), _indexing(x, test_indices)) for x in arrays))

当然,sklearn的实现支持分层k折叠,Pandas系列的分裂等。这一个只适用于分裂列表和numpy数组,我认为这将适用于您的情况。

mqkwyuun

mqkwyuun3#

这段代码应该可以工作(假设X_data是一个PandasDataFrame):

import numpy as np
num_of_rows = len(X_data) * 0.8
values = X_data.values
np.random_shuffle(values) #shuffles data to make it random
train_data = values[:num_of_rows] #indexes rows for training data
test_data = values[num_of_rows:] #indexes rows for test data

希望这对你有帮助!

cunj1qz1

cunj1qz14#

import numpy as np
import pandas as pd

X_data = pd.read_csv('house.csv')
Y_data = X_data["prices"]
X_data.drop(["offers", "brick", "bathrooms", "prices"], 
            axis=1, inplace=True) # important to drop prices as well

# create random train/test split

indices = range(X_data.shape[0])
num_training_instances = int(0.8 * X_data.shape[0])
np.random.shuffle(indices)
train_indices = indices[:num_training_indices]
test_indices = indices[num_training_indices:]

# split the actual data

X_data_train, X_data_test = X_data.iloc[train_indices], X_data.iloc[test_indices]
Y_data_train, Y_data_test = Y_data.iloc[train_indices], Y_data.iloc[test_indices]

假设您想要随机分割,我们会建立一个索引清单,长度与您拥有的数据点数目相同,即X_data的第一个轴然后,我们将它们按随机顺序排列,并仅将这些随机索引的前80%作为训练数据,其余用于测试。[:num_training_indices]仅从列表中选择前num_training_indices。之后,您只需使用随机索引列表从数据中提取行,然后您的数据被拆分。如果您希望拆分可重复(开始时为np.random.seed(some_integer)),请记住从X_data中删除价格,并设置种子。

tnkciper

tnkciper5#

这个解决方案只使用Pandas和numpy

def split_train_valid_test(data,valid_ratio,test_ratio):
    shuffled_indcies=np.random.permutation(len(data))
    valid_set_size= int(len(data)*valid_ratio)
    valid_indcies=shuffled_indcies[:valid_set_size]
    test_set_size= int(len(data)*test_ratio)
    test_indcies=shuffled_indcies[valid_set_size:test_set_size+valid_set_size]
    train_indices=shuffled_indcies[test_set_size:]
    return data.iloc[train_indices],data.iloc[valid_indcies],data.iloc[test_indcies]

train_set,valid_set,test_set=split_train_valid_test(dataset,valid_ratio=0.2,test_ratio=0.2)
print(len(train_set),len(valid_set),len(test_set))

## out: (16512, 4128, 4128)

相关问题