tensorflow 用一个序列设置一个数组元素,它说我的列表是非齐次的,但我认为它不是

z9smfwbn  于 2022-12-30  发布在  其他
关注(0)|答案(1)|浏览(149)

我试图创建一个简单的聊天机器人与tensorflow我有一个简单的意图文件在.json格式是这样的东西

{"intents": [
  {"tag": "greeting",
    "patterns": ["Hi", "hello" ....],
    "responses": ["greeting"],
  },
  {"tag": "goodbye",
    "patterns": ["bye", "goodbye"],
    "responses": [""goodbye],
  },
]

首先,我使用nltk中的wordnetllemetizer对它们进行标记
并从所有类(标签)中创建了一个列表

`lemmatizer = WordNetLemmatizer()

intents = json.loads(open("intents.json").read())

words = []
classes = []
documents = []
ignore_letters = ["?", "!", ".", ","]

for intent in intents["intents"]:
    for pattern in intent["patterns"]:
        word_list = nltk.word_tokenize(pattern)
        words.extend(word_list)
        documents.append(((word_list), intent["tag"]))
        if intent["tag"] not in classes:
            classes.append(intent["tag"])

words = [lemmatizer.lemmatize(word) for word in words if word not in ignore_letters]
words = sorted(set(words))

classes = sorted(set(classes))

pickle.dump(words, open("trainingMenuAiWords.pkl", "wb"))
pickle.dump(classes, open("trainingMenuAiclasses.pkl", "wb"))`

然后,因为我需要以某种方式将数据馈送到模型中,所以我创建了一个1和0的列表,该列表响应消息是否具有特定令牌(如果具有,则为1;如果没有,则为0)

`training = []
output_empty = [0] * len(classes)

for document in documents:
    bag = []
    word_patterns = document[0]
    word_patterns = [lemmatizer.lemmatize(word.lower()) for word in word_patterns]
    for word in words:
        if word in word_patterns:
            bag.append(1)
        else:
            bag.append(0)
    output_row = list(output_empty)
    output_row[classes.index(document[1])] = 1
    training.append([bag, output_row])

random.shuffle(training)
training = np.array(training)`

然后我把它们输入一个NN模型

`model = Sequential()
model.add(Dense(128, input_shape=(len(train_x[0]),), activation="relu"))
model.add(Dropout(0.5))
model.add(Dense(64, activation="relu"))
model.add(Dropout(0.5))
model.add(Dense(len(train_y[0]), activation="softmax"))

sgd = SGD(learning_rate=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=["accuracy"])
hist = model.fit(np.array(train_x), np.array(train_y), epochs=NUMBER_OF_TRAINING, batch_size=5, verbose=1)
model.save("menuAimodel.h5", hist)`

所以训练文件的完整代码是这样的:

`import random
import json
import pickle
import numpy as np
import nltk
from nltk.stem import WordNetLemmatizer
from time import perf_counter

from tensorflow.python.keras.models import Sequential
from tensorflow.python.keras.layers import Dense, Activation, Dropout
from tensorflow.python.keras.optimizer_v2.gradient_descent import SGD

NUMBER_OF_TRAINING = 3_000

start = perf_counter()

lemmatizer = WordNetLemmatizer()

intents = json.loads(open("intents.json").read())

words = []
classes = []
documents = []
ignore_letters = ["?", "!", ".", ","]

for intent in intents["intents"]:
    for pattern in intent["patterns"]:
        word_list = nltk.word_tokenize(pattern)
        words.extend(word_list)
        documents.append(((word_list), intent["tag"]))
        if intent["tag"] not in classes:
            classes.append(intent["tag"])

words = [lemmatizer.lemmatize(word) for word in words if word not in ignore_letters]
words = sorted(set(words))

classes = sorted(set(classes))

pickle.dump(words, open("trainingMenuAiWords.pkl", "wb"))
pickle.dump(classes, open("trainingMenuAiclasses.pkl", "wb"))

training = []
output_empty = [0] * len(classes)

for document in documents:
    bag = []
    word_patterns = document[0]
    word_patterns = [lemmatizer.lemmatize(word.lower()) for word in word_patterns]
    for word in words:
        if word in word_patterns:
            bag.append(1)
        else:
            bag.append(0)
    output_row = list(output_empty)
    output_row[classes.index(document[1])] = 1
    training.append([bag, output_row])

random.shuffle(training)
training = np.array(training)

train_x = list(training[:, 0])
train_y = list(training[:, 1])

model = Sequential()
model.add(Dense(128, input_shape=(len(train_x[0]),), activation="relu"))
model.add(Dropout(0.5))
model.add(Dense(64, activation="relu"))
model.add(Dropout(0.5))
model.add(Dense(len(train_y[0]), activation="softmax"))

sgd = SGD(learning_rate=0.01, decay=1e-6, momentum=0.9, nesterov=True)
model.compile(loss='categorical_crossentropy', optimizer=sgd, metrics=["accuracy"])
hist = model.fit(np.array(train_x), np.array(train_y), epochs=NUMBER_OF_TRAINING, batch_size=5, verbose=1)
model.save("menuAimodel.h5", hist)

end = perf_counter()

print(f"training time in sec : {end - start}")
print(f"training time in min : {(end - start)/60}")

print("Done")`

现在的问题是在这一行,我试图创建一个numpy数组的列表,其中包含那些1和0的列表,它说他们不是同质的,但我不明白为什么,因为在代码中,它检查每个令牌,并添加0或1,所以它不应该有任何空值或更长或更短的列表,事实上,它工作得很好,我确实用它创建了一些测试模型,但几天前我换成了一个康达虚拟环境来使用我的CUDA内核,它停止了工作,我想不出我还做了什么改变,我错过了什么吗?
给出错误的代码行

`training = np.array(training)`

这就是错误

Traceback (most recent call last):
  File "C:\Users\iparsw\Desktop\python learning\plotting\trainingMenuAi.py", line 60, in <module>
    training = np.array(training)
ValueError: setting an array element with a sequence. The requested array has an inhomogeneous shape after 2 dimensions. The detected shape was (554, 2) + inhomogeneous part.

我试图检查哪里的名单得到不均匀,但我不能undrstand(我是一个ml新手),我试图打印它,并检查不均匀的一部分,由眼睛,但名单得到大到快,我无法找到它

dxpyg8gm

dxpyg8gm1#

我设法修复了更改的问题

training = np.array(training)

到训练= np.数组(训练,数据类型=对象)
显然是因为我改变了我使用的麻木版本

相关问题