我在nodejs中使用worker_threads
读取多个文件并处理行,以便稍后插入数据库(使用oracle-db
)。
大约我处理的行超过一百万,每个文件都有超过10,000行。
当我试图创建“绑定”插入到数据库的每一行字段,而使用forEachAsyncParallel
或承诺的每一行在.map
我得到了以下错误:
错误
RangeError: Value undefined out of range for undefined options property undefined
at Set.add (<anonymous>)
at AsyncHook.init (internal/inspector_async_hook.js:23:25)
at PromiseWrap.emitInitNative (internal/async_hooks.js:182:43)
at /app/services/workers/PushMultiValueFile.worker.js:1:1
at new Promise (<anonymous>)
at /app/services/workers/PushMultiValueFile.worker.js:20:84
at Array.map (<anonymous>)
at /app/services/workers/PushMultiValueFile.worker.js:20:57
at new Promise (<anonymous>)
at AssignKeysNameOnValues (/app/services/workers/PushMultiValueFile.worker.js:16:12)
RangeError: Value undefined out of range for undefined options property undefined
at Set.add (<anonymous>)
at AsyncHook.init (internal/inspector_async_hook.js:23:25)
at PromiseWrap.emitInitNative (internal/async_hooks.js:182:43)
at Promise.then (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:97:5)
-----------------------
***Custom code to read error:***
[2021-03-10T19:34:47.331Z]
{
captured_at: 2021-03-10T19:34:47.000Z,
messages: '"Error: Worker stopped with exit code 1"',
stacktrace: '"Error: Error: Worker stopped with exit code 1\\n at /app/services/oracle_database.js:434:36\\n at processTicksAndRejections (internal/process/task_queues.js:97:5)"',
execution_detail_id: 1203
}
有办法解决吗?
带有Promises的Map
function AssignKeysNameOnValues(keys, values) {
return new Promise( async (resolve, reject) => {
try {
let result = {};
if (Object.entries(keys).length > 0) {
const promises = (Object.entries(keys)).map( async (item,index) => new Promise( async (resolve, reject) => {
try{
if ([(oracledb.DB_TYPE_NUMBER)].includes(item[1].type)) {
result[item[0]] = values[index] ? parseFloat(values[index]) : null;
resolve()
}
else {
result[item[0]] = values[index];
resolve()
}
} catch (e) {
// throw await logError(new Error(error),(workerData.executionID));
reject();
}
}));
Promise.all(promises)
.then(() => {
resolve(result);
})
.catch((err) => {
reject(err);
});
}
else {
resolve();
}
} catch (err) {
reject(err);
}
});
}
参数接收:
按键:
{
ARRANGEMENT_ID: {
type: 2001,
maxSize: 255,
},
OWNER: {
type: 2001,
maxSize: 255,
},
LINKED_APPL: {
type: 2001,
maxSize: 255,
}
}
取值:
{
"0": "ARR16@#$%3423C0",
"1": "1",
"2": "42"
}
其他类型迭代相同的值,但我得到了相同的错误。
forEachAsyncParallel
Array.prototype.forEachAsyncParallel = async function (fn) {
await Promise.all(this.map(fn));
}
版本:
- 节点:v12.18.4
- Npm:6.14.6
1条答案
按热度按时间2lpgd9681#
当Set或Map达到元素的最大限制时会导致上述错误。参见参考:Node.js Map中的最大条目数?
如果仍然遇到这个问题,你可以尝试
large-set
和large-map
,这是一个允许你存储和处理大量元素的包,而不用担心Set或Map中16.777.216(2^24)的限制,通过分区成更小的,使它能够存储和访问比内置的Set或Map更多的元素。