regexp\u替换异常

iyzzxitl  于 2021-06-04  发布在  Hadoop
关注(0)|答案(1)|浏览(398)

我尝试在cloudera界面中运行一个配置单元查询,它可以很好地处理几百条记录。当我在更大的数据集上运行它时,它失败了。我试图在互联网上搜索它,但它看起来有很多类似的错误,但不是确切的解决方案,我正在寻找。我在配置单元查询中使用redexp\u replace,我不认为这会导致任何异常(我的印象是它可以轻松处理字符串和空类型)
我得到的错误是java.util.regex.patternsyntaxexception:索引12附近的unmatched closing')
更新:这是导致问题的记录:
columna:readdata (或列表目录)
columnb:listdirectory)
列C:空
columnd:空
我的查询:regexp\u replace(columna,columnb,“”)为columna,columnb regexp\u replace(columnc,columnd,“”)为columnc,
请告诉我哪里出了问题。
下面是日志中有趣的部分…*[记录]。。。

at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:565)
                              at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
                              at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)
                              at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)
                              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
                              at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
                              at java.security.AccessController.doPrivileged(Native Method)
                              at javax.security.auth.Subject.doAs(Subject.java:396)
                              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
                              at org.apache.hadoop.mapred.Child.main(Child.java:262)
                              Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException: DFSOutputStream is closed
                              at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:620)
                              at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
                              at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
                              at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
                              at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
                              at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
                              at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
                              at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
                              at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
                              at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
                              ... 9 more
                              Caused by: java.io.IOException: DFSOutputStream is closed
                              at org.apache.hadoop.hdfs.DFSOutputStream.isClosed(DFSOutputStream.java:1239)
                              at org.apache.hadoop.hdfs.DFSOutputStream.writeChunk(DFSOutputStream.java:1407)
                              at org.apache.hadoop.fs.FSOutputSummer.writeChecksumChunk(FSOutputSummer.java:161)
                              at org.apache.hadoop.fs.FSOutputSummer.write1(FSOutputSummer.java:104)
                              at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:90)
                              at org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:54)
                              at java.io.DataOutputStream.write(DataOutputStream.java:90)
                              at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat$1.write(HiveIgnoreKeyTextOutputFormat.java:86)
                              at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:606)
                              ... 18 more

                              2013-05-31 16:35:20,090 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3 finished. closing...
                              2013-05-31 16:35:20,090 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3 forwarded 90478 rows
                              2013-05-31 16:35:20,090 INFO org.apache.hadoop.hive.ql.exec.MapOperator: DESERIALIZE_ERRORS:0
                              2013-05-31 16:35:20,090 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 finished. closing...
                              2013-05-31 16:35:20,090 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 forwarded 90478 rows
                              2013-05-31 16:35:20,090 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 1 finished. closing...
                              2013-05-31 16:35:20,090 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 1 forwarded 90478 rows
                              2013-05-31 16:35:20,090 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: 2 finished. closing...
                              2013-05-31 16:35:20,090 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: 2 forwarded 0 rows
                              2013-05-31 16:35:20,090 INFO org.apache.hadoop.hive.ql.exec.FileSinkOperator: TABLE_ID_1_ROWCOUNT:90478
                              2013-05-31 16:35:20,090 INFO org.apache.hadoop.hive.ql.exec.SelectOperator: 1 Close done
                              2013-05-31 16:35:20,090 INFO org.apache.hadoop.hive.ql.exec.TableScanOperator: 0 Close done
                              2013-05-31 16:35:20,090 INFO org.apache.hadoop.hive.ql.exec.MapOperator: 3 Close done
                              2013-05-31 16:35:20,090 INFO ExecMapper: ExecMapper: processed 90477 rows: used memory = 10815536
                              2013-05-31 16:35:20,097 INFO org.apache.hadoop.mapred.TaskLogsTruncater: Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
                              2013-05-31 16:35:20,099 ERROR org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException as:hdfs (auth:SIMPLE) cause:java.io.IOException: Filesystem closed
                              2013-05-31 16:35:20,099 WARN org.apache.hadoop.mapred.Child: Error running child
                              java.io.IOException: Filesystem closed
                              at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:552)
                              at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:522)
                              at java.io.FilterInputStream.close(FilterInputStream.java:155)
                              at org.apache.hadoop.util.LineReader.close(LineReader.java:149)
                              at org.apache.hadoop.mapred.LineRecordReader.close(LineRecordReader.java:195)
                              at org.apache.hadoop.hive.ql.io.CombineHiveRecordReader.doClose(CombineHiveRecordReader.java:72)
                              at org.apache.hadoop.hive.ql.io.HiveContextAwareRecordReader.close(HiveContextAwareRecordReader.java:96)
                              at org.apache.hadoop.hive.shims.HadoopShimsSecure$CombineFileRecordReader.close(HadoopShimsSecure.java:273)
                              at org.apache.hadoop.mapred.MapTask$TrackedRecordReader.close(MapTask.java:223)
                              at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:422)
                              at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)
                              at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
                              at java.security.AccessController.doPrivileged(Native Method)
                              at javax.security.auth.Subject.doAs(Subject.java:396)
                              at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
                              at org.apache.hadoop.mapred.Child.main(Child.java:262)
                              2013-05-31 16:35:20,102 WARN org.apache.hadoop.mapred.Task: Parent died.  Exiting attempt_201305300036_0011_m_000000_1
ugmeyewa

ugmeyewa1#

我不完全理解您要实现什么,但是我可以告诉您regex\u replace的第二个参数需要是一个有效的正则表达式。
您传递的值不是有效的表达式。明确地, ListDirectory)) 是正则表达式中的保留符号。

相关问题