关于fileoutputstream的java outofmemoryerror？

htrmnn0y 于 2021-07-09 发布在 Java

关注(0)|答案(2)|浏览(334)

谢谢大家，问题是solved:there is 一行太大（超过400米…我下载了一个损坏的文件，而我没有意识到），所以抛出outofmemoryerror

我想用java拆分一个文件，但总是抛出内存错误：java堆空间，我搜索了整个互联网，但似乎没有帮助：(
文件的大小是600米，它有超过30000000行，每行不超过100个字符(也许你可以生成这样的“级别文件：{id:0000000001，level:1 id:0000000002,level:2 ....（超过3000万）}）
pss公司。将jvm内存大小设置得更大是不可行的：(
psss公司。我换了另一台电脑，问题依然存在/(ㄒoㄒ)/~~
不管我设置的-xms或-xmx有多大，outputfile的大小总是相同的（runtime.getruntime（）.totalmemory（）确实发生了变化）
以下是堆栈跟踪：

Heap Size = 2058027008
    Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
        at java.util.Arrays.copyOf(Arrays.java:2882)
        at java.lang.AbstractStringBuilder.expandCapacity(AbstractStringBuilder.java:100)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:515)
        at java.lang.StringBuffer.append(StringBuffer.java:306)
        at java.io.BufferedReader.readLine(BufferedReader.java:345)
        at java.io.BufferedReader.readLine(BufferedReader.java:362)
        at com.xiaomi.vip.tools.ptupdate.updator.Spilt.main(Spilt.java:39)
    ...

这是我的密码：

package com.updator;

import java.io.BufferedReader;
import java.io.File;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.FileReader;

public class Spilt {
    public static void main(String[] args) throws Exception {
        long heapSize = Runtime.getRuntime().totalMemory();

        // Print the jvm heap size.
        System.out.println("Heap Size = " + heapSize);

        String mainPath = "/home/work/bingo/";
        File mainFilePath = new File(mainPath);
        FileInputStream inputStream = null;
        FileOutputStream outputStream = null;
        try {
            if (!mainFilePath.exists())
                mainFilePath.mkdir();

            String sourcePath = "/home/work/bingo/level.txt";
            inputStream = new FileInputStream(sourcePath);
            BufferedReader bufferedReader = new BufferedReader(new FileReader(
                    new File(sourcePath)));

            String savePath = mainPath + "tmp/";
            Integer i = 0;
            File file = new File(savePath + "part"
                    + String.format("%0" + 5 + "d", i) + ".txt");
            if (!file.getParentFile().exists())
                file.getParentFile().mkdir();
            file.createNewFile();
            outputStream = new FileOutputStream(file);
            int count = 0, total = 0;
            String line = null;
            while ((line = bufferedReader.readLine()) != null) {
                line += '\n';
                outputStream.write(line.getBytes("UTF-8"));
                count++;
                total++;
                if (count > 4000000) {
                    outputStream.flush();
                    outputStream.close();
                    System.gc();
                    count = 0;
                    i++;
                    file = new File(savePath + "part"
                            + String.format("%0" + 5 + "d", i) + ".txt");
                    file.createNewFile();
                    outputStream = new FileOutputStream(file);
                }
            }

            outputStream.close();
            file = new File(mainFilePath + "_SUCCESS");
            file.createNewFile();
            outputStream = new FileOutputStream(file);
            outputStream.write(i.toString().getBytes("UTF-8"));
        } finally {
            if (inputStream != null)
                inputStream.close();
            if (outputStream != null)
                outputStream.close();
        }
    }
}

我想可能是：当outputstream.close（）时，内存没有释放？

Java out-of-memory FileOutputStream

来源：https://stackoverflow.com/questions/41458900/java-outofmemoryerror-about-fileoutputstream

2条答案

按热度按时间

jc3wubiy1#

所以打开原始文件并创建一个 BufferedReader 还有一个柜台。

char[] buffer = new char[5120];
BufferedReader reader = Files.newBufferedReader(Paths.get(sourcePath), StandardCharsets.UTF_8);
int lineCount = 0;

现在读入缓冲区，并在输入时写入字符。

int read;

BufferedWriter writer = Files.newBufferedWriter(Paths.get(fileName), StandardCharsets.UTF_8);
while((read = reader.read(buffer, 0, 5120))>0){
    int offset = 0;
    for(int i = 0; i<read; i++){
        char c = buffer[i];
        if(c=='\n'){
           lineCount++;
           if(lineCount==maxLineCount){
              //write the range from 0 to i to your old writer.
              writer.write(buffer, offset, i-offset);
              writer.close();
              offset=i;
              lineCount=0;
              writer = Files.newBufferedWriter(Paths.get(newName), StandarCharset.UTF_8);
           }
        }
        writer.write(buffer, offset, read-offset);
    }
    writer.close();
}

这样可以降低内存使用率，避免一次读取太多的行。您可以不使用缓冲写入程序来控制内存，但我不认为这是必要的。

赞(0）回复(0）举报 2021-07-09

46qrfjad2#

我已经测试了大文本文件。（250mb）
效果很好。
您需要为文件流添加try-catch异常代码。

public class MyTest {
    public static void main(String[] args) {
        String mainPath = "/home/work/bingo/";
        File mainFilePath = new File(mainPath);
        FileInputStream inputStream = null;
        FileOutputStream outputStream = null;
        try {
            if (!mainFilePath.exists())
                mainFilePath.mkdir();

            String sourcePath = "/home/work/bingo/level.txt";
            inputStream = new FileInputStream(sourcePath);
            Scanner scanner = new Scanner(inputStream, "UTF-8");

            String savePath = mainPath + "tmp/";
            Integer i = 0;
            File file = new File(savePath + "part" + String.format("%0" + 5 + "d", i) + ".txt");
            if (!file.getParentFile().exists())
                file.getParentFile().mkdir();
            file.createNewFile();
            outputStream = new FileOutputStream(file);
            int count = 0, total = 0;

            while (scanner.hasNextLine()) {
                String line = scanner.nextLine() + "\n";
                outputStream.write(line.getBytes("UTF-8"));
                count++;
                total++;
                if (count > 4000000) {
                    outputStream.flush();
                    outputStream.close();
                    count = 0;
                    i++;
                    file = new File(savePath + "part" + String.format("%0" + 5 + "d", i) + ".txt");
                    file.createNewFile();
                    outputStream = new FileOutputStream(file);
                }
            }

            outputStream.close();
            file = new File(mainFilePath + "_SUCCESS");
            file.createNewFile();
            outputStream = new FileOutputStream(file);
            outputStream.write(i.toString().getBytes("UTF-8"));
        } catch (FileNotFoundException e) {
            System.out.println("ERROR: FileNotFoundException :: " + e.getStackTrace());
        } catch (IOException e) {
            System.out.println("ERROR: IOException :: " + e.getStackTrace());
        } finally {
            if (inputStream != null)
                try {
                    inputStream.close();
                    if (outputStream != null)
                        outputStream.close();

                } catch (IOException e) {
                    e.printStackTrace();
                }
        }
    }
}

如果问题仍然存在，请在shell提示符下使用以下命令更改java堆内存大小。
例如）xmx1g:1gb堆内存大小，mytest:类名
java-xmx1g mytest

赞(0）回复(0）举报 2021-07-09

我来回答

关于fileoutputstream的java outofmemoryerror？

谢谢大家，问题是solved:there is 一行太大（超过400米…我下载了一个损坏的文件，而我没有意识到），所以抛出outofmemoryerror

2条答案

相关问题

热门标签

最新问答