q:spark提交之后,找不到logger(org.apache.spark.deploy.sparksubmit$$anon$2)的appender

pbgvytdp  于 2021-05-27  发布在  Spark
关注(0)|答案(1)|浏览(760)

使用apachespark一步一步快速开始,但最后显示此警告消息

20/05/25 09:43:05 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
log4j:WARN No appenders could be found for logger (org.apache.spark.deploy.SparkSubmit$$anon$2).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.

我的密码是

package firstmaven;

import org.apache.spark.sql.SparkSession;
import org.apache.spark.sql.Dataset;
/**
 * Hello world!
 *
 */
public class App 
{
    public static void main( String[] args )
    {
        String logFile = "/data/spark/README.md";
        SparkSession spark = SparkSession.builder().appName("Simple Application")
                            .config("spark.master","local").getOrCreate();
        Dataset<String> logData = spark.read().textFile(logFile).cache();
        long numAs = logData.filter(s -> s.contains("a")).count();
        long numBs = logData.filter(s -> s.contains("b")).count();
        System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);

        System.out.println("Hello world");

        spark.stop();
    }
}

我该怎么做才能让它工作?谢谢。

yh2wf1be

yh2wf1be1#

您需要指定要筛选的列的值。请检查以下代码段:

import org.apache.spark.sql.Dataset;
import org.apache.spark.sql.SparkSession;

import java.util.List;

public class App {
    public static void main(String[] args) {
        String logFile = "/Users/ajay/input.txt";
        SparkSession spark = SparkSession.builder().appName("Simple Application")
                .config("spark.master", "local").getOrCreate();
        Dataset<String> logData = spark.read().textFile(logFile).cache();
        List<String> rowList = logData.collectAsList();
        System.out.println("rowList is = " + rowList);
        Dataset<String> rowDatasetWithA = logData.filter((logData.col("value").contains("a")));
        List<String> rowWithA = rowDatasetWithA.collectAsList();
        System.out.println("rowWithA is = " + rowWithA);

        Dataset<String> rowDatasetWithB = logData.filter((logData.col("value").contains("b")));
        List<String> rowWithB = rowDatasetWithB.collectAsList();
        System.out.println("rowWithB is = " + rowWithB);

        long numAs = rowWithA.size();
        long numBs = rowWithB.size();
        System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);

        spark.stop();
    }
}

假设 input.txt 是这样吗

a
b
c
a
aa

上面代码段的输出如下

rowList is = [a, b, c, a, aa]
rowWithA is = [a, a, aa]
rowWithB is = [b]
Lines with a: 3, lines with b: 1
enter code here

希望这有帮助。

相关问题