使用java将应用程序发布到spark时遇到问题

jhkqcmku  于 2021-05-27  发布在  Spark
关注(0)|答案(0)|浏览(269)

当我运行spark示例时,我得到这个错误
20/09/04 17:06:35警告taskschedulerimpl:初始作业未接受任何资源;检查集群ui以确保worker已注册并且具有足够的资源
我在本地运行这个应用程序,我的spark服务器部署在 47.111.185.105 ```
public class WordCountApp {

private static final Logger logger = LoggerFactory.getLogger(WordCountApp.class);

public static void main(String[] args) {
    // Should be some file on your system
    String logFile = "src/main/resources/people.json";
    SparkConf conf = new SparkConf().set("spark.shuffle.service.enabled", "false")
            .set("spark.dynamicAllocation.enabled", "false")
            .set("spark.cores.max", "1")
            .set("spark.executor.instances","2")
            .set("spark.executor.memory","500m")
            .set("spark.executor.cores","1")
            .setMaster("spark://47.111.185.105:7077");
            //.set("deploy-mode", "client");
    SparkSession spark = SparkSession.builder()
            .appName("Word Count Application")
            .config(conf)
            .getOrCreate();
    Dataset<String> logData = spark.read().textFile(logFile).cache();
    System.out.println("Spark version = " + spark.version());
    logger.info("Spark version = " + spark.version());
    long numAs = logData.filter((FilterFunction<String>) s -> s.contains("a")).count();
    long numBs = 3;//logData.filter((FilterFunction<String>) s -> s.contains("b")).count();

    System.out.println("Lines with a: " + numAs + ", lines with b: " + numBs);
    logger.info("Lines with a: " + numAs + ", lines with b: " + numBs);
    spark.stop();
}

}

打包并上传jar文件,使用sparkshell效果很好
检查剩余资源是否足够
尝试重新启动spark
![](https://i.stack.imgur.com/MwnZC.png)

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题