使用Java1.8和spark2.2创建sparksession

我对分布式计算非常陌生。我能够在本地机器上运行spark 2.2中的类似代码，没有任何问题。现在我正试图用java做同样的事情，我不断地得到这个错误。我可以创建javasparkcontext并在代码中使用该对象。问题是当我使用builder创建sparksession时。我的本地环境是：windows7、spark2.2、java1.8
我的代码看起来像

public static void main(String[] args) {
        SparkConf sparkConf = new SparkConf();          
        sparkConf.setAppName("Java-Spark");
        sparkConf.setMaster("local");

        SparkSession sc= SparkSession.builder().appName("test").master("local[*]").config("spark.driver.cores", 1).getOrCreate();

/*  this commented code works fine. 
    JavaSparkContext context = new JavaSparkContext(sparkConf);

    JavaRDD<Integer> numbersRDD = context.parallelize(Arrays.asList(1,2,3));

    JavaRDD<Integer> squaresRDD = numbersRDD.map( n -> n*n );
    System.out.println("1 :");
    System.out.println(squaresRDD.collect().toString());

    JavaRDD<Integer> evenRDD = squaresRDD.filter( n -> n%2==0 );
    System.out.println("2 :");
    System.out.println(evenRDD.collect().toString());

    JavaRDD<Integer> multipliedRDD = numbersRDD.flatMap( n->Arrays.asList(n,n*2,n*3).iterator());
    System.out.println("3 :");
    System.out.println(multipliedRDD.collect().toString());

    context.close();
  */
  sc.close(); 
  }

我正在使用maven来管理依赖项。xml看起来像

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>
  <groupId>ICC_POC</groupId>
  <artifactId>spark</artifactId>
  <version>0.0.1-SNAPSHOT</version>
    <name>spark</name>
    <dependencies>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>com.bazaarvoice.jolt</groupId>
            <artifactId>jolt-complete</artifactId>
            <version>0.1.0</version>
        </dependency>
        <dependency>
            <groupId>com.bazaarvoice.jolt</groupId>
            <artifactId>jolt-core</artifactId>
            <version>0.1.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-launcher_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-network-common_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-network-shuffle_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-unsafe_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-tags_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-catalyst_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sketch_2.11</artifactId>
            <version>2.2.0</version>
        </dependency>
        <dependency>
            <groupId>org.apache.xbean</groupId>
            <artifactId>xbean-asm5-shaded</artifactId>
            <version>4.5</version>
        </dependency>
    </dependencies>
    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.4.3</version>
            </plugin>
        </plugins>
    </build>
</project>

在过去的两天里，我一直在与这个错误作斗争，无法找出我在这里遗漏了什么。需要另一双眼睛来发现问题。谢谢
编辑：添加mvn依赖项树的输出

ICC_POC:spark:jar:0.0.1-SNAPSHOT
+- org.apache.spark:spark-core_2.11:jar:2.2.0:compile
|  +- org.apache.avro:avro:jar:1.7.7:compile
|  |  +- org.codehaus.jackson:jackson-core-asl:jar:1.9.13:compile
|  |  +- org.codehaus.jackson:jackson-mapper-asl:jar:1.9.13:compile
|  |  +- com.thoughtworks.paranamer:paranamer:jar:2.3:compile
|  |  \- org.apache.commons:commons-compress:jar:1.4.1:compile
|  |     \- org.tukaani:xz:jar:1.0:compile
|  +- org.apache.avro:avro-mapred:jar:hadoop2:1.7.7:compile
|  |  +- org.apache.avro:avro-ipc:jar:1.7.7:compile
|  |  \- org.apache.avro:avro-ipc:jar:tests:1.7.7:compile
|  +- com.twitter:chill_2.11:jar:0.8.0:compile
|  |  \- com.esotericsoftware:kryo-shaded:jar:3.0.3:compile
|  |     +- com.esotericsoftware:minlog:jar:1.3.0:compile
|  |     \- org.objenesis:objenesis:jar:2.1:compile
|  +- com.twitter:chill-java:jar:0.8.0:compile
|  +- org.apache.hadoop:hadoop-client:jar:2.6.5:compile
|  |  +- org.apache.hadoop:hadoop-common:jar:2.6.5:compile
|  |  |  +- commons-cli:commons-cli:jar:1.2:compile
|  |  |  +- xmlenc:xmlenc:jar:0.52:compile
|  |  |  +- commons-httpclient:commons-httpclient:jar:3.1:compile
|  |  |  +- commons-io:commons-io:jar:2.4:compile
|  |  |  +- commons-collections:commons-collections:jar:3.2.2:compile
|  |  |  +- commons-lang:commons-lang:jar:2.6:compile
|  |  |  +- commons-configuration:commons-configuration:jar:1.6:compile
|  |  |  |  +- commons-digester:commons-digester:jar:1.8:compile
|  |  |  |  |  \- commons-beanutils:commons-beanutils:jar:1.7.0:compile
|  |  |  |  \- commons-beanutils:commons-beanutils-core:jar:1.8.0:compile
|  |  |  +- com.google.protobuf:protobuf-java:jar:2.5.0:compile
|  |  |  +- com.google.code.gson:gson:jar:2.2.4:compile
|  |  |  +- org.apache.hadoop:hadoop-auth:jar:2.6.5:compile
|  |  |  |  \- org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15:compile
|  |  |  |     +- org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15:compile
|  |  |  |     +- org.apache.directory.api:api-asn1-api:jar:1.0.0-M20:compile
|  |  |  |     \- org.apache.directory.api:api-util:jar:1.0.0-M20:compile
|  |  |  +- org.apache.curator:curator-client:jar:2.6.0:compile
|  |  |  \- org.htrace:htrace-core:jar:3.0.4:compile
|  |  +- org.apache.hadoop:hadoop-hdfs:jar:2.6.5:compile
|  |  |  +- org.mortbay.jetty:jetty-util:jar:6.1.26:compile
|  |  |  \- xerces:xercesImpl:jar:2.9.1:compile
|  |  |     \- xml-apis:xml-apis:jar:1.3.04:compile
|  |  +- org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.6.5:compile
|  |  |  +- org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.6.5:compile
|  |  |  |  +- org.apache.hadoop:hadoop-yarn-client:jar:2.6.5:compile
|  |  |  |  \- org.apache.hadoop:hadoop-yarn-server-common:jar:2.6.5:compile
|  |  |  \- org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.6.5:compile
|  |  +- org.apache.hadoop:hadoop-yarn-api:jar:2.6.5:compile
|  |  +- org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.6.5:compile
|  |  |  \- org.apache.hadoop:hadoop-yarn-common:jar:2.6.5:compile
|  |  |     +- javax.xml.bind:jaxb-api:jar:2.2.2:compile
|  |  |     |  \- javax.xml.stream:stax-api:jar:1.0-2:compile
|  |  |     +- org.codehaus.jackson:jackson-jaxrs:jar:1.9.13:compile
|  |  |     \- org.codehaus.jackson:jackson-xc:jar:1.9.13:compile
|  |  +- org.apache.hadoop:hadoop-mapreduce-client-jobclient:jar:2.6.5:compile
|  |  \- org.apache.hadoop:hadoop-annotations:jar:2.6.5:compile
|  +- net.java.dev.jets3t:jets3t:jar:0.9.3:compile
|  |  +- org.apache.httpcomponents:httpcore:jar:4.3.3:compile
|  |  +- org.apache.httpcomponents:httpclient:jar:4.3.6:compile
|  |  +- commons-codec:commons-codec:jar:1.8:compile
|  |  +- javax.activation:activation:jar:1.1.1:compile
|  |  +- mx4j:mx4j:jar:3.0.2:compile
|  |  +- javax.mail:mail:jar:1.4.7:compile
|  |  +- org.bouncycastle:bcprov-jdk15on:jar:1.51:compile
|  |  \- com.jamesmurty.utils:java-xmlbuilder:jar:1.0:compile
|  |     \- net.iharder:base64:jar:2.3.8:compile
|  +- org.apache.curator:curator-recipes:jar:2.6.0:compile
|  |  +- org.apache.curator:curator-framework:jar:2.6.0:compile
|  |  +- org.apache.zookeeper:zookeeper:jar:3.4.6:compile
|  |  \- com.google.guava:guava:jar:16.0.1:compile
|  +- javax.servlet:javax.servlet-api:jar:3.1.0:compile
|  +- org.apache.commons:commons-lang3:jar:3.5:compile
|  +- org.apache.commons:commons-math3:jar:3.4.1:compile
|  +- com.google.code.findbugs:jsr305:jar:1.3.9:compile
|  +- org.slf4j:slf4j-api:jar:1.7.16:compile
|  +- org.slf4j:jul-to-slf4j:jar:1.7.16:compile
|  +- org.slf4j:jcl-over-slf4j:jar:1.7.16:compile
|  +- log4j:log4j:jar:1.2.17:compile
|  +- org.slf4j:slf4j-log4j12:jar:1.7.16:compile
|  +- com.ning:compress-lzf:jar:1.0.3:compile
|  +- org.xerial.snappy:snappy-java:jar:1.1.2.6:compile
|  +- net.jpountz.lz4:lz4:jar:1.3.0:compile
|  +- org.roaringbitmap:RoaringBitmap:jar:0.5.11:compile
|  +- commons-net:commons-net:jar:2.2:compile
|  +- org.scala-lang:scala-library:jar:2.11.8:compile
|  +- org.json4s:json4s-jackson_2.11:jar:3.2.11:compile
|  |  \- org.json4s:json4s-core_2.11:jar:3.2.11:compile
|  |     +- org.json4s:json4s-ast_2.11:jar:3.2.11:compile
|  |     \- org.scala-lang:scalap:jar:2.11.0:compile
|  |        \- org.scala-lang:scala-compiler:jar:2.11.0:compile
|  |           +- org.scala-lang.modules:scala-xml_2.11:jar:1.0.1:compile
|  |           \- org.scala-lang.modules:scala-parser-combinators_2.11:jar:1.0.1:compile
|  +- org.glassfish.jersey.core:jersey-client:jar:2.22.2:compile
|  |  +- javax.ws.rs:javax.ws.rs-api:jar:2.0.1:compile
|  |  +- org.glassfish.hk2:hk2-api:jar:2.4.0-b34:compile
|  |  |  +- org.glassfish.hk2:hk2-utils:jar:2.4.0-b34:compile
|  |  |  \- org.glassfish.hk2.external:aopalliance-repackaged:jar:2.4.0-b34:compile
|  |  +- org.glassfish.hk2.external:javax.inject:jar:2.4.0-b34:compile
|  |  \- org.glassfish.hk2:hk2-locator:jar:2.4.0-b34:compile
|  |     \- org.javassist:javassist:jar:3.18.1-GA:compile
|  +- org.glassfish.jersey.core:jersey-common:jar:2.22.2:compile
|  |  +- javax.annotation:javax.annotation-api:jar:1.2:compile
|  |  +- org.glassfish.jersey.bundles.repackaged:jersey-guava:jar:2.22.2:compile
|  |  \- org.glassfish.hk2:osgi-resource-locator:jar:1.0.1:compile
|  +- org.glassfish.jersey.core:jersey-server:jar:2.22.2:compile
|  |  +- org.glassfish.jersey.media:jersey-media-jaxb:jar:2.22.2:compile
|  |  \- javax.validation:validation-api:jar:1.1.0.Final:compile
|  +- org.glassfish.jersey.containers:jersey-container-servlet:jar:2.22.2:compile
|  +- org.glassfish.jersey.containers:jersey-container-servlet-core:jar:2.22.2:compile
|  +- io.netty:netty-all:jar:4.0.43.Final:compile
|  +- io.netty:netty:jar:3.9.9.Final:compile
|  +- com.clearspring.analytics:stream:jar:2.7.0:compile
|  +- io.dropwizard.metrics:metrics-core:jar:3.1.2:compile
|  +- io.dropwizard.metrics:metrics-jvm:jar:3.1.2:compile
|  +- io.dropwizard.metrics:metrics-json:jar:3.1.2:compile
|  +- io.dropwizard.metrics:metrics-graphite:jar:3.1.2:compile
|  +- com.fasterxml.jackson.core:jackson-databind:jar:2.6.5:compile
|  |  \- com.fasterxml.jackson.core:jackson-core:jar:2.6.5:compile
|  +- com.fasterxml.jackson.module:jackson-module-scala_2.11:jar:2.6.5:compile
|  |  +- org.scala-lang:scala-reflect:jar:2.11.7:compile
|  |  \- com.fasterxml.jackson.module:jackson-module-paranamer:jar:2.6.5:compile
|  +- org.apache.ivy:ivy:jar:2.4.0:compile
|  +- oro:oro:jar:2.0.8:compile
|  +- net.razorvine:pyrolite:jar:4.13:compile
|  +- net.sf.py4j:py4j:jar:0.10.4:compile
|  +- org.apache.commons:commons-crypto:jar:1.0.0:compile
|  \- org.spark-project.spark:unused:jar:1.0.0:compile
+- com.bazaarvoice.jolt:jolt-complete:jar:0.1.0:compile
|  \- com.bazaarvoice.jolt:json-utils:jar:0.1.0:compile
+- com.bazaarvoice.jolt:jolt-core:jar:0.1.0:compile
|  \- javax.inject:javax.inject:jar:1:compile
+- org.apache.spark:spark-launcher_2.11:jar:2.2.0:compile
+- org.apache.spark:spark-network-common_2.11:jar:2.2.0:compile
|  +- org.fusesource.leveldbjni:leveldbjni-all:jar:1.8:compile
|  \- com.fasterxml.jackson.core:jackson-annotations:jar:2.6.5:compile
+- org.apache.spark:spark-network-shuffle_2.11:jar:2.2.0:compile
+- org.apache.spark:spark-unsafe_2.11:jar:2.2.0:compile
+- org.apache.spark:spark-tags_2.11:jar:2.2.0:compile
+- org.apache.spark:spark-sql_2.11:jar:2.2.0:compile
|  +- com.univocity:univocity-parsers:jar:2.2.1:compile
|  +- org.apache.parquet:parquet-column:jar:1.8.2:compile
|  |  +- org.apache.parquet:parquet-common:jar:1.8.2:compile
|  |  \- org.apache.parquet:parquet-encoding:jar:1.8.2:compile
|  \- org.apache.parquet:parquet-hadoop:jar:1.8.2:compile
|     +- org.apache.parquet:parquet-format:jar:2.3.1:compile
|     \- org.apache.parquet:parquet-jackson:jar:1.8.2:compile
+- org.apache.spark:spark-catalyst_2.11:jar:2.2.0:compile
+- org.apache.spark:spark-sketch_2.11:jar:2.2.0:compile
\- org.apache.xbean:xbean-asm5-shaded:jar:4.5:compile

使用Java1.8和spark2.2创建sparksession

1条答案

相关问题

热门标签

最新问答