我正在用spark和scala在intellij中创建maven项目，但是intellij无法识别sparksession、sparkcontext和sqlcontext。随附图片。
错误：无法解析符号getsqlcontext错误：无法解析符号GetSparkContext
我对pom的理解是一个项目对象模型，或者pom是maven的基本工作单元。它是一个xml文件，包含有关项目的信息和maven用于构建项目的配置详细信息。
我是否需要更改pom属性或依赖项中的某些内容。
我的pom文件看起来像这样。
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>com.IPM</groupId>
<artifactId>lambda</artifactId>
<packaging>pom</packaging>
<version>1.0-SNAPSHOT</version>
<modules>
<module>spark-lambda</module>
</modules>
<properties>
<spark.version>2.4.4</spark.version>
<chill.version>0.7.2</chill.version>
<algebird.version>0.11.0</algebird.version>
<kafka.clients.version>0.8.2.1</kafka.clients.version>
<avro.version>1.7.7</avro.version>
<spark-cassandra-connector.version>1.6.1</spark-cassandra-connector.version>
<cassandra-driver-core.version>3.0.1</cassandra-driver-core.version>
<nscala-time.version>2.6.0</nscala-time.version>
</properties>
<dependencies>
<dependency>
<groupId>com.typesafe</groupId>
<artifactId>config</artifactId>
<version>1.3.0</version>
</dependency>
<dependency>
<groupId>com.datastax.spark</groupId>
<artifactId>spark-cassandra-connector_2.11</artifactId>
<version>${spark-cassandra-connector.version}</version>
</dependency>
<dependency>
<groupId>com.datastax.cassandra</groupId>
<artifactId>cassandra-driver-core</artifactId>
<version>${cassandra-driver-core.version}</version>
</dependency>
<dependency>
<groupId>com.github.nscala-time</groupId>
<artifactId>nscala-time_2.11</artifactId>
<version>${nscala-time.version}</version>
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka-clients</artifactId>
<version>${kafka.clients.version}</version>
</dependency>
<dependency>
<groupId>org.slf4j</groupId>
<artifactId>slf4j-api</artifactId>
<version>1.7.7</version>
</dependency>
<dependency>
<groupId>com.twitter</groupId>
<artifactId>chill_2.11</artifactId>
<version>${chill.version}</version>
</dependency>
<dependency>
<groupId>com.twitter</groupId>
<artifactId>chill-avro_2.11</artifactId>
<version>${chill.version}</version>
</dependency>
<dependency>
<groupId>com.twitter</groupId>
<artifactId>algebird-core_2.11</artifactId>
<version>${algebird.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-core_2.11</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
<!-- provided -->
</dependency>
<dependency>
<groupId>org.apache.kafka</groupId>
<artifactId>kafka_2.11</artifactId>
<version>${kafka.clients.version}</version>
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-streaming_2.11</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
<!-- provided -->
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-sql_2.11</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
<!-- provided -->
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-mllib_2.11</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
<!-- provided -->
</dependency>
<dependency>
<groupId>org.apache.spark</groupId>
<artifactId>spark-hive_2.11</artifactId>
<version>${spark.version}</version>
<scope>compile</scope>
<!-- provided -->
</dependency>
<dependency>
<groupId>joda-time</groupId>
<artifactId>joda-time</artifactId>
<version>2.8.1</version>
</dependency>
<dependency>
<groupId>org.joda</groupId>
<artifactId>joda-convert</artifactId>
<version>1.7</version>
</dependency>
</dependencies>
<build>
<sourceDirectory>src/main/scala</sourceDirectory>
<plugins>
<plugin>
<groupId>net.alchim31.maven</groupId>
<artifactId>scala-maven-plugin</artifactId>
<version>3.1.6</version>
<executions>
<execution>
<phase>compile</phase>
<goals>
<goal>compile</goal>
<goal>testCompile</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-shade-plugin</artifactId>
<version>2.3</version>
<executions>
<execution>
<phase>package</phase>
<goals>
<goal>shade</goal>
</goals>
<configuration>
<shadedArtifactAttached>true</shadedArtifactAttached>
<filters>
<filter>
<artifact>*:*</artifact>
<excludes>
<exclude>META-INF/*.SF</exclude>
<exclude>META-INF/*.DSA</exclude>
<exclude>META-INF/*.RSA</exclude>
</excludes>
</filter>
</filters>
<artifactSet>
<includes>
<include>*:*</include>
</includes>
</artifactSet>
<transformers>
<transformer
implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
<resource>reference.conf</resource>
</transformer>
</transformers>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>在此处输入图像描述
1条答案
按热度按时间cgh8pdjw1#
spark中没有这样的函数，您需要正确地创建上下文，如下所示（下面是完整的示例）：
第二，您的依赖项不正确-您使用的是spark cassandra connector 1.6.1，它是为spark 1.6设计的，而您正在尝试使用spark 2.4.4。
您需要使用spark cassandra connector 2.5.0（最新版本）-它与spark 2.4兼容，还具有许多新功能。
您还需要删除此依赖项：
最有可能的是：
如果要将spark与kafka一起使用，则需要使用正确的软件包，例如：
在这里你可以找到一个使用Kafka+Spark+Cassandra的例子。