hdinsight/spark活动没有为spark submit指定--files参数的选项

q9yhzks0 于 2021-05-18 发布在 Spark

关注(0)|答案(1)|浏览(535)

我已在azure中创建了一个hdinsight群集（v4，spark 2.4），并希望通过azure data factory v2活动在此群集上运行spark.ne应用程序。在spark活动中，可以指定jar的路径，--类参数和要传递给spark应用程序的参数。参数在运行时自动以“-args”作为前缀。但是能够设置“--files”是必要的，因为它告诉spark提交哪些文件需要部署到工作节点。在本例中，它用于分发带有自定义项定义的dll。这些文件是spark运行所必需的。由于自定义项是激发应用程序的关键组件，我本以为这应该是可能的。
spark活动设置
如果我ssh到集群并直接运行spark submit命令并指定--files参数，那么spark应用程序可以工作，因为文件被分发到工作节点。

spark-submit --deploy-mode cluster --master yarn --files wasbs://xxx@yyy.blob.core.windows.net/SparkJobs/mySparkApp.dll --class org.apache.spark.deploy.dotnet.DotnetRunner wasbs://xxx@yyy.blob.core.windows.net/SparkJobs/microsoft-spark-2.4.x-0.12.1.jar wasbs://xxx@yyy.blob.core.windows.net/SparkJobs/publish.zip mySparkApp

以下是已遵循的指南：
https://docs.microsoft.com/en-us/dotnet/spark/how-to-guides/deploy-worker-udf-binaries
https://docs.microsoft.com/en-us/dotnet/spark/how-to-guides/hdinsight-deploy-methods
https://docs.microsoft.com/en-us/dotnet/spark/tutorials/hdinsight-deployment

yarn apache-spark azure-hdinsight azure-data-factory-2 .net-spark

来源：https://stackoverflow.com/questions/64588909/hdinsight-spark-activity-in-azure-data-factory-v2-does-not-have-option-to-specif

1条答案

按热度按时间

ar5n3qh51#

您可以在azure数据工厂中将参数/参数传递给pyspark脚本，如下所示：

代码：

{
    "name": "SparkActivity",
    "properties": {
        "activities": [
            {
                "name": "Spark1",
                "type": "HDInsightSpark",
                "dependsOn": [],
                "policy": {
                    "timeout": "7.00:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "rootPath": "adftutorial/spark/script",
                    "entryFilePath": "WordCount_Spark.py",
                    "arguments": [
                        "--input-file",
                        "wasb://sampledata@chepra.blob.core.windows.net/data",
                        "--output-file",
                        "wasb://sampledata@chepra.blob.core.windows.net/results"
                    ],
                    "sparkJobLinkedService": {
                        "referenceName": "AzureBlobStorage1",
                        "type": "LinkedServiceReference"
                    }
                },
                "linkedServiceName": {
                    "referenceName": "HDInsight",
                    "type": "LinkedServiceReference"
                }
            }
        ],
        "annotations": []
    },
    "type": "Microsoft.DataFactory/factories/pipelines"
}

如何在adf中传递参数：

在azure数据工厂中传递参数的一些示例：

{
    "name": "SparkSubmit",
    "properties": {
        "description": "Submit a spark job",
        "activities": [
            {
                "type": "HDInsightMapReduce",
                "typeProperties": {
                    "className": "com.adf.spark.SparkJob",
                    "jarFilePath": "libs/spark-adf-job-bin.jar",
                    "jarLinkedService": "StorageLinkedService",
                    "arguments": [
                        "--jarFile",
                        "libs/sparkdemoapp_2.10-1.0.jar",
                        "--jars",
                        "/usr/hdp/current/hadoop-client/hadoop-azure-2.7.1.2.3.3.0-3039.jar,/usr/hdp/current/hadoop-client/lib/azure-storage-2.2.0.jar",
                        "--mainClass",
                        "com.adf.spark.demo.Demo",
                        "--master",
                        "yarn-cluster",
                        "--driverMemory",
                        "2g",
                        "--driverExtraClasspath",
                        "/usr/lib/hdinsight-logging/*",
                        "--executorCores",
                        "1",
                        "--executorMemory",
                        "4g",
                        "--sparkHome",
                        "/usr/hdp/current/spark-client",
                        "--connectionString",
                        "DefaultEndpointsProtocol=https;AccountName=<YOUR_ACCOUNT>;AccountKey=<YOUR_KEY>",
                        "input=wasb://input@<YOUR_ACCOUNT>.blob.core.windows.net/data",
                        "output=wasb://output@<YOUR_ACCOUNT>.blob.core.windows.net/results"
                    ]
                },
                "inputs": [
                    {
                        "name": "input"
                    }
                ],
                "outputs": [
                    {
                        "name": "output"
                    }
                ],
                "policy": {
                    "executionPriorityOrder": "OldestFirst",
                    "timeout": "01:00:00",
                    "concurrency": 1,
                    "retry": 1
                },
                "scheduler": {
                    "frequency": "Day",
                    "interval": 1
                },
                "name": "Spark Launcher",
                "description": "Submits a Spark Job",
                "linkedServiceName": "HDInsightLinkedService"
            }
        ],
        "start": "2015-11-16T00:00:01Z",
        "end": "2015-11-16T23:59:00Z",
        "isPaused": false,
        "pipelineMode": "Scheduled"
    }
}

赞(0）回复(0）举报 2021-05-19

我来回答

hdinsight/spark活动没有为spark submit指定--files参数的选项

1条答案

相关问题

热门标签

最新问答