通过livy获得完整响应

bgtovc5b  于 2021-05-27  发布在  Spark
关注(0)|答案(0)|浏览(231)

我正在尝试通过livy使用交互式会话读取文件。我创建了一个会话并提交了一个要执行的语句。语句执行得很好,但响应在一定长度后被截断。例如,这是我的回答get:-

{
    "id": 17,
    "code": "import org.apache.spark.sql.types.StringType; val dataframe = new org.apache.spark.sql.SQLContext(sc).read.parquet(\"/ClickstreamInput\"); dataframe.select(dataframe.columns.map(c => col(c).cast(StringType)): _*).na.fill(\"null\").toJSON.rdd.collect().toList.map(_.toString)",
    "state": "available",
    "output": {
        "status": "ok",
        "execution_count": 17,
        "data": {
            "text/plain": "warning: there was one deprecation warning; re-run with -deprecation for details\nimport org.apache.spark.sql.types.StringType\ndataframe: org.apache.spark.sql.DataFrame = [value: string]\nres16: List[String] = List({\"value\":\"m6pHQD5Qf0HWiLI,2254023209288530,2069219912,product_view,facebook,desktop\"}, {\"value\":\"pwSyumNQcy0o3s1,2253999803834925,2081337593,product_view,facebook,ios\"}, {\"value\":\"KVsFm9dd49TlmGA,2253999782497840,533382031,product_view,google,android\"}, {\"value\":\"y6AgiVTOpqE4l5q,2253999749428795,533382031,product_view,facebook,android\"}, {\"value\":\"XXKYip1uDMeUXF1,2254032037691786,1174233544,purchase,facebook,desktop\"}, {\"value\":\"KJ7nUudfdXUMMdB,2254024802941761,1585974151,cart_add,google,android\"}, {\"value\":\"vRB1O17qA1sE0JL,2253999779673921,571768064,product_view,instagram,android\"}, {\"value\":\"Bw2R1MuAGNBuG31,2254009356680570,1773864570,product_view,faceboo..."
        }
    },
    "progress": 1.0
}

有没有办法增加这个字符限制并得到完整的响应?

暂无答案!

目前还没有任何答案,快来回答吧!

相关问题