2
spark历史服务配置
source link: https://jimolonely.github.io/2019/10/07/spark/009-spark-history-conf/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
spark历史服务配置
本文配置spark的历史服务,记录历史日志。
写日志配置
conf/spark-defaults.conf
spark.master spark://localhost:7077
spark.eventLog.enabled true
spark.eventLog.dir /home/jack/workspace/spark/log-dir
读日志配置
spark-env.sh
SPARK_HISTORY_OPTS="-Dspark.history.ui.port=18080 -Dspark.history.retainedApplication=30 -Dspark.history.fs.logDirectory=/home/jack/workspace/spark/log-dir"
开启历史服务
$ sbin/start-history-server.sh
starting org.apache.spark.deploy.history.HistoryServer, logging to /home/jack/workspace/spark/spark-2.4.4-bin-hadoop2.7/logs/spark-jack-org.apache.spark.deploy.history.HistoryServer-1-jack.out
$ jps
30340 HistoryServer
30506 Jps
查看日志可以看到成功了:
16 19/10/07 14:50:51 INFO FsHistoryProvider: History server ui acls disabled; users with admin permissions: ; groups with admin permissions
17 19/10/07 14:50:51 INFO Utils: Successfully started service on port 18080.
18 19/10/07 14:50:52 INFO HistoryServer: Bound HistoryServer to 0.0.0.0, and started at http://10.13.112.16:18080
启动spark
$ sbin/start-all.sh
跑一个应用
$ bin/spark-submit \
--class org.apache.spark.examples.SparkPi \
--master spark://localhost:7077 \
--executor-memory 1G \
--total-executor-cores 2 \
./examples/jars/spark-examples_2.11-2.4.4.jar \
100
然后去看界面就有了,同时日志目录下有一个日志文件:
~/workspace/spark/log-dir$ ll
总用量 240
drwxrwxr-x 2 jack jack 4096 10月 7 14:57 ./
drwxrwxr-x 4 jack jack 4096 10月 7 14:40 ../
-rwxrwx--- 1 jack jack 236566 10月 7 14:57 app-20191007145723-0000*
- spark10
Recommend
About Joyk
Aggregate valuable and interesting links.
Joyk means Joy of geeK