CentOS6.5

JDK1.8.0

Hadoop2.7.1

Zookeeper3.7.1

Scala-2.13.0

二，下载及解压

机器：hadoop01

下载地址​ ​http://spark.apache.org/releases/spark-release-2-2-2.html​​

解压到/home/software下

机器：hadoop01

进入conf目录

1，复制spark-env.sh

cp spark-env.sh.template spark-env.sh

 复制缓存的文件spark-env.sh.template改为spark识别的文件spark-env.sh，在文件尾部编辑如下内容：

export JAVA_HOME=/home/software/jdk1.8.0

export SCALA_HOME=/home/software/scala-2.13.0

export HADOOP_HOME=/home/hadoop-2.7.1

export HADOOP_CONF_DIR=/home/hadoop-2.7.1/etc/hadoop

export SPARK_MASTER_IP=hadoop01

export SPARK_WORKER_MEMORY=2g

export SPARK_WORKER_CORES=2

export SPARK_WORKER_INSTANCES=1

2，复制slaves

cp slaves.template slaves

编辑如下内容：

hadoop02

hadoop03

3，复制到hadoop02和hadoop03上

[root@hadoop01 software]# scp -r spark-2.2.2 hadoop02:/home/software/

[root@hadoop01 software]# scp -r spark-2.2.2 hadoop03:/home/software/

1，进入spark/sbin下

sh start-all.sh

2，在web上访问

​​http://hadoop01:8080​​

3，进入bin/spark-shell

a，启动并加载jar包准备进入hive库

[root@hadoop01 bin]# sh spark-shell --master local[3] --jars /home/software/hive-1.2.0/lib/mysql-connector-java-5.1.38-bin.jar

b，查看库

scala> spark.sql("show databases").show;

c，创建库

scala> spark.sql("create database test").show;

d，使用库

scala> spark.sql("use test").show;

e，创建表

scala> spark.sql("create table student(id int, name string,age int) row format delimited fields terminated by ','").show;

f，增加数据

spark.sql("insert into student values(1,'Zenghaoxuan',10)").show;

g，查看数据

scala> spark.sql("select * from student").show;

+---+---------+---+

| id|     name|age|

+---+---------+---+

|  1|Zenghaoxuan| 10|

+---+---------+---+