加装spark-3.5.3

news/2024/10/7 19:51:39/文章来源:https://www.cnblogs.com/used-conduit-onion/p/18450494

集群版本

hadoop-3.4.0
hive-3.1.3
zookeeper-3.9.2
hbase-2.6.0（1.0.0以上需要zookeeper-3.4.0以上）
spark-3.5.3（只能选2.13.0）
scala-2.13.0（jdk8仅支持x.x.0系）总结一下：JDK8和scala-2.13.0必选。

1.安装scala

1.1 下载解压

tar zxvf scala-2.13.0.tgz

1.2 配置环境变量

vi /etc/profile ，添加内容如下

#scala
export SCALA_HOME=/root/scala-2.13.0
export PATH=$PATH:$SCALA_HOME/bin

1.3 分发

scp -r scala-2.13.0 root@clone1:/root/
scp -r scala-2.13.0 root@clone2:/root/
scp -r etc/profile root@clone1:/etc/profile
scp -r etc/profile root@clone2:/etc/profile

source /etc/profile
scala -version

2.安装spark

2.1 下载解压

tar zxvf spark-3.5.3-bin-hadoop3.tgz

2.2 配置环境变量

#spark
export SPARK_HOME=/root/spark-3.5.3-bin-hadoop3
export PATH=$PATH:$SPARK_HOME/bin

2.3 修改配置文件

2.3.1 修改spark-env.sh

cp /root/spark-3.5.3-bin-hadoop3/conf/spark-env.sh.template spark-env.shvi spark-env.sh   #添加如下内容export SPARK_MASTER_IP=master
export SCALA_HOME=/root/scala-2.13.0
export SPARK_WORKER_MEMORY=2g
export JAVA_HOME=/root/jdk1.8.0_401
export HADOOP_HOME=/root/hadoop-3.4.0
export HADOOP_CONF_DIR=/root/hadoop-3.4.0/etc/hadoop

2.3.2 配置从节点

cp /root/spark-3.5.3-bin-hadoop3/conf/works.template works    #在末尾添加如下内容clone1
clone2

2.4 分发

scp -r /root/spark-3.5.3-bin-hadoop3 root@clone1:/root/
scp -r /root/spark-3.5.3-bin-hadoop3 root@clone2:/root/scp /etc/profile root@clone1:/etc/profile
scp /etc/profile root@clone2:/etc/profile

source /etc/profile

激动人心的时刻

3. 集群群起

#master/root/hadoop-3.4.0/sbin/start-all.sh
/root/spark-3.5.3-bin-hadoop3/start-all.sh

jps

web效果图：

版本对应
1.spark on hive
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark: Getting Started#:~:text=Hive on Spark provides Hive with the ability to utilize
2.spark on hadoop

https://spark.apache.org/docs/latest/building-spark.html
3.spark on scala

https://mvnrepository.com/artifact/org.apache.spark/spark-core
4.scala on jdk

https://docs.scala-lang.org/overviews/jdk-compatibility/overview.html#:~:text=Scala’s primary platform is the Java Virtual Machine (JVM). (Other
5.hbase on Hadoop&JDK
https://hbase.apache.org/book.html#java
6.hbase on zookeeper
https://issues.apache.org/jira/browse/HBASE-16598
7.hive on hadoop
https://hive.apache.org/general/downloads/