8

Spark笔记(九九)-Spark开发常见故障

 3 years ago
source link: http://timebusker.top/2020/01/25/2099-Spark%E7%AC%94%E8%AE%B0(%E4%B9%9D%E4%B9%9D)-Spark%E5%BC%80%E5%8F%91%E5%B8%B8%E8%A7%81%E6%95%85%E9%9A%9C/
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client
Spark笔记(九九)-Spark开发常见故障 - timebusker的博客|BY Blog

Spark IDE: System memory 251396096 must be at least 471859200

spark运行内存不够导致
conf.set("spark.testing.memory", "1G")

SparkSQL-Hive通过scala API建表异常 For direct MetaStore DB connections, we don't support retries at the client level.

Hive元数据编码异常导致
hive元数据库编码默认要采用 latin1 
若中文乱码再单独解决特定表，不可全部改变编码格式为UTF-8

posgresql:
CREATE DATABASE maggieprod WITH ENCODING = 'LATIN1' LC_CTYPE = 'en_CA' LC_COLLATE = 'en_CA' TEMPLATE template0;

解决以上问题的办法是修改hive元数据TBLS表结构，设计为IS_REWRITE_ENABLED可为空或者设置默认值

ALTER TABLE "public"."TBLS" ALTER COLUMN "IS_REWRITE_ENABLED" DROP NOT NULL;

ALTER TABLE "public"."TBLS" ALTER COLUMN "IS_REWRITE_ENABLED" set default(true);

report

Recommend

About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK