5

HSql执行报错:Hive internal error inside isAssignableFromSettablePrimitiveOI voi...

 2 years ago
source link: https://www.oschina.net/question/5532927_2325020
Go to the source link to view the article. You can view the picture content, updated content and better typesetting reading experience. If the link is broken, please click the button below to view the snapshot at that time.
neoserver,ios ssh client

HSql执行报错:Hive internal error inside isAssignableFromSettablePrimitiveOI void not supported yet

osc_89275544 发布于 01/27 16:14

这是一段简单的sql,需求是根据通过关联表获取union_account_id并按此字段分组进行计算,只需保留在关联表中能查到union_account_id的记录。

代码如下:

set hive.exec.mode.local.auto=false;
set hive.auto.convert.join=false;
set hive.optimize.skewjoin = true;
set hive.groupby.skewindata = true;
set hive.skewjoin.key = 100000;
set mapreduce.map.memory.mb=4096;
set mapreduce.reduce.memory.mb=4096;
set hive.exec.compress.output=true;
set mapred.output.compress=true;
set mapred.output.compression.codec=org.apache.hadoop.io.compress.GzipCodec;
set io.compression.codecs=org.apache.hadoop.io.compress.GzipCodec;
drop table if exists ycf_dwh_mdm.m_act_user_portrait_mid2;
create table if not exists ycf_dwh_mdm.m_act_user_portrait_mid2
as
select
tt2.union_account_id
,concat_ws('|',collect_set(tt1.list_product_and_package)) as list_product_and_package
,concat_ws('|',collect_set(nvl(tt1.pay_type_names,''))) as pay_type_names
,concat_ws('^',collect_set(tt1.order_product_ids)) as order_product_ids
,concat_ws('^',collect_set(tt1.order_product_names)) as order_product_names
,concat_ws('^',collect_set(tt1.order_package_ids)) as order_package_ids
,concat_ws('^',collect_set(tt1.order_package_names)) as order_package_names
,concat_ws(',',collect_set(tt1.label_name)) as label_name
from ycf_dwh_mdm.m_act_user_portrait_mid1 tt1
left join ycf_dwh_fdm.f_usr_mag_user_union_account_mapping_full tt2
on tt1.user_id=tt2.user_id and tt2.row_id=1
where tt2.union_account_id is not null
and (
tt1.list_product_and_package<>''
or tt1.pay_type_names<>''
or tt1.order_product_ids<>''
or tt1.order_product_names<>''
or tt1.order_package_ids<>''
or tt1.order_package_names<>''
or tt1.label_name<>''
)
group by tt2.union_account_id
;

数据量:TT1:1600w ;TT2:80w

Hive版本:hive-common-1.1.0

错误日志:

Query ID = hdfs_20220127155252_844fbc68-7d2d-4b31-9a8f-86a90eb15d0d
Total jobs = 5
Launching Job 1 out of 5
Number of reduce tasks not specified. Estimated from input data size: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
  set mapreduce.job.reduces=<number>
Starting Job = job_1643268609681_0075, Tracking URL = http://xxx-big1.xxx.com:8088/proxy/application_1643268609681_0075/
Kill Command = /opt/cloudera/parcels/CDH-5.14.0-1.cdh5.14.0.p0.24/lib/hadoop/bin/hadoop job  -kill job_1643268609681_0075
Hadoop job information for Stage-1: number of mappers: 2; number of reducers: 1
2022-01-27 15:52:28,397 Stage-1 map = 0%,  reduce = 0%
2022-01-27 15:52:46,978 Stage-1 map = 100%,  reduce = 100%
Ended Job = job_1643268609681_0075 with errors
Error during job, obtaining debugging information...
Examining task ID: task_1643268609681_0075_m_000001 (and more) from job job_1643268609681_0075

Task with the most failures(4):
-----
Task ID:
  task_1643268609681_0075_m_000000

URL:
  http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1643268609681_0075&tipid=task_1643268609681_0075_m_000000
-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:455)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
    at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1917)
    at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 9 more
Caused by: java.lang.RuntimeException: Error in configuring object
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)
    at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
    at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)
    ... 14 more
Caused by: java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)
    ... 17 more
Caused by: java.lang.RuntimeException: Map operator initialization failed
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:147)
    ... 22 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeException: Hive internal error inside isAssignableFromSettablePrimitiveOI void not supported yet.
    at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:323)
    at org.apache.hadoop.hive.ql.exec.MapOperator.setChildren(MapOperator.java:333)
    at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:116)
    ... 22 more
Caused by: java.lang.RuntimeException: Hive internal error inside isAssignableFromSettablePrimitiveOI void not supported yet.
    at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettablePrimitiveOI(ObjectInspectorUtils.java:1191)
    at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.isInstanceOfSettableOI(ObjectInspectorUtils.java:1200)
    at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorUtils.hasAllFieldsSettable(ObjectInspectorUtils.java:1238)
    at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:224)
    at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:252)
    at org.apache.hadoop.hive.serde2.objectinspector.ObjectInspectorConverters.getConvertedOI(ObjectInspectorConverters.java:188)
    at org.apache.hadoop.hive.ql.exec.MapOperator.getConvertedOI(MapOperator.java:305)
    ... 24 more

FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.mr.MapRedTask
MapReduce Jobs Launched:
Stage-Stage-1: Map: 2  Reduce: 1   HDFS Read: 0 HDFS Write: 0 FAIL
Total MapReduce CPU Time Spent: 0 msec


About Joyk


Aggregate valuable and interesting links.
Joyk means Joy of geeK