我正在使用sqoop import将utf8数据从mysql导入hdfs。它工作正常,但当数据是utf-8时会遇到问题。源mysql表与utf-8兼容,但看起来sqoop在导入期间正在转换数据。示例-源值-л.с. 加载为ð».ñ. 到hdfs。
目前,mysql(v5.6.10)字符集和排序规则如下:
+--------------------------+-----------------------------------------+
| Variable_name | Value |
+--------------------------+-----------------------------------------+
| character_set_client | latin1 |
| character_set_connection | latin1 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | latin1 |
| character_set_server | utf8 |
| character_set_system | utf8 |
| collation_connection | latin1_swedish_ci |
| collation_database | latin1_swedish_ci |
| collation_server | utf8_unicode_ci |
+--------------------------+-----------------------------------------+
-- Table Structure
CREATE TABLE utf_test_cases_ms
(
test_case varchar(50) NOT NULL,
english_lang varchar(250) NOT NULL,
language_name varchar(50) NOT NULL,
utf8_lang varchar(300) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ;
-- Mysql
select * from utf_test_cases_ms;
+--------------------+--------------+---------------+-----------+
| test_case | english_lang | language_name | utf8_lang |
+--------------------+--------------+---------------+-----------+
| Multiple Character | hp | Russian | л.с. |
+--------------------+--------------+---------------+-----------+
-- Sqoop Import Command
sqoop import --connect jdbc:mysql://<<IP_ADDRESS_WITH_PORT>>/<<DB_NAME>>
--table utfmb_test_cases_ms --username sqoop_user --password sqoop_pwd
--hive-import --hive-table utf_ms_db.utfmb_test_cases_ms
--create-hive-table --null-string '\\N' --null-non-string '\\N'
--fields-terminated-by '|' --lines-terminated-by '\n' -m 1
-- Hive (HDFS)
select * from utfmb_test_cases_ms;
Multiple Character hp Russian л.Ñ.
我是否需要更改mysql配置文件中的字符集和排序规则?通过sqoop导入数据时是否需要传递额外的unicode/utf8参数?
请提供解决方案。提前谢谢!
(来自评论)
CREATE TABLE utf_test_cases_ms (
test_case varchar(50) NOT NULL,
english_lang varchar(250) NOT NULL,
language_name varchar(50) NOT NULL,
utf8_lang varchar(300) NOT NULL
) ENGINE=MyISAM DEFAULT CHARSET=utf8 ;
2条答案
按热度按时间dohp0rv51#
在sqoop命令中传递字符集参数
qacovj5a2#
也许@marjun的建议应该包含utf8?
不管怎样,
л.с.
以…的身份出现л.Ñ.
是“莫吉巴克”吗http://stackoverflow.com/questions/38363566/trouble-with-utf8-characters-what-i-see-is-not-what-i-stored请提供
SHOW CREATE TABLE
所以我们可以验证设置是否正确。如果连接参数不起作用,请使用
SET NAMES utf8
.