【CentOS】flume 报错：“SinkRunner-PollingRunner-DefaultSinkProcessor“

【CentOS】flume 报错：“SinkRunner-PollingRunner-DefaultSinkProcessor“

在执行flume任务的时候报了错，如下:

Exception in thread "SinkRunner-PollingRunner-DefaultSinkProcessor" java.lang.ExceptionInInitializerError at org.apache.hadoop.tracing.SpanReceiverHost.get(SpanReceiverHost.java:79) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:634) at org.apache.hadoop.hdfs.DFSClient.(DFSClient.java:619) at org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:149) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371) at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:243) at org.apache.flume.sink.hdfs.BucketWriter$1.call(BucketWriter.java:235) at org.apache.flume.sink.hdfs.BucketWriter$9$1.run(BucketWriter.java:679) at org.apache.flume.auth.SimpleAuthenticator.execute(SimpleAuthenticator.java:50) at org.apache.flume.sink.hdfs.BucketWriter$9.call(BucketWriter.java:676) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)Caused by: java.lang.IllegalStateException: Shutdown in progress at java.lang.ApplicationShutdownHooks.add(ApplicationShutdownHooks.java:66) at java.lang.Runtime.addShutdownHook(Runtime.java:211) at org.apache.hadoop.util.ShutdownHookManager.(ShutdownHookManager.java:47)

找了一下，说是将flume的lib下的低版本guava-11.0.2.jar换成Hadoop下的高版本guava-27.0-jre.jar

然后找到本环境下的flume、hadoop找到guava.jar文件发现连个版本相同！？

最后找到的原因是：hdfs上的目标文件目录不存在，需要先创建

[hadoop@master lib]$ hdfs dfs -ls /tmp/flume[hadoop@master lib]$ cd /usr/local/src/flume/[hadoop@master flume]$ ll总用量 136drwxr-xr-x. 2 hadoop hadoop 62 2月 20 14:31 bin-rw-r--r--. 1 hadoop hadoop 69856 5月 9 2015 CHANGELOGdrwxr-xr-x. 2 hadoop hadoop 147 2月 20 14:37 conf-rw-r--r--. 1 hadoop hadoop 6172 5月 9 2015 DEVNOTESdrwxr-xr-x. 10 hadoop hadoop 4096 5月 12 2015 docsdrwxr-xr-x. 2 hadoop hadoop 4096 2月 20 14:31 lib-rw-r--r--. 1 hadoop hadoop 25903 5月 9 2015 LICENSE-rw-r--r--. 1 hadoop hadoop 249 5月 9 2015 NOTICE-rw-r--r--. 1 hadoop hadoop 1779 5月 9 2015 README-rw-r--r--. 1 hadoop hadoop 1585 5月 9 2015 RELEASE-NOTES-rw-rw-r--. 1 hadoop hadoop 507 2月 20 14:38 simple-hdfs-flume.confdrwxr-xr-x. 2 hadoop hadoop 68 2月 20 14:31 tools[hadoop@master flume]$ flume-ng agent --conf-file simple-hdfs-flume.conf --name a1Warning: No configuration directory set! Use --conf

to override.Info: Including Hadoop libraries found via (/usr/local/src/hadoop/bin/hadoop) for HDFS accessInfo: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-api-1.7.10.jar from classpathInfo: Excluding /usr/local/src/hadoop/share/hadoop/common/lib/slf4j-log4j12-1.7.10.jar from classpathInfo: Including Hive libraries found via (/usr/local/src/hive) for Hive access+ exec /usr/local/src/java/bin/java -Xmx20m -cp '/usr/local/src/flume/lib/*:/usr/local/src/hadoop/etc/hadoop:/usr/local/src/hadoop/share/hadoop/common/lib/activation-1.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/apacheds-i18n-2.0.0-M15.jar:/usr/local/src/hadoop/share/hadoop/common/lib/apacheds-kerberos-codec-2.0.0-M15.jar:/usr/local/src/hadoop/share/hadoop/common/lib/api-asn1-api-1.0.0-M20.jar:/usr/local/src/hadoop/share/hadoop/common/lib/api-util-1.0.0-M20.jar:/usr/local/src/hadoop/share/hadoop/common/lib/asm-3.2.jar:/usr/local/src/hadoop/share/hadoop/common/lib/avro-1.7.4.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-beanutils-1.7.0.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-beanutils-core-1.8.0.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-cli-1.2.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-codec-1.4.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-collections-3.2.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-compress-1.4.1.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-configuration-1.6.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-digester-1.8.jar:/usr/local/src/hadoop/share/hadoop/common/lib/commons-flume]$ hdfs dfs -ls /tmp/flume #查看hdfs上传输的flume日志文件Found 67 items-rw-r--r-- 3 hadoop supergroup 1568 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575970-rw-r--r-- 3 hadoop supergroup 1643 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575971-rw-r--r-- 3 hadoop supergroup 1707 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575972-rw-r--r-- 3 hadoop supergroup 1864 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575973-rw-r--r-- 3 hadoop supergroup 1732 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575974-rw-r--r-- 3 hadoop supergroup 1365 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575975-rw-r--r-- 3 hadoop supergroup 1379 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575976-rw-r--r-- 3 hadoop supergroup 1376 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575977-rw-r--r-- 3 hadoop supergroup 1707 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575978-rw-r--r-- 3 hadoop supergroup 1864 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575979-rw-r--r-- 3 hadoop supergroup 814 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575980-rw-r--r-- 3 hadoop supergroup 1665 2022-02-20 15:02 /tmp/flume/FlumeData.1645340575981

版权声明：本文内容由网络用户投稿，版权归原作者所有，本站不拥有其著作权，亦不承担相应法律责任。如果您发现本站中有涉嫌抄袭或描述失实的内容，请联系我们jiasou666@gmail.com 处理，核实后本网站将在24小时内删除侵权内容。