Install fluentd Agent : Log Data Collection For Hadoop

Our previous two guides were on how to install Apache Hadoop and how to install Apache Spark. Next step technically is to get data from source like server log files. Here is How To Install fluentd Agent on Ubuntu 16.04 As Intermediate Step of Log Data Collection For Hadoop. We Can Receive Data on Laptop. This step is on your servers to monitor, not on other servers.

Install fluentd Agent : Preface For Log Data Collection For Hadoop

Here is official website of fluentd :

It is a beautiful software written in Ruby. Fluentd is a Big Data tool and can work with unstructured data in real time. Exactly like an another tool Kafka, it analyzes the event logs, application logs, and clickstreams. It can simply output the collected data over HTTPS with settings to secure the transport. It is better to get used with the usage of such tools before jumping to analyse data with Apache Hadoop, Spark or plain Elastic Search. You must not use the public IP of the web server associated with domain under question for higher security except for debugging, initial setup etc.

The above illustration shows our plan of tutorials. In future we will stream this data to :

1. Local computer like MacBook Pro
2. Server running Big Data Specific tool like Apache Hadoop, Spark alone or in combination.
3. Server running primarily basic analysis tools like ElasticSearch rather ELK Stack.
4. Server running Big Data Specific tool like Apache Hadoop, Spark alone or in combination and ElasticSearch and Kibana.

This way of learning will help you to never fumble with – “will I use ElasticSearch or Apache Hadoop?”. Elastic analytics stack is gaining popularity for various reasons. ElasticSearch uses JSON based query language which is much easier to master than Hadoop’s MapReduce. Also the Developers are more comfortable maintaining ElasticSearch instance over Hadoop. Of course, this architecture would have data loss issue on the ElasticSearch side. You need Treasure Data in such case in front of ElasticSearch to “buffer” it. We are simply using fluentd to maintain backward and forward compatibility without disturbing the main server too much. Where there’s a shell, there is a way. The way is for the hackers too.

Steps To Install fluentd Agent on Ubuntu 16.04

It is recommended to set up ntpd on the main server under question of logging to prevent invalid timestamps in logs. Increase the maximum number of file descriptors, run :

Expect value like 65535 against ulimit -n. If you get value like 1024, open :

add/modify :

Exit session after saving and reboot. open :

add/modify :

Exit session after saving and reboot. open :

add/modify :

Exit session after saving and reboot. open :

add/modify :

Exit session after saving and reboot. For Ubuntu 16.04, run this :

You can control the software with :

Carefully run this command after changing IP, port :

log-ip is your log server ip, 24224 is your port.

open :

add/modify :

Run :

You can run cURL to POST :

There are more documentation on port part :

Here is plugin management :

fluentd has one of the worst documentation on this earth. I ran the command many steps before :

else you would get error. They actually have lot of stuffs :

including web UI.