Install Elastic Stack on Ubuntu 16.04, CentOS 7 Single Cloud Server

Three softwares – Elasticsearch, Logstash, and Kibana better known before as ELK Stack. Here is How to Install Elastic Stack on Ubuntu 16.04, CentOS 7 on Single Cloud Server Instance For Server Log Analysis, Big Data Processing. In this tutorial, we will show how to installa of the mentioned softwares on Ubuntu 16.04 and CentOS 7. Additionally we will show you to configure, visualize the syslogs. The server will run Nginx web server.

Why Install Elastic Stack Instead of Apache Hadoop or Spark With Elastic Search?

Previously we discussed how to install Apache Hadoop, how to install Apache Spark and how to install fluentd on single cloud server instance. This guide to install Elastic Stack is not on Apache Hadoop as not all users are used to handle real Big Data software. In separate guide we will show how to integrate Elasticsearch with Apache Hadoop. This kind of setups and examples are practical usage of the software collections.

We are showing how to install Elastic Stack on Ubuntu and CentOS in this easy manner so that any level go users can install and use for basic log analysis purpose without going in to the fuss of Big Data softwares.

Why Install Elastic Stack on Ubuntu 16.04, CentOS 7 Single Cloud Server?

You can install Elastic Stack on same dedicated server where your web software or database software is/are running. But Cloud Server can be cheap like VPSDime 6GB OpenVZ instance costs only $7/month. Such resources are great for the small to medium business owners and webmasters. ES crash on OpenVZ not uncommon but a workaround, low load average does make it working. We are installing on a single cloud server instance for cost saving and isolation purpose. We suggest to use a server with 4GB of RAM for running Elastic Stack. 1GB RAM instance can run it somehow, 1GB RAM VMWare Cloud Server from Aruba Cloud will cost just 1 Euro per month.

The individual components does these works :

1. Logstash collects, enrich and send data it to Elasticsearch
2. ElasticSearch stores incoming logs from Logstash and provides the ability to search in real time.
3. Kibana provides the way of visualization of logs. There are other softwares as alternative but Kibana is commonly used.
4. Beats/Filebeat/fluentd when installed on client, sends logs to Logstash.
5. Nginx works as web server, reverse proxy.

Steps to Install Elastic Stack on Ubuntu 16.04, CentOS 7 Single Cloud Server

Step 1 : SELinux and Install Nginx

First we need to make sure that SELINUX is disabled. Open :

Change SELINUX value :

then reboot the server :

SSH to the server again and run this command to check the SELinux state :

Install Nginx :

Make sure to allow traffic through TCP port 9200, port 5044, port 80, port 443 in your firewall.

Step 2 : Install Oracle Java

For Ubuntu 16.04 :

For CentOS 7 :

For both of them :

Check the version of Java :

Example output :

Step 3 : Install ElasticSearch

For Ubuntu 16.04 :

If you need to restrict access to Elasticsearch instance at port 9200, do this:

Edit :

Save, exit, start Elasticsearch, configure to start Elasticsearch on boot up::

Test elasticsearch :

For CentOS 7 :

If you need to restrict access to Elasticsearch instance at port 9200, do this:

Edit :

Save, exit. This disables memory swapping for Elasticsearch :

Uncomment :

Save and exit. Edit :

Save and exit. To start Elasticsearch, configure to start Elasticsearch on boot up::

Test :

Step 4 : Install Kibana

For Ubuntu 16.04 :

Install the Kibana using the following command.

Start and enable kibana on system startup :

Access the Kibana using the following URL :

For CentOS 7 :

Now edit the Kibana configuration file :

Uncomment the configuration lines for server.port, server.host and elasticsearch.url.

Save and exit. Add Kibana to run at boot and start it :

Kibana will run on port 5601 as node application.

For both of them :

Now configure with Nginx :

This is typical example of nginx config:

Save and exit. Test restart Nginx :

Step 5 : Install Logtash

For Ubuntu 16.04 :

For CentOS 7 :

Open :

add :

Install the Logstash package:

For Both of them :

We are showing for subdomain/domain installation, which is easy :

Add entry with real values :

Then run after changing “server.your.local” :

If you want to use IP address then :

Edit to your IP :

Logstash configuration found at /etc/logstash/conf.d/. logstash configuration file consists of 3 parts – input, filter, and output. It is better to use a single file for these three.

Keep in mind to change /etc/ssl/logstash-forwarder.crt :

Save and exit. Now run :

You can run cat :

Step 6 : Install Filebeat

There are four beats clients available. Packetbeat – Analyzes network packet data, Filebeat gives insight into log data, Topbeat gives insights from infrastructure data, Metricbeat delivers metrics to Elasticsearch.

For Ubuntu 16.04 :

For CentOS 7 :

For both :

Filebeat uses SSL certificate for validating logstash server identity, so we need to copy the logstash-forwarder.crt from the logstash server to the client :

Now :

Add two files ‘/var/log/secure’ for ssh activity and ‘/var/log/messages’ for the server log :

Also :

Under output uncomment line with logstash, edit the IP address of ELK server and port where Logstash is listening hosts, make sure the path to the certificate points to the actual file. Thereafter restart :

Now go to the IP address where we can get Kibana. Installing Elastic Stack is difficult work.

Fully optional testing :

For TESTING purpose, you can download sample Kibana dashboards and Beats index patterns :

Download the filebeat index template

Run the following CURL command :