Installing the ELK Stack on Ubuntu - Elasticsearch, Logstash, Kibana and Nginx

This article forms part of our series on setting up a simple, open source network security monitoring system on Linux suitable for a small office network.

In this post we will get the basics in place to manage our data and provide indexed search facilities. We will use the ELK stack of Elasticsearch, Logstash and Kibana and put it all behind an Nginx reverse proxy server to provide HTTPS and give us authentication options.

All commands relate to a default installation of Ubuntu 14.04.3 LTS and the versions of Java, ELK, etc are current at the time of the post.

Install Java 8

$ sudo add-apt-repository -y ppa:webupd8team/java
$ sudo apt-get update
$ echo debconf shared/accepted-oracle-license-v1-1 select true | sudo debconf-set-selections
$ echo debconf shared/accepted-oracle-license-v1-1 seen true | sudo debconf-set-selections
$ sudo apt-get -y install oracle-java8-installer

If all goes well you should have Java 8 installed.

$ java -version
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

Install Elasticsearch

$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ echo "deb http://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
$ sudo apt-get update && sudo apt-get install elasticsearch
$ sudo update-rc.d elasticsearch defaults 95 10
 Adding system startup for /etc/init.d/elasticsearch ...
   /etc/rc0.d/K10elasticsearch -> ../init.d/elasticsearch
   /etc/rc1.d/K10elasticsearch -> ../init.d/elasticsearch
   /etc/rc6.d/K10elasticsearch -> ../init.d/elasticsearch
   /etc/rc2.d/S95elasticsearch -> ../init.d/elasticsearch
   /etc/rc3.d/S95elasticsearch -> ../init.d/elasticsearch
   /etc/rc4.d/S95elasticsearch -> ../init.d/elasticsearch
   /etc/rc5.d/S95elasticsearch -> ../init.d/elasticsearch

Now start it up and perform a quick test to see if it’s ok.

$ sudo /etc/init.d/elasticsearch start
$ curl 'http://localhost:9200'
{
  "name" : "Topaz",
  "cluster_name" : "elasticsearch",
  "version" : {
    "number" : "2.3.3",
    "build_hash" : "218bdf10790eef486ff2c41a3df5cfa32dadcfde",
    "build_timestamp" : "2016-05-17T15:40:04Z",
    "build_snapshot" : false,
    "lucene_version" : "5.5.0"
  },
  "tagline" : "You Know, for Search"
}

Install Logstash

$ echo "deb https://packages.elastic.co/logstash/2.3/debian stable main" | sudo tee -a /etc/apt/sources.list
$ sudo apt-get update && sudo apt-get install logstash

Enable Logstash on startup.

$ sudo update-rc.d logstash defaults 95 10
 Adding system startup for /etc/init.d/logstash ...
   /etc/rc0.d/K10logstash -> ../init.d/logstash
   /etc/rc1.d/K10logstash -> ../init.d/logstash
   /etc/rc6.d/K10logstash -> ../init.d/logstash
   /etc/rc2.d/S95logstash -> ../init.d/logstash
   /etc/rc3.d/S95logstash -> ../init.d/logstash
   /etc/rc4.d/S95logstash -> ../init.d/logstash
   /etc/rc5.d/S95logstash -> ../init.d/logstash

And finally, start it up.

$ sudo /etc/init.d/logstash start

Install Kibana

$ echo "deb http://packages.elastic.co/kibana/4.5/debian stable main" | sudo tee -a /etc/apt/sources.list
$ sudo apt-get update && sudo apt-get install kibana
$ sudo update-rc.d kibana defaults 95 10

Optional

It is our intention to configure an Nginx reverse proxy in front of our ELK stack so for this reason we are going to change our Kibana instance to bind to localhost only, to prevent direct network access.

This is achieved by editing /opt/kibana/config/kibana.yml and setting the following:

server.host: "127.0.0.1"

Now fire her up.

$ sudo /etc/init.d/kibana start

Nginx

Now to make things nice and tidy, let’s put all of this behind an Nginx reverse proxy.

$ sudo apt-get install nginx

We’re going to use HTTPS of course. Creating a valid key and certificate are beyond the scope of this article but the following commands may be useful to generate a new key and then a Certificate Signing Request you can use with whichever internal or external certificate authority you prefer.

$ cd /etc/nginx
$ sudo openssl genrsa -out cert.key 2048
$ sudo openssl req -new -sha256 -key cert.key -out cert.csr

When you’ve got a key and cert, put them in /etc/nginx/cert.key and /etc/nginx/cert.pem (or somewhere else but update the following config) and then create a file /etc/nginx/sites-available/nsm with the following content:

server {
	listen 80 default_server;
	return 301 https://$host$request_uri;
}

server {
        listen 443 ssl;
        server_name nsm; <-- edit this to match your hostname

        root html;
        index index.html index.htm;

        ssl on;
        ssl_certificate cert.pem;
        ssl_certificate_key cert.key;
	ssl_session_timeout 1d;
	ssl_session_cache shared:SSL:50m;

	ssl_protocols TLSv1.2;
	ssl_ciphers 'ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256';
	ssl_prefer_server_ciphers on;

	location / {
		proxy_pass http://127.0.0.1:5601;
		include proxy_params;
	}
}

Delete the default Nginx config file and enable our new one, then restart the web server.

$ sudo rm /etc/nginx/sites-enabled/default && sudo ln -s /etc/nginx/sites-available/nsm /etc/nginx/sites-enabled/nsm
$ sudo /etc/init.d/nginx restart

If everything’s gone ok, that should be the core search infrastructure in place and it should be accessible at https:///.

At this point you will likely find that Kibana, the graphical search tool will insist you create an index pattern but not be able to. This is simply because there is no data in Elasticsearch yet. We’ll start populating this in future articles, stay tuned.