Deploying and Configuring ELK (elasticsearch,logstash,kibana)

It gave me headache to make the combination of elasticsearch, logstash, kibana and logstash-forwarder work togather properly. The main problems I faced was to compile the go code for logstash-forwarder and the x509v3 self-signed certificate for logstash.

You will need to get the following files:


Use “tar xvf file.tar.gz” to extract them.

Elasticsearch and Kibana have config file so we just need to edit, but for logstash create config/logstash.yml file inside.

mkdir logstash-1.5.0.beta1/config/
touch logstash-1.5.0.beta1/config/logstash.yml
mkdir logstash-1.5.0.beta1/

Edit all these file according to the contents you will find in appendix 1,2 and 3.

vi logstash-1.5.0.beta1/config/logstash.yml
vi elasticsearch-1.4.2/config/elasticsearch.yml
vi kibana-4.0.0-beta3/config/kibana.yml

Then we need to create a certificate and private key for logstash:

mkdir cert
cd cert/
touch ssl.conf (use appendix 5 content)
openssl req -x509 -batch -nodes -newkey rsa:2048 -keyout server.key -out server.crt -config ssl.conf -days 1825

And finally run them

logstash-1.5.0.beta1/bin/logstash agent -f ~/logstash-1.5.0.beta1/config/logstash.yml

Deploying logstash_forwarder:

To deploy logstash forwarder we need to intsall go and gem-fpm. We are basically creating a rpm or deb installer file.

yum install golang ruby ruby-devel rubygems
git clone
cd logstash-forwarder
go build
gem install fpm
make rpm
sudo rpm -ivh logstash-forwarder-*.x86_64.rpm

Once it is installed we need to deal with keys:

sudo cp server.key /usr/local/etc/logstash-forwarder/server.key
sudo cp server.crt /usr/local/etc/logstash-forwarder/server.crt
sudo openssl x509 -in server.crt -text >> /etc/pki/tls/certs/ca-bundle.crt

To configure logstash forwarder create a file and copy the contents of appendix 4. We used /etc/logstash_forwarder.yml file.

And then run it:

/opt/logstash-forwarder/bin/logstash-forwarder -config /etc/logstash_forwarder.yml

Appendix1: Elasticsearch configuration to be added

script.disable_dynamic: true localhost
http.cors.allow-origin: "/.*/"
http.cors.enabled: true

Appendix2: Logstash configuration

input {
  lumberjack {
    port => 5000
    type => "logs"
    ssl_certificate => "~/cert/server.crt"
    ssl_key => "~/cert/server.key"

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }

Appendix3: Kibana configuration to be edited

port: 8080
host: ""
elasticsearch: "http://localhost:9200

Appendix4: Logstash_forwarder configurations

    "network": {
        "servers": [ "" ],
        "ssl certificate": "/usr/local/etc/logstash-forwarder/server.crt",
        "ssl key": "/usr/local/etc/logstash-forwarder/server.key",
        "timeout": 15
    "files": [
        "paths": [
        "fields": { "type": "syslog" }
        "paths": [
        "fields": { "type": "apache" }

Appendix 5: OpenSSL configuration file for creating certificate:

distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

C = TG
ST = Togo
L =  Lome
O = Private company
CN = *

subjectKeyIdentifier = hash
authorityKeyIdentifier = keyid,issuer
basicConstraints = CA:TRUE
subjectAltName = @alt_names

DNS.1 = *
DNS.2 = *.*
DNS.3 = *.*.*
DNS.4 = *.*.*.*
DNS.5 = *.*.*.*.*
DNS.6 = *.*.*.*.*.*
DNS.7 = *.*.*.*.*.*.*
IP.1 =
IP.2 =
IP.3 =

Appendix 6: Alternatives for /etc/pki/tls/certs/ca-bundle.crt


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s