Hands on workshop with ELK: Elasticsearch, Logstash and Kibana

Marco Bertani-√łkland

Sigmund Hansen

Half-day workshop - in English

ELK consist of three open source projects — Elasticsearch, Logstash, and Kibana — designed to take data from any source and search, analyze, and visualize it in real time. The philosophy behind these tools is that getting immediate, actionable insight from data matters.

During this workshop we will teach you how to:

  • Set up and configure ELK
  • Define two pipelines in ELK:
    1. Visualize criminal records from 2014 from LAPD: https://data.lacity.org/A-Safe-City/LAPD-Crime-and-Collision-Raw-Data-2014/eta5-h8qx 
      The original link has been deprecated, but we have the data in the github repo
    2. Visualize data from a http_access log (available in the github repo, can be done as "homework")
  • Configure logstash to send the information to elasticsearch
  • Define a mapping template in elasticsearch to shape our data, so that you can both search and aggregate.
  • Visualize the data in Kibana. Create a dashboard with a map to get insights from the data
  • If time allows, do a small time series analysis with Kibanas Timelion plugin. Build an outlier detector for a search query.

For more information, take a look at https://www.elastic.co/

What we expect from you:

  • Not afraid to use a shell, though commands will be served
  • Hands on
  • Eager to learn a new technology
  • Not afraid to ask questions

Get prepared

  1. Clone our github repo branch master at https://github.com/mbertani/elk-workshop/
    git clone --recursive https://github.com/mbertani/elk-workshop/

    # or if you have already cloned with "git clone https://github.com/mbertani/elk-workshop/"
    # you have to initialize the reveal.js submodule with this command inside the elk-workshop directory:
    git submodule update --init --recursive

    # switch to branch booster2016
    git checkout booster2016
  2. Follow instructions at Readme.md
  3. The presentation is in https://github.com/mbertani/elk-workshop/blob/booster2016/presentation/presentation.html
    After you clone the repository, you can run the presentation in your browser 
  4. Probably you should run a "git pull" the day of the workshop to make sure you get the latest changes.
  5. This workshop has been tested before in windows and linux. There are some issues with Ruby if you are using OSX.

Primarily for: Developers, Architects, Managers, Scrum masters

Participant requirements: Own laptop with 5GB free disk space. Unix or Windows. Make sure you also have at least 4 GB RAM available. Java 8 installed A good text editor (like Notepad++ or similar)