Centralized Logging with the Elastic Stack

This month I launched my newest Pluralsight course, Centralized Logging with the Elastic Stack: Getting Started. If you’ve ever wanted to understand how Elasticsearch, Kibana, Beats, and Logstash all work together. This is definitely the course you want to start with.

It’s more than just learning the basics of the individual parts, though. My course will show you how to create a full fledged monitoring and alerting system, all from the open source Elastic components that make up the Elastic Stack. Each component works on Windows, Linux, or macOS so this course has something for everyone. Here’s a breakdown of what’s covered:

Read More

Build Your Own 32 Core Home Lab Server

While working on my newest Pluralsight course this past Fall, I decided I wanted an easier way to create virtual machines on my local home network. Sure, it’s easy enough to spin up a few VMs on my iMac, but at some point the limitations of 16 gb of RAM and 4 processor cores was going to catch up with me. For this course in particular, I wanted to be able to run a half dozen or more VMs simultaneously to simulate a real corporate network with all or most components of the Elastic Stack running on it. What I really needed was a cheap virtual machine host with a lot of CPU cores and RAM and some reasonably fast disks. Before getting started on my course, I set out to build a server exclusively for the purpose of serving virtual machines on my network.

The centerpiece of the server build is the Intel Xeon E5-2670 CPU. The 2670 was released in 2012 with 8 cores (16 with Hyperthreading), and a full 20 mb L3 cache. By any measure it was a 64 bit x86 workhorse. Originally 2670s were priced at around $1,500 each which is far above the ~$1000 price tag I was hoping to keep the cost of this server under. As of this writing, however, you can find them on eBay for around $80. If you combine two of them, you end up with 32 logical cores which is an awful lot to spread around for virtual machines.

The Intel Xeon E5-2670 2.6 ghz

Read More

My Elasticsearch Tools List

[updated]

Working with Elasticsearch is wonderful but it can be a challenge to lasso together a coherent set of tools with which to manage and interact with your cluster. There is no one “do-everything” utility that includes 100% of the Elasticsearch query, operations, and administrative API functionality. There are a few good tools and plugins that I use, however, that round out a complete toolbox for using Elasticsearch effectively.

Operations

elasticsearch-head
The Elasticsearch head plugin is written by Ben Birch and is one of the old stalwarts that nearly everyone has used at one point to help manage an Elasticsearch cluster. It has a nice basic overview of indexes and includes some rudimentary administrative functionality like creating index aliases, running optimize, and easily viewing read-only data and cluster health. I still use this plugin pretty regularly but it has since been superseded by some more modern interfaces. It also falls down a bit, interface-wise, if you have a lot of indices in your cluster. You can install elasticsearch-head as a plugin directly from the Elasticsearch plugin command.

Elasticsearch Head
Elasticsearch Head Demonstration

Read More

log4net.Elasticsearch Rolling Indexes

Today I made a small update to log4net.Elasticsearch so that users can optionally choose to have new indexes created each day. This was suggested in a Github issue and I think it’s a good idea. Many Elasticsearch indexes are managed on a rotating basis. It’s simple to create index aliases so that if any log readers or applications are reading directly from the Elasticsearch index, the index alias can be established nightly with a routine API call.

The connection string parameter is optional. Simply add

rolling=true

to your usual application connection string and the logger will take care of the rest. You can see an example of the setting on the sample App.config file in the Github repository.

Happy logging!

Build a Geckoboard TV Dashboard with a Raspberry Pi

Part of the work I do on Postmark is helping make sure we know what’s going on inside the system at any given time. We built a real-time(ish) monitoring system using Geckoboard and a variety of Windows Performance Counters, API endpoints, and database calls. I’ll save those details for another post.

Geckoboard provides an easy-to-use suite of widgets that speak JSON and XML so if you shoot your data at it on an interval (or provide it an API to poll), it’ll display the data through charts and graphs in ways that are useful to an audience. We use it extensively at Wildbit to keep an eye on all the various Postmark systems.

Having this information on display at our office for everyone to see is important so we arranged to show our Geckobard dashboards on 40 inch televisions, 2 for Beanstalk and 1 for Postmark. We bought 3 Rikomagic MK802 Android devices which are convenient since they come equipped with HDMI out and can be powered by the USB ports on the TVs.

The original MK802 device.
The original MK802 device.

For the most part this worked well but there were some drawbacks to using the Android devices. The biggest problem was that we couldn’t seem to get a browser to run in full-screen or kiosk mode. A lot of screen real estate was taken up by the browser chrome and menus which meant less room was available for Geckoboard graphs. Last week the Android device powering the Postmark Geckoboard suffered some kind of disk encryption failure and stopped working. After a few failed attempts at restoring the MK802, I opted to try out my new Raspberry Pi to display the dashboard.

The way I saw it, there were no downsides to using the Pi instead of another MK802:

  • Cost is about the same.
  • Also comes with HDMI output and can be powered by the USB port on the TV.
  • Runs a full fledged Linux distro (Raspbian, among others) so it would be a familiar configuration setup.
  • Possible to run X11 and Chrome in kiosk mode for a more professional looking setup.
Raspberry Pi Model B
Raspberry Pi Model B

Following are the steps I took to create a fully automatic Raspberry Pi Geckoboard TV dashboard:

Hardware:
Software Setup:
  1. First you’ll need to install an operating system on your Pi. I’m using Raspbian ‘Wheezy’ which is a Debian derivative and one of the recommended choices. I won’t go into the installation details here. For that, I’ll direct you to their setup guide (PDF). You can go ahead and get through that first. It’s cool. I’ll wait. Make sure to get your wireless (or wired) networking going before trying to continue.
  2. Set your Pi to boot to ‘desktop mode’ by default. This will go straight into the X desktop and window manager when you turn your Pi on. To do that, you’ll need to go into your raspi-config tool by typing:
    sudo raspi-config
    

    Setting up to boot direction to desktop mode.
    Setting up to boot direction to desktop mode.

     

  3. Once that’s done, we’ll need to install a few things:
    sudo apt-get update && apt-get upgrade -y
    sudo apt-get install chromium x11-xserver-utils unclutter

    Now that we’ve got a few utilities installed and, more importantly, the Chromium browser, let’s make some configuration tweaks: (source)

  4. Edit the /etc/xdg/lxsession/LXDE/autostart file
    sudo nano /etc/xdg/lxsession/LXDE/autostart

    Comment out the @screensaver line with a “#”
    Add the following lines:

    @xset s off
    @xset -dpms
    @xset s noblank
    @chromium --kiosk --incognito http://link.to.dashboard/9894389439
    

    That last line contains the link to which you want Chromium to be directed on bootup so put in the public link to your Geckoboard (or other dashboard).

  5. Finally, you may need to force your Pi to use HDMI out if you are getting power from the TV. The Pi doesn’t detect the HDMI signal from the TV fast enough when you turn it on and disables the port on some TVs (this happened to me).
    sudo nano /boot/config.txt

    Remove the comment from the line:

    hdmi_force_hotplug=1

    That should keep your Pi outputting video to your HDMI output regardless if it detected the TV at first or not.

Final Product:

Our command center consists of 3 TVs, the middle is for Postmark. I still need to mount our Raspberry Pi properly since this was just a prototype setup. So far so good, however. I think we’ll replace the last two MK802s with Pis soon since it offers a superior dashboard experience.

Beanstalk and Postmark
Beanstalk and Postmark
Current setup. A little temporary.
Current setup. A little temporary.