The 7 Log Management Tools Java Developers Should Know

By Tal Weiss —  April 23, 2014 — 19 Comments

blog boat 1 The 7 Log Management Tools Java Developers Should Know

Splunk vs. Sumo Logic vs. LogStash vs. GrayLog vs. Loggly vs. PaperTrails vs. Splunk>Storm

 

Splunk, Sumo Logic, LogStash, GrayLog, Loggly, PaperTrails – did I miss someone? I’m pretty sure I did. Logs are like fossil fuels – we’ve been wanting to get rid of them for the past 20 years, but we’re not quite there yet. Well, if that’s the case I want a BMW!

To deal with the growth of log data a host of log management & analysis tools have been built over the last few years to help developers and operations make sense of the growing data. I thought it’d be interesting to look at our options and what are each tools’ selling point, from a developer’s standpoint.

Splunk

As the biggest tool in this space, I decided to put Splunk in a category of its own. That’s not to say it’s the best tool for what you need, but more to give credit to a product who essentially created a new category.

Pros

Splunk is probably the most feature rich solution in the space. It’s got hundreds of apps (I counted 537) to make sense of almost every format of log data, from security to business analytics to infrastructure monitoring. Splunk’s search and charting tools are feature rich to the point that there’s probably no set of data you can’t get to through its UI or APIs.

Cons

Splunk has two major cons. The first, that is more subjective, is that it’s an on-premise solution which means that setup costs in terms of money and complexity are high. To deploy in a high-scale environment you will need to install and configure a dedicated cluster. As a developer, it’s usually something you can’t or don’t want to do as your first choice.

Splunk’s second con is that it’s expensive. To support a real-world application you’re looking at tens of thousands of dollars, which most likely means you’ll need sign offs from high-ups in your organization, and the process is going to be slow. If you’ve got a new app and you want something fast that you can quickly spin up and ramp as things progress – keep reading.

Some more enterprise log analyzers can be found here.

SaaS Log Analyzers

Sumo Logic

Sumo was founded as a SaaS version of Splunk, going so far as to imitate some of splunk’s features and visuals early on. Having said that, SL has developed to a full fledged enterprise class log management solution.

Pros

SL is chock-full of features to reduce, search and chart mass amounts of data. Out of all the SaaS log analyzers, it’s probably the most feature rich. Also, being a SaaS offering it inherently means setup and ongoing operation are easier. One of Sumo Logic’s main points of attraction is the ability to establish baselines and to actively notify you when key metrics change after an event such as a new version rollout or a breach attempt.

Cons

This one is shared across all SaaS log analyzers, which is you need to get the data to the service to actually do something with it. This means that you’ll be looking at possible GBs (or more) uploaded from your servers. This can create issues on multiple fronts –

  1. As a developer, if you’re logging sensitive or PII you need to make sure it’s redacted.
  2. There may be a lag between the time data is logged and the time it’s visible to to the service.
  3. There’s additional overhead on your machines transmitting GBs of data, which really depends on your logging throughput.

Sumo’s pricing is also not transparent, which means you might be looking at a buying process which is more complex than swiping your team’s credit card to get going.

Update – I just got a note from Brandon at the Sumo Logic team letting us know you can purchase the product directly using your credit card from within the Free version. Not as easy as going through the web site, but quite close.

Loggly

Loggly is also a robust log analyzer, focusing on simplicity and ease of use for a devops audience.

Loggly The 7 Log Management Tools Java Developers Should Know

Pros

Whereas Sumo Logic has a strong enterprise and security focus, Loggly is geared more towards helping devops find and fix operational problems. This makes it very developer-friendly. Things like creating custom performance and devops dashboards are super-easy to do. Pricing is also transparent, which makes start of use easier.

Cons

Don’t expect Loggly to scale into a full blown infrastructure, security or analytics solution. If you need forensics or infrastructure monitoring you’re in the wrong place. This is a tools mainly for devops to parse data coming from your app servers. Anything beyond that you’ll have to build yourself.

PaperTrails

PaperTrails is a simple way to look and search through logs from multiple machines, in one consolidated easy-to-use interface. Think of it like tailing your log in the cloud, and you won’t be too far off.

PaperTrails The 7 Log Management Tools Java Developers Should Know

Pros

PT is what it is. A simple way to look at log files from multiple machines in a singular view in the cloud. The UX itself is very similar to looking at a log on your machine, and so are the search commands. It aims to do something simple and useful, and does it elegantly. It’s also very affordable.

Cons

PT is mostly text based. Looking for any advanced integrations, predictive or reporting capabilities? You’re barking up the wrong tree.

Splunk>Storm

This is Splunk’s little (some may say step) SaaS brother. It’s a pretty similar offering that’s hosted on Splunk’s servers.

Pros

Storm lets you experiment with Splunk without having to install the actual software on-premise, and contains much of the features available in the full version.

Cons

This isn’t really a commercial offering, and you’re limited in the amount of data you can send. It seems to be more of an online limited version of Splunk meant to help people test out the product without having to deploy first. A new service called Splunk Cloud is aimed at providing a full-blown Splunk SaaS experience.

Open Source Analyzers

Logstash

Logstash is an open source tool for collecting and managing log files. It’s part of an open-source stack which includes ElasticSearch for indexing and searching through data and Kibana for charting and visualizing data. Together they form a powerful Log management solution.

Logstash The 7 Log Management Tools Java Developers Should Know

Pros

Being an open-source solution means you’re inherently getting a lot of a control and a very good price. Logstash uses three mature and powerful components, all heavily maintained, to create a very robust and extensible package. For an open-source solution it’s also very easy to install and start using. We use Logstash and love it.

Cons

As Logstash is essentially a stack, it means you’re dealing with three different products. That means that extensibility also becomes complex. Logstash filters are written in Ruby, Kibana is pure javascript and ElasticSearch has its own REST API as well as JSON templates.

When you move to production, you’ll also need to separate the three into different machines, which adds to the complexity.

Graylog2

A fairly new player in the space, GL2 is an open-source log analyzer backed by MongoDB as well as ElasticSearch (similar to Logstash) for storing and searching through log errors. It’s mainly focused on helping developers detect and fix errors in their apps.

Also in this category you can find fluentd and Kafka whose one of its main use-cases is also storing log data. Phew, so many choices!

Takipi for Logs

logScreenshot The 7 Log Management Tools Java Developers Should Know

While this post is not about Takipi, I thought there’s one feature it has which you might find relevant to all of this.

The biggest disadvantage in all log analyzers and log files in general, is that the right data has to be put there by you first. From a dev perspective, it means that if an exception isn’t logged, or the variable data you need to understand why it happened isn’t there, no log file or analyzer in the world can help you. Production debugging sucks icon sad The 7 Log Management Tools Java Developers Should Know

One of the things we’ve added to Takipi is the ability to jump into a recorded debugging session straight from a log file error. This means that for every log error you can see the actual source code and variable values at the moment of error. You can learn more about it here.

This is one post where I would love to hear from you guys about your experiences with some of the tools mentioned (and some that I didn’t). I’m sure there are things you would disagree with or would like to correct me on – so go ahead, the comment section is below and I would love to hear from you.

 

blog running 300x142 The 7 Log Management Tools Java Developers Should Know

The Logging Olympics – A Race Between Today’s Top 5 Java Logging Frameworks – read more

Duke T shirt 268x300 The 7 Log Management Tools Java Developers Should Know

Takipi detects all your exceptions and errors and tells you why they happen. Even across multiple threads and machines. Installs in 1min. Less than 2% overhead – Deploy Takipi now and get a free T-shirt

Tal Weiss

Posts Twitter

Tal is the CEO of Takipi. Tal has been designing scalable, real-time Java and C++ applications for the past 15 years. He still enjoys analyzing a good bug though, and instrumenting code. In his free time Tal plays Jazz drums.
  • https://www.dataloop.io Colin Hemmings

    There is Splunk cloud, so you dont have to run it on-premise : http://www.splunk.com/view/cloud/SP-CAAAG58

    • http://www.takipi.com/ Tal Weiss

      Hi Colin,

      Thanks for the comment. It’s right there with Splunk>Storm.

      • Johannes Nicolai

        AFAIK, Splunk Cloud and Splunk Storm are different products. Splunk Cloud is very feature compatible to Splunk on premise and does not have any index size restrictions (as long as you pay for it).

  • logscape

    Hi Tal, You can also take a look at http://logscape.com

    • http://www.takipi.com/ Tal Weiss

      Hey Logscape, thanks for the comment – looks great!

      I Would love to hear more about the tool’s special features and advantages.

  • Francis DB

    Another option: https://logentries.com/

  • cdukes

    Please consider LogZilla (http://www.logzilla.net) as well. It scales to 1B events per day on a single server and is about 1/10th of the cost of the other tools in its class. There’s also a free version for small networks.

  • Ashish Mohindroo

    Try the new Cloud based log management platform for Java: http://www.oohlalog.com. It’s Free! And it offers Non-Blocking I/O and stacktraces.

  • Da Beave

    You might want to consider checking out “Sagan”. While archiving and being able to search mass amounts of logs is very powerful, knowing what to search for and doing it in real time is also important from a security monitoring standpoint. Sagan basically “watches” you logs and detects security related events based off them (malware detection, brute force attacks, suspicious traffic, etc). It was a lot like a Snort IDS system, but for logs. In fact, the rule syntax is very similar to Snort and Sagan can even write to Snort/Suricata graphical interfaces (ie – Snorby, Sguil, etc). Oh, and it’s a open source project. More information is at:

    http://sagan.io

  • Kurt

    In regards to some of the cons associated w/ SaaS based log management solutions, Sumo Logic does the best job in regards to security, data collection, and real-time ingest. Their founders are from Arcsight (SIEM tool purchased by HP), so their service is very secure and extremely robust (real-time data ingest of up to 1TB per day). Their collectors encrypt data via SSL and compress 10x before sending to their service (something Splunk and other competitors DO NOT offer).

  • Julian Cohen

    Hey, this is Julian from Logentries. If you are looking for an easy-to-use service for centralizing, managing and analyzing your log data, check out our free account at http://logentries.com. We have built the service for the cloud so that you can get to the important data you need, in seconds, at a very cost-effective price. Let us know your feedback or technical questions at support@logentries.com

  • Kattant

    Great article with useful pros and cons. I think that Open source tools have a major disadvantage: they become ineffective above several GB/day log data input, because they have general regex logic; for every query they go through the log data again and again. Commercial products generally solve this by relying on a massive infrastructure – e.g. Splunk is able to work in a distributed way, or in case of Arcsight or QRadar parsed logs with metadata can be put in an expensive high performance database plus you can use high-end HP/IBM servers for data processing. But for your big money you can get some extra features and nice GUI too. (Cloud is not so effective if you need to transfer log data, or the data is sensitive) I am writing because there is a different approach:
    http://www.logdrill.com : you can find a free software here if you register. They created a domain specific easy to use descriptive language for effective parsing, which is capable of around 130000 EPS on one CPU depending on the input and the rule complexity. LogDrill developed an in-memory OLAP framework, which gives instant results on a drag&drop GUI, and one click drilldown to original logs. A few TB log analysis can run on a laptop.

  • Praveen D Kumar

    Hey Tal, have you heard/tried http://www.alienvault.com/ please provide your inputs on it.

  • Jim Sherman

    Thanks Tal, great list. One that you missed and I really like is Stackify (www.stackify.com). We use it both for the error and logs which are integrated and give you a lot more info per exception, but they also integrated the monitoring piece so you can monitor the app, the servers (VMs and physical), DB etc but also frequency of occurrence of an errors and if one that I’ve resolved came back etc.

  • http://mrjarichard.info Jesse Andrew

    There’s also Logentries (http://logentries.com), another great SaaS log management platform.

  • Sébastien Lorber

    Afaik Kafka is a log management system but not exactly like others. Kafka’s primary purpose is to collect applicative logs, ie applicative events, to be processed in a CQRS/Lambda architecture. This is not really like logging string statements in a text file. But it still can be used that way anyway (Loggly uses Kafka)

    • Jens Rantil

      LinkedIn also uses Kafka. Based on their initial posts, it sounds as if they are using it for their log infrastructure specifically, too.

  • Scott Wilkerson

    And an inexpensive commercial offering from and industry leader can now be added to the list Nagios Log Server – http://www.nagios.com/products/nagios-log-server