Benchmark

You can edit this page on GitHub

Logging should have no significant impact on performance. This is why multiple benchmarks are part of the tinylog project. These are helpful for comparing different logging frameworks. Additionally, benchmarks are used during development for testing performance optimizations and detecting performance bugs at an early stage.

The benchmarks are based on JMH and hosted on GitHub. The first benchmark measures how many log entries per second can be output to a log file. Usually, there are also trace and debug log statements that should be ignored in production. How many log statements can be discarded per second is measured in the second benchmark.

To ensure statistically reliable data, each benchmark for each logging framework was executed sequentially in 10 forks, with 10 warm-up iterations and 10 relevant iterations. In the benchmark diagrams, you see the average value of the total 100 relevant iterations (10 forks × 10 iterations). The test machine was an Intel Core i5-1145G7 with 32 GB RAM and a 512 GB NVMe SSD from Samsung (model PM991a). Java Runtime Environment 11.0.6 was used to execute the benchmarks.

Writing Log Entries to a File

In these benchmarks, all logging frameworks have to write info log entries to a log file. Firstly, the log entries are written synchronously and unbuffered (blue bars). This supports every logging framework and is the default for all of them. For preventing an application from being blocked by slow I/O operations, all logging frameworks (except java.util.logging) can also write log entries asynchronously and buffered. The recommended configuration for asynchronous output for each logging framework has been benchmarked in the second step (yellow bars).

The output of caller information such as class and method name can have a big impact on performance. Therefore, there are three different benchmarks: output of class and method name, output of class or category, and no output of any caller information.

Class and Method

In this benchmark, all logging frameworks use {date:yyyy-MM-dd HH:mm:ss} - {thread} - {class}.{method}() - {level}: {message} or an equivalent as format pattern for outputting log entries.

FrameworkProcessed Log Entries per Second
tinylog 2.5.0
147,918
with writing thread
378,362
tinylog 1.3.6
148,859
with writing thread
344,066
Log4j 2.17.1
90,868
with async logger
110,718
Log4j 1.2.17
72,955
with async appender
74,379
Logback 1.2.7
68,167
with async appender
72,831
java.util.logging
84,386

Class or Category only

In this benchmark, all logging frameworks use {date:yyyy-MM-dd HH:mm:ss} - {thread} - {class} - {level}: {message} or an equivalent as format pattern for outputting log entries. Only tinylog 1 and 2 use the caller class. All other logging frameworks use the logger category instead, which is according to the documentation of Logback and Apache Log4j significantly faster and ensures a fair benchmark.

FrameworkProcessed Log Entries per Second
tinylog 2.5.0
194,707
with writing thread
704,309
tinylog 1.3.6
158,089
with writing thread
379,281
Log4j 2.17.1
224,501
with async logger
418,652
Log4j 1.2.17
236,161
with async appender
984,089
Logback 1.2.7
254,570
with async appender
1,308,555
java.util.logging
197,616

No Caller Information

In this benchmark, all logging frameworks use {date:yyyy-MM-dd HH:mm:ss} - {thread} - {level}: {message} or an equivalent as format pattern for outputting log entries. No caller information is output.

FrameworkProcessed Log Entries per Second
tinylog 2.5.0
262,614
with writing thread
889,343
tinylog 1.3.6
265,076
with writing thread
1,474,315
Log4j 2.17.1
226,551
with async logger
481,154
Log4j 1.2.17
245,565
with async appender
1,137,043
Logback 1.2.7
262,495
with async appender
1,722,066
java.util.logging
200,827

Discarding Log Entries

Debug and trace log entries are helpful for developers in their development environment, but are not usually output in the production environment. This benchmark measures how many log entries can be discarded per second by the logging frameworks.

FrameworkProcessed Log Entries per Second
tinylog 2.5.0
2,231,935,582
with writing thread
2,122,058,548
tinylog 1.3.6
855,800,963
with writing thread
805,882,427
Log4j 2.17.1
623,411,151
with async logger
628,710,595
Log4j 1.2.17
692,210,955
with async appender
692,155,371
Logback 1.2.7
603,475,131
with async appender
608,650,058
java.util.logging
1,079,547,928

For comparison, here is a no-op JMH benchmark that runs an empty method that does simply nothing:

No-Op BenchmarkInvocations per Second
Empty Method
2,235,396,971

Conclusion

All logging frameworks but java.util.logging can benefit a lot from using buffered and asynchronous output. It is practically always useful to enable it to negate the performance impact of slow I/O operations, regardless of which logging framework is used.

tinylog is multiple times faster than all other logging frameworks when outputting log entries along with caller information such as class and method name. With tinylog, caller information – even such as the method name – can be output with good conscience and without noticeable impact on the performance in real projects.

When outputting the class or category name as the only caller information, tinylog’s output performance is between Apache Log4j and Logback. Surprisingly, tinylog 1 and Logback seem to be the fastest logging frameworks when outputting no caller information at all. This is because tinylog 1 and Logback perform the rendering of log entries synchronously in the caller thread and perform only the slow I/O operations asynchronously in the writing thread and async appender respectively. In contrast, tinylog 2 also performs the rendering of log entries asynchronously in the writing thread, which on the one hand reduces the benchmark performance, but on the other hand leads to very low latency of issuing log entries in real projects, since the caller thread is blocked for significantly shorter time.

tinylog 2 is very good at discarding log entries with disabled severity levels. Calling a logging method for a disabled severity level is effectively a no-op. This is because tinylog 2 loads the visibility of the severity levels while class loading, and stores them as final boolean values. Each logging method firstly checks against this boolean value whether a severity level is enabled or disabled. This allows the JVM to identify which logging methods will never output anything at runtime and eliminate these calls completely. Hence, the performance of tinylog 2 for disabled severity levels is identical to an invocation of an empty no-op method that does nothing.