reaction v2 : Rust & NLnet

Cet article est aussi disponible en 🇫🇷 français.

For an introduction to reaction, see this previous article

I'm pleased to announce reaction's v2.0.0 release!

This new version brings few new features, but it opens the way for a bright future for reaction 🌞

TL;DR🔗

In this article:

🦀 Rust rewrite🔗

I began one year ago a complete Rust rewrite of reaction, with the following goals:

After a lot of twists, I finally have a fully satisfying reaction reimplementation. The first functional version was ready last October, but this rewrite has been an occasion to test different internal systems.

I notably spent the months of February and May testing several database systems, which were not satisfying. I ended up rewriting my own implementation. There is now (once again) an embedded mini-redis in reaction!

📊 Benchmark🔗

Because we all love data, here's the result of a benchmark.

This benchmark isn't realistic. It sends 4 streams of 1 million lines of logs each, as fast as possible, requiring reaction to process all of it. reaction will execute ~6000 actions (a sleep 0.1). This allows us to have an idea of how reaction is reacting to a huge stress.

Of course, those tests should be taken with a grain of salt, because they don't test real life.

Statistics were produced by systemd.

Go version, v1.4.1 :

Service runtime: 2min 10.024s
CPU time consumed: 6min 47.643s
Memory peak: 13.6G (swap: 6.6G)
IO bytes read: 18.5G written: 21.3G

Rust version, v2.0.0 :

Service runtime: 42.235s
CPU time consumed: 4min 27.828s
Memory peak: 4.3G (swap: 0B)
IO bytes written: 499.9M

Those tests were executed on a PC with brand new NVME SSD and 16G RAM. The disk IO differences would have a bigger impact on a slower disk. Total service runtime differences would probably be less important if reaction Go hadn't had to swap.

As a note, performance differences aren't just a result of the languages themselves, but stem most importantly from technical choices for the internal architecture and communication between components, from the database (which have the same general structure but implementation differences), etc.

As a piece of evidence, the first version of the rewrite, close to the goroutine and channels proposed by Go, and less adapted to Rust, had much worse performance than the Go version.

⚙️ Incompatibilities🔗

Behavior differences between the v1 and v2 are few.

But they can lead to side effects in some cases:

🧑‍🏫 Configuration evaluation🔗

Configuration loading is more strict in reaction v2. It includes detecting misnamed fields, which were previously ignored.

In case of an incorrect configuration, a human-friendly error will be shown.

A new command allows users to test the config upfront: reaction test-config.

📏 Regex engine🔗

Go and Rust regex libraries have some syntax differences on advanced features.

⛲ Streams🔗

Previously, reaction only read the stdout of launched commands.

Now, both stdout and stderr are read.

📂 Database🔗

Database formats between v1 and v2 are incompatible.

Stored data (matches, current actions) will be lost.

In reaction's state directory (probably /var/lib/reaction), the following files can be deleted:

The v2 uses a standalone file called reaction.db, in JSONL format (a JSON object per line)

🎉 New features🔗

The rewrite was the main goal, but some additions came up in this new version!

Configuration directory🔗

Main real addition for sysadmins, reaction now supports splitting configuration across multiple files. Convenient when configuration is generated with Infra as Code tools, or when you just like having multiple files.

This feature is documented in the wiki's FAQ

🐚 Command Line🔗

The CLI is the same, but more flexible:

🐚🐚 Shell completions & man pages🔗

Shell completions are now available for bash, zsh and fish. Man pages for the CLI, a bit more detailed than --help, are available too.

📦 Packaging🔗

reaction binaries are now available for 2 architectures: amd64 (x86-64) and arm64.

For each architecture, a .tar archive and a Debian package are available. Both of them contain:

Additionnally, the .tar contains a Makefile to install the other files.

🔮 Upcoming features🔗

Missing features from issues🔗

Now that the foundations are here, I'll finally be able to add new features, including the most requested by reaction users:

Documentation🔗

At the same time, I want to improve documentation, by writing:

Plugin system🔗

Once this is done, I'll work on a second big step: adding a plugin system.

This system will allow anyone to extend reaction, without modifying the core code.

It's already possible to do what you want with reaction, by using custom scripts in streams' and actions' commands. But it can be inefficient or hacky for complicated things, or to bridge reaction to other systems.

For example, if you want to write IPs in a PostgreSQL database, you can write something along this:

{
  insert_psql: {
    cmd: ['psql',  '-h', 'HOST', '-U', 'user', '-c', 'INSERT INTO ips (ip) VALUES(<ip>)'],
  }
}

It works, but it isn't optimized. This example will:

A PostgreSQL plugin could allow for keeping a session open, and to just insert the line.

Fictional example:

{
  insert_psql: {
    plugin: 'psql',
    options: {
      host: 'HOST',
      user: 'user',
      query: 'INSERT INTO ips (ip) VALUES(%1)',
      params: {
        '%1': '<ip>'
      }
    }
  }
}

I don't plan to write database plugins! This will be left to others but should be easy.

JSON🔗

A lot of programs can log structured data, whether it's in JSON, or by using journald format (which journalctl can then print in JSON).

It's possible to write regexes to process JSON, but it's tedious and error-prone.

A JSON plugin would permit to replace a reaction filter's regex by more advanced methods.

Clustering🔗

reaction works alone for now.

But it would be great if it could work in a cluster of multiple reaction instances, on different machines. They could exchange messages, like IPs to ban.

A clustering plugin will enable this.

I guess this will be the most work intensive plugin for me ^^

HTTP soft ban🔗

When responding to attacks, for now reaction completely bans the IP at the firewall level. It's effective, but it can be a source of misunderstanding when there are false positives and a legit user finds themself banned.

To fix this, I want to write a plugin that spawns an HTTP middleware server, which places itself between the reverse proxy and the application server:

I've wanted to do this for some time, and some people were skeptical on this implementation choice. Since then, the same choice has been made for Anubis, a bot protection software which can be used alongside reaction!

Firewall🔗

iptables and nftables commands aren't efficient when a lot of them are launched at the same time. This can cause an additional pressure on a server when it's attacked.

A plugin that'd start one command, able to stream firewall actions would reduce this load a lot.

💰 NLnet🔗

I applied for an NLnet grant in June 2024, to be able to give more time to reaction. I received an agreement in January!

All upcoming features described in this article will be financed by the foundation. I'll be able to give a lot to reaction, while still managing to pay my rent 😄

That's great news 🎉

As a treat, the NLnet fund includes a security audit, which is really important for such software.

👉 The NLnet foundation

👉 Reaction's project Page @ NLnet

🤝 How to contribute🔗

Use reaction 😄

If you have a problem, or ideas, several spaces are available to talk about it:

And your configurations are gladly welcomed on the wiki! Don't hesitate to create a merge request (or even to put your code in an issue if you're lazy) on the Git repo.

🫶 Thanks🔗

Thanks to @bcareil who wrote the configuration directory feature, and corrected other problems in the Rust implementation!

Thanks to M for their help and advice on the advanced features of Rust (it's complicated ^^).

Thanks to Bertille for her listening and her advice on technical choices and on project management.

Thanks to the Rustaceans of #reaction-dev-fr:club1.fr for their views on the database choice.

Thanks to Luc for help on Debian packaging.

Thanks to all people contributing ideas and reporting problems in the issues.