reaction v2 : Rust & NLnet
Cet article est aussi disponible en ๐ซ๐ท franรงais.
For an introduction to reaction, see this previous article
I'm pleased to announce reaction's v2.0.0 release!
This new version brings few new features, but it opens the way for a bright future for reaction ๐
TL;DR๐
In this article:
- Rust rewrite
- Benchmark
- Incompatibilities from v1 to v2
- New features
- Upcoming features
- NLnet support
๐ฆ Rust rewrite๐
I began one year ago a complete Rust rewrite of reaction, with the following goals:
- Better performance and robustness
- A language with more hype, to attract contributors and enjoy a better library ecosystem
- A language better adapted to the creation of a fast plugin system (see below)
- Writing tests for most of the codebase.
After a lot of twists, I finally have a fully satisfying reaction reimplementation. The first functional version was ready last October, but this rewrite has been an occasion to test different internal systems.
I notably spent the months of February and May testing several database systems, which were not satisfying. I ended up rewriting my own implementation. There is now (once again) an embedded mini-redis in reaction!
๐ Benchmark๐
Because we all love data, here's the result of a benchmark.
This benchmark isn't realistic. It sends 4 streams of 1 million lines of logs each, as fast as possible, requiring reaction to process all of it. reaction will execute ~6000 actions (a
sleep 0.1
). This allows us to have an idea of how reaction is reacting to a huge stress.Of course, those tests should be taken with a grain of salt, because they don't test real life.
Statistics were produced by systemd.
Go version, v1.4.1
:
Service runtime: 2min 10.024s
CPU time consumed: 6min 47.643s
Memory peak: 13.6G (swap: 6.6G)
IO bytes read: 18.5G written: 21.3G
Rust version, v2.0.0
:
Service runtime: 42.235s
CPU time consumed: 4min 27.828s
Memory peak: 4.3G (swap: 0B)
IO bytes written: 499.9M
Those tests were executed on a PC with brand new NVME SSD and 16G RAM. The disk IO differences would have a bigger impact on a slower disk. Total service runtime differences would probably be less important if reaction Go hadn't had to swap.
As a note, performance differences aren't just a result of the languages themselves, but stem most importantly from technical choices for the internal architecture and communication between components, from the database (which have the same general structure but implementation differences), etc.
As a piece of evidence, the first version of the rewrite, close to the goroutine and channels proposed by Go, and less adapted to Rust, had much worse performance than the Go version.
โ๏ธ Incompatibilities๐
Behavior differences between the v1 and v2 are few.
But they can lead to side effects in some cases:
๐งโ๐ซ Configuration evaluation๐
Configuration loading is more strict in reaction v2. It includes detecting misnamed fields, which were previously ignored.
In case of an incorrect configuration, a human-friendly error will be shown.
A new command allows users to test the config upfront: reaction test-config
.
๐ Regex engine๐
Go and Rust regex libraries have some syntax differences on advanced features.
โฒ Streams๐
Previously, reaction only read the stdout of launched commands.
Now, both stdout and stderr are read.
๐ Database๐
Database formats between v1 and v2 are incompatible.
Stored data (matches, current actions) will be lost.
In reaction's state directory (probably /var/lib/reaction
), the following files can be deleted:
reaction-matches.db
,reaction-flushes.db
(v1)data
directory (v2.0.0-rc2)
The v2 uses a standalone file called reaction.db
, in JSONL format (a JSON object per line)
๐ New features๐
The rewrite was the main goal, but some additions came up in this new version!
Configuration directory๐
Main real addition for sysadmins, reaction now supports splitting configuration across multiple files. Convenient when configuration is generated with Infra as Code tools, or when you just like having multiple files.
This feature is documented in the wiki's FAQ
๐ Command Line๐
The CLI is the same, but more flexible:
- Argument order is less strict,
- Each subcommand has its own
--help
.
๐๐ Shell completions & man pages๐
Shell completions are now available for bash, zsh and fish.
Man pages for the CLI, a bit more detailed than --help
, are available too.
๐ฆ Packaging๐
reaction binaries are now available for 2 architectures: amd64 (x86-64) and arm64.
For each architecture, a .tar
archive and a Debian package are available.
Both of them contain:
reaction
,ip46tables
andnft46
binaries- man pages and shell completions
- a systemd service file.
Additionnally, the .tar contains a Makefile to install the other files.
๐ฎ Upcoming features๐
Missing features from issues๐
Now that the foundations are here, I'll finally be able to add new features, including the most requested by reaction users:
- Better IP address support (#79)
- Configurable duplicates management (#68)
- "One shot" actions (including for alerting) (#92)
- Ban command (#93)
- More complete support for the NixOS distribution (upstreaming a module)
Documentation๐
At the same time, I want to improve documentation, by writing:
- a complete reference of configuration options
- a step-by-step guide for writing one's own configuration
- more configuration examples for common services
Plugin system๐
Once this is done, I'll work on a second big step: adding a plugin system.
This system will allow anyone to extend reaction, without modifying the core code.
It's already possible to do what you want with reaction, by using custom scripts in streams' and actions' commands. But it can be inefficient or hacky for complicated things, or to bridge reaction to other systems.
For example, if you want to write IPs in a PostgreSQL database, you can write something along this:
{
insert_psql: {
cmd: ['psql', '-h', 'HOST', '-U', 'user', '-c', 'INSERT INTO ips (ip) VALUES(<ip>)'],
}
}
It works, but it isn't optimized. This example will:
- Execute a PostgreSQL client, which:
- Opens a session with the server
- Authenticates
- Inserts a line (โ ๏ธ sensitive to SQL injections in this minimal example: see the wiki for a secured version)
- Closes the session.
A PostgreSQL plugin could allow for keeping a session open, and to just insert the line.
Fictional example:
{
insert_psql: {
plugin: 'psql',
options: {
host: 'HOST',
user: 'user',
query: 'INSERT INTO ips (ip) VALUES(%1)',
params: {
'%1': '<ip>'
}
}
}
}
I don't plan to write database plugins! This will be left to others but should be easy.
JSON๐
A lot of programs can log structured data, whether it's in JSON, or by using journald format (which journalctl
can then print in JSON).
It's possible to write regexes to process JSON, but it's tedious and error-prone.
A JSON plugin would permit to replace a reaction filter's regex by more advanced methods.
Clustering๐
reaction works alone for now.
But it would be great if it could work in a cluster of multiple reaction instances, on different machines. They could exchange messages, like IPs to ban.
A clustering plugin will enable this.
I guess this will be the most work intensive plugin for me ^^
HTTP soft ban๐
When responding to attacks, for now reaction completely bans the IP at the firewall level. It's effective, but it can be a source of misunderstanding when there are false positives and a legit user finds themself banned.
To fix this, I want to write a plugin that spawns an HTTP middleware server, which places itself between the reverse proxy and the application server:
- It would serve content transparently for legit IPs, from the reverse proxy to the application server.
- It would show a user-friendly ban message to banned IPs, effectively blocking traffic from the reverse proxy to the application server.
I've wanted to do this for some time, and some people were skeptical on this implementation choice. Since then, the same choice has been made for Anubis, a bot protection software which can be used alongside reaction!
Firewall๐
iptables
and nftables
commands aren't efficient when a lot of them are launched at the same time.
This can cause an additional pressure on a server when it's attacked.
A plugin that'd start one command, able to stream firewall actions would reduce this load a lot.
๐ฐ NLnet๐
I applied for an NLnet grant in June 2024, to be able to give more time to reaction. I received an agreement in January!
All upcoming features described in this article will be financed by the foundation. I'll be able to give a lot to reaction, while still managing to pay my rent ๐
That's great news ๐
As a treat, the NLnet fund includes a security audit, which is really important for such software.
๐ The NLnet foundation
๐ Reaction's project Page @ NLnet
๐ค How to contribute๐
Use reaction ๐
If you have a problem, or ideas, several spaces are available to talk about it:
-
Matrix development rooms :
-
Matrix support rooms :
-
Ping @ppom@mamot.fr on the Fediverse
And your configurations are gladly welcomed on the wiki! Don't hesitate to create a merge request (or even to put your code in an issue if you're lazy) on the Git repo.
๐ซถ Thanks๐
Thanks to @bcareil who wrote the configuration directory feature, and corrected other problems in the Rust implementation!
Thanks to M for their help and advice on the advanced features of Rust (it's complicated ^^).
Thanks to Bertille for her listening and her advice on technical choices and on project management.
Thanks to the Rustaceans of #reaction-dev-fr:club1.fr for their views on the database choice.
Thanks to Luc for help on Debian packaging.
Thanks to all people contributing ideas and reporting problems in the issues.