2018-08-27

new packages available at obs

Currently we are working on new rsyslog packages using the SUSE OBS tool. While this is still experimental, we have now published the first batch of packages. You can find them here.

We have decided to first focus on Debian, since there are no other rsyslog official packages for this distribution. However, these are still in the testing phase and we can not guarantee everything is working perfectly. The available Debian packages are similar to the currently existing ones for Ubuntu and are available in roughly the same scope of modules.

In obs, there are also some packages for other distributions, such as CentOS. These are also experimental and considered a lower priority than Debian, so they will probably have some problems.

If you try out these new packages, we would be glad to hear your feedback. It would also be great if you tell us which packages we should focus on next. Please do this by commenting on this github issue.

2018-03-20

loadable rainerscript functions

As I said in my last post, I was working on making rainerscript functions into loadable modules.

This had multiple reasons. For one, adding new functions may add more and more dependencies, which in the end takes up a lot of memory on your computer, even if you don't plan on using any of the functions. A good example of this are hash-functions. If they were implemented to always be available, it would bloat rsyslog dependencies. However, not many users actually require hash functions, so the loss of memory would not bring any advantage.

This also led to functions (http_request being an example) not being available on many distributions, since it would require too many built-in libraries. Because of this, distributions did not provide the function, because of the dependencies (in case of http_request the curl library), causing the functions to not be usable. With the new loadable interface, this will no longer be a problem, since dedicated packages can be created. So users can decide what functions to load and can then add the needed libraries (if not already present) to their installation.

Another reason is consistency across the rsyslog project. Almost anything with the exception of script functions is loadable. Before now, making them loadable would have been unnecessary,considering that the functions are so small that loading them would take longer and make the program take up more space than just making them built in. Now however, with good options to load bigger and more complicated functions without influencing smaller systems, basically everything is loadable, as was the goal since the initial design in 2007.

However, functions can still be built in, just like omfile and other built in output modules. This is the case for all functions that existed in 8.33.1 with the exception of http_request (now available via module fmhttp). This has multiple reasons:
  1. they have a very small memory footprint 
  2. changing these functions into modules would increase the footprint, since the overhead for modules would be necessary
  3. it would break existing configurations (else missing load statements in configuration)
Reason 3. does not apply to http_request, since it has only recently been implemented (8.32.0) and hasn't been picked up by any distros at all. So it is unlikely that users will be affected.

Future blogpost will describe the implementation and usage of loadable function modules.

2018-03-05

Making rainerscript functions loadable

Right now i am working on making rsyslog script functions loadable instead of statically built in at compile time. Currently, them being built in is not a problem as they do not have many dependenies. But as more functions get added, the bulk of dependencies required for script functions grows.

One recent example is the newly added function HTTP_request(), which requires libcurl to work. This will be a problem in the future, since there are plans to expand script functions. For example, if cryptographic functions were to be implemented, they would create many new dependencies and thereby inflate rsyslog.

To counteract this, we have decided to enable the script functions to be loaded when needed instead of being built in when compiled. This process will happen in partial steps.

2017-11-12

anonymizing IPv6 with embedded IPv4

Apart from the normal format for IPv6, it is also possible to format an IPv6 with an embedded IPv4 instead of the last two blocks. This is described in the RFC 4291 section 2.2.3, for example:
::FFFF:129.144.52.38
The format is used for systems working with both IPv6 and IPv4 addresses. However, in the RFC it is not made clear if this format can also be used to display normal IPv6 addresses or not.

It is important that anonymizers recignize this format as well, since privacy regulations also require it to be anonymized. Therefore, a solution should be found that can do that.

2017-10-12

SLFA release

As I mentioned in my recent post about the mmanon rewrite, I have been working on an offline anonymization tool. It is called SLFA (short for simple log file anonymizer) and I have now finished the first usable version.

SLFA is a java application that allows you to anonymize files from the command line. Currently, it can anonymize IPv4 addresses and IPv6 addresses (similar to the mmanon module of rsyslog) and can be configured to work with regular expressions.

It will be released shortly and  would be glad  about feedback once it has been released.

2017-09-02

Mmanon rewrite finished for the time being

I have recently finished the rewrite of the mmanon module. Now I have also finished implementing support for IPv6. This includes the same parameters as IPv4, but the mode parameter is now called anonmode, to make a later implementation of different output format modes for ipv6 easier. This might be in the future, but it would be possible to add another parameter. It could be named outputmode and would set the format of IPv6 output format. One thing that could be configured would be if abbreviations (::) should be used.

Also, IPv6 does not support anonymization in simple mode because of the different formats possible. It is now also possible to only anonymize IPv4 or IPv6 by using the new "enable" parameters. These have been added because mmanon now does not only work for IPv4.

Right now,  I consider the rewrite and the implementation of IPv6 finished until there is more feedback from users. It would also be great if people could test the changes once they are merged and post their feedback.

I will now work on an offline tool for log anonymization.

2017-08-25

IPv6 anonymization portability problems

IPv6-addresses are represented by 128 bits. This makes it possible to provide far more addresses than IPv4.

However, this also causes problems when working with IPv6. In this case, I am currently working on an IPv6-anonymiation function for rsyslog.

There are some systems that support an unsigned 128 bit integer when using GCC or clang compilers. However, many systems do not support this datatype.

Since rsyslog tries to cater to as many systems as possible, the implementation has to work on all platforms. As such, we have to use two unsigned 64 bit integers instead of one with 128 bits.

The main problem this causes is that this implementation is on software-level as opposed to he hardware-level a 128 bit integer is implemented on. This brings up a conflict of portability vs. speed, since an implementation on software-level is slower than one on hardware-level.

I have decided to make the first implementation as portable as possible, but might also later on try to speed the anonymization up. This might be possible by checking whether the 128 bit integer is supported by the system and if it is using it instead of two 64 bit integers.