I have an rsyslog stream that accepts RFC-5424 messages from a wide range of applications and forwards them to a storage backend in RFC-5424 format. Since they originate from so many sources, I have limited control over what is sent. I need to limit the overall length of the log message to 64MB. Messages that exceed the 64MB limit mostly contain an "exception" property in the structured data that typically contains a Java stack trace, which may be quite long. Some messages also contain a "message" property in the structured data that is a duplicate and redundant to the log message's payload. Hence, to limit the overall message to 64MB, I need to delete the "messages" property and truncate the "exception" text in the structured data, if they exist. I won't know what other properties exist in the structured data, but, whatever they are, they should remain unchanged in the out-bound message. After doing that, I can be relatively confident that limiting the overall message to 64MB will truncate somewhere in the message payload, rather than in the structured data, which would produce an illegally formatted message that would be dropped. I have not found a good way to parse and manipulate structured data properties. Instead, I have tried to use regular expression matches and replacements, but it is occasionally causing separate messages to be merged and concatenated together. I can only assume that is caused by some multithreading synchronization problem in the rsyslog code. Is there a better way to accomplish what I need?
