Centralize our servers' logs
Originally created by @intrigeri on #11880 (Redmine)
Right now we have volatile Journal + some persistent log files managed by rsyslog and individual applications. This sometimes makes it painful to debug problems since one has to cross-match info from various sources. It would be nicer if all our logs landed in a single place.
An initial idea to start brainstorming about it would be:
- on each of our systems, send all logs to journald
- configure these journalds to have volatile storage only (that’s the default and what we currently do) and to forward them to a single journald instance running in a central place
- in the central logging location, either have journald store logs in a persistent manner, or forward them to a fancy system like Graylog (that seems much easier to setup than an ELK stack)
Note: For S11, this fits in:
-
B.2 - Keep our infrastructure up-to-date and secure
: Having centralized logs makes it easier for Sysadmins to have visibility on the current state of systems and services, thus making it easier to conduct diagnoses and spot problems.