Logging Service | Continuous Learning

Graylog2 – это open source решение для централизованного сбора, хранения и анализа логов. Cостоит из:
– Сервер написанный на Java, принимающий syslog messages через TCP, UDP или AMQP
– Логи хранятся в ElasticSearch
– MongoBD для статистики и графиков
– Кавайный Web interface

Graylog2 принимает логи через TCP/UDP как и обычный syslog и GELF через UDP. Почитать чем хорош GELF. Можно отправлять логи используя оба формата через AMQP

Сервер на котором будем устанавливать Graylog

# uname -a
Linux centos.bezha.od.ua 2.6.32-220.el6.x86_64 #1 SMP x86_64 GNU/Linux
# cat /etc/issue
CentOS release 6.2 (Final)

Устанавливаем graylog2-server
Подключим репозиторий EPEL

# rpm -ivh http://download.fedoraproject.org/pub/epel/6/i386/epel-release-6-5.noarch.rpm

Устанавливаем Java и gcc и curl-devel zlib-devel

# yum -y install lsb gcc make java-1.6.0-openjdk curl-devel zlib-devel openssl-devel

Устанавливаем mongodb-server

# yum -y install mongodb mongodb-server

Создаем пользователя:

# mongo
MongoDB shell version: 1.8.2
connecting to: test

> use admin
switched to db admin

> db.addUser('admin', 'Password')
{
        "user" : "admin",
        "readOnly" : false,
        "pwd" : "Password"
}

> db.auth('admin', 'Password')
1

> use graylog2
switched to db graylog2

> db.addUser('grayloguser', 'Password')
{
        "user" : "grayloguser",
        "readOnly" : false,
        "pwd" : "Password"
}

> db.auth('grayloguser', 'Password')
1

> exit
bye

Делаем Mongodb более безопастной

/etc/mongodb.conf
auth = true

Перестартуем Mongodb

# service mongod restart

Добавляем в автозагрузку

# chkconfig mongod on
# chkconfig --list | grep mongod
mongod          0:off   1:off   2:on    3:on    4:on    5:on    6:off

Проверяем что все работает:

# netstat -nlp | grep mongod
tcp        0      0 0.0.0.0:27017   0.0.0.0:*    LISTEN      3326/mongod
tcp        0      0 0.0.0.0:28017   0.0.0.0:*    LISTEN      3326/mongod

Устанавливаем Elasticsearch Скачиваем самую последнюю стабильную версию

# curl -L -O https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.0.zip
# unzip elasticsearch-0.17.2.zip
# mv elasticsearch-0.17.2 /usr/local/elasticsearch
# rm -Rf elasticsearch*.zip

Устанавливаем Service wrapper

# cd /usr/local/elasticsearch
# curl -kL http://github.com/elasticsearch/elasticsearch-servicewrapper/tarball/master | tar -xz
# mv *servicewrapper*/service /usr/local/elasticsearch/bin/
# rm -Rf *servicewrapper*

Устанавливаем elasticsearch как сервис по этой документации

# /usr/local/elasticsearch/bin/service/elasticsearch install
Detected RHEL or Fedora:
Installing the ElasticSearch daemon..

Перестартуем Elasticsearch

# service elasticsearch restart
Stopping ElasticSearch...
ElasticSearch was not running.
Starting ElasticSearch...
Waiting for ElasticSearch......
running: PID:2122

Добавляем в автозагрузку

# chkconfig elasticsearch on
# chkconfig --list | grep elasticsearch
elasticsearch   0:off   1:off   2:on    3:on    4:on    5:on    6:off

Скачиваем последнюю стабильную версию Graylog2-server 0.9.6

# wget https://github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.6.tar.gz 
# tar xfvz graylog2-server-0.9.6.tar.gz
# mv graylog2-server-0.9.6 /usr/local/graylog2-server

Копируем

# cd /usr/local/graylog2-server
# cp graylog2.conf.example /etc/graylog2.conf
# mcedit /etc/graylog2.conf

Редактируем /etc/graylog2.conf подставляем пользователя и его пароль mongodb, которые мы создавали ранее.

# grep -v -E "#|^$" graylog2.conf
syslog_listen_port = 514
syslog_protocol = udp
elasticsearch_url = http://localhost:9200/
elasticsearch_index_name = graylog2
force_syslog_rdns = false

mongodb_useauth = true
mongodb_user = grayloguser
mongodb_password = Password
mongodb_host = localhost
mongodb_database = graylog2
mongodb_port = 27017

mq_batch_size = 4000
mq_poll_freq = 1
mq_max_size = 0
mongodb_max_connections = 100
mongodb_threads_allowed_to_block_multiplier = 5
use_gelf = true
gelf_listen_address = 0.0.0.0
gelf_listen_port = 12201
amqp_enabled = false
amqp_subscribed_queues = somequeue1:gelf,somequeue2:gelf,somequeue3:syslog
amqp_host = localhost
amqp_port = 5672
amqp_username = guest
amqp_password = guest
amqp_virtualhost = /
forwarder_loggly_timeout = 3

Создаем init скрипт для запуска Graylog2-server /etc/init.d/graylog2

#!/bin/bash
#
# graylog2-server - This script starts and stops the graylog2-server daemon
#
# chkconfig:   - 85 15
# description: graylog2 is a syslog implementation that stores logs in MongoDB

### BEGIN INIT INFO
# Provides: 
# Required-Start:      $network
# Required-Stop:       $network
# Default-Stop:	       0 1 2 6
# Short-Description:   Start/stop the graylog2-server daemon
# Description:	       A syslog implementation that stores logs in MongoDB
### END INIT INFO

# Source function library.
. /etc/init.d/functions

# Check networking
. /etc/sysconfig/network
[[ $NETWORKING == no ]] && exit 0

prog="graylog2-server"
jar="/usr/local/$prog/$prog.jar"
lockfile="/var/lock/$prog"
logfile="/var/log/graylog2.log"
pidfile="/var/run/$prog.pid"

JAVA_BIN="/usr/bin/java"

unset pid

gl2_run() {
  # Disable csh style history expansion for safety
  set +H
	daemon "( $JAVA_BIN -jar \"$jar\" >\"$logfile\" 2>&1 & echo \"\$!\" >\"$pidfile\" )"
	pid=$(<"$pidfile")
  sleep 1
	if kill -0 "$pid" >/dev/null 2>&1
	then
		return 0
	else
		return 1
	fi
}

gl2_status() {
	[[ -f $pidfile ]] && pid=$(<"$pidfile")
	[[ -z $pid ]] && return 1
	if kill -0 "$pid" >/dev/null 2>&1
	then
		return 0
	else
		return 1
	fi
}

status() {
	if gl2_status
	then
		echo $"$prog (pid $pid) is running..."
		return 0
	else
		if [[ -f $lockfile ]]
		then
			echo $"$prog dead but subsys locked"
			return 2
		fi
		echo $"$prog is stopped"
		return 3
	fi
}

start() {
  [[ -r $jar ]] || exit 5
	gl2_run >/dev/null 2>&1
	if [[ $? -eq 0 ]]
	then
		retval=0
		touch "$lockfile"
		success
	else
		retval=1
		failure
	fi

	echo $"Starting $prog: "
  return "$retval"
}

stop() {
	killproc -p "$pidfile"
	if [[ $? -eq 0 ]]
	then
		retval=0
		rm -f "$lockfile"
    success
	else
		retval=1
    failure
	fi

  echo $"Stopping $prog: "
  return "$retval"
}

restart() {
  stop
  start
}

reload() {
  restart
}

force_reload() {
  restart
}

case "$1" in
  start)
      gl2_status && exit 0
      $1
      ;;
  stop)
      gl2_status || exit 0
      $1
      ;;
  restart)
      $1
      ;;
  reload)
      gl2_status || exit 7
      $1
      ;;
  force-reload)
      force_reload
      ;;
  status)
      $1
      ;;
  condrestart|try-restart)
      gl2_status || exit 0
      restart
      ;;
  *)
   echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}"
   exit 2
esac
exit $?

Добавляем Graylog2-server в автозагрузку

# chmod +x /etc/init.d/graylog2
# chkconfig --add graylog2
# chkconfig graylog2 on
# chkconfig --list |grep graylog2
graylog2        0:off   1:off   2:on    3:on    4:on    5:on    6:off

Запускаем Graylog2-server

# service graylog2 start
Usage: /etc/init.d/graylog2 {start|stop|status|restart|condrestart|try-restart|reload|force-reload}

Проверяем

# ps waux | grep graylog2
root 2907  /usr/bin/java -jar /usr/local/graylog2-server/graylog2-server.jar

Устанавливаем Web interface

Устанавливаем RVM

# bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)
Downloading RVM from wayneeseguin branch stable
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  829k  100  829k    0     0   250k      0  0:00:03  0:00:03 --:--:--  438k

Installing RVM to /usr/local/rvm/
    Creating RVM system user group 'rvm'

# RVM:  Shell scripts enabling management of multiple ruby environments.
# RTFM: https://rvm.beginrescueend.com/
# HELP: http://webchat.freenode.net/?channels=rvm (#rvm on irc.freenode.net)
# Screencast: http://screencasts.org/episodes/how-to-use-rvm

# In case of any issues read output of 'rvm requirements' and/or 'rvm notes'

Installation of RVM in /usr/local/rvm/ is almost complete:

  * First you need add all users that will be using rvm to 'rvm' group,
    anyone using rvm will be operating with `umask g+w`.

  * To start using RVM you need to run `source /etc/profile.d/rvm.sh`
    in all your open shell windows, in rare cases you need to reopen all shell windows.

  * Optionally you can run `rvm tools rvm-env ruby bash` which will generate
    shebang wrappers for easier selecting ruby in scripts.

# root,
#
#   Thank you for using RVM!
#   I sincerely hope that RVM helps to make your life easier and more enjoyable!!!
#
# ~Wayne

После перезапуска терминала устанавливаем Ruby 1.9.2

# rvm install 1.9.2
# rvm alias create default 1.9.2

Посмотреть, какие версии доступны в данный момент в системе и какие из них используются по умолчанию

# rvm list

rvm rubies

=* ruby-1.9.2-p290 [ x86_64 ]
# => - current
# =* - current && default
#  * - default

Скачиваем последнюю версию версию Graylog2-web-interface

# wget https://github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.6.tar.gz
# tar xzvf graylog2-web-interface-0.9.6.tar.gz
# mv graylog2-web-interface-0.9.6/ /usr/local/graylog2-web-interface
# adduser -m graylog

Устанавливаем gem bundle

# cd /usr/local/graylog2-web-interface
# gem install bundler
Successfully installed bundler-1.0.22
1 gem installed
# bundle install
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.

Выполняем конфигурацию Graylog2-web-interface

# cd /usr/local/graylog2-web-interface/config
# mcedit mongoid.yml
production:
 host: localhost
 port: 27017
 username: grayloguser
 password: Password
 database: graylog2

Устанавливаем passenger-standalone

# gem install passenger
# gem install file-tail
# cd /urs/local/graylog2-web-interface
# mkdir tmp log
# chmod -R 777 tmp log

Создаем init скрипт для запуска Graylog2-web-interface Источник

#!/bin/bash
#
# graylog2-web-interface:   graylog2 web interface
#
# chkconfig: - 98 02
# description:  Starts graylog2-web-interface using passenger-standalone. \
#       Uses RVM to use switch to a specific ruby version.
#

# config
USER=graylog
APP_DIR=/usr/local/graylog2-web-interface
RVM_RUBY=1.9.2
ADDR=127.0.0.1
PORT=3000
ENVIRONMENT=production
LOG_FILE=/var/log/graylog2-web-interface.log

# --

CMD_START="cd $APP_DIR; rvm use $RVM_RUBY; passenger start -d \
                    -a $ADDR \
                    -p $PORT \
                    -e $ENVIRONMENT \
                    --user $USER"
CMD_STOP="cd $APP_DIR; rvm use $RVM_RUBY; passenger stop -p $PORT"

CMD_STATUS="cd $APP_DIR; rvm use $RVM_RUBY; passenger status -p $PORT"

. /lib/lsb/init-functions
case "$1" in
  start)
    echo "Starting graylog2-web-interface"
    su - $USER -c "$CMD_START"
    ;;
  stop)
    echo "Stopping graylog2-web-interface"
    su - $USER -c "$CMD_STOP"
    ;;
  status)
   su - $USER -c "$CMD_STATUS"
   ;;
  *)
    echo "Usage: $0 start|stop|status"
    exit 3
    ;;
esac

Устанавливаем graylog2-web-interface как сервис

# chmod +x /etc/init.d/graylog2-web
# chkconfig --add graylog2-web
# chkconfig graylog2-web on

Устанавливаем Apache

# yum -y install httpd
# chkconfig httpd on

Прописываем Virtual host /etc/httpd/conf.d/graylog2.conf (Убрать пробелы между <>)

< VirtualHost *:80 >
        ServerName graylog2.bezha.od.ua
        ProxyPreserveHost On
        ProxyPass        / http://127.0.0.1:3000/
        ProxyPassReverse / http://127.0.0.1:3000/
        CustomLog /var/log/httpd/log.mydomain.com-access_log common
< /VirtualHost >

Перезапускаем httpd

# service httpd configtest
Syntax OK
# service httpd restart
Stopping httpd:                                            [  OK  ]
Starting httpd:                                            [  OK  ]

Заходим в Web-Interface создаем первого пользователя

Настройки ротации логов:

Добавим первый хост в Graylog через rsyslog /etc/rsyslog.conf
Через UDP

*.* @127.0.0.1

Через TCP

*.* @@127.0.0.1

Перезапускаем Rsyslog

# service rsyslog restart

Поиск по сообщениям. Use case: Посмотреть с каких ip адресов и в какое время происходил заход на сервер пользователем root

Если кликнуть по сообщению появляется дополнительная информация

Можно создать stream, который будет фильтровать только определенные сообщения. Например stream который показывает сообщения Severity Critical

Нагрузка при реальном использовании на VPS:
10 minutes

CPU[|||||||                                                   7.2%]     Tasks: 184 total, 1 running
Mem[|||||||||||||||||||||||||||||||||||||||||||||||||||||691/996MB]     Load average: 0.18 0.17 0.27
Swp[||||                                                 97/2015MB]     Uptime: 36 days, 03:25:18

Graph

Continuous Learning

"Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live." ― John F. Woods

Tag Archives: Logging Service

CentOS 6.2 x64 Graylog2 v0.9.6