mongodb | Continuous Learning

Graylog2 – это open source решение для централизованного сбора, хранения и анализа логов. Cостоит из:
– Сервер написанный на Java, принимающий syslog messages через TCP, UDP или AMQP
– Логи хранятся в ElasticSearch
– MongoBD для статистики и графиков
– Кавайный Web interface

Graylog2 принимает логи через TCP/UDP как и обычный syslog и GELF через UDP. Почитать чем хорош GELF. Можно отправлять логи используя оба формата через AMQP

Сервер на котором будем устанавливать Graylog2

# uname -a
Linux debian 2.6.32-5-amd64 #1 SMP UTC 2012 x86_64 GNU/Linux
# lsb_release -a
Distributor ID:	Debian
Description:	Debian GNU/Linux 6.0.4 (squeeze)
Release:	6.0.4
Codename:	squeeze

Обновим репозитории

# vi /etc/apt/sources.list
deb http://mirror.yandex.ru/debian/ squeeze main contrib non-free
deb-src http://mirror.yandex.ru/debian/ squeeze main contrib non-free
deb http://security.debian.org/ squeeze/updates main contrib non-free
deb http://mirror.yandex.ru/debian/ squeeze-updates main contrib non-free
deb-src http://mirror.yandex.ru/debian/ squeeze-updates main contrib non-free
# apt-get update

Устанавливаем Java

# aptitude install openjdk-6-jre
# aptitude install openjdk-6-jre-headless

Устанавливаем Mongodb

# aptitude install mongodb-server mongodb

Создаем пользователя:

# mongo
MongoDB shell version: 1.4.4
url: test
connecting to: test
type "help" for help

> use admin
switched to db admin

> db.addUser('admin', 'Password')
{
        "user" : "admin",
        "readOnly" : false,
        "pwd" : "Password"
}

> db.auth('admin', 'Password')
1

> use graylog2
switched to db graylog2

> db.addUser('grayloguser', 'Password')
{
        "user" : "grayloguser",
        "readOnly" : false,
        "pwd" : "Password"
}

> db.auth('grayloguser', 'Password')
1

> exit
bye

Делаем Mongodb более безопастной

/etc/mongodb.conf
auth = true

Перестартуем монгу:

# service mongodb restart

Проверяем что все работает:

# netstat -nlp | grep mongo
tcp        0      0 127.0.0.1:28017         0.0.0.0:*               LISTEN      4408/mongod
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      4408/mongod

Устанавливаем Elasticsearch Скачиваем самую последнюю стабильную версию

# wget https://github.com/downloads/elasticsearch/elasticsearch/elasticsearch-0.19.1.deb
# dpkg -i elasticsearch-0.19.1.deb

Запускаем ElasticSearch

# service elasticsearch start
Starting ElasticSearch Server:.

# sysv-rc-conf –list | grep elasticsearc
elasticsearc 0:off 1:off 2:on 3:on 4:on 5:on 6:off

Скачиваем последнюю версию Graylog2-server 0.9.6 RC1

# wget https://github.com/downloads/Graylog2/graylog2-server/graylog2-server-0.9.6p1-RC1.tar.gz
# tar xvf graylog2-server-0.9.6p1-RC1.tar.gz
# mv graylog2-server-0.9.6p1-RC1 /usr/local/graylog2-server

Копируем

# cd /usr/local/graylog2-server
# cp graylog2.conf.example /etc/graylog2.conf
# mcedit /etc/graylog2.conf

Редактируем /etc/graylog2.conf подставляем пользователя и его пароль mongodb, которые мы создавали ранее.

# grep -v -E "#|^$" graylog2.conf
syslog_listen_port = 514
syslog_protocol = udp
elasticsearch_url = http://localhost:9200/
elasticsearch_index_name = graylog2
force_syslog_rdns = false
allow_override_syslog_date = true
mongodb_useauth = true
mongodb_user = grayloguser
mongodb_password = Password
mongodb_host = localhost
mongodb_database = graylog2
mongodb_port = 27017
mq_batch_size = 4000
mq_poll_freq = 1
mq_max_size = 0
mongodb_max_connections = 100
mongodb_threads_allowed_to_block_multiplier = 5
use_gelf = true
gelf_listen_address = 0.0.0.0
gelf_listen_port = 12201
amqp_enabled = false
amqp_subscribed_queues = somequeue1:gelf,somequeue2:gelf,somequeue3:syslog
amqp_host = localhost
amqp_port = 5672
amqp_username = guest
amqp_password = guest
amqp_virtualhost = /
forwarder_loggly_timeout = 3

Проверим что все у нас запускаеться правильно и без ошибок:

# cd /usr/local/graylog2-server
# java -jar graylog2-server.jar --debug

Создаем Debian init скрипт для запуска Graylog2-server

# vi /etc/init.d/graylog2

#!/bin/bash
### BEGIN INIT INFO
# Provides:          graylog2
# Required-Start:    $all
# Required-Stop:    $all
# Default-Start:    2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Starts graylog2
# Description:      Starts graylog2 using start-stop-daemon
### END INIT INFO

NAME=graylog2
GL_HOME=/usr/local/graylog2-server
GL_PID=/var/run
CMD=$1

start() {
        echo "Starting $NAME ..."
        java -jar $GL_HOME/graylog2-server.jar &
}

stop() {
        PID=`cat $GL_PID/$NAME.pid`
        echo "Stopping $NAME ($PID) ..."
        kill $PID
}

restart() {
        echo "Restarting graylog2-server ..."
        stop
        start
}

case "$CMD" in
        start)
                start
                ;;
        stop)
                stop
                ;;
        restart)
                restart
                ;;
        *)
                echo "Usage $0 {start|stop|restart}"
esac

Добавляем Graylog2-server в автозагрузку

# chmod +x graylog2
# update-rc.d graylog2 defaults
# sysv-rc-conf --list | grep graylog2
graylog2     0:off	1:off	2:on	3:on	4:on	5:on	6:off

Проверяем

# ps waux | grep graylog2
root     25967  0.4  1.0 1419064 39916 pts/0 java -jar /usr/local/graylog2-server/graylog2-server.jar

Устанавливаем RVM

# bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)
Downloading RVM from wayneeseguin branch stable
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  919k  100  919k    0     0   312k      0  0:00:02  0:00:02 --:--:--  567k

Installing RVM to /usr/local/rvm/
    Creating RVM system user group 'rvm'

# RVM:  Shell scripts enabling management of multiple ruby environments.
# RTFM: https://rvm.beginrescueend.com/
# HELP: http://webchat.freenode.net/?channels=rvm (#rvm on irc.freenode.net)
# Screencast: http://screencasts.org/episodes/how-to-use-rvm

# In case of any issues read output of 'rvm requirements' and/or 'rvm notes'

Installation of RVM in /usr/local/rvm/ is almost complete:

  * First you need to add all users that will be using rvm to 'rvm' group,
    anyone using rvm will be operating with `umask g+w`.

  * To start using RVM you need to run `source /etc/profile.d/rvm.sh`
    in all your open shell windows, in rare cases you need to reopen all shell windows.

  * Optionally you can run `rvm tools rvm-env ruby bash` which will generate 
    shebang wrappers for easier selecting ruby in scripts.

# root,
#
#   Thank you for using RVM!
#   I sincerely hope that RVM helps to make your life easier and more enjoyable!!!
#
# ~Wayne

Перезапускаем и устанавливаем Ruby 1.9.3

# source /etc/profile.d/rvm.sh
# rvm install 1.9.3
# rvm alias create default 1.9.3
# rvm use 1.9.3

Скачиваем последнюю версию версию Graylog2-web-interface

# wget https://github.com/downloads/Graylog2/graylog2-web-interface/graylog2-web-interface-0.9.6p1-RC1.tar.gz
# tar xvf graylog2-web-interface-0.9.6p1-RC1.tar.gz
# mv graylog2-web-interface-0.9.6p1-RC1 /usr/local/graylog2-web

Устанавливаем gem bundle

# cd /usr/local/graylog2-web
# gem install bundler
Successfully installed bundler-1.0.22
1 gem installed
# bundle install
Your bundle is complete! Use `bundle show [gemname]` to see where a bundled gem is installed.

Выполняем конфигурацию Graylog2-web-interface

# cd /usr/local/graylog2-web/config
# mcedit mongoid.yml
production:
 host: localhost
 port: 27017
 username: grayloguser
 password: Password
 database: graylog2

Проверяем работу Graylog2-Web

# cd /usr/local/graylog2-web/
# script/rails server -e production

Устанавливаем Apache и Apache-passenger

# apt-get install apache2-dev apache2
# cd /usr/local/graylog2-web
# gem install passenger
# passenger-install-apache2-module

Редактируем файл /etc/apache2/apache2.conf добавим:
LoadModule passenger_module /usr/local/rvm/gems/ruby-1.9.3-p125/gems/passenger-3.0.11/ext/apache2/mod_passenger.so
PassengerRoot /usr/local/rvm/gems/ruby-1.9.3-p125/gems/passenger-3.0.11
PassengerRuby /usr/local/rvm/wrappers/ruby-1.9.3-p125/ruby

Создадим VirtualHost в /etc/apache2/conf.d

# vi /etc/apache2/conf.d/graylog2.conf

Убрать пробелы между < >

< VirtualHost debian.bezha.od.ua:80 >
ServerAdmin email@gmail.com
DocumentRoot /usr/local/graylog2-web/public
< Directory /usr/local/graylog2-web/public >
Allow from all
Options -MultiViews
< /Directory >
ErrorLog /var/log/apache2/error.log
LogLevel warn
CustomLog /var/log/apache2/access.log combined
< /VirtualHost >

Перезапускаем apache2

# service apache2 restart

Congratulation

Version Graylog2

Добавим первый хост в Graylog через rsyslog /etc/rsyslog.conf
Через UDP

*.* @127.0.0.1

Через TCP

*.* @@127.0.0.1

Перезапускаем Rsyslog

# service rsyslog restart

Continuous Learning

"Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live." ― John F. Woods

Tag Archives: mongodb

Debian 6.0 x64 Graylog2 v0.9.6 RC1