使用logstash分析Apache日志

网站刚刚上线,就发现一些扫描信息,因此,有必要搭一个日志分析系统,对web访问日 志进行分析,或许能够发现一些有意思的东西。logstash是一个非常灵活的日志分析引擎, 使用强大的elasticsearch作为数据存储、索引的数据库。

118.26.224.17 - - [10/Jun/2012:18:26:46 +0000] "GET /w00tw00t.at.blackhats.romanian.anti-sec:) HTTP/1.1" 404 490 "-" "ZmEu"
118.26.224.17 - - [10/Jun/2012:18:26:47 +0000] "GET /phpMyAdmin/scripts/setup.php HTTP/1.1" 404 479 "-" "ZmEu"
118.26.224.17 - - [10/Jun/2012:18:26:47 +0000] "GET /phpmyadmin/scripts/setup.php HTTP/1.1" 404 479 "-" "ZmEu"
118.26.224.17 - - [10/Jun/2012:18:26:48 +0000] "GET /pma/scripts/setup.php HTTP/1.1" 404 475 "-" "ZmEu"
118.26.224.17 - - [10/Jun/2012:18:26:48 +0000] "GET /myadmin/scripts/setup.php HTTP/1.1" 404 478 "-" "ZmEu"
118.26.224.17 - - [10/Jun/2012:18:26:49 +0000] "GET /MyAdmin/scripts/setup.php HTTP/1.1" 404 478 "-" "ZmEu"

创建日志分析环境

#!/bin/bash
# 创建日志环境的目录
mkdir loganalysis
cd loganalysis
 
# 下载logstash,目前是1.1.0版本
mkdir bin
cd bin
wget http://semicomplete.com/files/logstash/logstash-1.1.0-monolithic.jar
 
# 创建启动脚本
cat << EOF >>run.sh
#!/bin/bash
java -jar $LOGSTASH_HOME/bin/logstash-1.1.0-monolithic.jar agent -f $LOGSTASH_HOME/conf/apache-parse.conf -- web --backend elasticsearch:///?local
EOF
 
# 环境变量设置
cd ..
cat << EOF >>env.sh
#!/bin/bash
export LOGSTASH_HOME=`pwd`
EOF
 
# 下载Apache日志解析配置文件
mkdir conf
cd conf
wget http://logstash.net/docs/1.1.0/tutorials/10-minute-walkthrough/apache-parse.conf

修改apache-parse.conf

apache-parse.conf默认不使用elasticsearch,所以需要加上 elasticsearch { embedded => true }

input {
  tcp { 
    type => "apache"
    port => 3333
  } 
}
 
filter {
  grok {
    type => "apache"
    # See the following URL for a complete list of named patterns
    # logstash/grok ships with by default:
    # https://github.com/logstash/logstash/tree/master/patterns
    #
    # The grok filter will use the below pattern and on successful match use
    # any captured values as new fields in the event.
    pattern => "%{COMBINEDAPACHELOG}"
  }
 
  date {
    type => "apache"
    # Try to pull the timestamp from the 'timestamp' field (parsed above with
    # grok). The apache time format looks like: "18/Aug/2011:05:44:34 -0700"
    timestamp => "dd/MMM/yyyy:HH:mm:ss Z"
  }
}
 
output {
  # Use stdout in debug mode again to see what logstash makes of the event.
  stdout {
    debug => true
  }
  elasticsearch { embedded => true }
}

安装grok

apache-parse.conf使用grok作为输入的接口,但是grok与ubuntu主流版本(只测试过 11.10)并不兼容,按照INSTALL说明的安装无法成功。幸好,我们只要libgrok.so即可。

cd /tmp
git clone https://github.com/jordansissel/grok.git
cd grok
make libgrok.so
sudo cp libgrok.so /usr/local/lib
sudo ldconfig -v

运行logstash

$ . env.sh   #导入环境变量
$ bin/run.sh

启动完成之后,将apache的访问日志导入logstash:

nc localhost 3333 </var/log/apache2/access.log*

最后使用浏览器打开http://localhost:9292/ 访问logstash的web界面。

还没有评论,快来抢沙发!

发表评论

  • 😉
  • 😐
  • 😡
  • 😈
  • 🙂
  • 😯
  • 🙁
  • 🙄
  • 😛
  • 😳
  • 😮
  • emoji-mrgree
  • 😆
  • 💡
  • 😀
  • 👿
  • 😥
  • 😎
  • ➡
  • 😕
  • ❓
  • ❗
  • 65 queries in 0.375 seconds