Sunday, 13 January 2013

Github Repo of Pcaps

I became a little tired manually downloading pcaps from the various, freely available resources on the net so I added a repo to my very basic github repo so that I could simply `git clone` from wherever I am - assuming decent bandwidth and Github don't get to pissed at storing pcaps - as it's only 1.1gb at present, I doubt they will (/me hoping :) ).

I'm not really sure what I plan to do with this repo, I guess I'll extend it and add more as I get around to it. My sole goal at the minute is to use the awesome functionality of Github to manage and modify the content, as well as making it easier for me to remember where the pcaps are ;-)

Tuesday, 8 January 2013

Separate MongoDB Syslog by Facility

In my last post, I showed how you can set up MongoDB v2.2 to syslog its logs off to a remote syslog server. As my `tcpdump` snippets show, the syslog messages hit the syslog server tagged as "user.info", which means that they're assigned to the "user" facility with a severity level of "info".

I've received a few questions regarding the possiblity of splitting out syslog messages by facility, however, as everything is currently sent to a "user.info" bucket, so-to-speak, this is not possibility. There is a current feature request for this capability and work will be done on this but if this is important for you, I'd strongly encourage you to vote for this feature.

In the meantime, however, (whilst not ideal) you can still do some host filtering with rsyslog as outlined here.

Tuesday, 1 January 2013

MongoDB Logging to Remote Syslog Server

As per the MongoDB 2.2 release notes, log output for MongoDB can now be redirected to a remote syslog server.

Here is an example configuration.

MongoDB Instance

MongoDB is started as follows (note the extra `syslog` switch):

 $ mongod --dbpath=/data/db/syslog --fork --syslog  

The local "/etc/syslog.conf" file (i.e. on the `mongod` instance) is configured to send everything to the syslog server (10.7.100.20):

@10.7.100.20:514

Syslog Server

I ran my Syslog server on Ubuntu 12.04. There are a tonne of links out there describing how to install syslog on Ubuntu - see here. The syslog "facilities" are configured in the server's `/etc/syslog.conf` file (I left this as default):

#################################################################################
#
# First some standard logfiles.  Log by facility.
#
auth,authpriv.*            /var/log/auth.log
*.*;auth,authpriv.none        -/var/log/syslog

#cron.*                /var/log/cron.log
daemon.*            -/var/log/daemon.log
kern.*                -/var/log/kern.log
lpr.*                -/var/log/lpr.log
mail.*                -/var/log/mail.log
user.*                -/var/log/user.log
.....
.....
*.=info;*.=notice;*.=warning;\
        auth,authpriv.none;\
        cron,daemon.none;\
        mail,news.none          -/var/log/messages
#################################################################################

We then need to enable the syslog server to accept remote syslog messages as follows:

more /etc/default/syslogd 
#################################################################################
#
# Top configuration file for syslogd
#
# Full documentation of possible arguments are found in the manpage
# syslogd(8).
#
# For remote UDP logging use SYSLOGD="-r"
#
SYSLOGD="-r"
#################################################################################

Using `tcpdump`, we can see the syslog messages arriving at the syslog server from the `mongod` instance:

#################################################################################
01:27:40.675624 IP 10.7.100.6.55318 > 10.7.100.20.514: SYSLOG user.info, length: 111

    0x0000:  4500 008b 335e 0000 4011 6adc 0a07 6406  E...3^..@.j...d.

    0x0010:  0a07 6414 d816 0202 0077 0a1e 3c31 343e  ..d......w..<14>
    0x0020:  4465 6320 3134 2031 313a 3537 3a31 3720  Dec.14.11:57:17.
    0x0030:  6d61 726b 2d6d 6270 2e6c 6f63 616c 206d  mark-mbp.local.m
    0x0040:  6f6e 676f 642e 3135 3030 315b 3433 3032  ongod.15001[4302
    0x0050:  5d3a 2046 7269 2044 6563 2031 3420 3131  ]:.Fri.Dec.14.11
    0x0060:  3a35 373a 3137 205b 696e 6974 616e 646c  :57:17.[initandl
    0x0070:  6973 7465 6e5d 2072 6563 6f76 6572 2063  isten].recover.c
    0x0080:  6c65 616e 696e 6720 7570 0a              leaning.up.
01:27:40.675703 IP 10.7.100.6.55318 > 10.7.100.20.514: SYSLOG user.info, length: 110
    0x0000:  4500 008a 0d49 0000 4011 90f2 0a07 6406  E....I..@.....d.
    0x0010:  0a07 6414 d816 0202 0076 df4a 3c31 343e  ..d......v.J<14>
    0x0020:  4465 6320 3134 2031 313a 3537 3a31 3720  Dec.14.11:57:17.
    0x0030:  6d61 726b 2d6d 6270 2e6c 6f63 616c 206d  mark-mbp.local.m
    0x0040:  6f6e 676f 642e 3135 3030 315b 3433 3032  ongod.15001[4302
    0x0050:  5d3a 2046 7269 2044 6563 2031 3420 3131  ]:.Fri.Dec.14.11
    0x0060:  3a35 373a 3137 205b 696e 6974 616e 646c  :57:17.[initandl
    0x0070:  6973 7465 6e5d 2072 656d 6f76 654a 6f75  isten].removeJou
    0x0080:  726e 616c 4669 6c65 730a                 rnalFiles.#################################################################################

The logs from the `mongod` instance will typically be located in `/var/logs/messages` on the syslog server:

#################################################################################
Nov 16 01:27:40 10.7.100.6 mark-mbp.local mongod.15001[4302]: Fri Dec 14 11:57:17 [initandlisten] recover create file /data/db/syslog/syslog.ns 16MB 
Nov 16 01:27:40 10.7.100.6 mark-mbp.local mongod.15001[4302]: Fri Dec 14 11:57:17 [initandlisten] recover create file /data/db/syslog/syslog.0 64MB 
Nov 16 01:27:40 10.7.100.6 mark-mbp.local mongod.15001[4302]: Fri Dec 14 11:57:17 [initandlisten] recover cleaning up 
Nov 16 01:27:40 10.7.100.6 mark-mbp.local mongod.15001[4302]: Fri Dec 14 11:57:17 [initandlisten] removeJournalFiles 
Nov 16 01:27:40 10.7.100.6 mark-mbp.local mongod.15001[4302]: Fri Dec 14 11:57:17 [initandlisten] recover done 
Nov 16 01:27:40 10.7.100.6 mark-mbp.local mongod.15001[4302]: Fri Dec 14 11:57:17 [websvr] admin web console waiting for connections on port 16001 
Nov 16 01:27:40 10.7.100.6 mark-mbp.local mongod.15001[4302]: Fri Dec 14 11:57:17 [initandlisten] waiting for connections on port 15001 
#################################################################################
So as you can see, it's quite simple to syslog your logs off MongoDB to a centralised syslog server. If you want to keep an eye on "logging" related MongoDB feature requests and bugs, check out this JIRA link.