wiki:BroCluster

Setting Up a Bro Cluster on a Ganeti Cluster

Assume a three-node Ganeti cluster running one bro VM on each Ganeti node: bro0, bro1, and bro2.

A normal Ganeti cluster has two LANs, one the global internet to which all VMs are bridged, and a second, normally used only by the Ganeti nodes themselves for inter-node DRBD replication. In this example, it looks like this

                  Global LAN
                147.28.0.0/24
     +---------------+---------------+-------->  Global
     |               |               |          Internet
     |               |               |
+----+----+     +----+----+     +----+----+
|  eth0   |     |  eth0   |     |  eth0   |
|         |     |         |     |         |
|         |     |         |     |         |
|  bro0   |     |  bro1   |     |  bro2   |
|         |     |         |     |         |
|         |     |         |     |         |
|  eth1   |     |  eth1   |     |  eth1   |
+----+----+     +----+----+     +----+----+
     |               |               |
     |               |               |
     +---------------+---------------+
              DRBD Closed LAN
                10.0.0.0/24

We will monitor the global segment and use the Closed LAN for inter-bro traffic.

On Each Bro VM, Create a Second Interface

We do not want the inter-bro traffic over the main LAN or we will have bro watching itself watching itself watching itself watching itself ...

So we will use the Ganeti cluster's DRBD private LAN for the inter-bro traffic.

Bridge Each VM onto the DRBD LAN

On the Ganeti master, add the DRDB Closed LAN to each Bro instance.

gnt-instance modify --net 1:add,link=br-hack bro0.sea.rg.net
gnt-instance modify --net 1:add,link=br-hack bro1.sea.rg.net
gnt-instance modify --net 1:add,link=br-hack bro2.sea.rg.net

Tell Each Bro Node About the Backdoor LAN

Edit each of the bro node's /etc/network/interfaces to add the new interface.

auto eth1
iface eth1 inet static
  address 10.0.0.10/24

and so for each bro node.

On all bro nodes, add entries in /etc/hosts so the back LAN will have names

# BRO Backdoor LAN
#
10.0.0.10       bro0.backlan
10.0.0.11       bro1.backlan
10.0.0.12       bro2.backlan

Reboot the Instances so they Get the New Configurations

The instances must be rebooted from the ganeti master, not from within the instance

gnt-instance reboot bro0.sea.rg.net bro1.sea.rg.net bro2.sea.rg.net

Log in to each and ping the others to make sure the configuration has been successful.

Create bro User and Give it Perms

On each of the bro VMs, as root set up a bro user and copy over the basic credentials and dot files.

adduser bro
rsync -vlpPStgoHxr .ssh .bashrc .emacs .exrc .forward .inputrc ~bro
chown -R bro:bro ~bro

On all nodes, add bro user to /etc/sudoers

bro     ALL=(ALL) NOPASSWD: ALL

Set Up Credentials

Log on one of the VMs as the bro user, let's use bro0, and create a passwordless ssh key set to be used between the nodes, add it to the keyring, and push it to the other bro VMs.

ssh-keygen -t ed25519 -N "" -f .ssh/id_ed25519
cat .ssh/id_ed25519.pub >> .ssh/authorized_keys
rsync -vPaHxRSzr .ssh bro1.backlan:
rsync -vPaHxRSzr .ssh bro2.backlan:

Introduce the accounts to each other. On all three nodes, run the following and confirm the host keys

ssh -i .ssh/ed25519.pub bro0.backlan
ssh -i .ssh/ed25519.pub bro1.backlan
ssh -i .ssh/ed25519.pub bro2.backlan

Install Bro on the Bro Manager Node

I use the excellent Bro Doc

sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev
git clone --recursive git://git.bro.org/bro
./configure
make
make install

This takes a while.

Fix the $PATH in .bashrc or whatever

export PATH=/usr/local/bro/bin:$PATH

Configure the Cluster

This was my three node cluster, with the first node playing all four roles, manager, logger, proxy, and worker-0.

cat /usr/local/bro/etc/node.cfg << EOF
# Example BroControl node configuration.
#
# This example has a standalone node ready to go except for possibly changing
# the sniffing interface.

# This is a complete standalone configuration.  Most likely you will
# only need to change the interface.
#[bro]
#type=standalone
#host=localhost
#interface=eth0

## Below is an example clustered configuration. If you use this,
## remove the [bro] node above.

[logger]
type=logger
host=bro0.backlan
#
[manager]
type=manager
host=bro0.backlan
#
[proxy-1]
type=proxy
host=bro0.backlan
#
[worker-0]
type=worker
host=bro0.backlan
interface=eth0
#
[worker-1]
type=worker
host=bro1.backlan
interface=eth0
#
[worker-2]
type=worker
host=bro2.backlan
interface=eth0
EOF

Configure broctl.cfg

Make it so that Bro can be Promiscuous on all Nodes

Hack the following into /usr/local/bro/etc/broctl.cfg on the master.

###############################################
# Hacks

### clean up setcap problem
### https://github.com/PingTrip/broctl-setcap
#
setcap.enabled=1
setcap.command=sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro && sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/capstats

Bro 2.5 Forgot Sendmail Configuration

Hack the following into /usr/local/bro/etc/broctl.cfg on the master.

### sendmail not configured
#
SendMail = /usr/sbin/sendmail

And you probably want to fix up the MailTo

MailTo = randy@psg,com

Give bro User Access to the Ethernet

Allow the bro user to control network devices. The setcap will be done later.

gpasswd -a bro netdev

Configure networks.cfg for the LAN You Want to Monitor

cat > /usr/local/bro/etc/networks.cfg << EOF
# List of local networks in CIDR notation, optionally followed by a
# descriptive tag.
# For example, "10.0.0.0/8" or "fe80::/64" are valid prefixes.

147.28.0.0/24
EOF

Prepare the Worker Nodes

Make it so bro user can write to /usr/local/bro on all nodes

sudo mkdir /usr/local/bro
sudo chown bro:bro /usr/local/bro

Test

Go for broke

broctl deploy

And start debugging.

It is Working, so Cron Watcher

Add the following to the bro user's crontab:

*/5 * * * * /usr/local/bro/bin/broctl cron

Note that you can disable and enable the cron watcher

broctl cron disable
broctl cron enable
Last modified 3 years ago Last modified on Feb 14, 2017, 1:25:23 AM