Setting Up a Bro Cluster on a Ganeti Cluster

Assume a three-node Ganeti cluster running one bro VM on each Ganeti node: bro0, bro1, and bro2.

A normal Ganeti cluster has two LANs, one the global internet to which all VMs are bridged, and a second, normally used only by the Ganeti nodes themselves for inter-node DRBD replication. In this example, it looks like this

                  Global LAN
     +---------------+---------------+-------->  Global
     |               |               |          Internet
     |               |               |
+----+----+     +----+----+     +----+----+
|  eth0   |     |  eth0   |     |  eth0   |
|         |     |         |     |         |
|         |     |         |     |         |
|  bro0   |     |  bro1   |     |  bro2   |
|         |     |         |     |         |
|         |     |         |     |         |
|  eth1   |     |  eth1   |     |  eth1   |
+----+----+     +----+----+     +----+----+
     |               |               |
     |               |               |
              DRBD Closed LAN

We will monitor the global segment and use the Closed LAN for inter-bro traffic.

On Each Bro VM, Create a Second Interface

We do not want the inter-bro traffic over the main LAN or we will have bro watching itself watching itself watching itself watching itself ...

So we will use the Ganeti cluster's DRBD private LAN for the inter-bro traffic.

Bridge Each VM onto the DRBD LAN

On the Ganeti master, add the DRDB Closed LAN to each Bro instance.

gnt-instance modify --net 1:add,link=br-hack
gnt-instance modify --net 1:add,link=br-hack
gnt-instance modify --net 1:add,link=br-hack

Tell Each Bro Node About the Backdoor LAN

Edit each of the bro node's /etc/network/interfaces to add the new interface.

auto eth1
iface eth1 inet static

and so for each bro node.

On all bro nodes, add entries in /etc/hosts so the back LAN will have names

# BRO Backdoor LAN
#       bro0.backlan       bro1.backlan       bro2.backlan

Reboot the Instances so they Get the New Configurations

The instances must be rebooted from the ganeti master, not from within the instance

gnt-instance reboot

Log in to each and ping the others to make sure the configuration has been successful.

Create bro User and Give it Perms

On each of the bro VMs, as root set up a bro user and copy over the basic credentials and dot files.

adduser bro
rsync -vlpPStgoHxr .ssh .bashrc .emacs .exrc .forward .inputrc ~bro
chown -R bro:bro ~bro

On all nodes, add bro user to /etc/sudoers


Set Up Credentials

Log on one of the VMs as the bro user, let's use bro0, and create a passwordless ssh key set to be used between the nodes, add it to the keyring, and push it to the other bro VMs.

ssh-keygen -t ed25519 -N "" -f .ssh/id_ed25519
cat .ssh/ >> .ssh/authorized_keys
rsync -vPaHxRSzr .ssh bro1.backlan:
rsync -vPaHxRSzr .ssh bro2.backlan:

Introduce the accounts to each other. On all three nodes, run the following and confirm the host keys

ssh -i .ssh/ bro0.backlan
ssh -i .ssh/ bro1.backlan
ssh -i .ssh/ bro2.backlan

Install Bro on the Bro Manager Node

I use the excellent Bro Doc

sudo apt-get install cmake make gcc g++ flex bison libpcap-dev libssl-dev python-dev swig zlib1g-dev
git clone --recursive git://
make install

This takes a while.

Fix the $PATH in .bashrc or whatever

export PATH=/usr/local/bro/bin:$PATH

Configure the Cluster

This was my three node cluster, with the first node playing all four roles, manager, logger, proxy, and worker-0.

cat /usr/local/bro/etc/node.cfg << EOF
# Example BroControl node configuration.
# This example has a standalone node ready to go except for possibly changing
# the sniffing interface.

# This is a complete standalone configuration.  Most likely you will
# only need to change the interface.

## Below is an example clustered configuration. If you use this,
## remove the [bro] node above.


Configure broctl.cfg

Make it so that Bro can be Promiscuous on all Nodes

Hack the following into /usr/local/bro/etc/broctl.cfg on the master.

# Hacks

### clean up setcap problem
setcap.command=sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/bro && sudo /sbin/setcap cap_net_raw,cap_net_admin=eip /usr/local/bro/bin/capstats

Bro 2.5 Forgot Sendmail Configuration

Hack the following into /usr/local/bro/etc/broctl.cfg on the master.

### sendmail not configured
SendMail = /usr/sbin/sendmail

And you probably want to fix up the MailTo

MailTo = randy@psg,com

Give bro User Access to the Ethernet

Allow the bro user to control network devices. The setcap will be done later.

gpasswd -a bro netdev

Configure networks.cfg for the LAN You Want to Monitor

cat > /usr/local/bro/etc/networks.cfg << EOF
# List of local networks in CIDR notation, optionally followed by a
# descriptive tag.
# For example, "" or "fe80::/64" are valid prefixes.

Prepare the Worker Nodes

Make it so bro user can write to /usr/local/bro on all nodes

sudo mkdir /usr/local/bro
sudo chown bro:bro /usr/local/bro


Go for broke

broctl deploy

And start debugging.

It is Working, so Cron Watcher

Add the following to the bro user's crontab:

*/5 * * * * /usr/local/bro/bin/broctl cron

Note that you can disable and enable the cron watcher

broctl cron disable
broctl cron enable
Last modified 3 years ago Last modified on Feb 14, 2017, 1:25:23 AM