Tuesday, December 8, 2015

Cumulus VX on Libvirt (CentOS 6)

You may have seen my earlier post on Cumulus VX in libvirt. I'm going to go a step further and try to emulate the environment described here:

https://docs.cumulusnetworks.com/display/VX/Using+Cumulus+VX+with+KVM

There are some pretty significant differences though, because libvirt on CentOS 6 is pretty "feature light". Nevertheless, this should result in something to work with.

You will need to build a system that can serve DHCP and HTTP on your 'default' network and then disable the dhcp section (you could delete it) in /etc/libvirt/qemu/networks/default.xml . This is because while libvirt can create manual DHCP entries (sometimes called "static dhcp" entries) using the 'host' declaration, you will not be able to pass DHCP option 239 which is required for your Zero Touch Provisioning. Queue sad trombone noise.

Next, you will want to build 4 additional networks in the Virtual Machine Manager ( localhost>details>Virtual Networks ). I built isolated networks named testnet1 thru testnet4, but you don't need to follow my silliness. Be sure to restart libvirtd after you've completed this to be sure that it has picked up all your changes. Maybe backup your XML files too, because there are situations where libvirt can blow them away. Once your DHCP server is running on the 'default' network then build 4 manual DHCP entries using the MAC addresses that you'll find below for eth0 (the ones that are on virbr0).

You will follow the instructions in the cumulus docs but instead of the KVM commands you can use virt-install to initiate the guests (adjust path to qcow2 files you copied):

sudo virt-install --os-variant=generic --ram=256 --vcpus=1 --network bridge=virbr0,model=virtio,mac=00:01:00:00:01:01 --network network=testnet1,model=virtio,mac=00:00:02:00:00:11 --network network=testnet2,model=virtio,mac=00:00:02:00:00:12 --boot hd --disk path=/home/user1/leaf1.qcow2,format=qcow2 --name=leaf1

sudo virt-install --os-variant=generic --ram=256 --vcpus=1 --network bridge=virbr0,model=virtio,mac=00:01:00:00:01:02 --network network=testnet3,model=virtio,mac=00:00:02:00:00:21 --network network=testnet4,model=virtio,mac=00:00:02:00:00:22 --boot hd --disk path=/home/user1/leaf2.qcow2,format=qcow2 --name=leaf2

sudo virt-install --os-variant=generic --ram=256 --vcpus=1 --network bridge=virbr0,model=virtio,mac=00:01:00:00:01:03 --network network=testnet1,model=virtio,mac=00:00:02:00:00:31 --network network=testnet3,model=virtio,mac=00:00:02:00:00:32 --boot hd --disk path=/home/user1/spine1.qcow2,format=qcow2 --name=spine1

sudo virt-install --os-variant=generic --ram=256 --vcpus=1 --network bridge=virbr0,model=virtio,mac=00:01:00:00:01:04 --network network=testnet2,model=virtio,mac=00:00:02:00:00:41 --network network=testnet4,model=virtio,mac=00:00:02:00:00:42 --boot hd --disk path=/home/user1/spine2.qcow2,format=qcow2 --name=spine2

 I will admit that the configs that are described in the Cumulus docs indicate that swp1 thru swp3 would be provisioned, but the virtual hardware configured is only swp1 thru swp2. I'm going to ignore swp3 for now.

If you built option 239 correctly on the DHCP scope, then you should be able to point all these instances to your ZTP script. Just remember that you do not need to run your web server on the deafult port 80, as ZTP will accept a path like 'http://192.168.122.101:1234/my_ztp_script.sh'

I haven't setup a configuration management tool on my instances yet, but that will be upcoming via my ZTP script.


Friday, December 4, 2015

Boot Cumulus VX in a libvirt KVM

Why would I want to do that?

Well, if you are running CentOS 6  and want an easy way to play with the Cumulus VX, here is a good start.

On a CentOS 6 system running a GUI
#yum groupinstall "Virtualization*"

(this will install all your libvirt bits and pieces)

Then download the Cumulus KVM image (it is a qcow2 image) and remember the path

#sudo virt-install --os-variant=generic --ram=256 --vcpus=1 --network bridge=virbr0,model=virtio,mac=00:01:00:00:01:00 --boot hd --disk path=/home/user/cumulus.qcow2 --name=cumulus1

This only provisions the eth0 interface (mgmt) but it should get you a start to playing with the product.

Enjoy!!!

Tuesday, September 15, 2015

Backup pfsense firewall (via SSH) using ONE script

I know there are other methods out there to backup a pfsense config. These do work but I'm just not a fan of relying on the gui to perform my config backups. Plus, SSH is my encrypted session of choice because its secure, flexible and available.

Once again I'm using Fabric to perform this backup. If you haven't already done it, go to the effort of setting up Fabric on your platform of choice...you won't regret it. If you want to do interesting things with Fabric then get warmed up on your python at Codecadamy (https://www.codecademy.com/tracks/python).

This script uses an interactive prompt for you to enter the password, but you can simply provide the password either in the script itself (shame shame) or via another secure method.

Backups are pulled back to the system running the script into a directory named 'my_pfsense_backups' and are given a directory for each day. You can tweak this to suit your needs.

       

#!/usr/bin/python
#
# Designed and tested on pfsense v2.2
#
import urllib2, base64, getpass, json, re, sys, os
from fabric.api import *
from datetime import datetime
#
myname = ('root')
# NOTE: pfsense uses root user that has same password as admin - required for sftp file access
theList = ['pfsense1.company.com','pfsense2.company.com']
#
i = datetime.now()
now_is = i.strftime('%Y%m%d-%H%M%S')
today_is = i.strftime('%Y%m%d')
print now_is
#
print ('')
print ('Username is ' + myname)
pw = getpass.getpass()
print ('')
#
how_many = len(theList)
#
print("This will backup " + str(how_many) + " systems:\n")
print (theList)
print ('')
#
env.user = myname
env.hosts = theList
env.password = pw
#
#@parallel(pool_size=5)
#
# generate the backup file on the pfsense system itself, this will take some time
def generate_and_pull_backup():
        env.warn_only = True
#       run( "8", shell=False )
        backup_command_output = run( "/etc/rc.create_full_backup", shell=False )
# parse the output of the create_full_backup command
        file_generated_full_path = backup_command_output.rsplit(None, 1)[-1]
        filename_generated = file_generated_full_path.split('/')[-1]
# pull the backup home to me
        get("%s" % file_generated_full_path,"./my_pfsense_backups/%s/%s-%s" % (today_is,env.host,filename_generated))
# NOTE: configs can be restored via /etc/rc.restore_full_backup
#
# delete config backup just generated so disk does not fill
        run( "rm -f %s" % file_generated_full_path, shell=False )
#
if __name__ == '__main__':
        execute(generate_and_pull_backup)
       
 

Hope you enjoy this as much as I have! Backing up my pfsense systems has always been far too manual and problem prone so I'm looking forward to putting that behind me.

Monday, September 14, 2015

Backup F5 units (LTM or GTM) using one script!

So in case you have gone down this rabbit hole, the BigIP series are great but certain things about them require finesse. Case in point - the management interface does not always play well with other published services like TACACS or NTP. Sometime they require custom routes and that can become a hassle.

So backing up the config on an F5 can sometimes encounter these challenges too. I want to:

1. Generate a config backup on my F5 unit using tmsh commands
2. Securely transfer that config backup somewhere else, preferably on a secure network
3. I don't want to deploy a webserver or entire development suite just to accomplish goals 1 or 2

Enter Python Fabric.

Yes, getting Fabric will take some setup....but this is an environment that you will be able to use over & over again (not just for F5 backups)

The script you can grab below permits you to use Fabric to open an SSH session, generate the config backup and then pull it back (via SFTP) to the system where you originally ran the fabric script.

Be sure to edit the user ("myname" variable) and the list of F5 units ("theList" variables) that you want to run this on. This can be used to be fully automated and headless but I'm going to let you work out those details yourself. Files will be stored on the local box in ./my_F5_backups under a date directory for each day or you could customize this script to go hand in glove with logrotate.

       
#!/usr/bin/python
#
# Designed and tested on LTM v11.x
#
import urllib2, base64, getpass, json, re, sys, os
from fabric.api import *
from datetime import datetime
#
i = datetime.now()
now_is = i.strftime('%Y%m%d-%H%M%S')
today_is = i.strftime('%Y%m%d')
print now_is
#
#
#
myname = ('my_admin_user')
print ('')
print ('Username is ' + myname)
pw = getpass.getpass()
print ('')
#
#
theList = ['ltm01.company.com','gtm01.company.com']
#
#
#
how_many = len(theList)
#
# I am using someone else's yes/no logic here
#
def query_yes_no(question, default="yes"):
        valid = {"yes":"yes",   "y":"yes",  "ye":"yes",
                "no":"no",     "n":"no"}
        if default == None:
                prompt = " [y/n] "
        elif default == "yes":
                prompt = " [Y/n] "
        elif default == "no":
                prompt = " [y/N] "
        else:
                raise ValueError("invalid default answer: '%s'" % default)

        while 1:
                sys.stdout.write(question + prompt)
                choice = raw_input().lower()
                if default is not None and choice == '':
                        return default
                elif choice in valid.keys():
                        return valid[choice]
                else:
                        sys.stdout.write("Please respond with 'yes' or 'no' "\
                                "(or 'y' or 'n').\n")
#
keep_going = query_yes_no("This will backup " + str(how_many) + " systems. Would you like to continue?\n (No will list systems that it would have run on and exit)")
if keep_going == "no":
        print (theList)
        sys.exit()
print ('')
#
env.user = myname
env.hosts = theList
env.password = pw
#
#@parallel(pool_size=5)
#
# generate the backup file (ucs) on the F5 unit itself - will create some load on F5 while running
def run_on_f5_first():
        run( "tmsh save sys ucs %s" % now_is, shell=True )
# copy backup file (ucs) back to local system
def file_get():
    get("/var/local/ucs/%s.ucs" % now_is,"./my_F5_backups/%s/%s-%s.ucs" % (today_is,now_is,env.host))
# delete config backup just genrated so disk does not fill
def run_on_f5_last():
        run( "rm -f /var/local/ucs/%s.ucs" % now_is, shell=True )
#
if __name__ == '__main__':
        execute(run_on_f5_first)
        execute (file_get)
        execute(run_on_f5_last)
       
 

And there you go. One script that does something that I've meant to do for years and just didn't get around to until now. Boy am I glad that I waited until Fabric came along!

Python Fabric read hostnames from Foreman

I don't think that it comes as any surprise that I love Fabric. It just works. Yeah, you have to know a bit of python to make things work the way you want it, but this isn't rocket science. If you have problems running a python script or a fabfile then you will find a large set of examples online.

One thing that you may find challenging in Fabric (at least I have) is keeping your host lists up to date. This is now not a problem is you leverage Foreman to store your system information!

This is an interactive script that pulls your system hostnames out of Foreman (using the rest API) and then allows you to run a regex against the hostnames to narrow that list down. I've used someone else's yes/no logic so I can't take credit for that one (sorry, I can't recall where I pulled it from - I wrote this a few weeks ago)

I've named this sprinkler.py but you can name it what you want. Be sure to edit your Foreman FQDN and mark the file as executable....and remember that you can do great & terrible things with this script:



#!/usr/bin/python
import urllib2, base64, sys, getpass, json, os, re
from fabric.api import *
from fabric.tasks import execute
##
#Set variables we need to use
##
string_o_machines = ""
if len(sys.argv) == 1:
        print ("\n   Usage ./sprinkler.py  is required\n")
        sys.exit(0)
else:
        myname = sys.argv[1]
#
foreman_fqdn = "my_foreman_hostname.company.com"
#
def query_yes_no(question, default="yes"):
        valid = {"yes":"yes",   "y":"yes",  "ye":"yes",
                "no":"no",     "n":"no"}
        if default == None:
                prompt = " [y/n] "
        elif default == "yes":
                prompt = " [Y/n] "
        elif default == "no":
                prompt = " [y/N] "
        else:
                raise ValueError("invalid default answer: '%s'" % default)

        while 1:
                sys.stdout.write(question + prompt)
                choice = raw_input().lower()
                if default is not None and choice == '':
                        return default
                elif choice in valid.keys():
                        return valid[choice]
                else:
                        sys.stdout.write("Please respond with 'yes' or 'no' "\
                                "(or 'y' or 'n').\n")
#
print ("")
print ("Username is " + myname)
pw = getpass.getpass() #Get password from person running script
print ("")
#
print ("What command should I issue?")
send_command = raw_input('')
print ("")
#
print ("What systems should I send this to? Use regex like .*domain.com app0?.*.com")
domain = raw_input('')
print ("")
#
needs_sudo = query_yes_no("Does this command require sudo rights?")
#
request = urllib2.Request("https://" + foreman_fqdn + "/api/hosts?per_page=10000")
#
#
#
# You need the replace to handle encodestring adding a trailing newline
# (https://docs.python.org/2/library/base64.html#base64.encodestring)
base64string = base64.encodestring('%s:%s' % (myname, pw)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
json_obj = urllib2.urlopen(request)
#
#
#
data = json.load(json_obj)
hostlist = []
for item in data:
        hostname = item['host']['name']
        hostlist.append(hostname)
x=re.compile(domain)
sub_list = filter(x.match, hostlist)
#print sub_list
this_many = len(sub_list)
#
#
print ('')
keep_going = query_yes_no("Command will be send to " + str(this_many) + " systems. Would you like to continue?\n (No will list systems that it would have run on and exit)")
if keep_going == "no":
        print (sub_list)
        sys.exit()
#
#
env.user = myname
env.hosts = sub_list
env.password = pw
#
@parallel(pool_size=5)
def command_to_run():
        if needs_sudo == "no":
                with settings(warn_only=True):
                        run( send_command )
        else:
                with settings(warn_only=True):
                        sudo( send_command )
#
if __name__ == '__main__':
        execute(command_to_run)


Have fun using this! (You may also customize how many parallel systems this runs on at any given time by adjusting the pool size on that.)

Tuesday, December 17, 2013

Setup Trigger on CentOs\RHEL

Setting up trigger (https://github.com/trigger/trigger) is no small task on a RHEL\CentOs system.

Here is a basic setup bash script which you can use or modify to get up and running. I haven't tested it yet on a completely new system, but it should work without too much tweaking.


#!/bin/bash
## run this as root on a CentOs/RHEL system (tested on Centos 6.4 with rpmforge repos already setup)
## grab epel repository and make it available in yum
wget http://mirrors.solfo.com/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -i epel-release-6-8.noarch.rpm
## update yum cache so it knows to use epel rpms
yum update
## install everything you need to install trigger
yum -y install libxml2-devel libxml++-devel python-devel python-setuptools python-simpleparse python-twisted python-pip redis automake gcc
## start redis in case you need it
chkconfig redis on
service redis start
## use pip to install trigger
pip install trigger
## use pip to install twister (may be done already...but JIC)
pip install twister
## build config directory and populate with something (pip copies trigger files into /tmp/pip-build-root/trigger/)
mkdir -p /etc/trigger
cp /tmp/pip-build-root/trigger/conf/netdevices.xml /etc/trigger/netdevices.xml
cp /tmp/pip-build-root/trigger/conf/bounce.py /etc/trigger/bounce.py
cp /tmp/pip-build-root/trigger/conf/trigger_settings.py /etc/trigger/settings.py
cp /tmp/pip-build-root/trigger/conf/autoacl.py /etc/trigger/autoacl.py
cp /tmp/pip-build-root/trigger/tests/data/tackf /etc/trigger/.tackf
##create a few dirs that you may need
mkdir -p /data/firewalls
mkdir -p /data/tftproot
## now you need to edit your /etc/trigger/settings.py and set your DB type and details
## run init_task_db to populate your DB with the basic tables that you'll need

Don't forget, this is for my benefit as well. Now I'll have this post to look back on to see how I did this before.





Wednesday, January 25, 2012

Securely backup a Cisco Firewall (ASA & FWSM)

I have a problem with Cisco. Let me put it simply in this list:

1. Cisco does not support SSH keys (at least not very well) across its fleet of firewalls and security devices.
2. Config backups: TFTP is convenient but NOT secure. FTP is convenient but NOT secure. HTTP can be convenient but still NOT secure. HTTPS can be secure but can be unwieldy to setup and support. SCP is pretty convenient on a *nix machine, and is definitely secure, but there is a catch.
3. SCP on Cisco security devices: While these security devices usually do support SCP, they cannot SCP their configs to a destination (from the security device itself). Also, you cannot pull the running-config directly via SCP.

Addressing point #1 - using a password in an expect script is not extremely secure, but it beats the alternatives in almost every case. There are ways to secure strings (like passwords) in expect but you can decide for yourself whether these are needed.
Addressing point #3 - SCP can be used in the following way: Using expect, SSH to the device and save the running-config to a predefined location (root of the disk) and logout. Open a second shell and SCP the saved config back (from predefined location on device to predefined location on expect host).

My script isn't the shining example of a secure backup, but I'm unimpressed with the alternatives to this.

Here is an example of my firewall_backup.exp:

#!/usr/bin/expect
#######################################################
# Written by paklids 2012 01 12 #
# Expect script to securely backup Cisco ASAs & FWSMs #
# using SSH and SCP. #
# #
# This script can be run under a wrapper shell script #
# and then used (if desired) to checkin the firewall #
# configurations to SVN or any other change managment #
# or archiving solution - I will post my wrapper once #
# properly sanitized. #
#######################################################
#
##set an overall timeout if individual commands hang
set timeout 15
#
##set the directory you'd like the script to work out of
set -- /home/user/*
#
##open a pre-built list of IPs/FQDN hostnames delimited by line returns
##Note: it may be necessary to SSH to each host manually as serviceaccountname
##to save SSH host key
set f [open "/home/user/firewall_list"]
#
##for each entry in firewall_list, SSH to host and move to enable
##Note: enable may not be required if you customize the permissions
##on the serviceaccountname
while {[gets $f line] != -1} {
set pw1 "password_1"
set pw2 "password_2"
spawn "/bin/bash"
send "ssh serviceaccountname@$line\r"
expect "\npassword:"
send "$pw1\r"
expect -re "(\[^\r]*)>" {
set name $expect_out(1,string)
}
send "enable\r"
expect "\nPassword:"
send "$pw2\r"
expect "#"
##determine whether this is an FWSM or an ASA and appropriately save
##running-config to disk
send "sho version\r"
expect {
"FWSM" {
send \x20
set devtype "FWSM"
expect "#"
send "copy /noconfirm running-config disk:/$name\r"
}
"Adaptive Security Appliance" {
send \x20
set devtype "ASA"
expect "#"
send "copy /noconfirm running-config disk0:/$name\r"
}
}
expect "#"
sleep 2
send "logout\r"
#
##open a new bash shell and SCP the config that had been saved to disk
##back to the host this script is running on
spawn "/bin/bash"
send "scp serviceaccountname@$line:$name /home/user/workdir/$name\r"
expect -re ".*password:.*"
send "$pw1\r"
expect "100%"
sleep 1
}
#