Interesting Ansible ipify problem

I had this interesting problem recently and was able to work around that problem (once I discovered it)

I was using the ansible functionality of 'ipify' to determine the public ip address of the host I was provisioning. It worked great and I used that fact to find the ethernet interface that was attached to that public ip address.

I ran it on a new host and it just didn't work.

After far too long of troubleshooting I discovered my problem. The ip address provided by 'ipify` was not the public ip address of the host I ran the playbook on. It was the public ip address of the http proxy that the system was using (...doh!!!). Don't get me wrong, this is the correct behavior of an application that uses http by respecting the proxy settings. However, ipify has no way to override the proxy setting.

I was able to solve it:

- name: delete public ip address file
  file:
    state: absent
    path: /tmp/public_ip_address

- name: set_facts | get my public IP
  get_url:
    url: http://ifconfig.co/ip
    use_proxy: no
    dest: /tmp/public_ip_address

- name: Slurp file with public ip address
  slurp:
    src: /tmp/public_ip_address
  register: slurpfile
 
- name: set_facts | set fact from ip address in slurped file
  set_fact:
    _public_ip_address: "{{ slurpfile['content'] | b64decode | ipaddr }}"

- name: set_facts | interface for public ip
  set_fact:
    public_interface: "{{ item }}"
  when: >
    (hostvars[inventory_hostname]['ansible_%s' % item]|default({}))
    .get('ipv4', {}).get('address') == _public_ip_address
    or
    _public_ip_address in ((hostvars[inventory_hostname]['ansible_%s' % item]|default({}))
    .get('ipv4_secondaries'))|map(attribute='address')|list
  with_items:
    - "{{ ansible_interfaces }}"



Many thanks to user larsks that published a good way to iterate through the interfaces.

StackOverFlow discussion about Ansible iterating interface details

And shoutout to Flowroute ( https://www.flowroute.com/ )

Using Docker to solve my JAVA woes

Occasionally you encounter a problem that resurfaces every 6 months or so that you wish never would come back again. One of those problems is running multiple versions of Java (for various reasons) and toggling between those versions. "Jenv" works pretty well on a Mac, but sometimes when you are talking about libraries and java plugins it goes back to being complex.

For this last round, I took a different approach. What if I created a docker container that had everything I needed for that task? Then I could reuse that container whenever I needed and did not need to make any changes to my host system. My challenge this time around was connecting to an Avocent KVM switch that could only support a very old version of JAVA webstart.

I started with a basic Debian Wheezy container and added all the JAVA parts that I needed, then setup VNC to connect to the container. Downloading and launching the container is pretty easy if you follow the directions here https://hub.docker.com/r/paklids/jnlp-helper/ .

After that, there are a few other steps to follow.

#1 VNC to the container. I'm using RealVNC on a Mac, but you may use another method (My coworker used the Safari browser)

#2 After the window opens (using the correct password) then launch firefox from the text console

#3 Use firefox to browse to your Avocent KVM (or whatever site you need the older JAVA for - like those that use JNLP's)

#4 Accept any SSL exceptions. Even if you resolve the SSL problems with your appliance, you will still need to use an older version of JAVA which in itself is likely insecure. Now login to your device:

#5 Click on any of the links that uses Java Webstart (like JNLP links). Use the "Open with" context and select browse:

#6 Show other applications and then select "IcedTea Java Web Start"

At this point your Java Webstart application should start. I'll try to publish my Dockerfile so that if you need to tweak for your own purposes then you have a good base to start from. Enjoy!

My first iPXE adventure!

I recently encountered the need to rebuild some bare-metal servers in one of our datacenters and fell into a strange requirement. I've built tons of PXE boot systems before (all different variations of kickstart) so I was stubborn when the recommendation for iPXE came along. 

"I can do this with standard PXE!" I said...

I was wrong.

This time around I needed iPXE for passing arguments to a dynamic build system. Initially, I was resistant. Then I bit the bullet and jumped into using iPXE. Boy was I glad, because it gave me flexibility that I hadn't used before. I'll document the process I used to configure my build box, but I wanted first to post my working iPXE boot config (also its the boot menu)

       

#!ipxe
# pulled from http://boot.ipxe.org/undionly.kpxe
set store mybuildserver.example.com

prompt --key 0x02 --timeout 1000 Press Ctrl-B for the iPXE command line... && shell ||

:boot_menu
menu iPXE Boot Menu
item localboot  Boot From Local Disk
item --gap --   --------- Operating Systems -------------
item ubuntu1604 Wipe and Install Ubuntu 16.04.3
item coreos     Wipe and Install CoreOS
item --gap --   ---------     Utilities     -------------
item gparted    GParted Partition Manager
item DBAN       Dariks Boot and Nuke       NOTE: at end press Alt-F4 and reboot
item --gap --   ---------     iPXE tools    -------------
item shell      iPXE Shell
item reboot     Reload iPXE

choose --default localboot --timeout 30000 target && goto ${target} ||
echo __NOTE: Cancel Enter Select Menu, Exit
exit

:localboot
sanboot --no-describe --drive 0x80 || goto boot_menu

:ubuntu1604
echo Starting Ubuntu Xenial installer for ${mac}
kernel http://${store}/ubuntu/install/netboot/ubuntu-installer/amd64/linux
initrd http://${store}/ubuntu/install/netboot/ubuntu-installer/amd64/initrd.gz
imgargs linux auto=true url=http://${store}/auto-16-preseed.cfg priority=critical preseed/interactive=false netcfg/choose_interface=enp3s0 vga=788 live-installer/net-image=http://${store}/ubuntu/install/filesystem.squashfs  
boot || goto boot_menu

:coreos
kernel http://${store}/coreos/coreos_production_pxe.vmlinuz
initrd http://${store}/coreos/coreos_production_pxe_image.cpio.gz 
imgargs coreos_production_pxe.vmlinuz coreos.first_boot=1 coreos.autologin coreos.config.url=http://${store}:8080/ignition?mac=${mac:hexhyp}
boot || goto boot_menu

:gparted
kernel http://${store}/gparted/live/vmlinuz
initrd http://${store}/gparted/live/initrd.img
imgargs vmlinuz boot=live config components union=overlay username=user noswap noeject ip= vga=788 fetch=http://${store}/gparted/live/filesystem.squashfs
boot || goto boot_menu

:DBAN
kernel http://${store}/dban/dban.bzi
#imgargs dban.bzi nuke="dwipe"
imgargs dban.bzi nuke="dwipe --autonuke --method=zero" silent
boot || goto boot_menu

:shell
echo __NOTE: Type 'config' enter iPXE config setting, 'exit' return to boot menu.
shell
goto boot_menu

A reliable public API for (close to) free!

Sorry for the lack of posts lately, but I've been busy churning out projects (both personal and business related).

I recently went through an exercise that will definitely benefit some of the communities that I interact with (HAD and CoffeeOps) and felt obligated to share because I'm super excited about it.

In the world of IOT and scalable applications, I don't need to spend much time hyping the benefits of a simple API. Along those lines I've been playing around with the Serverless Framework to spin up services (in my case AWS Lambda) but then asked myself 3 distinct questions:

1. How small is "too small" for an API?
2. How cheap could I make a small API?
3. How reliable (or performant) could I make an API?

This seems very similar to the argument about hiring a contractor ("You can have cheap, good or fast....but you can only pick 2 of the 3 options), but it turns out that when you go small you can get pretty close to all three.

With Serverless Framework, in AWS it is pretty easy to setup an API Gateway that passes input to a Lambda function. It's really small and granular and AWS can host it on their infrastructure really easily. That covers the small stuff.

And here is the plus for us - it's really cheap! You pay only for the number of hits to your API Gateway and most AWS accounts give you a minimum number of Lambda functions for free (usually in the thousands or millions). But my problem was the last part - where do you store your data? You could spin up an RDS like PostgreSQL or MySQL, a Redis database or even a DynamoDB. But can you go cheaper than that? Yes....yes you can. 

AWS offers a simple key value store that is most often used for configuration data called SSM Parameter store. In that KV store, you can store strings of up to 4096 characters (I don't think it is bytes according to what I've found online) which is more than enough for the purpose I have. Using parameter store should be a free built in service, so that covers the whole cheap argument.

So finally we get to the reliable and fast part. Depending on how you write your function, you can have an API that completes well under 100ms. As for reliability, AWS has a long history of reliable services, and that isn't even counting that this could run in multiple regions (as long as you could have a way to sync your data). Reliable....check.

So how do you do it? Here is my very basic working example (no authentication in this example BTW...only a proof of concept):

Step 1: 
Learn about and install Serverless Framework here  https://serverless.com/ . Assumed is that you already have an AWS account to work with, but if not, you'll need to set one up.

Step 2:  
You can create a serverless project using the example templates, but to start out with you only need 3 things....a directory to work in, a serverless.yml file and a handler (I'm using python for mine)

Step 3: 
Create a Parameter in the SSM Parameter Store. Its in the AWS console under the EC2 section. Not hard to find and you only need to set an un-encrypted string. I'm using the name 'myParam'.

Step 4: A bit of code (edit your account & region as needed):

serverless.yml 

       
# Welcome to Serverless!
#

service: param-api # NOTE: update this with your service name

# frameworkVersion: "=X.X.X"

provider:
  name: aws
  runtime: python2.7
  stage: dev
  region: us-west-2
  iamRoleStatements:
    - Effect: "Allow"
      Action: 
        - "ssm:GetParameters"
        - "ssm:PutParameter"
      Resource:
        - "arn:aws:ssm:${self:provider.region}:${accountId}:parameter/myParam"

functions:
  get_myParam:
    handler: handler.get_myParam
    events:
      - http:
          path: get
          method: get

  put_myParam:
    handler: handler.put_myParam
    events:
      - http:
          path: put
          method: put
      
 

and my python functions in handler.py

       
import json
import boto3

def get_myParam(event, context):
    ssmResponse = boto3.client('ssm').get_parameters(
        Names=['myParam'],
        WithDecryption=False
    )
    
    body = {
        "message": ssmResponse['Parameters'],
        "input": event['body']
    }

    response = {
        "statusCode": 200,
        "body": json.dumps(body)
    }

    return response


def put_myParam(event, context):
    

    ssmResponse = boto3.client('ssm').put_parameter(
        Name='myParam',
        Description='this is a test',
        Value=event['body'],
        Type='String',
        Overwrite=True
    )

    response = {
        "statusCode": 200,
        "body": json.dumps(ssmResponse)
    }

    return response



      

 Step 5: 
Deploy using serverless. At CLI "serverless deploy -v"

Step 6: 
Test. Curl against the URL that you were given after your deploy completed. This API now supports get and put, but you could add whatever you want as far as HTTP method and authentication.

Step 7: 
Profit!

What could you do with this? Any number of things. Need to share data across IOT devices? Check. Need to build a global service and you don't have datacenters across the globe? Check. The advantage with this is that the investment is very low and this scales to extremely high (just on the micro level, like owning 50 million insects that you control all over the world).

Many thanks to my employer (https://www.flowroute.com/ ) for letting me work on this and my coworker Reed for helping me through the Python bits and bobs. 
Also, thanks to the Seattle CoffeeOps meetup  https://www.meetup.com/Seattle-CoffeeOps/ for listening to me rant about this stuff even though they are asking "What is he talking about ?!?"

Setting up an apt-mirror for Cumulus Linux

You may want to setup a local apt repository to pull packages from for your Cumulus infrastructure. 

I followed the instructions found here: 
https://www.howtoforge.com/local_debian_ubuntu_mirror 

But I did find a few gotcha's that  I had to fix before I was able to use my local repo. Here is a sample of my /etc/apt/mirror.list

       
############# config ##################
#
# set base_path    /var/spool/apt-mirror
#
# set mirror_path  $base_path/mirror
# set skel_path    $base_path/skel
# set var_path     $base_path/var
# set cleanscript $var_path/clean.sh
# set defaultarch  
# set postmirror_script $var_path/postmirror.sh
# set run_postmirror 0
set nthreads     20
set _tilde 0
#
############# end config ##############

deb http://repo.cumulusnetworks.com CumulusLinux-2.5 main addons updates
deb http://repo.cumulusnetworks.com CumulusLinux-2.5 security-updates

# Uncomment the next line to get access to the testing component
# deb http://repo.cumulusnetworks.com CumulusLinux-2.5 testing

# Uncomment the next line to get access to the Cumulus community repository
# deb http://repo.cumulusnetworks.com/community/ CumulusLinux-Community-2.5  main addons updates

deb http://ftp.us.debian.org/debian wheezy-backports main
deb http://ftp.us.debian.org/debian/ wheezy main

# mirror additional architectures
#deb-alpha http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-amd64 http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-armel http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-hppa http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-i386 http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-ia64 http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-m68k http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-mips http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-mipsel http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-powerpc http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-s390 http://ftp.us.debian.org/debian unstable main contrib non-free
#deb-sparc http://ftp.us.debian.org/debian unstable main contrib non-free

clean http://ftp.us.debian.org/debian

I was able to use the instructions in the howto to build the mirror and present it via http. I did have some problems using it at first. Notice the error:

W: Failed to fetch http://mydebianrepo.mycompany.com/debian/dists/wheezy-backports/main/i18n/Translation-en  Hash Sum mismatch

And then I read this:

http://askubuntu.com/questions/217502/speed-up-apt-get-update-by-removing-known-ignored-translation-en

So to fix this put the line: 

Acquire::Languages "none"; 

on the Cumulus node at the end of the conf file /etc/apt/apt.conf.d/70debconf

Here is my /etc/apt/sources.list

#
#

deb http://mydebianrepo.mycompany.com/CumulusLinux CumulusLinux-2.5 main addons updates
deb http://mydebianrepo.mycompany.com/CumulusLinux CumulusLinux-2.5 security-updates

# Uncomment the next line to get access to the testing component
# deb http://repo.cumulusnetworks.com CumulusLinux-2.5 testing

# Uncomment the next line to get access to the Cumulus community repository
# deb http://repo.cumulusnetworks.com/community/ CumulusLinux-Community-2.5  main addons updates
deb http://mydebianrepo.mycompany.com/debian wheezy-backports main
deb http://mydebianrepo.mycompany.com/debian wheezy main

Puppet 3 module for Cumulus Linux License

Here is a Puppet 3 module for applying the Cumulus Linux license to your nodes. You can change the logic to use $hostname or $fqdn or any other facter fact, but you will want to put some logic in there for future growth (ie: will you ever move to 40Gb switches?).

License file is dropped on the node at /etc/license . Be sure to name your directory the SAME AS the name of the class in ../modulename/manifests/init.pp 

       
# manifests/init.pp
class cumulus_license {

## BASIC CHECK - ARE 10GB SWITCHES THIS SETUP? THEN USE 10GB LICENSE
## OTHERWISE USE OTHER FACTER FACT TO DECIDE
if ($architecture == 'x86_64') and ($operatingsystem == 'CumulusLinux') {
  file { '/etc/license':
    ensure => present,
    owner  => root,
    group  => root,
    source => 'puppet:///modules/cumulus_license/my_10Gb_license',
    }

## AND USE THIS IDENTIFIER TO DECIDE ON 1GB LICENSE
if ($architecture == 'ppc') and ($operatingsystem == 'CumulusLinux') {
  file { '/etc/license':
    ensure => present,
    owner  => root,
    group  => root,
    source => 'puppet:///modules/cumulus_license/my_1Gb_license',
    }
  }
}

## CHANGE TO THE LICENSE FILE NOTIFIES THIS EXEC
## NOW RUN THE LICENSE COMMAND AND IF IT SUCCEEDS THEN DO NOTHING
## IF FAILS THEN HARDWARE IS UNLICENSED AND WILL INSTALL FROM PUPPET 
exec { cl-license-command :
  command     => ['/usr/cumulus/bin/cl-license -i /etc/license'],
  unless      => ['/usr/cumulus/bin/cl-license'],
  subscribe   => File['/etc/license'],
  refreshonly => true,
  }
}
       
 

A few notes about operation:

1. First the puppet run validates that there is an existing valid license on the switch. If there is, then do nothing. If you want to apply a new license then you will update puppet and run the cli-license command manually

2. If there is no active license then it tries to apply the license found at /etc/license (pushed there by puppet). You can rename this to whatever you need.
3. Make sure your Cumulus License files are in the files section of the module. These are just simple text files that get pulled in by the cli_license command during the run
4. You /may/ have to reboot the switch after the new license is applied. Please refer to Cumulus docs for guidance there.

Cumulus VX on Libvirt (CentOS 6)

You may have seen my earlier post on Cumulus VX in libvirt. I'm going to go a step further and try to emulate the environment described here:

https://docs.cumulusnetworks.com/display/VX/Using+Cumulus+VX+with+KVM

There are some pretty significant differences though, because libvirt on CentOS 6 is pretty "feature light". Nevertheless, this should result in something to work with.

You will need to build a system that can serve DHCP and HTTP on your 'default' network and then disable the dhcp section (you could delete it) in /etc/libvirt/qemu/networks/default.xml . This is because while libvirt can create manual DHCP entries (sometimes called "static dhcp" entries) using the 'host' declaration, you will not be able to pass DHCP option 239 which is required for your Zero Touch Provisioning. Queue sad trombone noise.

Next, you will want to build 4 additional networks in the Virtual Machine Manager ( localhost>details>Virtual Networks ). I built isolated networks named testnet1 thru testnet4, but you don't need to follow my silliness. Be sure to restart libvirtd after you've completed this to be sure that it has picked up all your changes. Maybe backup your XML files too, because there are situations where libvirt can blow them away. Once your DHCP server is running on the 'default' network then build 4 manual DHCP entries using the MAC addresses that you'll find below for eth0 (the ones that are on virbr0).

You will follow the instructions in the cumulus docs but instead of the KVM commands you can use virt-install to initiate the guests (adjust path to qcow2 files you copied):

sudo virt-install --os-variant=generic --ram=256 --vcpus=1 --network bridge=virbr0,model=virtio,mac=00:01:00:00:01:01 --network network=testnet1,model=virtio,mac=00:00:02:00:00:11 --network network=testnet2,model=virtio,mac=00:00:02:00:00:12 --boot hd --disk path=/home/user1/leaf1.qcow2,format=qcow2 --name=leaf1

sudo virt-install --os-variant=generic --ram=256 --vcpus=1 --network bridge=virbr0,model=virtio,mac=00:01:00:00:01:02 --network network=testnet3,model=virtio,mac=00:00:02:00:00:21 --network network=testnet4,model=virtio,mac=00:00:02:00:00:22 --boot hd --disk path=/home/user1/leaf2.qcow2,format=qcow2 --name=leaf2

sudo virt-install --os-variant=generic --ram=256 --vcpus=1 --network bridge=virbr0,model=virtio,mac=00:01:00:00:01:03 --network network=testnet1,model=virtio,mac=00:00:02:00:00:31 --network network=testnet3,model=virtio,mac=00:00:02:00:00:32 --boot hd --disk path=/home/user1/spine1.qcow2,format=qcow2 --name=spine1

sudo virt-install --os-variant=generic --ram=256 --vcpus=1 --network bridge=virbr0,model=virtio,mac=00:01:00:00:01:04 --network network=testnet2,model=virtio,mac=00:00:02:00:00:41 --network network=testnet4,model=virtio,mac=00:00:02:00:00:42 --boot hd --disk path=/home/user1/spine2.qcow2,format=qcow2 --name=spine2

 I will admit that the configs that are described in the Cumulus docs indicate that swp1 thru swp3 would be provisioned, but the virtual hardware configured is only swp1 thru swp2. I'm going to ignore swp3 for now.

If you built option 239 correctly on the DHCP scope, then you should be able to point all these instances to your ZTP script. Just remember that you do not need to run your web server on the deafult port 80, as ZTP will accept a path like 'http://192.168.122.101:1234/my_ztp_script.sh'

I haven't setup a configuration management tool on my instances yet, but that will be upcoming via my ZTP script.

Boot Cumulus VX in a libvirt KVM

Why would I want to do that?

Well, if you are running CentOS 6  and want an easy way to play with the Cumulus VX, here is a good start.

On a CentOS 6 system running a GUI
#yum groupinstall "Virtualization*"

(this will install all your libvirt bits and pieces)

Then download the Cumulus KVM image (it is a qcow2 image) and remember the path

#sudo virt-install --os-variant=generic --ram=256 --vcpus=1 --network bridge=virbr0,model=virtio,mac=00:01:00:00:01:00 --boot hd --disk path=/home/user/cumulus.qcow2 --name=cumulus1

This only provisions the eth0 interface (mgmt) but it should get you a start to playing with the product.

Enjoy!!!

Backup pfsense firewall (via SSH) using ONE script

I know there are other methods out there to backup a pfsense config. These do work but I'm just not a fan of relying on the gui to perform my config backups. Plus, SSH is my encrypted session of choice because its secure, flexible and available.

Once again I'm using Fabric to perform this backup. If you haven't already done it, go to the effort of setting up Fabric on your platform of choice...you won't regret it. If you want to do interesting things with Fabric then get warmed up on your python at Codecadamy (https://www.codecademy.com/tracks/python).

This script uses an interactive prompt for you to enter the password, but you can simply provide the password either in the script itself (shame shame) or via another secure method.

Backups are pulled back to the system running the script into a directory named 'my_pfsense_backups' and are given a directory for each day. You can tweak this to suit your needs.

       

#!/usr/bin/python
#
# Designed and tested on pfsense v2.2
#
import urllib2, base64, getpass, json, re, sys, os
from fabric.api import *
from datetime import datetime
#
myname = ('root')
# NOTE: pfsense uses root user that has same password as admin - required for sftp file access
theList = ['pfsense1.company.com','pfsense2.company.com']
#
i = datetime.now()
now_is = i.strftime('%Y%m%d-%H%M%S')
today_is = i.strftime('%Y%m%d')
print now_is
#
print ('')
print ('Username is ' + myname)
pw = getpass.getpass()
print ('')
#
how_many = len(theList)
#
print("This will backup " + str(how_many) + " systems:\n")
print (theList)
print ('')
#
env.user = myname
env.hosts = theList
env.password = pw
#
#@parallel(pool_size=5)
#
# generate the backup file on the pfsense system itself, this will take some time
def generate_and_pull_backup():
        env.warn_only = True
#       run( "8", shell=False )
        backup_command_output = run( "/etc/rc.create_full_backup", shell=False )
# parse the output of the create_full_backup command
        file_generated_full_path = backup_command_output.rsplit(None, 1)[-1]
        filename_generated = file_generated_full_path.split('/')[-1]
# pull the backup home to me
        get("%s" % file_generated_full_path,"./my_pfsense_backups/%s/%s-%s" % (today_is,env.host,filename_generated))
# NOTE: configs can be restored via /etc/rc.restore_full_backup
#
# delete config backup just generated so disk does not fill
        run( "rm -f %s" % file_generated_full_path, shell=False )
#
if __name__ == '__main__':
        execute(generate_and_pull_backup)
       
 

Hope you enjoy this as much as I have! Backing up my pfsense systems has always been far too manual and problem prone so I'm looking forward to putting that behind me.

Backup F5 units (LTM or GTM) using one script!

So in case you have gone down this rabbit hole, the BigIP series are great but certain things about them require finesse. Case in point - the management interface does not always play well with other published services like TACACS or NTP. Sometime they require custom routes and that can become a hassle.

So backing up the config on an F5 can sometimes encounter these challenges too. I want to:

1. Generate a config backup on my F5 unit using tmsh commands
2. Securely transfer that config backup somewhere else, preferably on a secure network
3. I don't want to deploy a webserver or entire development suite just to accomplish goals 1 or 2

Enter Python Fabric.

Yes, getting Fabric will take some setup....but this is an environment that you will be able to use over & over again (not just for F5 backups)

The script you can grab below permits you to use Fabric to open an SSH session, generate the config backup and then pull it back (via SFTP) to the system where you originally ran the fabric script.

Be sure to edit the user ("myname" variable) and the list of F5 units ("theList" variables) that you want to run this on. This can be used to be fully automated and headless but I'm going to let you work out those details yourself. Files will be stored on the local box in ./my_F5_backups under a date directory for each day or you could customize this script to go hand in glove with logrotate.

       
#!/usr/bin/python
#
# Designed and tested on LTM v11.x
#
import urllib2, base64, getpass, json, re, sys, os
from fabric.api import *
from datetime import datetime
#
i = datetime.now()
now_is = i.strftime('%Y%m%d-%H%M%S')
today_is = i.strftime('%Y%m%d')
print now_is
#
#
#
myname = ('my_admin_user')
print ('')
print ('Username is ' + myname)
pw = getpass.getpass()
print ('')
#
#
theList = ['ltm01.company.com','gtm01.company.com']
#
#
#
how_many = len(theList)
#
# I am using someone else's yes/no logic here
#
def query_yes_no(question, default="yes"):
        valid = {"yes":"yes",   "y":"yes",  "ye":"yes",
                "no":"no",     "n":"no"}
        if default == None:
                prompt = " [y/n] "
        elif default == "yes":
                prompt = " [Y/n] "
        elif default == "no":
                prompt = " [y/N] "
        else:
                raise ValueError("invalid default answer: '%s'" % default)

        while 1:
                sys.stdout.write(question + prompt)
                choice = raw_input().lower()
                if default is not None and choice == '':
                        return default
                elif choice in valid.keys():
                        return valid[choice]
                else:
                        sys.stdout.write("Please respond with 'yes' or 'no' "\
                                "(or 'y' or 'n').\n")
#
keep_going = query_yes_no("This will backup " + str(how_many) + " systems. Would you like to continue?\n (No will list systems that it would have run on and exit)")
if keep_going == "no":
        print (theList)
        sys.exit()
print ('')
#
env.user = myname
env.hosts = theList
env.password = pw
#
#@parallel(pool_size=5)
#
# generate the backup file (ucs) on the F5 unit itself - will create some load on F5 while running
def run_on_f5_first():
        run( "tmsh save sys ucs %s" % now_is, shell=True )
# copy backup file (ucs) back to local system
def file_get():
    get("/var/local/ucs/%s.ucs" % now_is,"./my_F5_backups/%s/%s-%s.ucs" % (today_is,now_is,env.host))
# delete config backup just genrated so disk does not fill
def run_on_f5_last():
        run( "rm -f /var/local/ucs/%s.ucs" % now_is, shell=True )
#
if __name__ == '__main__':
        execute(run_on_f5_first)
        execute (file_get)
        execute(run_on_f5_last)
       
 

And there you go. One script that does something that I've meant to do for years and just didn't get around to until now. Boy am I glad that I waited until Fabric came along!

Python Fabric read hostnames from Foreman

I don't think that it comes as any surprise that I love Fabric. It just works. Yeah, you have to know a bit of python to make things work the way you want it, but this isn't rocket science. If you have problems running a python script or a fabfile then you will find a large set of examples online.

One thing that you may find challenging in Fabric (at least I have) is keeping your host lists up to date. This is now not a problem is you leverage Foreman to store your system information!

This is an interactive script that pulls your system hostnames out of Foreman (using the rest API) and then allows you to run a regex against the hostnames to narrow that list down. I've used someone else's yes/no logic so I can't take credit for that one (sorry, I can't recall where I pulled it from - I wrote this a few weeks ago)

I've named this sprinkler.py but you can name it what you want. Be sure to edit your Foreman FQDN and mark the file as executable....and remember that you can do great & terrible things with this script:

#!/usr/bin/python
import urllib2, base64, sys, getpass, json, os, re
from fabric.api import *
from fabric.tasks import execute
##
#Set variables we need to use
##
string_o_machines = ""
if len(sys.argv) == 1:
        print ("\n   Usage ./sprinkler.py  is required\n")
        sys.exit(0)
else:
        myname = sys.argv[1]
#
foreman_fqdn = "my_foreman_hostname.company.com"
#
def query_yes_no(question, default="yes"):
        valid = {"yes":"yes",   "y":"yes",  "ye":"yes",
                "no":"no",     "n":"no"}
        if default == None:
                prompt = " [y/n] "
        elif default == "yes":
                prompt = " [Y/n] "
        elif default == "no":
                prompt = " [y/N] "
        else:
                raise ValueError("invalid default answer: '%s'" % default)

        while 1:
                sys.stdout.write(question + prompt)
                choice = raw_input().lower()
                if default is not None and choice == '':
                        return default
                elif choice in valid.keys():
                        return valid[choice]
                else:
                        sys.stdout.write("Please respond with 'yes' or 'no' "\
                                "(or 'y' or 'n').\n")
#
print ("")
print ("Username is " + myname)
pw = getpass.getpass() #Get password from person running script
print ("")
#
print ("What command should I issue?")
send_command = raw_input('')
print ("")
#
print ("What systems should I send this to? Use regex like .*domain.com app0?.*.com")
domain = raw_input('')
print ("")
#
needs_sudo = query_yes_no("Does this command require sudo rights?")
#
request = urllib2.Request("https://" + foreman_fqdn + "/api/hosts?per_page=10000")
#
#
#
# You need the replace to handle encodestring adding a trailing newline
# (https://docs.python.org/2/library/base64.html#base64.encodestring)
base64string = base64.encodestring('%s:%s' % (myname, pw)).replace('\n', '')
request.add_header("Authorization", "Basic %s" % base64string)
json_obj = urllib2.urlopen(request)
#
#
#
data = json.load(json_obj)
hostlist = []
for item in data:
        hostname = item['host']['name']
        hostlist.append(hostname)
x=re.compile(domain)
sub_list = filter(x.match, hostlist)
#print sub_list
this_many = len(sub_list)
#
#
print ('')
keep_going = query_yes_no("Command will be send to " + str(this_many) + " systems. Would you like to continue?\n (No will list systems that it would have run on and exit)")
if keep_going == "no":
        print (sub_list)
        sys.exit()
#
#
env.user = myname
env.hosts = sub_list
env.password = pw
#
@parallel(pool_size=5)
def command_to_run():
        if needs_sudo == "no":
                with settings(warn_only=True):
                        run( send_command )
        else:
                with settings(warn_only=True):
                        sudo( send_command )
#
if __name__ == '__main__':
        execute(command_to_run)

Have fun using this! (You may also customize how many parallel systems this runs on at any given time by adjusting the pool size on that.)

Setup Trigger on CentOs\RHEL

Setting up trigger (https://github.com/trigger/trigger) is no small task on a RHEL\CentOs system.

Here is a basic setup bash script which you can use or modify to get up and running. I haven't tested it yet on a completely new system, but it should work without too much tweaking.


#!/bin/bash
## run this as root on a CentOs/RHEL system (tested on Centos 6.4 with rpmforge repos already setup)
## grab epel repository and make it available in yum
wget http://mirrors.solfo.com/epel/6/i386/epel-release-6-8.noarch.rpm
rpm -i epel-release-6-8.noarch.rpm
## update yum cache so it knows to use epel rpms
yum update
## install everything you need to install trigger
yum -y install libxml2-devel libxml++-devel python-devel python-setuptools python-simpleparse python-twisted python-pip redis automake gcc
## start redis in case you need it
chkconfig redis on
service redis start
## use pip to install trigger
pip install trigger
## use pip to install twister (may be done already...but JIC)
pip install twister
## build config directory and populate with something (pip copies trigger files into /tmp/pip-build-root/trigger/)
mkdir -p /etc/trigger
cp /tmp/pip-build-root/trigger/conf/netdevices.xml /etc/trigger/netdevices.xml
cp /tmp/pip-build-root/trigger/conf/bounce.py /etc/trigger/bounce.py
cp /tmp/pip-build-root/trigger/conf/trigger_settings.py /etc/trigger/settings.py
cp /tmp/pip-build-root/trigger/conf/autoacl.py /etc/trigger/autoacl.py
cp /tmp/pip-build-root/trigger/tests/data/tackf /etc/trigger/.tackf
##create a few dirs that you may need
mkdir -p /data/firewalls
mkdir -p /data/tftproot
## now you need to edit your /etc/trigger/settings.py and set your DB type and details
## run init_task_db to populate your DB with the basic tables that you'll need

Don't forget, this is for my benefit as well. Now I'll have this post to look back on to see how I did this before.

Securely backup a Cisco Firewall (ASA & FWSM)

I have a problem with Cisco. Let me put it simply in this list:

1. Cisco does not support SSH keys (at least not very well) across its fleet of firewalls and security devices.

2. Config backups: TFTP is convenient but NOT secure. FTP is convenient but NOT secure. HTTP can be convenient but still NOT secure. HTTPS can be secure but can be unwieldy to setup and support. SCP is pretty convenient on a *nix machine, and is definitely secure, but there is a catch.

3. SCP on Cisco security devices: While these security devices usually do support SCP, they cannot SCP their configs to a destination (from the security device itself). Also, you cannot pull the running-config directly via SCP.

Addressing point #1 - using a password in an expect script is not extremely secure, but it beats the alternatives in almost every case. There are ways to secure strings (like passwords) in expect but you can decide for yourself whether these are needed.

Addressing point #3 - SCP can be used in the following way: Using expect, SSH to the device and save the running-config to a predefined location (root of the disk) and logout. Open a second shell and SCP the saved config back (from predefined location on device to predefined location on expect host).

My script isn't the shining example of a secure backup, but I'm unimpressed with the alternatives to this.

Here is an example of my firewall_backup.exp:

#!/usr/bin/expect

#######################################################

# Written by paklids 2012 01 12 #

# Expect script to securely backup Cisco ASAs & FWSMs #

# using SSH and SCP. #

# #

# This script can be run under a wrapper shell script #

# and then used (if desired) to checkin the firewall #

# configurations to SVN or any other change managment #

# or archiving solution - I will post my wrapper once #

# properly sanitized. #

#######################################################

#

##set an overall timeout if individual commands hang

set timeout 15

#

##set the directory you'd like the script to work out of

set -- /home/user/*

#

##open a pre-built list of IPs/FQDN hostnames delimited by line returns

##Note: it may be necessary to SSH to each host manually as serviceaccountname

##to save SSH host key

set f [open "/home/user/firewall_list"]

#

##for each entry in firewall_list, SSH to host and move to enable

##Note: enable may not be required if you customize the permissions

##on the serviceaccountname

while {[gets $f line] != -1} {

set pw1 "password_1"

set pw2 "password_2"

spawn "/bin/bash"

send "ssh serviceaccountname@$line\r"

expect "\npassword:"

send "$pw1\r"

expect -re "(\[^\r]*)>" {

set name $expect_out(1,string)

}

send "enable\r"

expect "\nPassword:"

send "$pw2\r"

expect "#"

##determine whether this is an FWSM or an ASA and appropriately save

##running-config to disk

send "sho version\r"

expect {

"FWSM" {

send \x20

set devtype "FWSM"

expect "#"

send "copy /noconfirm running-config disk:/$name\r"

}

"Adaptive Security Appliance" {

send \x20

set devtype "ASA"

expect "#"

send "copy /noconfirm running-config disk0:/$name\r"

}

}

expect "#"

sleep 2

send "logout\r"

#

##open a new bash shell and SCP the config that had been saved to disk

##back to the host this script is running on

spawn "/bin/bash"

send "scp serviceaccountname@$line:$name /home/user/workdir/$name\r"

expect -re ".*password:.*"

send "$pw1\r"

expect "100%"

sleep 1

}

#

Old BBQ = water rocket launcher!

P.S. I'll be adding more build pics as I upload them....

So a friend was having a "water rocket party" and we, as dads, were going to build water rocket launchers. It wasn't a competition, per se, but rather a way to have enough launchers that even if one or two failed, every kid would have a chance to launch their rocket.

His idea was to take a bunch of 2 litre bottles and spray paint them white, then allow the kids to use Sharpie markers, duct tape and other craft items to customize (decorate) their rockets. Great fun, because every kid has a chance to do their own thing and yet have fun with the other kids, and a good chance of getting really wet when we launched the rockets.

During my design phase, I researched a bunch of launchers on the internet. Trust me when I tell you that there are alot of different designs to choose from. They all were practical and easy to build, but they lacked 2 things:

1. They were not built very "heavy duty". They were built to be used a few times and then discarded or rebuilt.
2. They were slow to load. Most of the designs required pouring the water into the bottle prior to being put on the launcher, then required connecting the compressed air and pressurizing.

My problem was that kids wanted to enjoy this, so it had to handle a certain amount of abuse. It had to be quick to load, because kids aren't typically known for having a great deal of patience.

When I was in high school shop class, we built this monstrosity of a water rocket launcher, with a welded steel frame and plumbed water and air feeds. It was great for a launching alot of rockets quickly and certainly was tough, but I didn't have the resources to build something like that in my garage.

My solution?

Retrofit my old BBQ to be used as a water rocket launcher! I'd use the BBQ as a frame and then use common hardware store items to create a "simple to use" water rocket launcher.

First, I separated the BBQ from the rest of the BBQ parts. Then, I used a drill and bolts to connect one square pipe running end to end.

Next, I confirmed that an outer square pipe fit around that first square pipe. This second pipe (they call it C channel?) would be my sliding mechanism and eventually act as a release for the bottle.

Next, I drilled a hole in the inner square pipe and installed a T fitting upside down. One input would be water, one would be air and the output (pointed up) would be the bottle. When building this T piece, I took a 2L bottle with me to the hardware store (in my case Lowes) to make sure what I was building actually fit. This would now act as my launch mount.

I bolted the launch mount loosely in place. At the hardware store, I purchased 2 ball valves, for controlling the input of both the water and the air, as well as some air hose (which would handle the necessary pressure) and connect my inputs to my launch mount.

Adding a bit more control, I installed a pressure gauge just beyond the input air valve so that I could measure the air pressure I'd be adding to the system.

Placing the bottle on my launch mount, I took an old metal tool and trimmed it with a hacksaw to act as a fork that would be able to hold and lock the bottle onto the launch mount. Using a carefully crafted piece of 2X4 wood, I was able to attach the fork onto the sliding mechanism using wire ties (AKA zip ties). I now had a working lock and release and decent inputs.

A few wire ties later (to attach the ball valve inputs to the frame) and I had a working mock up that I could actually demonstrate.



Please note: If you attempt anything like this own, you do so at your own risk. For example, the pressures involved can be unsafe for the 2L bottle, not to mention if you accidentally pressurize your home's water system. Understand the risks before proceeding with anything.

After I tested the mockup, I decided to change my launch mount so that it withstood slightly more pressure. After a trip to Lowes & a helpful employee named Vern, I used a few additional PVC fittings and a stack of rubber O rings and was able to get a pretty decent seal on the bottle. This required me using a taller 2X4 wood piece.

In the end, I tore everything down and painted the frame with some good Rustoleum paint (hammerlite finish to hide the imperfections). Putting it back together, I used wire ties to secure the air hose to the frame and used the right size of hose clamps to hold the input valves to the frame, the fork to the sliding mechanism, and each air hose on its respective fitting. Two more hose clamps to connect a handle to the sliding mechanism and I was ready! And I didn't forget to leave an air compressor quick connect so that we could easily connect and disconnect the compressor.

Shameless plug: Tire repair system that I think rocks:

Gryyp Emergency Puncture Tire Repair Systems

ARM Toolchain for libspmp8000

Here is a toolchain builder that I edited from http://www.gnuarm.com that will help people build an ARM toolchain to start building programs for the spmp8000 series devices like the JXD1000 JXD2000 and any of those types of devices made by Suncom. This is designed on and built for Ubuntu 10.04 LTS but should work on newer versions of Fedora (sorry...haven't tested that yet).

All directions are found within the script.

#!/bin/sh

## Script largely based on gnu-arm-installer but customized for building a toolchain for the smpm8000
## script customized by Paklids June 14 2010
## *** for future be aware of the versions of source archives used (like gcc and binutils) and try to match those as closely as possible to the versions running natively on your build machine ***

## Prerequisites
## download and install these to your distribution (based on Ubuntu for now)
## 1. apt-get install build-essential
## 2. apt-get install libpng3 libpng12-dev libncurses5-dev
## 3. apt-get install subversion
## (hope there isn't enything else I'm missing...)


## Steps to build toolchain
## 1. create a parent directory for your toolchain (using mkdir) and cd into it
## 2. create 3 subdirectories 'src' 'build' & 'install'
## 3. Download these source archives into the 'src' subdirectory but don't unpack
## ftp://ftp.gnu.org/gnu/gcc/gcc-4.4.3/gcc-4.4.3.tar.bz2
## ftp://ftp.gnu.org/gnu/binutils/binutils-2.20.1.tar.bz2
## ftp://sources.redhat.com/pub/newlib/newlib-1.18.0.tar.gz
## ftp://sourceware.org/pub/insight/releases/insight-6.8.tar.bz2
## 4. move this script to the toolchain parent directory & make this script executable (using something like chmod u+x ./custom-arm-toolchain)
## 5. Launch script & let it run for a long while
## 6. Make sure the ?/parent-dir/install/bin is in your $PATH. do an 'echo $PATH', then try an 'export PATH=/fullpathto/parent-dir/install/bin:$PATH', then test again using 'echo $PATH'
## 7. After this you should be able to issue the command 'arm-elf-gcc' and it should NOT complain about 'command not found'.


ROOT=`pwd`
SRCDIR=$ROOT/src
BUILDDIR=$ROOT/build
PREFIX=$ROOT/install

GCC_SRC=gcc-4.4.3.tar.bz2
GCC_VERSION=4.4.3
GCC_DIR=gcc-$GCC_VERSION

BINUTILS_SRC=binutils-2.20.1.tar.bz2
BINUTILS_VERSION=2.20.1
BINUTILS_DIR=binutils-$BINUTILS_VERSION

NEWLIB_SRC=newlib-1.18.0.tar.gz
NEWLIB_VERSION=1.18.0
NEWLIB_DIR=newlib-$NEWLIB_VERSION

INSIGHT_SRC=insight-6.8.tar.bz2
INSIGHT_VERSION=6.8
INSIGHT_DIR=insight-$INSIGHT_VERSION

echo "I will build an arm-elf cross-compiler:

Prefix: $PREFIX
Sources: $SRCDIR
Build files: $BUILDDIR

Press ^C now if you do NOT want to do this."
read IGNORE

#
# Helper functions.
#
unpack_source()
{
(
cd $SRCDIR
ARCHIVE_SUFFIX=${1##*.}
if [ "$ARCHIVE_SUFFIX" = "gz" ]; then
tar zxvf $1
elif [ "$ARCHIVE_SUFFIX" = "bz2" ]; then
tar jxvf $1
else
echo "Unknown archive format for $1"
exit 1
fi
)
}

# Create all the directories we need.
#mkdir -p $SRCDIR $BUILDDIR $PREFIX

(
cd $SRCDIR

# Unpack the sources.
unpack_source $(basename $GCC_SRC)
unpack_source $(basename $BINUTILS_SRC)
unpack_source $(basename $NEWLIB_SRC)
unpack_source $(basename $INSIGHT_SRC)
)

# Set the PATH to include the binaries we're going to build.
OLD_PATH=$PATH
export PATH=$PREFIX/bin:$PATH

#
# Stage 1: Build binutils
#
(
(
# autoconf check.
cd $SRCDIR/$BINUTILS_DIR

) || exit 1

# Now, build it.
mkdir -p $BUILDDIR/$BINUTILS_DIR
cd $BUILDDIR/$BINUTILS_DIR

$SRCDIR/$BINUTILS_DIR/configure --target=arm-elf --prefix=$PREFIX --disable-werror && make all install

) || exit 1

#
# Stage 2: Patch the GCC multilib rules, then build the gcc compiler only
#
(
MULTILIB_CONFIG=$SRCDIR/$GCC_DIR/gcc/config/arm/t-arm-elf

echo "

MULTILIB_OPTIONS += mno-thumb-interwork/mthumb-interwork
MULTILIB_DIRNAMES += normal interwork

" >> $MULTILIB_CONFIG

mkdir -p $BUILDDIR/$GCC_DIR
cd $BUILDDIR/$GCC_DIR

$SRCDIR/$GCC_DIR/configure --target=arm-elf --prefix=$PREFIX \
--enable-languages="c" --with-gnu-as --with-gnu-ld --with-cpu=arm926ej-s \
--disable-nls --disable-werror --disable-libssp --disable-libgomp --disable-libmudflap --disable-werror && make all-gcc install-gcc

) || exit 1

#
# Stage 3: Build and install newlib
#
(
(
# Same issue, we have to patch to support makeinfo >= 4.11.
cd $SRCDIR/$NEWLIB_DIR

) || exit 1

# And now we can build it.
mkdir -p $BUILDDIR/$NEWLIB_DIR
cd $BUILDDIR/$NEWLIB_DIR

$SRCDIR/$NEWLIB_DIR/configure --target=arm-elf --prefix=$PREFIX --disable-newlib-supplied-syscalls --disable-werror && make all install

) || exit 1

#
# Stage 4: Build and install the rest of GCC.
#
(
cd $BUILDDIR/$GCC_DIR
echo "*** Re-CONFIGUREing GCC ***"
$SRCDIR/$GCC_DIR/configure --target=arm-elf --prefix=$PREFIX \
--enable-languages="c,c++" --with-gnu-as --with-gnu-ld --with-cpu=arm926ej-s \
--disable-nls --disable-werror --disable-libssp --disable-libgomp \
--disable-libmudflap --with-newlib --with-headers=$SRCDIR/$NEWLIB_DIR/newlib/libc/include
sleep 60
echo "*** Re-MAKEing GCC ***"
make
make install
sleep 60

) || exit 1

#
# Stage 5: Build and install INSIGHT.
#

(
# Now, build it.
mkdir -p $BUILDDIR/$INSIGHT_DIR
cd $BUILDDIR/$INSIGHT_DIR

$SRCDIR/$INSIGHT_DIR/configure --target=arm-elf --prefix=$PREFIX --disable-werror && make all install

) || exit 1


export PATH=$OLD_PATH

echo "
Build complete! Add $PREFIX/bin to your PATH to make arm-elf-gcc and friends
accessible directly.
"