Autocode

Automation with Python

Anthony Briggs

Intrepid Travel

Melbourne Python User Group

Automation with Python

Why automate?

Get more stuff done

Do more interesting things

Get more free time - automate the things which are keeping you busy

Increase the number of things you can do without effort

The ultimate example

Remapping your keyboard layout with an evolutionary algorithm

http://www.visi.com/~pmk/evolved.txt

. u y p q k l d c g Best evolved layout

e a i n w r h t s o 9640479

' , ; f z j m v b x

(dvorak was 12189785, qwerty 25390660)

Why Python?

I started out automating with Bash...

Python is better than bash because...

Python has many, many libraries - far more than bash or command.com

Python is more consistent in syntax

quick! how do bash if/then statements work? for loops?

Bash task

Given a bunch of zip files, extract them all into their own directories:

for ITEM in `ls` ; do mkdir $ITEM-dir ; cd $ITEM-dir ; tar -xzf ../$ITEM ; cd .. ; done

A python version

#!/usr/bin/python

# Given a bunch of zip files,

# extract them all into their own directories:

# for ITEM in `ls` ; do mkdir $ITEM ; cd $ITEM ; \

# tar -xzf ../$ITEM ; cd .. ; done

import os, sys

files = sys.argv[1:]

for tarName in files: # eg. foo.tar.gz

dirName = tarName.split('.')[0]

os.mkdir( dirName )

os.system( 'cd %s' % dirName )

os.system( 'tar -xzf ../%s' % tarName )

os.system( 'cd ..' )

Bash continued...

Python gives you a better path from quick hack to production script

Or even from quick hack to slightly slower hack

Add unit tests, refactor, create objects

Slap on a GUI with wxWindows

Programming styles

Automation works well with particular programming styles:

Extreme programming - simplest thing that could possibly work

Bottom up - particularly under Linux where you can accumulate scripts

Protoyping

Flamebait Warning

Python is better than Perl because...

Very easy to understand 6 months later

Easy for python novices to understand and maintain

Easier for python novices to write maintainable code

Still need to comment your code

Types of automation

Creating

Wrapping

Iterating

Summarising

Creating

eg. test directory data

Example:

A test script that I created when I thought I was going to write a cvs replacement in python

Test Repository

def _createSubDir( self, numberOfFiles, path ):

''' Create a subdirectory with numberOfFiles files	and directories in it. '''

# create the files for the directory

# wig out if we've finished

if numberOfFiles <= 0:

return

# randomly create a number of subdirectories

# and assign a random number of files to each

# randomly assign amounts to each dir

Test Repository

# recurse for each directory

for index, amount in enumerate(numDirArray):

dirName = self._randomFileName(path)

os.mkdir(path+'/'+dirName)

if amount != 0:

self._createSubDir( amount,

path+'/'+dirName )

Wrapping

Making complex applications (or series of applications) easier to use

Example:

merging remote repository changes with mergegap

mergegap

#!/usr/local/bin/python

import sys, os

try:

revNumber = int(sys.argv[1])

except:

print "invalid revision Number"

sys.exit(1)

try:

commit = sys.argv[2]

except:

print "invalid commit message"

sys.exit(1)

mergegap

command = "svn merge -r%s:%s https://altundev1.gap.ca/svn/ projects/compass/releases/1.12" % (revNumber-1, revNumber)

os.system( command )

commitMsg = "Merged r%s (%s) from GAP" % (revNumber, commit)

command = "svn commit -m \"%s\"" % commitMsg

os.system( command )

Iterating

Over directories and files

creating html galleries

searching for files

acting on a large number of files at once

Example:

Directory Checker

Directory Checker

# Quick script to compare two directories

import os

dir1 = "E:\My Pictures"

dir2 = "S:\photos"

offset1 = len(dir1)

offset2 = len(dir2)

laptop = [item for item in os.walk(dir1)]

server = [item for item in os.walk(dir2)]

laptop.sort()

server.sort()

#assert len(laptop) == len(server)

Directory Checker

for index, item in enumerate(laptop):

laptopName = laptop[index][0]

serverName = server[index][0]

if laptopName[offset1:] != serverName[offset2:]:

print "Directory names don't match:", \

laptopName, "vs.", serverName

continue

print laptopName, "matches OK"

# now check the files within the directory

files1 = laptop[index][2]

files1.sort()

files2 = server[index][2]

files2.sort()

if files1 != files2:

print "Directory files don't match:", \

files1, "vs.", files2

Summarising/Parsing/Collating (Figuring out stuff)

Polling

a 'control panel' (eg. web page, simple GUI app) which tells you status/what's broken

Interrupts

eg. an email/SMS/paging script

Collect information

from log files

Examples:

Mango development

keeping an eye on remote development with monitor.py

Parsing unit test output

I inherited a unit test suite with a large number of failures

Crunch time meant that the remote development team got slack with keeping tests up to date

Mission: you need to fix unit tests as quickly as possible

Unit test output looks like this:

Loading tests from name [test.ctrl.test_cascading]

.E..

======================================================================

ERROR: testMoreComplicatedAddingSubtrip (test.ctrl.test_child_management.TestChild)

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/anthony/dev/repositories/trunk/test/ctrl/test_child_management.py", line 146, in testMoreComplicatedAddingSubtrip

assert parent.addPaxToUnit( parent.units[0], [ pax.id for pax in b1.contacts ] )

IndexError: list index out of range

----------------------------------------------------------------------

Ran 4 tests in 4.373s

FAILED (errors=1)

Loading tests from name [test.ctrl.test_child_management]

.....

----------------------------------------------------------------------

Ran 5 tests in 0.429s

OK

And some more...

Loading tests from name [test.ctrl.test_company]

....

----------------------------------------------------------------------

Ran 4 tests in 1.833s

OK

Loading tests from name [test.ctrl.test_company_fee]

...F........

======================================================================

FAIL: testFailedPassport (test.ctrl.test_contact.TestContact)

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/anthony/dev/repositories/trunk/test/ctrl/test_contact.py", line 259, in testFailedPassport

self.assertEquals( page.getLastStatus().className(), "Fail" )

File "src/compass/util/unittest.py", line 303, in failUnlessEqual

raise self.failureException, \

AssertionError: 'Success' != 'Fail'

----------------------------------------------------------------------

Ran 12 tests in 6.442s

FAILED (failures=1)

Loading tests from name [test.ctrl.test_contact]

..

----------------------------------------------------------------------

Ran 2 tests in 0.129s

I've got about 100k of this...

OK

Loading tests from name [test.ctrl.test_contact_booking]

E

======================================================================

ERROR: testSearchOptimize (test.ctrl.test_contact_finder.TestContactFinder)

----------------------------------------------------------------------

Traceback (most recent call last):

File "/home/anthony/dev/repositories/trunk/test/ctrl/test_contact_finder.py", line 21, in testSearchOptimize

page.doSearch()

File "src/compass/web/mockweb.py", line 259, in __getattr__

raise AttributeError("Mock '%s' does not have attribute '%s'" % (self.pageName, name) )

AttributeError: Mock 'ContactFinderPage' does not have attribute 'doSearch'

----------------------------------------------------------------------

Ran 1 test in 0.287s

FAILED (errors=1)

Loading tests from name [test.ctrl.test_contact_finder]

.

----------------------------------------------------------------------

Ran 1 test in 0.370s

So where do I start?

Obviously not by scanning through and picking out what looks 'good' – 100k of logs will send me even further over the edge

Parse and summarise

The solution

#!/usr/local/bin/python

# Find out which test suites are the worst in terms of time and/or failures

import sys

testInput = sys.stdin.readlines()

testResults = {}

longestKey = 0

time = None

errors = None

for line in testInput:

if line.startswith('OK'):

errors = 0

if line.startswith('FAILED '): # FAILED (errors=5)

errors = line.strip().split('=')[-1][:-1]

errors = int( errors )

if line.startswith('Ran '): # Ran 7 tests in 19.508s

time = float( line.strip().split()[-1][:-1] )

# Loading tests from name [test.test_employee]

if line.startswith("Loading tests from name "):

testName = line.strip().split()[-1]

testName = testName[1:-1]

if len(testName) > longestKey:

longestKey = len(testName)

testResults[testName] = (errors, time)

outputList = [ key for key in testResults.keys() if key.startswith('test.') ]

outputList.sort( cmp = lambda x,y: testResults[y][0] - testResults[x][0] )

print "The 20 worst tests in terms of failures:"

for item in outputList[:20]:

print item.ljust(longestKey),'\t', \t

testResults[item][0], '\t',

testResults[item][1]

The output

The 20 worst tests in terms of failures:

test.ctrl.test_invoice	7	4.149

test.ctrl.test_security	6	0.143

test.test_mta	5	0.022

test.test_employee	5	0.249

...

The 20 worst tests in terms of time:

test.test_late_change	104.066	0

test.ctrl.test_triptemplate	78.318	3

test.ctrl.test_refunds	52.933	1

test.ctrl.test_payment	47.025	0

...

This now happens at 5am every day

Finding errors in web pages

We get an XML feed for availability

Availability *must* be right, or else we have very unhappy passengers

You could check it by hand and make a note of the dodgy ones, or...

Write some code!

#!/usr/local/bin/python

# A quick hack to check that the trip pages work on the intrepid web site

# looking for a line like: "XML error at line n. 1897 - undefined entity"

import os

import cookielib, urllib2

debug = False

def makeOpener():

" return an opener with http_auth and cookie handling "

# create a cookie jar

cj = cookielib.CookieJar()

# create an auth_handler and set username and password

auth_handler = urllib2.HTTPBasicAuthHandler()

auth_handler.add_password('Expressive Interface', 'http://www.intrepidtravel.com/', 'username', 'passwd')

# create the actual opener

opener = urllib2.build_opener(auth_handler, urllib2.HTTPCookieProcessor(cj))

# go to cart-test page to set cart_on PHP session var

r = opener.open("http://www.intrepidtravel.com/cart-test/startcart.php?cart_on=yes")

return opener

def getGAPCodes():

""" return a list of GAP codes, pulled from gapcodes.txt,

with the \n's chopped off the end """

return [ code.strip() for code in \

file('gapcodes.txt','r').readlines() ]

def checkCode( tripCode, opener ):

""" Does the page for a trip code render properly? """

# Load page from intrepidtravel.com; save as page.html

url = "http://www.intrepidtravel.com/trip.php?”

“code=%s&debug=yes" % tripCode

html = opener.open( url ).readlines()

# looking for "XML error at line n. 1897..."

errorLines = [line for line in html \

if 'XML error at line' in line]

return errorLines == []

if __name__ == "__main__":

opener = makeOpener()

gapcodes = getGAPCodes()

#print gapcodes

for gapcode in gapcodes:

passed = checkCode( gapcode, opener )

if not passed:

print "*** Code %s failed! ***" % gapcode

else:

if debug:

print "Code %s is ok..." % gapcode

Platforms

Linux/UNIX

You can do pretty much anything that you would do normally through a command line...

...especially if you also know other UNIX tools - grep, sed, awk, bash, etc.

Short scripts are better - ~/bin files are indistinguishable from system commands

"secret scripts" - cory/boingboing/lifehacks

Linux Examples:

~/bin/count - used for adding up the number of unit tests - tied in with a pipe

#!/usr/local/bin/python

# given a series of numbers, one per line, return the total

import sys

sum = 0

for line in sys.stdin.readlines():

try:

amount = int( line.strip() )

sum += amount

#print amount, sum

except ValueError:

pass # a non-integer line -- ignore

print sum

converting line endings

#!/usr/bin/env python

# Convert files to UNIX encoding

import sys, os

for fileName in sys.argv[1:]:

if os.access(fileName, os.F_OK) != 1:

print "%s: no such file" % fileName

contents = file(fileName, 'r')

input = contents.readlines()

contents.close()

output = []

for line in input:

output.append( line.rstrip() +'\n' )

replace = file(fileName, 'w+')

replace.writelines( output )

replace.close()

Windows

Harder to automate, since a lot of Windows apps are GUI-based

Some still have command line options though: "foo.exe /?" should tell you

Win32 extensions for Python

I'm not really a Windows-head, but I think you can automate Outlook and IE (with COM?), and do pretty much everything you can do manually

Windows

Can you replace Word macros?

Unixutils is pretty good if you're a UNIX fanboy - grep, ls, cp, mv, etc. Available from sourceforge

Example: Directory Checker

# Quick script to compare two directories

import os

dir1 = "E:\My Pictures"

dir2 = "S:\photos"

offset1 = len(dir1)

offset2 = len(dir2)

laptop = [item for item in os.walk(dir1)]

server = [item for item in os.walk(dir2)]

laptop.sort()

server.sort()

#assert len(laptop) == len(server)

for index, item in enumerate(laptop):

if laptop[index][0][offset1:] != server[index][0][offset2:]:

print "Directory names don't match:", laptop[index][0], "vs.", server[index][0]

continue

print laptop[index][0], "matches OK"

# now check the files within the directory

files1 = laptop[index][2]

files1.sort()

files2 = server[index][2]

files2.sort()

if files1 != files2:

print "Directory files don't match:", files1, "vs.", files2

Heuristics of Automation

ie. General rules of thumb, not hard and fast algorithms

When to automate

When not to automate

When to automate

Pretty much any time you're bored

Any time there's a simple series of repetitive steps

Whenever something important needs to be done

(I tend to forget things)

When you need to get real work done!

When not to automate

Haven't quite nailed down the process

Runs are different each time

Script needs slight alterations that aren't easy to figure out

When there's still 'fiddling' to be done

Automation Tools

Cron

os.system - get all of the unix tools

config files & little languages

when your program starts getting larger

just import a python configuration file

Automation Tools

Parsing

dictionary lookup – python has functions as 1st class objects

Web stuff - cookies and geturl

Iterating over directories

recursion – Test Directory Creator

os.walk – Directory Checker

Crazy Ideas

I hate paperwork

it should be possible to use a web-scraping script to automate your bank accounts and paste them into a spreadsheet for budgeting or allocations

what about a bank with an RSS feed?

Crazy Ideas

Perl/PyNomic – automating laws

A web script where players vote on modifications (ie. patches) to the running system

www.perlnomic.org

www.venomic.net

http://pn.jgb.ca/forum/index.php

2005-05-26T20:18:11