Automated GUI Testing
One of the buggest deals that people in my office seem to make about XP is the emphasis on Testing. (We could care less about Pair Programming, Iteration driven stuff, and the way they do story management, none of us including me really get disagree that what we have in those areas seems to work). But from all of the reading most of the supprot I have seen is foducsing on either object components or the lowest of the gui portions if even that. For the longest time I have been trying to find a good GUI test too I can integrate with the build, I think I found a hidden gem. But let's look at what's out there and why I didn't use it...
- Abbot - Serious GUI bug in 1.4, that pane on the right you are supposed to put stuff like classpath and class name in? Doesn't work, won't remember the values or persist it to the model. Hence it is unusable for record.
- Pounder - I forgot exactly why, but it behaved poorly for me out of the box. I seem to recall it used java.awt.Robot to do alot of the work, which would have been good or bad depending on what you are trying to test. But the use on Linux and the fact component sized change frequently would have been bad news for it. It also takes over the computer to take over the test and does it in real time, good or bad based on how you look at it. The other problem is that it records every stinking possible awt event that exists! Too much clean up on the test when you are done.
- JFCUnit - The lack of a record option is a killer here. If I'm not going to design the grid bag container by hand I sure am not going to write the test cases by hand. Especialy since some of the GUI is dynamicaly generated and the structure of those components beneath that pane are difficult to derive by hand. Yea, there is aFAQ item that involved wriring your own but it took less effort to keep on looking
- Jemmy - (a) Lack of JUnit Integration, and (b) Lack of a record option, see my entry on JFCUnit. (a) might not be such a big deal if (b) wasn't there and (c) I had infinite time.
- Any commercial test suite - I don't know if anyone noticed out there but there is a recession going on. The cost for a autometed GUI test suite to use on the scale we want to (every developer that checks in code) it would cost us at leas one head count. I don't want to see any more of my colleagues leave, we've already gotten rid of the stinkers (and some of the good ones). That's why I'm being cheap about it.
Well, that seems to be the extent of the GUI test tools out there, and they've all got there blockers that encourage me to look elsewhere. Or is that all of them? In doing a google search on "Java GUI Test Tools" I stumbeled upon the mention of a java-gui-testing group on Yahoo Groups, where I learned about the hidden gem I previously mentioned:
- Marathon - not perfect but Good Enough. Record option, junit playback, was able to get it working with my detateched test harness. Dosen't record random key strokes like I would like it to, but it referes to the JComponents principally by the setName(...) value you gave to it. It uses Jython to record the test and play back the test script. The API for the types of tests it has is a little limited, but the whole set of Java APIs are exposed so you can do whatever you need to inside of the jython script, so to add some custom valdiation logic you don't have to write a whole new stinking java class. And it integrates into my JUnit setup, with the tests showing up in the junitreport.
Good Enough, and much better than what I had before. What I had before was my memory and intuition of what I thought would break. But that tends to be unreliable under time pressure. Doing the same tests over and over again, and when it's automated you don't have to worry about the Intern or Script Monkey (which is not the same as a QA person) finding a better job!
Update : Glen posted about this sunday, and I learned that after I wrote this. So I didn't get to "out" marathon in the java.blogs community. But I felt my selection reasons and some more exposure would be good for marathon.