Clirr is a tool that checks Java libraries for binary compatibility with older releases. Basically you give it two sets of jar files and Clirr dumps out a list of changes in the public api.