Getting Started with Java APT for Android in Eclipse

Fellow Googler Ian Ni-Lewis and I have recently been working on annotation-driven code generation tools for Android and I wanted to pass along a few tech notes on getting it all up and running. The tool I’m working on is a lightweight ORM for the SQLite database in the Android OS. In a nutshell, you add the @Entity to any POJO that you want to make persistable, and the annotation processor generates a corresponding DAO class and the necessary SQLiteOpenHelper.

In this post, I won’t focus on the ORM tool as it’s not quite ready, but rather on how to work with Eclipse to build an annotation processor of your own (or contribute to the ORM project once I make it available). Despite the annoyances of the Mirror API, I found it reasonably straightforward and rewarding to write my own annotation processor.

What can APT do?

Java 5 introduced an Annotation Processing Tool (apt) which can be run standalone. Eclipse 3.3 added support for annotation processing as part of the Java build process. In a nutshell, an annotation processor can find annotations in your source code, inspect the code using something like reflection, and generate new source code as a result. In addition, an annotation processor can throw errors which will show up as red squigglies in Eclipse in the original annotated source code. This is extremely powerful as it allows your annotation processor to alert the developer regarding the correct use of the annotation. For example, when you tag a POJO with @Entity, my forthcoming ORM tool shows you immediately if any field types in the POJO are unsupported.

What’s in an annotation processor?

1. The definition of your annotations using @interface.
2. A class that extends AbstractProcessor, part of the Java 6 annotation processing APIs. One processor class can process multiple annotations.
3. Code that inspects the annotated source code using the Mirror API.
4. META-INF/services/javax.annotation.processing.Processor, which contains merely the fully-qualified name of your AbstractProcessor class, like “com.example.storm.MainProcessor”.

How to run your annotation processor

To use your annotation processor, package it in a jar. There are several ways to run it:

1. On the command line, run Java apt. In Java 6 and later, javac will automatically search the classpath for annotation processors. You can bypass the default discovery mechanism using the -processor and -processorpath compiler options. You can also include your own AnnotationProcessorFactory to replace the default factory.
2. In ant, add the -processorPath and -processor options or use the ant apt task (pre-Java 6).
3. In Eclipse, add your annotation processor jar to the annotation factory classpath as described in the Eclipse guide to APT .

While the Eclipse JDT-APT integration is cool as it enables editor integration (hello, red squigglies), there is one limitation that makes it a pain to work on the annotation processor code itself. Whether by design or bug, when you make changes to your annotation processor, you must repackage as a jar and manually remove / add it to the annotation factory classpath of any projects using it. The packaging can be easily automated as I will show, but I haven’t yet found a way to get Eclipse to pick up changes to apt jars, even when they’re referenced as jars in an Eclipse project (vs. external jars).

Getting Started

During development, I found it useful to organize the annotation processor code into 3 separate projects: API, impl, and test.

The API project

The API project contains the @interface definitions for your annotations. Since this code will need to be on the classpath of any project that uses your annotations, it’s reasonable to also include other non-generated code that will be used at runtime. For example, my API project includes a DAO base class and other runtime code as well as the annotations. Since some of the runtime code has Android dependencies, I made this an Android library project by checking the Library box when running the New Android Application Project wizard. The project has no Activities, and the AndroidManifest.xml contains only the <uses-sdk> element.

As we noted earlier, the only way to run an annotation processor in Eclipse is from a jar or plugin. Since the API is required by the annotation processor, it must exist as a jar on the annotation factory classpath of any client projects such as the test project below. Therefore, it’s helpful to build the jar every time you make a change to the project. Eclipse does not do this for you automatically. However, you can set it up using ant as follows:

Once you’ve created the API project and configured the build path as required, export an ant build file using File | Export | Ant Buildfiles.

Edit the build.xml and add this inside the build-project target:

<jar destfile="your-api.jar" basedir="bin/classes"></jar>

Go to project properties | Builders and add a New Ant Builder. In the Main tab, click Browse Workspace… to choose the build.xml file, then finish. The default targets are OK. Now every time you save a change to the API, you’ll see it build your-api.jar in the project.

The APT implementation project

The impl project includes one or more AbstractProcessors, supporting model classes, and the META-INF file that specifies the name of your processor class. The APT impl project has no Android dependencies because the Android references exist only in the Freemarker templates. This project has on its build path the API project. You could depend on the API jar since it’s getting updated by your ant task, but depending on the project will facilitate refactoring. As for the API project, you’ll want to use an Ant Builder to update the impl jar every time the project is built.

My annotation processor uses the Freemarker template language to generate source code. The basic approach is to inspect the annotated code, populate a model class (like EntityModel) which simply has getters and setters for the fields that will be needed by the template, then invoke Freemarker, passing it the model and the template. This is my annotation processor:

package com.example.storm;

import java.io.IOException;
import java.io.Writer;
import java.util.ArrayList;
import java.util.List;
import java.util.Set;

import javax.annotation.processing.AbstractProcessor;
import javax.annotation.processing.RoundEnvironment;
import javax.annotation.processing.SupportedAnnotationTypes;
import javax.annotation.processing.SupportedSourceVersion;
import javax.lang.model.SourceVersion;
import javax.lang.model.element.Element;
import javax.lang.model.element.TypeElement;
import javax.tools.JavaFileObject;

import freemarker.cache.ClassTemplateLoader;
import freemarker.template.Configuration;
import freemarker.template.Template;
import freemarker.template.TemplateException;

@SupportedAnnotationTypes({ "com.example.storm.Entity","com.example.storm.Database" })
@SupportedSourceVersion(SourceVersion.RELEASE_6)
public class MainProcessor extends AbstractProcessor {
	private ProcessorLogger logger;
	private Configuration cfg = new Configuration();
	private List entities = new ArrayList();

	@Override
	public boolean process(Set<? extends TypeElement> annotations,
			RoundEnvironment roundEnv) {
		this.logger = new ProcessorLogger(processingEnv.getMessager());
		logger.info("Running MainProcessor...");

		cfg.setTemplateLoader(new ClassTemplateLoader(this.getClass(), "/res"));

//		for (TypeElement annotationType : annotations) {}

		for (Element element : roundEnv.getElementsAnnotatedWith(Entity.class)) {
			Entity annotation = element.getAnnotation(Entity.class);
			EntityModel em = new EntityModel(element);
			processTemplateForModel(em);
			// retain for DatabaseHelper class
			entities.add(em);
		}

		for (Element element : roundEnv.getElementsAnnotatedWith(Database.class)) {
			Database dbAnno = element.getAnnotation(Database.class);
			DatabaseModel dm = new DatabaseModel(element, dbAnno.name(), dbAnno.version());
			// Add entity DAOs
			for (EntityModel em : entities) {
				dm.addDaoClass(em.getQualifiedClassName());
			}
			processTemplateForModel(dm);
		}

		return true;
	}

	private void processTemplateForModel(ClassModel model) {
		JavaFileObject file;
		try {
			file = processingEnv.getFiler().createSourceFile(model.getQualifiedClassName());
			logger.info("Creating file  " + file.getName());
			Writer out = file.openWriter();
			Template t = cfg.getTemplate(model.getTemplatePath());
			logger.info("Processing template " + t.getName());
			t.process(model, out);
			out.flush();
			out.close();
		} catch (IOException e) {
			logger.error("EntityProcessor error", e, model.getElement());
		} catch (TemplateException e) {
			logger.error("EntityProcessor error", e, model.getElement());
		}
	}

}

The hardest part about using Freemarker was locating the templates on the classpath. I found it is easiest to use Freemarker’s ClassTemplateLoader and put my .ftl templates in src/res. The name happens to coincide with the standard Android resources folder; however, this is not required as the impl project is not an Android project. Furthermore, my res folder is under src, not a sibling directory as the standard Android folder. This is the easiest way for ClassTemplateLoader to find it.

Note the catch clauses in processTemplateForModel() that invoke logger.error(). Messages written to the ProcessorLogger show up in Eclipse’s Error Log view. In addition, by passing the Element which triggered the error, Eclipse can generate red squigglies in source code using the annotation.

In my current design, EntityModel and DatabaseModel correspond to the @Entity and @Database annotations and extend ClassModel, which does the heavy lifting using the Mirror API. Here are the key methods from ClassModel which show how to introspect on an annotated class and its fields using the Mirror API:

	public ClassModel(Element element) {
		TypeElement typeElement = (TypeElement) element;
		this.typeElement = typeElement;
		readFields(typeElement);
	}

	protected void readFields(TypeElement type) {
		// Read fields from superclass if any
		TypeMirror superClass = type.getSuperclass();
		if (TypeKind.DECLARED.equals(superClass.getKind())) {
			DeclaredType superType = (DeclaredType) superClass;
			readFields((TypeElement) superType.asElement());
		}
		for (Element child : type.getEnclosedElements()) {
				if (child.getKind() == ElementKind.FIELD) {
					VariableElement field = (VariableElement) child;
					Set modifiers = field.getModifiers();
					if (!modifiers.contains(Modifier.TRANSIENT) && !modifiers.contains(Modifier.PRIVATE)) {
						String javaType = getFieldType(field);
						addField(field.getSimpleName().toString(), javaType);
					}
				}
		}
	}

	protected String getFieldType(VariableElement field) {
		TypeMirror fieldType = field.asType();
		return fieldType.toString();
	}

I’m not completely happy with this design and plan to move all the mirror API code into processor classes, leaving the model classes pure. This should enable even more fine-grained error reporting. Here’s a bit of my Freemarker template src/res/EntityDao.ftl. All variable names inside ${} invoke the corresponding getters in the model class passed to the process() method above.

public class ${className} extends ${baseDaoClass}<${entityName}>{

	public static void onCreate(SQLiteDatabase db) {
		String sqlStmt =
			"CREATE TABLE ${tableName}(" +
				<#list fields as field>
				"${field.colName} ${field.sqlType}<#if field.primitive> NOT NULL<#if field_has_next>," +
				</#list>
			")";
		db.execSQL(sqlStmt);
	}
	...
}

The test project

The test project is an Android Application Project which has on its build path the API project. However, because the API project is an Android Library project, you must add it as a referenced library under project properties | Android, not the usual Java Build Path. In addition, you must add 3 jars to the annotation factory classpath in Eclipse: the Freemarker jar, the API jar, and the impl jar.

There are several ways to invoke Android unit tests. I found it easiest to include test instrumentation directly in an Android Application Project created using the Application Project wizard (not the Android Test Project wizard, as that requires yet another project under test). It has the default MainActivity, and you can simply add the test instrumentation to AndroidManifest.xml as follows:

    <instrumentation
        android:name="android.test.InstrumentationTestRunner"
        android:targetPackage="com.example.storm.test" />

    <application
        android:icon="@drawable/ic_launcher"
        android:label="@string/app_name"
        android:theme="@style/AppTheme" >
        <activity
            android:name=".MainActivity"
            android:label="@string/title_activity_main" >
            <intent-filter>
                <action android:name="android.intent.action.MAIN" />
                <category android:name="android.intent.category.LAUNCHER" />
            </intent-filter>
        </activity>

        <uses-library android:name="android.test.runner" />
    </application>

The activity should already be there; you just need to add the uses-library tag and the instrumentation tag pointing to the package which contains your Junit 3 tests (note: Android doesn’t use Junit 4). To create tests, simply run the New Junit Test wizard. To run them, right-click on the project and select Run As | Android Junit Test. This will run the test on an emulator or device and show results in the standard JUnit console.

Wrapping up

Creating an annotation processor has been a pretty fun learning project, and I trust the final product will be useful to a lot of Android developers. The most painful part is having to manually remove and re-add the jars from the test project’s annotation factory classpath every time I make a change to one of the jars. I have to go to project properties | Java Compiler > Annotation Processing > Factory Path, uncheck the boxes next to the api and impl jars, click OK and rebuild, then go back in to recheck the boxes and rebuild. Doesn’t take long with keyboard shortcuts, but still… I’ll be glad when then the ORM jars are stable, and (I hope), so will you.