TurboManage

David Chandler's Journal of Java Web and Mobile Development

  • David M. Chandler


    Web app developer since 1994 and former Developer Advocate with Google now residing in Colorado. Besides tech, I enjoy landscape photography and share my work at ColoradoPhoto.gallery.

  • Subscribe

  • Enter your email address to subscribe to this blog and receive notifications of new posts by email.

    Join 242 other followers

  • Sleepless Nights…

    March 2010
    S M T W T F S
    « Feb   Apr »
     123456
    78910111213
    14151617181920
    21222324252627
    28293031  
  • Blog Stats

    • 849,257 hits

A recipe for unit testing AppEngine task queues

Posted by David Chandler on March 3, 2010

One of the difficulties of testing AppEngine code has been the problem of running unit tests for code written against the Task Queue API. AppEngine provided no easy way to programatically control the task queue engine. In addition, tasks are servlets, so it’s not particularly easy to invoke them with the correct task payload in a unit test.

Fortunately, AppEngine 1.3.1 provides test helpers that greatly simplify writing unit tests against the Datastore and Task Queue APIs (as well as Memcache, Blobstore, and others). By default, the Datastore is configured with the NO_STORAGE property set to true (the default) so that each test method starts with a blank datastore. Also by default, the task queuing auto-run capability is disabled, which gives you the opportunity to programatically run a task once you have verified that it’s been enqueued properly.

The following BaseTest class initializes the Datastore and TaskQueue services. Once the AppEngine team fixes the bug that requires the LocalServerEnvironment override, it will be very simple, indeed. You can ignore the Guice injector stuff if you’re not using Guice.

BaseTest.java

package com.roa.test.helper;

import java.io.File;

import junit.framework.TestCase;

import com.google.appengine.tools.development.LocalServerEnvironment;
import com.google.appengine.tools.development.testing.LocalDatastoreServiceTestConfig;
import com.google.appengine.tools.development.testing.LocalServiceTestHelper;
import com.google.appengine.tools.development.testing.LocalTaskQueueTestConfig;
import com.google.inject.Guice;
import com.google.inject.Injector;
import com.roa.server.guice.ServerModule;
import com.turbomanage.gwt.server.guice.DispatchTestModule;

/**
 * A simple test base class that can be extended to build unit tests that
 * properly construct and tear down AppEngine test environments.
 */
public abstract class BaseTest extends TestCase
{

	private static Injector inj;
	protected LocalServiceTestHelper helper;
	private LocalDatastoreServiceTestConfig datastoreConfig;
	private LocalTaskQueueTestConfig taskQueueConfig;

	/**
	 * Sets up the AppEngine environment and initializes Guice.
	 */
	@Override
	protected void setUp() throws Exception
	{
		super.setUp();
		datastoreConfig = new LocalDatastoreServiceTestConfig();
		taskQueueConfig = new LocalTaskQueueTestConfig();
		helper = new LocalServiceTestHelper(datastoreConfig, taskQueueConfig)
		{
			// Temp workaround until 1.3.2 to help task queue find war/WEB-INF/queue.xml
			@Override
			protected LocalServerEnvironment newLocalServerEnvironment()
			{
				final LocalServerEnvironment lse = super.newLocalServerEnvironment();
				return new LocalServerEnvironment()
				{
					public File getAppDir()
					{
						return new File("war");
					}

					public String getAddress()
					{
						return lse.getAddress();
					}

					public int getPort()
					{
						return lse.getPort();
					}

					public void waitForServerToStart() throws InterruptedException
					{
						lse.waitForServerToStart();
					}
				};
			}
		};

		helper.setEnvAuthDomain("auth");
		helper.setEnvEmail("test@example.com");
		helper.setEnvIsAdmin(true);
		helper.setEnvIsLoggedIn(true);
		helper.setUp();

		inj = Guice.createInjector(new ServerModule(), new DispatchTestModule());
	}

	@Override
	protected void runTest() throws Throwable
	{
		super.runTest();
	}

	/**
	 * Deconstructs the AppEngine environment.
	 */
	@Override
	protected void tearDown() throws Exception
	{
		helper.tearDown();
		super.tearDown();
	}

	/**
	 * Provide Guice injector to tests
	 */
	protected static Injector getInj()
	{
		return inj;
	}

}

Code under test will now have access to the Datastore and the TaskQueue API just as in dev and production environments.

The new task queue helper (LocalTaskQueueTestConfig) provides methods to inspect the task queue and verify task creation. When you call runTask(), the task queue test service will attempt to invoke the servlet associated with your task; however, unless you can figure out a way to run your unit test in a servlet container, your task servlet won’t actually be reachable by the runTask() call (at least, I’m assuming this is why I’m seeing “connection refused” errors).

Fortunately, we can easily simulate what the AppEngine task queue does by invoking the task servlet using ServletUnit. The test case below creates a simple test case using Vince Bonfanti’s Deferred task servlet, verifies that it has been enqueued properly, and then invokes it using ServletUnit.

package com.roa.test;

import java.io.ByteArrayInputStream;
import java.io.IOException;
import java.io.Serializable;
import java.util.Map;

import javax.servlet.ServletException;

import org.xml.sax.SAXException;

import com.google.appengine.api.labs.taskqueue.dev.LocalTaskQueue;
import com.google.appengine.api.labs.taskqueue.dev.QueueStateInfo;
import com.google.appengine.api.labs.taskqueue.dev.QueueStateInfo.TaskStateInfo;
import com.google.appengine.api.users.User;
import com.google.appengine.api.users.UserServiceFactory;
import com.google.appengine.tools.development.testing.LocalTaskQueueTestConfig;
import com.meterware.httpunit.PostMethodWebRequest;
import com.meterware.servletunit.ServletRunner;
import com.meterware.servletunit.ServletUnitClient;
import com.newatlanta.appengine.taskqueue.Deferred;
import com.newatlanta.appengine.taskqueue.Deferred.Deferrable;
import com.roa.test.helper.BaseTest;

public class SimpleTaskTest extends BaseTest implements Serializable
{

	@Override
	protected void setUp() throws Exception
	{
		super.setUp();
		// Do additional setup here
	}

	public class HelloUserTask implements Deferrable
	{
		private User user;

		public HelloUserTask(User u)
		{
			this.user = u;
		}

		@Override
		public void doTask() throws ServletException, IOException
		{
			System.out.println("Hello, " + user.getNickname());
		}
	}

	public void testTaskInvocation() throws IOException, SAXException
	{
		User testUser = UserServiceFactory.getUserService().getCurrentUser();
		HelloUserTask task = new HelloUserTask(testUser);
		// Queue the task
		Deferred.defer(task);

		// Verify that one task has been enqueued
		LocalTaskQueue taskQueue = LocalTaskQueueTestConfig.getLocalTaskQueue();
		Map<String, QueueStateInfo> allQueues = taskQueue.getQueueStateInfo();
		QueueStateInfo deferredQueue = allQueues.get("deferred");
		assertEquals(1, deferredQueue.getCountTasks());
		TaskStateInfo taskInfo = deferredQueue.getTaskInfo().get(0);
		String queuedTask = taskInfo.getBody();
		String taskName = taskInfo.getTaskName();

		// Run task will fail because no servlet container running
		// taskQueue.runTask("deferred", taskName);

		// Run using ServletUnit instead
		ServletRunner sr = new ServletRunner();
		sr.registerServlet("/_ah/queue/deferred", "com.newatlanta.appengine.taskqueue.Deferred");
		ServletUnitClient client = sr.newClient();
		client.sendRequest(new PostMethodWebRequest("http:/_ah/queue/deferred",
			new ByteArrayInputStream(queuedTask.getBytes()), null));

		// Flush task queue (removes all tasks)
		taskQueue.flushQueue("deferred");
	}

}

When you run this, you’ll see “Hello, test@example.com” written to the console by the task.

Using the AppEngine test helpers and ServletUnit, we now have an easy way to verify that tasks have been queued as expected AND a way to invoke the task servlets in the same test case.

Happy testing!

13 Responses to “A recipe for unit testing AppEngine task queues”

  1. Max Ross (Google) said

    Nice post, I’m really happy to see LocalServiceTestHelper being put to such good use!

  2. I’ve been looking for something like this for the Python environment on GAE, but came up with nothing. Can you suggest anything?

  3. Brendan Doherty said

    When I run each of the tests individually they pass, but running as part of a test suite, all tests that use Task Queue except the first one fails.

    I’m using Deferrable tasks, and the first time the code in the second tests tries to add a task to the queue I get a TransactionalTaskException, which contains a DatastoreFailureException with the message “handle 1 not found”;

    My test cases extend this class shown at http://pastebin.com/bhuf9FdQ, based on the code here.
    Perhaps I’m setting up the Local…TestConfig’s incorrectly.

    • Hard to say without looking at code under test, but possibly one of your tests is expecting data from another test to be persisted? This won’t work as JUnit calls setUp / tearDown run for each test method and the LocalDatastoreTestConfig, by default, wipes all data at teardown.

      • Brendan said

        I tracked down the problem. I think it’s a problem in the appengine code (with an easy workaround).

        In some of my tests I was calling routines that created tasks in the deferred queue, but I wasn’t calling my processDeferredQueue function, as the test wasn’t checking the output of the tasks.

        The problem appears to be that the LocalServiceTestHelper::tearDown() wasn’t properly cleaning up the task queue (even though I think it should), and so the next test would fall over as soon as it tried to add a task to the queue.

        The workaround was just to add the following call to my tearDown.

        LocalTaskQueueTestConfig.getLocalTaskQueue().flushQueue(DEFERRED);

      • super, thanks for posting back

  4. ГОСТ said

    Thank you. I did’t know about defer

  5. Brendan Doherty said

    Now that appengine 1.4.3 provides DeferredTask, do you know what we need to do to be able to test it?

    I can’t use RunTask as it still fails because no servlet container is running, and I’m not sure what class to specify as the second parameter to registerServlet.

  6. Answer here: use taskInfo.getBodyAsBytes() instead of taskInfo.getBody()

    http://groups.google.com/group/google-appengine-java/msg/b57d325db50ca1c5

    Thanks, Brendan! 🙂

  7. Chris Ritter said

    Is this code considered Apache Licensed?

    Thanks, Chris

  8. M.S.Naidu said

    Thank you for your great post.

    If i want to reuse the same database(LocalDatastoreServiceTestConfig ) with other LocalServiceTestHelper, what we have to do?

    Thanks in advance

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

 
%d bloggers like this: