Correction: AppEngine Datastore and owned relationships

Last week, I mistakenly wrote that AppEngine Datastore would not retrieve Key fields (representing an unowned relationship) unless you annotated with defaultFetchGroup=”true”, but that it would automatically retrieve properties represented by an owned relationship. This was not correct, as AppEngine Datastore never allows joins. I’ve updated the original post accordingly.

 

AppEngine JDO tip regarding detachCopy()

In order to avoid JDO and/or serialization exceptions, I always call some form of persistenceManager.detachCopy() before returning objects from the Datastore back to GWT. I discovered today that you must set any non-persistent properties of the object after calling detachCopy(). My initial inclination is that detachCopy() does not copy properties annotated with @NotPersistent, but I have to do a little more testing…

How to backup or move Picasa albums

I’ve been very happily managing my photos with Picasa since version 2. Many machines ago, I began storing photos along with all my other portable data in a directory separate from the normal Windows user folder to make it easy to back up and move files between machines with an external hard drive. So far so good.

The problem is how to back up and/or move Picasa albums to a new machine, as these are not stored with your photo files like Picasa edits and captions are. Picasa 3 now offers a way to export the entire photo database; however, it is very slow and results in a very large file (multi-GB) when you have 10 years’ worth of digital photos. It’s not practical to export the entire database to make a weekly backup, so I was delighted to find this technique on a forum which allows you to move just your album data:

Restore Picasa albums

I’ve done it many times and it works. This way, you can continue to use an incremental backup solution for photo files and backup the tiny amount of data in C:\Users\username\AppData\Local\Google\Picasa2Albums\ to save your albums. The only catch is that when you rebuild the Picasa database as instructed in the link above, the photos referenced in the album must be in the same directory location as they were when the album was created.

In order to ensure that my photos stay in the same directory structure even when Microsoft changes the location of the Users folder between, say, XP and Vista, I keep all my photos in a directory outside the Windows user directory (I use C:\@My\Photos). When I get a new machine, all I have to do is:

1. Copy the whole C:\@My directory to the new machine
2. Install Picasa
3. Copy the Picasa2Albums directory into place under the Users folder (C:\Users\David\AppData\Local\Google\Picasa2Albums\ in Vista)
4. Rebuild the Picasa database as instructed above using the Shift+Ctrl+Alt combination.

Of course, this technique may not be supported forever, and hopefully won’t be needed as Google continues to improve Picasa. But for now, it allows me to backup my albums on a regular basis and know that I’ll be able to move my photos, albums and all, to a new machine when needed.

http://picasa.google.com/

Key fields are not retrieved by default

I’ve noticed that AppEngine does not populate Key fields by default when querying. I’m sure it’s documented, but I can’t remember where. At any rate, you can annotate with defaultFetchGroup to resolve this. Same goes for properties of type Text and Blob.

...
import com.google.appengine.api.datastore.Key;

@PersistenceCapable(identityType = IdentityType.APPLICATION)
public class PrayerItem implements Serializable
{
	@PrimaryKey
	@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
	private Long id;
	@Persistent(defaultFetchGroup="true")
	private Key list;
	...

}

In the case of owned relationships, where a property is represented by its actual domain class rather than a Key, it is also necessary to use defaultFetchGroup=”true”. Even though owned properties are in the same entity group and are therefore eligible for inclusion in a transaction, the AppEngine Datastore doesn’t allow joins, so owned relationships are not returned in the initial query. When you set defaultFetchGroup=”true” on an owned property, you will get this warning:

WARNING: Meta-data warning for com…your_owned_property_name: The datastore does not support joins and therefore cannot honor requests to place child objects in the default fetch group.  The field will be fetched lazily on first access.  You can modify this warning by setting the datanucleus.appengine.ignorableMetaDataBehavior property in your config.  A value of NONE will silence the warning.  A value of ERROR will turn the warning into an exception.

Lazy fetching does work as described in the message.

Eclipse keyboard shortcut for working with interfaces

Maybe everyone knows this already, but in case not…

Put your cursor in the name of an interface (say, a Display interface nested in a GWT presenter) and press Ctrl-T. Eclipse shows you all the classes that implement the interface and you can then navigate directly to an implementing class. I frequently use this in combination with Ctrl+Alt+H to find out how a method gets called.

You can find all of my favorite keyboard shortcuts in my Eclipse Google notebook listed in the left sidebar.

Free Web conferencing for Windows, Linux, and Mac

Yuuguu.com

I recently needed to share my Windows desktop with a Linux user over the Internet. It took only minutes to install the Yuuguu client and start sharing my screen. Remote viewers don’t need to install anything, as there is a Web viewer. But if the remote participants do install the Yuuguu client, they can also request control of the presenter’s screen.

Yuuguu also gives you a telephone conference line with global dial-in numbers. It is a free service in the US, although you still have to dial long distance.

Yuuguu lets you chat across several instant messaging networks. And it works on Windows, Linux, and Mac.

Funny name, great service!

AppEngine JDO example #1: retrieve an object by ID

This is the first of a series of posts that will demonstrate a variety of JDO mappings and queries with AppEngine.

For starters, let’s retrieve an object from the Datastore given its ID. Here’s the User object in our domain model:

package com.roa.client.domain;

import java.io.Serializable;

import javax.jdo.annotations.IdGeneratorStrategy;
import javax.jdo.annotations.IdentityType;
import javax.jdo.annotations.PersistenceCapable;
import javax.jdo.annotations.Persistent;
import javax.jdo.annotations.PrimaryKey;

@PersistenceCapable(identityType = IdentityType.APPLICATION)
public class User implements Serializable
{
	private static final long serialVersionUID = -1126191336687818754L;

	@PrimaryKey
	@Persistent(valueStrategy = IdGeneratorStrategy.IDENTITY)
	private Long id;
	@Persistent
	private String firstName;
	...
}

Now we want to retrieve a User object given the ID property obtained from getId(). Easy enough, you think, as the JDO PersistenceManager provides a method for this:

// Doesn't quite work
User attachedUser = (User) getPM().getObjectById(userId);

But there’s a catch. Confusingly, the ID needed by getObjectById is a JDO ID, not the object property named “id” that we’ve annotated as an IDENTITY. You can get the JDO ID one of two ways. Whenever the object is attached, such as just after you’ve called makePersistent() on it, you can call persistenceManager.getObjectId(obj) and keep it in a safe place for later use. Alternatively, you can use a JDO helper method at any time to get the JDO ID from the object ID:

		 // Must obtain the JDO ID first
		 Object idInstance = getPM().newObjectIdInstance(User.class, user.getId());
		 User attachedUser = (User) getPM().getObjectById(idInstance);

Of course, you can always just run a normal query on the object’s ID property, but it’s a bit ugly because the simplest query syntax always returns a Collection and we assume here that there will be only one matching object.

		Query q = getPM().newQuery(User.class, "id == :userId");
		List<User> users = (List<User>) q.execute(u.getId());
		User attachedUser = users.get(0);

Better is to let the query language know that we expect a unique object, but then we have to write out the whole query and declare our imports one way or another.

		Query q = getPM().newQuery("SELECT UNIQUE FROM com.roa.client.domain.User WHERE id == :userId");
		User attachedUser = (User) q.execute(u.getId());

All things considered, calling the JDO helper method to get a JDO ID from an object ID may be the least painful.

Notes from CloudCamp

There were about 100 folks at CloudCamp Atlanta last night. Participatns were mostly interested in Amazon EC2 or the upcoming Windows Azure platform (Microsoft was the primary sponsor). A handful of folks turned out for my “show me the code” presentation on AppEngine and GWT+MVP.

My main takeaway is that I’m really, really glad to be a Google AppEngine user. Most of the sessions and discussions centered around how to grow/shrink your Amazon server pool automatically and how to scale MySQL. These are things I don’t have to worry about at all as an AppEngine user. AppEngine is perfect for my application: launching a one-man startup that could go viral (not to mention it’s free to get started!) I’m just plain tired of sysadmin, and I’m willing to trade the  freedom to run any configuration of my choice for the scalability that comes with AppEngine’s constraints.

There was some discussion regarding the cloud and SLAs (or lack thereof). Someone said they needed five nines, not three. I don’t know where they got either number, but .999 reliability is definitely good enough for me. I have a consumer-facing app that doesn’t involve money, and I’ll be ecstatic if I have only 8 hrs 46 minutes downtime in the coming year. It’s clear that cloud offerings (especially AppEngine) appeal more to lone developers and small startups than to enterprises, although Amazon seems to have made great inroads in this area. It’s always going to be easier to start in the cloud than to move an existing data center. What excites me most about AppEngine and the like is that ONE developer can realistically create and deploy a scalable application. This has the potential to unleash a whole new wave of entrepreneurial energy on the Internet, and will make it possible for smaller companies that could never afford a data center to nevertheless get custom software for their needs. Which means we software engineers should get to do more coding and less sysadmin. Yay!

Other takeaways:

Lots of folks were happily using RightScale to manage Amazon EC2 instances and automatically grow or shrink the server pool based on load.

IBM provides Amazon machine images of its popular products like WebSphere FREE for development use (other than Amazon charges, of course).

How to clean up muddy water

in Photoshop, that is…

1. Select the muddy water using the quick selection tool.
2. Create a new Hue/Saturation adjustment layer using the selection as the mask.
3. In the Hue/Saturation dialog, select the Yellow channel.
4. Set saturation way down and luminance way up.
5. Now people will think your photo was taken in the pristine waters of the Rockies instead of Georgia…

 

 

CloudCamp Atlanta tonight

I’m planning to do a short unconference session on securing AppEngine services with gwt-dispatch and unit testing with AppEngine and gwt-dispatch at tonight’s CloudCamp Atlanta.

I’m looking forward to meeting some local AppEngine developers, as I’ve been feeling awfully close to the bleeding edge lately. I routinely find that solutions have been posted on AppEngine forums just eight days ago, and sometimes don’t exist yet. I really need TaskQueue in order to send out emails, which is still experimental in Labs, and would even more like to use deferred.defer (less than two weeks old), but alas, it’s currently only available for Python.