Delete Old Entities – Java Datastore

This is an ultra-simplified example of how to delete old entities from the App Engine Datastore. The first 3 lines of code retrieves the current date, then subtracts 60 days from the current time (the multiplication converts days to milliseconds). DATE_PROPERTY_ON_ENTITY is the date property on the entity – when first writing the entity to the datastore, add the current date as a property. ENTITY_KIND is the entity kind we’re deleting.

		//Calculate 60 days ago.
		long current_date_long = (new Date()).getTime();
		long past_date_long = current_date_long - (1000 * 60 * 60 * 24 * 60);
		Date past_date = new Date(past_date_long);
		
		DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();
		Query.Filter date_filter = new Query.FilterPredicate("DATE_PROPERTY_ON_ENTITY", Query.FilterOperator.LESS_THAN_OR_EQUAL, past_date);
		Query date_query = new Query("ENTITY_KIND").setFilter(date_filter);
		PreparedQuery date_query_results = datastore.prepare(date_query);
		
		Iterator<Entity> iterate_over_old_entities = date_query_results.asIterator();
		
		while (iterate_over_old_entities.hasNext()) {
			Entity old_entity = iterate_over_old_entities.next();
			
			System.out.println("Deleting: " + old_entity.getProperties());
			
			datastore.delete(old_entity.getKey());
		}

Note that is a simplified function – it’s useful if you have a handful of entities that need deleting, but if you have more than a handful, you should convert to using datastore cursors and paging through entities to delete.

DatastoreFailureException: Internal Error Exception Messages

On rare occasions, an application may spawn errors similar to the below:

com.google.appengine.api.datastore.DatastoreFailureException: 
    internal error.
at com.google.appengine.api.datastore.DatastoreApiHelper.translateError     (DatastoreApiHelper.java:50)
at com.google.appengine.api.datastore.DatastoreApiHelper$1.convertException     (DatastoreApiHelper.java:70)

As the exception message suggests, this log indicates that the datastore encountered an internal error while handling the datastore operation. Unfortunately, there’s nothing that a developer can do to fix this error since it’s an internal App Engine issue.

Generally this type of exception fixes itself sooner or later; if it persists, file an issue at the App Engine bug tracker: https://code.google.com/p/googleappengine/issues/entry?template=Production%20issue

Datastore Query Cursor: ExQ

While using JDO or another datastore helper library, applications may encounter a value of ExQ when requesting a query cursor. This value means that there are no more results (entities) to display for that query, and consequently that there are no more query pages to return.

All applications using query cursors should check for an invalid or end of query results marker, and handle this in an application-specific way. A query cursor string that is null or empty (zero length) generally denotes the end of query results. The cursor value ExQ can be handled in the same way.

A word of warning: some third-party datastore helper libraries spawn an exception or error condition when applications attempt to access invalid cursors, or attempt to request a cursor when end of results has already been reached. When using these libraries, it’s important to fully read the documentation and understand how these libraries interact with the App Engine datastore.

Counting Entities In The App Engine Datastore.

In SQL, the COUNT statement is available for counting the number of rows returned by a SELECT query. Here’s an example:

SELECT COUNT(*) FROM foods WHERE name="apple";

Unfortunately, the App Engine datastore offers no equivalent to the COUNT statement. To count the number of entities, an application has two options:

  1. Count the entities in a separate counter as they’re added to the datastore. For superior performance, the counter can be sharded and temporarily stored in memcache.
  2. Query for entities and count the number of entities returned. To help speed up this query, a keys-only request can be configured. However, this option can be slow and expensive.

Unable to find property ‘application’ on class: IndexYamlReader

A quick note for today: some developers are experiencing the below error while attempting to vacuum an application’s indexes:

Error Details:
Line 0, column 10: Unable to find property 'application' on class:
com.google.apphosting.utils.config.IndexYamlReader
.
com.google.appengine.tools.admin.AdminException: Unable to perform vacuum_indexes
    at com.google.appengine.tools.admin.AppAdminImpl.vacuumIndexes(AppAdminImpl.java:346)
    at com.google.appengine.tools.admin.AppCfg$VacuumIndexesAction.execute(AppCfg.java:1605)
    at com.google.appengine.tools.admin.AppCfg.executeAction(AppCfg.java:327)
    at com.google.appengine.tools.admin.AppCfg.<init>(AppCfg.java:210)
    at com.google.appengine.tools.admin.AppCfg.<init>(AppCfg.java:121)
    at com.google.appengine.tools.admin.AppCfg.main(AppCfg.java:117)

Google seems to have changed the index backend during the 1.8.6 App Engine release, and this is causing older SDK versions to break when they attempt to issue the vacuum indexes command. To fix this, update your SDK version to the latest available.

Retrieving All Entities Older Than An Arbitrary Date

Here’s a Java code example to search the datastore for all entities within a kind older than a given date.

The variable kind is the entity kind being searched, add_date is a property on each entity that is set to the date the entity was created, and entities is a java.util.List object containing the returned entities. The variable time_point represents a point in time; we query the datastore for all entities with a date less than that.

/**
 * Retrieve all entities older than a set amount of time.
 */
Query q = new Query(kind);
//Represents a point in time 48 hours ago.
Date time_point = new Date((new Date().getTime()) - (1000 * 60 * 60 * 48));
Query.Filter time_point_filter = new Query.FilterPredicate("add_date", Query.FilterOperator.LESS_THAN_OR_EQUAL, time_point);
q.setFilter(time_point_filter);
PreparedQuery pq = DatastoreServiceFactory.getDatastoreService().prepare(q);
List<Entity> entities = pq.asList(FetchOptions.Builder.withLimit(30));
System.out.println(entities.size() + " entities returned.");

Suppose you wanted to loop through all of the returned entities. Here’s an example:

//Loop through all entities
for (int i = 0; i < entities.size(); i++) {
    Entity entity = entities.get(i);
    System.out.println("Entity: " + entity.toString());
    //Do something with the entity variable.
}//end loop going through all entities

Querying The Datastore In Golang

Here’s a demonstration of how to query the datastore in Go.

In this example we filter on PropertyOne, requiring it to be equal to true. You can also set other inequalities such as greater than ( > ). Kind is the kind of the entities to query, and PropertyTwodemonstrates ordering by descending order. CustomStruct is the struct that was used to create the entity. Remember to put your entity processing code just before the last brace ( } ).

//Search the datastore for entities
q := datastore.NewQuery(kind).Filter("PropertyOne =", true).Order("-PropertyTwo")
//Loop through each returned entity.
for t := q.Run(c); ; {
    //This represents the entity currently being processed.
    var x CustomStruct
    key, err := t.Next(&x)
    if err == datastore.Done {
        //This "error" means that we're done going through entities
        //Since we're done, break out of this loop.
        break
    }
    if err != nil {
        //Some other error happened. Report error and panic.
        c.Infof("Error in querying datastore: %v", err)
        panic(err)
    }
    //DO SOMETHING HERE WITH ENTITY x
}//end for going through q.Run

Oops! We Couldn’t Retrieve Your List Of Kinds.

Occasionally you may see the error Oops! We couldn’t retrieve your list of kinds from the datastore viewer screen of the App Engine admin console:

Generally this is a transient error: it essentially means that the App Engine admin console is currently too busy to show a view of your datastore’s contents. Wait a few minutes and refresh the page, your datastore’s information should appear.

Seeing this error can also mean that the datastore is empty; for example, if it’s a just-created application.

Deleting An Entity In Golang

Here’s a short code example demonstrating how to delete an entity in Go. C is an appengine.Contextreference, kind is the entity kind, and entity_id_int represents the integer ID of the entity.

key := datastore.NewKey(c, kind, "", entity_id_int, nil)
datastore.Delete(c, key)

Retrieving An Entity By ID In Go

Here is a code snippet demonstrating how to retrieve a datastore entity by a known kind and integer ID. This code also retrieves the entity ID from a HTTP parameter.

R represents a http.Request reference. The entity ID to retrieve is accessed from the user’s submitted form under the name entity_idKind represents a string containing the name of the kind to retrieve from the datastore. The retrieved entity is stored into the variable entity, which is the struct CustomStruct.

c := appengine.NewContext(r)
//Retrieve the entity ID from the submitted form. Convert to an int64
entity_id := r.FormValue("entity_id")
entity_id_int, err := strconv.ParseInt(entity_id, 10, 64) 
if err != nil {
    fmt.Fprint(w, "Unable to parse key")
    return;
}
//We manufacture a datastore key based on the Kind and the 
//entity ID (passed to us via the HTTP request parameter.
key := datastore.NewKey(c, kind, "", entity_id_int, nil)
//Load the Entity this key represents.
//We have to state the variable first, then conduct the Get operation 
//so the datastore understands the struct representing the entity.
var entity CustomStruct
datastore.Get(c, key, &entity)

In short, this code pulls out the Entity ID from the HTTP request, converts it into an integer, manufactures the entity’s key using the kind and entity id, and then retrieves the corresponding entity. The code then translates the retrieved entity into the defined CustomStruct variable entity.