Tag: appengine

Posted on

HTTP POST URLFetch Using The Low Level Java API

Here’s a code snippet demonstrating how to generate a POST request using the low level urlfetch API.

The variables post_url and post_data represent the request URL and the content of the POST as strings. The response from the urlfetch is stored in response_content . If the request was unsuccessful (the target server returned a non-200 HTTP status code) a RuntimeException will be thrown.

HTTPRequest request = new HTTPRequest(new URL(post_url), HTTPMethod.POST);
request.setHeader(new HTTPHeader("User-Agent", "Example Application"));
request.setPayload(post_data.getBytes("UTF-8"));
HTTPResponse response = URLFetchServiceFactory.getURLFetchService().fetch(request);
//If the response wasn't successful, throw an exception.
if (response.getResponseCode() != 200) {
    throw new RuntimeException("Response code was not 200: " + response.getResponseCode());
}
String response_content = new String(response.getContent());

Remember to import the URLFetch library:

import com.google.appengine.api.urlfetch.*;
Posted on

CRLF In HTTP Response Headers

A quick note: App Engine will convert CRLF characters in headers (new line characters, such as /r and /n) into underscore characters.

For example, setting the following header should cause Line One and Line Two to appear on two different lines:

resp.setHeader("Example-Header-Name", "Line One \r\n Line Two");

App Engine will run this code, but it will convert the newline characters into two underscore characters before sending the header to the client browser.

While the HTTP 1.1 specification allows header fields to be split across multiple lines (RFC 2616 Section 4.2), App Engine apparently doesn’t allow this. It isn’t too much of an inconvenience, since response headers can be grouped up into one line, but it’s a good idea to keep this limitation in mind especially when communicating with APIs.

Posted on

Sleeping An Application In Java

Some applications need to pause execution for a short amount of time. In App Engine, there are two ways to do that.

For short pauses, most programming languages have the concept of sleep operations. Here’s how to sleep for 2,000 milliseconds (2 seconds) in Java:

try {
    Thread.sleep(2000);
}
catch (InterruptedException e) {
}

If you want to pause the application for a few seconds, using the sleep function is a good idea. But if you need a longer pause – more than a minute or so – it’s a better idea to use the Task Queue’s countdownMillis function.

Here’s how to create a task that will wait for 5 minutes, then launch off another request:

TaskOptions task = TaskOptions.Builder.withCountdownMillis(5 * 60 * 1000);

In short, whenever you need a long pause in execution, queue up a task with a countdownMillis() delay. Then the task can call a request handler which can continue any application logic needed.

Posted on

Favicon.ico In Errors Screen

Some applications may see the path /favicon.ico listed in the errors screen within the admin console. The error will look similar to the following image:

Browsers request favicon.ico to display in the address bar and as the icon in a bookmark. If no favicon is available, browsers will substitute a default icon – for example, Firefox displays an icon that looks like a blank page.

Setting a favicon isn’t required, but is useful for branding your web site. Once you create an icon, save it as favicon.ico in the root folder of your application.

Posted on

DeadlineExceededException at java.util.zip.ZipFile.open

During instance startup, large applications may see a DeadlineExceededException and the following stack trace:

Failed startup of context 
    com.google.apphosting.utils.jetty.RuntimeAppEngineWebAppContext
com.google.apphosting.api.DeadlineExceededException:
    This request started at (time) and was still executing at (time + 60 seconds).
at java.util.zip.ZipFile.open(Native Method)
at java.util.zip.ZipFile.<init>(ZipFile.java:227)
at java.util.zip.ZipFile.<init>(ZipFile.java:156)
at java.util.jar.JarFile.<init>(JarFile.java:153)
at java.util.jar.JarFile.<init>(JarFile.java:90)

This exception (including the reference to ZipFile.open ) means that App Engine was unable to extract the application’s JAR file within the 60 second request limit.

Generally, this means that the application is too large. Try to reduce the app’s size by removing unused code. Also, cut down on the size of included libraries by removing unused portions. Running the application through a code optimizer can also help.

This error can also appear during times when App Engine is experiencing an outage or is slow in general. In this case, the error will most likely resolve itself when App Engine returns to normal serving status.

Posted on

Extract All HTTP Headers In Java

Here’s a code snippet that extracts all HTTP request headers and loops through them.

The variable req represents a javax.servlet.http.HttpServletRequest reference, while header_name and header_value represent the name and value of the header. A single header name may have multiple values; if so, header_value will record the first value listed.

Enumeration<String> headers = req.getHeaderNames();
//Loop through all headers
while (headers.hasMoreElements()) {
    String header_name = headers.nextElement();
    String header_value = req.getHeader(header_name);
    //Do something with the header name and value
}//end while loop going through headers.

Remember that HTTP header names are case-insensitive. If you’re doing any comparisons, you may want to turn the name into a lowercase form:

String header_name_lowercase = header_name.toLowerCase();

Don’t forget to import the Enumeration class:

import java.util.Enumeration;
Posted on

Clearing Memcache

A quick note today: If you upload new data to the datastore via the bulk uploader or change your application’s data model, you should flush your application’s memcache to prevent stale data from being served to browsers. To do this, go to the Memcache Viewer screen (under the Data heading in the navigation bar) and press the button marked Flush Cache:

Posted on

Looking Up App Engine Issues

Found a bug within App Engine? Want to request a feature? The best way to notify Google is to file an issue within App Engine’s tracker, located at: https://code.google.com/p/googleappengine/issues/list .

You can file an issue by clicking the New Issue button in the top left corner of the page:

But before you file a new issue, make sure that someone else hasn’t already filed the same issue. You can search issues by using the search box on the upper portion of the page:

If you don’t want to search, you can browse through the issues by using the dropdown boxes in each header:

Here’s an example of what an issue looks like:

Posted on

Retrieving All Entities Older Than An Arbitrary Date

Here’s a Java code example to search the datastore for all entities within a kind older than a given date.

The variable kind is the entity kind being searched, add_date is a property on each entity that is set to the date the entity was created, and entities is a java.util.List object containing the returned entities. The variable time_point represents a point in time; we query the datastore for all entities with a date less than that.

/**
 * Retrieve all entities older than a set amount of time.
 */
Query q = new Query(kind);
//Represents a point in time 48 hours ago.
Date time_point = new Date((new Date().getTime()) - (1000 * 60 * 60 * 48));
Query.Filter time_point_filter = new Query.FilterPredicate("add_date", Query.FilterOperator.LESS_THAN_OR_EQUAL, time_point);
q.setFilter(time_point_filter);
PreparedQuery pq = DatastoreServiceFactory.getDatastoreService().prepare(q);
List<Entity> entities = pq.asList(FetchOptions.Builder.withLimit(30));
System.out.println(entities.size() + " entities returned.");

Suppose you wanted to loop through all of the returned entities. Here’s an example:

//Loop through all entities
for (int i = 0; i < entities.size(); i++) {
    Entity entity = entities.get(i);
    System.out.println("Entity: " + entity.toString());
    //Do something with the entity variable.
}//end loop going through all entities
Posted on

Hanging Memcache Calls

Recently there was a discussion in the App Engine forums about memcache calls that were hanging; in one instance, a memcache async put call was taking 2 hours to complete!

This was a particularly interesting issue, and I’d like to share a number of thoughts I had while solving it:

App Engine has a number of internal rate limiting/throttling controls on services. Moving large quantities of data around can quickly cause an application to hit these limits. In fact, I suspect that this was the actual problem – the original poster’s application was storing multiple megabytes of data into memcache in multiple asynchronous calls that occurred simultaneously; this design could easily be hitting a number of different rate limits. My suggestion for solving this problem (which ultimately worked) was to add a short delay after each memcache put call and to split the data amongst an increased number of memcache put calls. The reasons for which I suggested this fix are numerous:

  1. Adding a short delay after each memcache put call buys time for App Engine’s rate limit to reset; it prevents App Engine from thinking that the application is malfunctioning or attempting to overwhelm the memcache pipeline.
  2. Delays are easy to implement – in Python it’s one call to time.sleep(number of seconds to delay)and in Java it’s a simple call to Thread.sleep(number of seconds to delay). Note that in Java, you have to catch the potential InterruptedException. The Go call is similar to Python: call time.Sleep(delay duration). In PHP a delay is even simpler than in all of the above languages: all you need to do is call sleep(delay seconds).
  3. Increasing the number of memcache put calls means that a smaller amount of data is being stored for each memcache put. This contributes to point 1: preventing the pipeline to memcache from being overwhelmed with data.
  4. The delay doesn’t need to be long: two to five seconds is more than enough. In some cases, even a one second delay is enough to work.

Fortunately, the above fix worked in this case. But if it had not, I was prepared with a number of other possible fixes. For instance, I would have suggested the use of the task queue: split the data among multiple tasks, and then have each task store their data into memcache. Since each task would constitute a separate request and may be split amongst multiple instances, there’s less of a chance for any rate limiting to kick in. If that option wasn’t palatable for any reason, then another option would be to switch to dedicated memcache; it seems to be much more forgiving in regards to usage.

If none of the above options had worked, I would have suggested dumping memcache entirely and writing to the datastore/Cloud SQL. While memcache is a terrific service, it is not reliable – persisting the data through alternative sources is a much better way to manage large quantities of information.

The short version of this post: hanging or slow memcache calls can be fixed by inserting delays after each call and decreasing the amount of data handled in each memcache call.