App Engine Logos

Badge logos (sometimes called button logos) are used to advertise a website’s server software or platform. Here’s a few examples:

Apache Badge Logo

Apple Xserve:

PHP:

Personally, I always like seeing badge logos because they give an interesting insight into that website: which stack is powering the website, the preferred coding language of the webmaster, etc.

Here’s logos for App Engine:

For reference, these are the equivalent badge logos for Microsoft Azure and AWS:

 

Google Cloud Storage RetryHelper Failure

I have numerous applications which use Google Cloud Storage buckets to save files, maintain backups, etc. Most of these applications are Java apps using the official Google Cloud Storage library.

Yesterday one of these applications had difficulty accessing Cloud Storage and printed out the following error report:

The GCS library attempted to access Cloud Storage a total of 6 times; this image shows the last failure and the final “giving-up” error.

Here’s a copy of the error in text form:

com.google.appengine.tools.cloudstorage.RetryHelper doRetry: 
    RetryHelper(4.114 s, 6 attempts, com.google.appengine.tools.cloudstorage.GcsServiceImpl): 
    Attempt 6 failed, sleeping for 3140 ms: java.io.IOException: Response code 503, 
    retryable: Request: POST https://storage.googleapis.com/[bucket_url]/[file_name]
x-goog-resumable: start
x-goog-api-version: 2
Content-Type: image/jpg

no content

Response: 503 with 19 bytes of content
Content-Length: 19
Date: Thu, 01 Aug 2013 [time]
Server: HTTP Upload Server Built on Jul 21 2013 19:20:38 (1374459638)
Content-Type: text/html; charset=UTF-8
X-Google-Cache-Control: remote-fetch
Via: HTTP/1.1 GWA
Service Unavailable

And after 6 failures attempting to access the Google Cloud Storage bucket, the library reports an error and stops:

.<stderr>: IO Exception: RetryHelper(4.115 s, 6 attempts, 
    com.google.appengine.tools.cloudstorage.GcsServiceImpl): 
    Too many failures, giving up

Fortunately this seemed to be only a transient problem; the next upload to GCS succeeded.

Generate SHA1 Hash

Many web applications need to generate a SHA1 hash for verifying file integrity. If you use the Apache Commons libraries, the DigestUtils class offers convenience methods for generating hashes.

If you don’t use Apache Commons, the following function will generate SHA1 hashes for you. Simply pass the string to calculate the hash from into the function ( to_be_sha1 ); the function will then return the SHA1 hash. If an error is encountered (such as the passed-in string being null), a RuntimeException will be thrown.

/**
 * Calculates a SHA1 hash from a provided String. If 
 * to_be_sha1 is null, a RuntimeException will be thrown.
 * 
 * @param to_be_sha1 String to calculate a SHA1 hash from.
 * @return A SHA1 hash from the provided String.
 */
public static String generateSHA1(String to_be_sha1) {
    String sha1_sum = "";
    //if the provided String is null, throw an Exception.
    if (to_be_sha1 == null) {
        throw new RuntimeException("There is no String to calculate a SHA1 hash from.");
    }
    try {
        MessageDigest digest = MessageDigest.getInstance("SHA1");
        byte[] array = digest.digest(to_be_sha1.getBytes("UTF-8"));
        StringBuffer collector = new StringBuffer();
        for (int i = 0; i < array.length; i++) {
            collector.append(Integer.toString((array[i] & 0xff) + 0x100, 16).substring(1));
        }
        sha1_sum = collector.toString();
    }//end try
    catch (NoSuchAlgorithmException e) {
        throw new RuntimeException("Could not find a SHA1 instance: " + e.getMessage());
    }
    catch (UnsupportedEncodingException e) {
        throw new RuntimeException("Could not translate UTF-8: " + e.getMessage());
    }
    return sha1_sum;
}//end generateSHA1

Add these imports as well:

import java.io.UnsupportedEncodingException;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;

URL Path To Array In Golang

Here’s a short Go function that converts the URL path into an array; it’s useful if you want to implement REST style queries into a simple application.

func parseComponents(r *http.Request) []string {
    //Creates an App Engine Context - required to access App Engine specific services
    c := appengine.NewContext(r)
    //The URL that the user queried.
    path := r.URL.Path
    path = strings.TrimSpace(path)
    //Log the URL received:
    c.Infof("URL: ", path)
    //Cut off the leading and trailing forward slashes, if they exist.
    //This cuts off the leading forward slash.
    if strings.HasPrefix(path, "/") {
        path = path[1:]
    }
    //This cuts off the trailing forward slash.
    if strings.HasSuffix(path, "/") {
        cut_off_last_char_len := len(path) - 1
        path = path[:cut_off_last_char_len]
    }
    //We need to isolate the individual components of the path.
    components := strings.Split(path, "/")
    return components
}

Free Tier Performance

Two weeks ago I posted a picture of a free tier application serving close to 52,000 requests in 1 day. Soon after that, the same application exceeded that by processing over 53,000 requests within a single day. Here’s a few pictures demonstrating the capabilities of GAE’s free tier:

First off, here’s a picture of the Current Load screen:

A graph of milliseconds/request over the same time period:

The instances used:

And finally requests/second:

App Engine’s free tier is generally enough to host applications during development, testing, and low-level production traffic. With that said, it’s always a good idea to activate billing if you believe that your application will be receiving more than small amounts of traffic.

Reading In A File With Java

Here’s a short code example demonstrating how to read in a file within your application. This snippet assumes the file data.json is within your application’s root directory (change as necessary). The contents of the file are read into the variable json – insert your own processing code after the while code block. If the file cannot be found, or if there are other difficulties in reading from the file, an IOException will be thrown.

try {
    InputStream is = context.getResourceAsStream("/data.json");
    BufferedReader reader = new BufferedReader(new InputStreamReader(is));  
    String json = "";
    String line = "";   
    while (line != null) {
        json += line;
        line = reader.readLine();
    }   
    //Do something with the read-in data within the json variable.
}
catch (IOException e) {
    throw new RuntimeException("Unable to read in file.");
}

Inspecting Task Queue Logs

Here’s a sample view of a request from the Task Queue service. The log includes the name of the queue (in this case default), the task’s name (the long number starting with 1636460), and the instance that handled the request (the blue text on the bottom).

An important note: a quick and easy way to figure out if a request comes from the Task Queue is to look at the request’s IP address. All Task Queue requests come from the IP 0.1.0.2.

Generating MD5 Hashes In Java

Web applications frequently need to generate hashes for comparing data, verifying file integrity, generating keys, etc. If your application uses the Apache Commons libraries, you can use the included DigestUtils convenience class to create hashes. If you don’t use the Apache libraries, then you need an alternate way to generate MD5 hashes.

The below code snippet is a simple function that generates a MD5 hash from a given string. It’s simple to use: simply pass the string to be hashed into the function (to_be_md5), and the hash is returned. This function will throw a RuntimeException if an error is encountered (for example, if the passed-in String is null).

/**
 * Calculate a MD5 hash from the provided String. If the 
 * provided String is null, this method will throw a 
 * RuntimeException.
 * 
 * @param to_be_md5 A String to calculate a MD5 hash from.
 * @return A MD5 hash calculated from the provided String.
 * @throws RuntimeException If an error was encountered during calculating.
 */
public static String generateMD5(String to_be_md5) {
    String md5_sum = "";
    //If the provided String is null, then throw an Exception.
    if (to_be_md5 == null) {
        throw new RuntimeException("There is no string to calculate a MD5 hash from.");
    }
    try {
        MessageDigest md = MessageDigest.getInstance("MD5");
        byte[] array = md.digest(to_be_md5.getBytes("UTF-8"));
        StringBuffer collector = new StringBuffer();
        for (int i = 0; i < array.length; ++i) {
            collector.append(Integer.toHexString((array[i] & 0xFF) | 0x100).substring(1, 3));
        }
        md5_sum = collector.toString();
    }//end try
    catch (NoSuchAlgorithmException e) {
        throw new RuntimeException("Could not find a MD5 instance: " + e.getMessage());
    }
    catch (UnsupportedEncodingException e) {
        throw new RuntimeException("Could not translate UTF-8: " + e.getMessage());
    }
    return md5_sum;
}//end generateMD5

Don’t forget to import the following classes:

import java.io.UnsupportedEncodingException;
import java.security.MessageDigest;
import java.security.NoSuchAlgorithmException;

Retrieving All Recently Created Entities

Searching the datastore using date-based entity properties can often be confusing to new App Engine developers. Here’s how to search for all recently created entities.

First, each entity has to record the time it was created. Add the following line of code when constructing your entities (entity represents the Entity object being created):

entity.setProperty("add_date", new Date());//Records the date this entity was added.

Here’s the code for the query. This code example searches for all entities created within the last 24 hours with a kind of mail. The resulting entities are placed into the variable results (which is a standard java.util.List object).

/**
 * Creates a Date object that represents a point in time 
 * 24 hours ago.
 */
Date current_date = new Date();//The current date.
long current_date_in_ms = current_date.getTime();//The current date in milliseconds.
//Take the current date in milliseconds, and subtract 24 hours worth 
//of milliseconds (1000 milliseconds per second, times 60 seconds per 
//minute, times 60 minutes per hour, times 24 hours in a day).
long one_day_ago_in_ms = current_date_in_ms - (1000 * 60 * 60 * 24);
Date one_day_ago = new Date(one_day_ago_in_ms);//Create a new Date object representing one day ago.
//Search all entities with the kind "mail". 
//We create a Filter that requires that the add_date property be greater 
//than the Date we computed above.
Query q = new Query("mail");
Query.Filter date_filter = new Query.FilterPredicate("add_date", Query.FilterOperator.GREATER_THAN_OR_EQUAL, one_day_ago);
q.setFilter(date_filter);
PreparedQuery pq = DatastoreServiceFactory.getDatastoreService().prepare(q);
List<Entity> results = pq.asList(FetchOptions.Builder.withDefaults());

 

Error Code 202

On high traffic App Engine applications, you may occasionally see a request fail with error code 202. You’ll see the following text in your logs:

A problem was encountered with the process that handled this request, causing it to exit. This is likely to cause a new process to be used for the next request to your application. (Error code 202)

The request log will look similar to the picture below:

Error code 202 is an internal App Engine failure code: an unknown error occurred within App Engine or its associated services, not from your application. This is not an error that your application can catch or recover from, nor is the application at fault for this error. The current request will fail, but the client is free to retry the request. If this request is from the task queue service, the task queue will automatically retry the request if you set the appropriate settings while creating the task.

A note about error codes: the code 202 relates to App Engine; it is NOT a HTTP status code. When a request encounters this internal error, it fails with the HTTP status code 500 (as you can see from the above picture).