Tag: appengine

Posted on

Deadline exceeded while waiting for HTTP response from URL

Occasionally applications – even the best behaved applications – will get the error “Deadline exceeded while waiting for HTTP response from URL.”

Generally, this means that the web service you’re trying to connect to is down or slow. If the service is down, then you can continuously retry your URL fetches by queuing them up within a task.

If the web service is slow, then you have an alternative: setting the read and connect timeouts to a longer timeout point. By default, App Engine expects that an URL fetch will take – at most – 5 seconds. That’s 5 seconds to connect to the web service (resolve DNS and so forth), send the request data, allow the web service to process the request, and finally retrieve any response sent back. For the vast majority of applications, that’s more than enough. The popular web APIs such as Twitter, Facebook, Google, etc all process and return requests in much less than 5 seconds.

However, a slow or malfunctioning web service may take longer than 5 seconds to respond to a query. If your app is downloading a large amount of data (more than a few MB) you may also go past this limit. To tell App Engine to wait for a longer period of time, use this code (url_connection represents a HttpURLConnection object):

url_connection.setReadTimeout(milliseconds_to_wait_for_read);
url_connection.setConnectTimeout(milliseconds_to_wait_for_connect);

Remember that the time to wait is denoted in milliseconds, so do the appropriate conversions (for example, if you wanted the connection to wait 30 seconds, you would put 30000 milliseconds).

Posted on

Writing Files To Google Cloud Storage

Writing a file to Google Cloud Storage is easy in Java. All you need is the Google Cloud Storage library.

The following code writes a file into GCS. Bucket represents the name of the Cloud Storage bucket, object represents the filename, mime represents the MIME type, and data represents a byte array with the contents of the file you’re writing.

    String bucket = "your_bucket_name";
    String object = "file_name_here_including_extension";

    GcsFilename gcs_filename = new GcsFilename(bucket, object);
    //File Options
    GcsFileOptions.Builder options_builder = new GcsFileOptions.Builder();
    options_builder = options_builder.mimeType(mime);
    GcsFileOptions options = options_builder.build();       
    GcsOutputChannel output = GcsServiceFactory.createGcsService().createOrReplace(gcs_filename, options);
    output.write(ByteBuffer.wrap(data));
    output.close();

Remember to set bucket permissions in Google Cloud Storage so your app can access the files.

Posted on

Configuring A .Properties File In App Engine

Configuration settings for an application should never be hardcoded; instead, they should be configured through a separate properties file.

A properties file looks like this:

#This is a comment
key=value

The # symbol denotes a comment, while keys are separated from values with an = sign.

You can read this file into a Java application by using the following code. It reads in a properties file named app.properties located within the /WEB-INF/ folder (putting your properties files in /WEB-INF/ also keeps them private):

InputStream app_properties_stream = this.getServletContext().getResourceAsStream("/WEB-INF/app.properties");
Properties app_properties = new Properties();
//This will throw a NullPointerException if the properties file cannot be found, 
//and an IOException if an error occurs while reading.
app_properties.load(app_properties_stream);

From here you can retrieve values by calling:

String value = app_properties.getProperty("key");
Posted on

Storing To The Memcache In Golang

A short Golang example: how to store to memcache in Go. To use this, you’ll need to import appengine/memcache.

This code fragment generates a memcache.Item struct, to which you’ll need to supply a key and the value you want stored (a struct, array, map, etc).

memcache_item := &memcache.Item {
    Key: "key_to_this_object",
    Object: item_to_store,
}

//Store into Memcache
memcache.JSON.Set(c, memcache_item)
Posted on

Accessing POP3 & IMAP From AppEngine

A quick note about accessing IMAP and POP3 servers from AppEngine: you need to include the JavaMail libraries and import everything from javax.mail and javax.mail.internet.

If you need to streamline the amount of JARs bundled with your application, you can also select individual protocol provider JARs.

Here’s example code to extract emails from a POP3 store:

    Properties properties = new Properties();
    properties.put("mail.host", host);
    properties.put("mail.store.protocol", "pop3s");
    properties.put("mail.pop3s.auth", "true");
    properties.put("mail.pop3s.port", port);
    Session session = Session.getDefaultInstance(properties);
    Store store = session.getStore();
    //With a POP3 store available, connect to the given account.
    store.connect(host, user, password);
    //Open up the inbox folder, and give ourselves read/write privilegese.
    Folder folder = store.getFolder("inbox");
    folder.open(Folder.READ_WRITE);
    //Collect messages.
    Message[] messages = folder.getMessages();
    System.out.println(messages.length);
    for (int i = 0; i < messages.length; i++) {
        //Extract message.
        MimeMessage message = (MimeMessage)messages[i];

From here, you can extract the from address and the subject line from the MimeMessage. Remember that the Sockets API requires that you have billing enabled on your application.

Posted on

A Simple index.yaml File

The App Engine SDK developer server can automatically create datastore indexes for your application – just run the datastore querying code within the dev server. But sometimes that isn’t possible.

In that case, you can hand-write your index settings. Here is an example of a simple index.yaml file. It indexes all entities with a kind of Fact, on the property Approved and sorts on the property Add_date in descending order.

indexes:

- kind: Fact
  ancestor: no
  properties:
  - name: Approved
  - name: Add_date
    direction: desc

index.yaml goes in the root directory of your application.

Posted on

Sending Mail With Golang

Previously I published a Java code example using the low level Mail API to send a message to the registered admins of an application. Here’s sample code for a Golang application to send mail to app admins. C stands for an appengine.Context reference.

If you want to send mail to an arbitrary user, and not an admin, you can uncomment the To line and change SendToAdmins() to Send().

application_id := appengine.AppID(c)
separation_point := strings.Index(application_id, ":")
if separation_point > -1 {
    application_id = application_id[separation_point:]
}

//Create the message struct
msg := &mail.Message{
    Sender:  "donotreply@" + application_id + ".appspotmail.com",
    //To:    []string{"To-User <[email protected]>"},
    Subject: subject,
    Body:    email_body,
}
c.Infof("Sending message: %v", msg)

//Send an email to admins
err := mail.SendToAdmins(c, msg)
if err != nil {
    c.Errorf("Unable to send email: %v", err)
}
Posted on

Writing To The Blobstore Using The Files API

A short code snippet showing how to write data to the Blobstore using the experimental Files API.

FileService file_service = FileServiceFactory.getFileService();
AppEngineFile file = file_service.createNewBlobFile(mime_type, filename);
FileWriteChannel write_channel = file_service.openWriteChannel(file, true);
ByteBuffer buffer = ByteBuffer.wrap(byte_array);
write_channel.write(buffer);
write_channel.closeFinally();

 

Posted on

TaskTooLargeError

Google App Engine has an upper limit on how much data you can put into a task – a maximum of 100 KB. This includes everything within an individual task such as parameters, payload, the request URL, etc.

To get around this error, store any large pieces of information within an entity in the datastore. You can then pass the entity’s ID as a task parameter and pull out the entity’s data during task processing.

Posted on

Geolocation With App Engine

One of the best parts of hosting with App Engine is the free geolocation service it provides. Every incoming request has special AppEngine location headers added to it.

For instance, here’s how to find out the user’s city (req is the servlet’s HttpServletRequest object):

 String city = req.getHeader("X-AppEngine-City");

Other geolocation information is available as well: the user’s country (X-AppEngine-Country), the region of that country (X-AppEngine-Region), and the nearest city’s latitude and longitude (X-AppEngine-CityLatLong).