I was checking my Spam folder when I saw this message. Apparently Gmail thinks that emails from the Google Cloud Platform mailing list are spam.
A Simple Cron Declaration
Cron is a service to schedule tasks at predetermined intervals. For example, you could schedule a certain task to run every 10 minutes, or 10 hours, or every Monday. At every interval, App Engine will send a HTTP request to an URL that you specify.
Here’s an example of a cron file:
cron:
- description: Description Of Cron
url: /url
schedule: every 24 hours
timezone: America/Chicago
This tells App Engine to run the script located at /url once every 24 hours. You can name other schedules, such as every 10 minutes, or every Monday 9:00 (run every Monday 9 AM).
Cron files are saved as cron.yaml in the root directory of a Go app, or in the /war/WEB-INF folder of a Java application.
Configuring EdgeCache – A Cache For App Engine
App Engine has a distributed caching system called EdgeCache which can be configured to quickly serve static or rarely-changed assets. To tell EdgeCache to cache your files, you need to set the following two headers:
Cache-Control: public, max-age=[time in seconds to cache]
Pragma: Public
The max-age argument takes the number of seconds to cache the given file. It must be set to at least 61 seconds.
Here’s example Java code to implement these headers. The max age is set to 86,400 seconds (1 day). The resp object represents a HttpServletResponse class:
resp.setHeader("Cache-Control", "public, max-age=86400");
resp.setHeader("Pragma", "Public");
Here’s the same code implemented in Go ( w represents http.ResponseWriter ):
w.Header().Set("Cache-Control", "public, max-age=86400")
w.Header().Set("Pragma", "Public")
Setting Permissions To Access Cloud Storage From App Engine
First, go to http://cloud.google.com/console and log in. Select your project and then click the Cloud Storage link. You’ll see the screen below:
Check the checkboxes next to the Cloud Storage buckets you’re using, then press the button marked “Bucket Permissions”. You’ll see the following screen:
Add your App Engine application by setting the first drop down box to “User”, the textbox to “(your application id)@appspot.gserviceaccount.com”, and the second drop down box to “Owner”.
Creating Twitter Access Credentials
Integrating Twitter access to your application is fairly simple. But before you write a single line of code, you need OAuth access credentials to log into Twitter. Here’s how to get those credentials.
First, go to https://dev.twitter.com/apps. You’ll see the following screen:
Click on Create Application, then fill in the application details boxes in the next screen:
Scroll all the way down and click the button marked Create Application.
After that, go to the Settings tab of your new application:
Go down to the Application Type section:
Click on the “Read, Write and Access direct messages” option, and save your settings.
Go back to the details tab:
Scroll down and press the button marked “Create my access token”:
After a moment, you’ll see entries for the OAuth consumer key, consumer secret, access token, and access token secret (they’ll look like long strings of random characters). These tokens can be passed to twitter4j or another Twitter library to log into Twitter.
Uploading And Serving Files With The Blobstore In Java
Here’s a code example to let users upload a file to the Blobstore and then download it back again. This code uses a servlet and a JSP page: the JSP page shows a form and the servlet accepts the upload, then immediately sends the uploaded file back to the user.
Here’s the doPost and doGet code for the servlet (in this example, we aliased the servlet to /blobstoreexample):
public void doGet(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
String blob_key_string = req.getParameter("blob_key");
System.out.println("BLOBKEY: " + blob_key_string);
if (blob_key_string == null) {
resp.sendRedirect("/example.jsp");
System.out.println("No blobkey given, redirecting to upload page.");
}
else {
BlobKey blob_key = new BlobKey(blob_key_string);
BlobstoreServiceFactory.getBlobstoreService().serve(blob_key, resp);
System.out.println("Blobkey given, sending file.");
}
}//end doGet
public void doPost(HttpServletRequest req, HttpServletResponse resp)
throws IOException {
try {
Map<String, List<BlobKey>> files_sent = BlobstoreServiceFactory.getBlobstoreService().getUploads(req);
BlobKey file_uploaded_key = files_sent.get("file").get(0);
resp.sendRedirect("/blobstoreexample?blob_key=" + file_uploaded_key.getKeyString());
System.out.println("Document successfully POSTED, redirect to doGET");
}
catch (Exception e) {
resp.sendRedirect("/example.jsp");
System.out.println("Document failed to POST, redirecting back to upload.");
}
}//end doPost
Here is the code for example.jsp (remember to import the Blobstore library):
<form enctype="multipart/form-data" method="post" action="<%= BlobstoreServiceFactory.getBlobstoreService().createUploadUrl("/blobstoreexample") %>">
<input type="file" name="file" size="30" />
<input type="submit" /></form>
You can add additional form elements to the form if needed; the servlet will be able to access them.
An Introduction To The Datastore In Java
Storing Data
Essentially, the datastore is a highly replicated, highly reliable place to store data. However, it is not a SQL database that you may have experience with. Instead, the datastore stores Entities, which are grouped into categories called kinds. Each entity has a unique ID or name – Google App Engine can set a unique ID, or you can set a name in your application code.
To create an entity with a mail kind, we use this line (App Engine allocates a unique ID for us automatically):
Entity entity = new Entity("mail");
Entities can have properties, which are name/valid pairs. To add a property, do this:
entity.setProperty("subject", subject); //Key, then value.
Here’s a complete code example to create an entity, set properties, and store it:
Entity entity = new Entity("mail");//Creates the entity with a kind of "mail", GAE automatically allocates an ID
entity.setUnindexedProperty("from", from);
entity.setProperty("subject", subject);
entity.setProperty("add_date", new Date());//Records the date this entity was added.
DatastoreServiceFactory.getDatastoreService().put(entity);
Note the use of setUnindexedProperty. This method informs App Engine that we won’t be searching on this property, so don’t build a search index for it. This is useful to reduce the cost of storing entities.
Datastore Query
Querying the datastore is simple as well.
Here’s an example piece of code which queries for all Entities with the kind of mail, and requiring that all properties returned have the property add_date greater than the value stored in date_var :
Query q = new Query("mail");
Query.Filter query_filter = new Query.FilterPredicate("add_date", Query.FilterOperator.GREATER_THAN_OR_EQUAL, date_var);
q.setFilter(query_filter);
PreparedQuery pq = DatastoreServiceFactory.getDatastoreService().prepare(q);
List<Entity> results = pq.asList(FetchOptions.Builder.withDefaults());
From here, we can iterate through the List, retrieve each Entity, and conduct operations on each Entity:
for (int i = 0; i < results.size(); i++) {
Entity entity = results.get(i);
//Do anything needed with the information from each entity.
}
Deleting
And finally, here is how to delete an entity (entity is the Entity object being deleted).
Key key = entity.getKey();
DatastoreServiceFactory.getAsyncDatastoreService().delete(key);
Deadline exceeded while waiting for HTTP response from URL
Occasionally applications – even the best behaved applications – will get the error “Deadline exceeded while waiting for HTTP response from URL.”
Generally, this means that the web service you’re trying to connect to is down or slow. If the service is down, then you can continuously retry your URL fetches by queuing them up within a task.
If the web service is slow, then you have an alternative: setting the read and connect timeouts to a longer timeout point. By default, App Engine expects that an URL fetch will take – at most – 5 seconds. That’s 5 seconds to connect to the web service (resolve DNS and so forth), send the request data, allow the web service to process the request, and finally retrieve any response sent back. For the vast majority of applications, that’s more than enough. The popular web APIs such as Twitter, Facebook, Google, etc all process and return requests in much less than 5 seconds.
However, a slow or malfunctioning web service may take longer than 5 seconds to respond to a query. If your app is downloading a large amount of data (more than a few MB) you may also go past this limit. To tell App Engine to wait for a longer period of time, use this code (url_connection represents a HttpURLConnection object):
url_connection.setReadTimeout(milliseconds_to_wait_for_read);
url_connection.setConnectTimeout(milliseconds_to_wait_for_connect);
Remember that the time to wait is denoted in milliseconds, so do the appropriate conversions (for example, if you wanted the connection to wait 30 seconds, you would put 30000 milliseconds).
Serving URL For Images Stored In Cloud Storage
Creating a serving URL for images stored in GCS is slightly different than for images stored in the Blobstore. Here’s a code sample:
Bucket represents the name of your Google Cloud Storage bucket, and object represents the file name of the image (including extension).
String bucket = "bucket_name";
String object = "file_name_including_extension";
//Get serving url
String gs_blob_key = "/gs/" + bucket + "/" + object;
BlobKey blob_key = BlobstoreServiceFactory.getBlobstoreService().createGsBlobKey(gs_blob_key);
ServingUrlOptions serving_options = ServingUrlOptions.Builder.withBlobKey(blob_key);
String serving_url = ImagesServiceFactory.getImagesService().getServingUrl(serving_options);
System.out.println("Serving URL: " + serving_url);
resp.getWriter().println(serving_url);
Writing Files To Google Cloud Storage
Writing a file to Google Cloud Storage is easy in Java. All you need is the Google Cloud Storage library.
The following code writes a file into GCS. Bucket represents the name of the Cloud Storage bucket, object represents the filename, mime represents the MIME type, and data represents a byte array with the contents of the file you’re writing.
String bucket = "your_bucket_name";
String object = "file_name_here_including_extension";
GcsFilename gcs_filename = new GcsFilename(bucket, object);
//File Options
GcsFileOptions.Builder options_builder = new GcsFileOptions.Builder();
options_builder = options_builder.mimeType(mime);
GcsFileOptions options = options_builder.build();
GcsOutputChannel output = GcsServiceFactory.createGcsService().createOrReplace(gcs_filename, options);
output.write(ByteBuffer.wrap(data));
output.close();
Remember to set bucket permissions in Google Cloud Storage so your app can access the files.