Tag: cloudstorage

Posted on

Google Cloud Storage RetryHelper Failure

I have numerous applications which use Google Cloud Storage buckets to save files, maintain backups, etc. Most of these applications are Java apps using the official Google Cloud Storage library.

Yesterday one of these applications had difficulty accessing Cloud Storage and printed out the following error report:

The GCS library attempted to access Cloud Storage a total of 6 times; this image shows the last failure and the final “giving-up” error.

Here’s a copy of the error in text form:

com.google.appengine.tools.cloudstorage.RetryHelper doRetry: 
    RetryHelper(4.114 s, 6 attempts, com.google.appengine.tools.cloudstorage.GcsServiceImpl): 
    Attempt 6 failed, sleeping for 3140 ms: java.io.IOException: Response code 503, 
    retryable: Request: POST https://storage.googleapis.com/[bucket_url]/[file_name]
x-goog-resumable: start
x-goog-api-version: 2
Content-Type: image/jpg

no content

Response: 503 with 19 bytes of content
Content-Length: 19
Date: Thu, 01 Aug 2013 [time]
Server: HTTP Upload Server Built on Jul 21 2013 19:20:38 (1374459638)
Content-Type: text/html; charset=UTF-8
X-Google-Cache-Control: remote-fetch
Via: HTTP/1.1 GWA
Service Unavailable

And after 6 failures attempting to access the Google Cloud Storage bucket, the library reports an error and stops:

.<stderr>: IO Exception: RetryHelper(4.115 s, 6 attempts, 
    com.google.appengine.tools.cloudstorage.GcsServiceImpl): 
    Too many failures, giving up

Fortunately this seemed to be only a transient problem; the next upload to GCS succeeded.

Posted on

Creating A Google Cloud Storage Bucket

Creating a Google Cloud Storage bucket is relatively simple. Just follow these steps:

Go to https://cloud.google.com/console‎ and log in. You’ll see the following page. Create a project or click on an already-existing project (in this picture, Fact and Invalid are pre-existing projects).

Click on the Cloud Storage link in the next screen:

Now press the New Bucket button:

A prompt will open asking you to name your new bucket.

Your new bucket will be created. You can upload new files and folders to the bucket by clicking the Upload button in the middle of the screen:

Posted on

Setting Permissions To Access Cloud Storage From App Engine

First, go to http://cloud.google.com/console and log in. Select your project and then click the Cloud Storage link. You’ll see the screen below:

Check the checkboxes next to the Cloud Storage buckets you’re using, then press the button marked “Bucket Permissions”. You’ll see the following screen:

Add your App Engine application by setting the first drop down box to “User”, the textbox to “(your application id)@appspot.gserviceaccount.com”, and the second drop down box to “Owner”.

Posted on

Serving URL For Images Stored In Cloud Storage

Creating a serving URL for images stored in GCS is slightly different than for images stored in the Blobstore. Here’s a code sample:

Bucket represents the name of your Google Cloud Storage bucket, and object represents the file name of the image (including extension).

    String bucket = "bucket_name";
    String object = "file_name_including_extension";

    //Get serving url
    String gs_blob_key = "/gs/" + bucket + "/" + object;
    BlobKey blob_key = BlobstoreServiceFactory.getBlobstoreService().createGsBlobKey(gs_blob_key);
    ServingUrlOptions serving_options = ServingUrlOptions.Builder.withBlobKey(blob_key);
    String serving_url = ImagesServiceFactory.getImagesService().getServingUrl(serving_options);
    System.out.println("Serving URL: " + serving_url);
    resp.getWriter().println(serving_url);
Posted on

Writing Files To Google Cloud Storage

Writing a file to Google Cloud Storage is easy in Java. All you need is the Google Cloud Storage library.

The following code writes a file into GCS. Bucket represents the name of the Cloud Storage bucket, object represents the filename, mime represents the MIME type, and data represents a byte array with the contents of the file you’re writing.

    String bucket = "your_bucket_name";
    String object = "file_name_here_including_extension";

    GcsFilename gcs_filename = new GcsFilename(bucket, object);
    //File Options
    GcsFileOptions.Builder options_builder = new GcsFileOptions.Builder();
    options_builder = options_builder.mimeType(mime);
    GcsFileOptions options = options_builder.build();       
    GcsOutputChannel output = GcsServiceFactory.createGcsService().createOrReplace(gcs_filename, options);
    output.write(ByteBuffer.wrap(data));
    output.close();

Remember to set bucket permissions in Google Cloud Storage so your app can access the files.