Tag: googlecloudstorage

Posted on

Turning A Date Into A Folder Path – Node

This short code segment turns today’s date into a folder path. For example, today’s date of April 2, 2019 would be turned into 2019/04/02. I use it for categorizing date related documents into appropriate folders within GCS.

var datestamp_obj = new Date( (new Date()).getTime() + (-6) * 3600 * 1000 );
var datestamp_slash = datestamp_obj.getFullYear() + "/";
    datestamp_slash += datestamp_obj.getMonth().toString().padStart(2, "0") + "/" + datestamp_obj.getDate().toString().padStart(2, "0");
Posted on

Utility Functions To Save To Google Cloud Storage

Here are short utility functions used to save to Google Cloud Storage in a Java App Engine application.

These snippets require the Java Cloud Storage library ( com.google.appengine.tools.cloudstorage.GcsService ). The bucket string is the name of the bucket, no gs:// required. folder_path is the subdirectory you want to save the file under. It should end with a forward slash. For example this/is/a/subdirectory/ . If you do not want a subdirectory, pass an empty string (“”). filename is the filename of the file you want to save, while contents is the data being saved.

There are two functions here. The top is intended to quickly save simple text/html/xml/etc files, while the bottom is intended to save binary files such as images, executables, and so forth. The bottom function includes a filetype argument, which should be filled with the appropriate MIME type. For example: image/png, image/jpg, application/octet-stream.

Reminder: GCS innately views files as individual objects. When you specify a subfolder, the filename for the object itself includes the folder path. When you read the file back from Google cloud storage, you need to include the folder path: for example this/is/a/subdirectory/example.png .

	public static void saveStringToGCS(String bucket, String folder_path, String filename, String contents) throws IOException {
		GcsService storage_service = GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
		GcsFilename to_file = new GcsFilename(bucket, folder_path + filename);
		GcsFileOptions file_options = GcsFileOptions.getDefaultInstance();
		storage_service.createOrReplace(to_file, file_options, ByteBuffer.wrap(contents.getBytes(StandardCharsets.UTF_8)));
	}
	
	public static void saveByteBufferToGCS(String bucket, String folder_path, String filename, ByteBuffer contents, String filetype) throws IOException {
		GcsService storage_service = GcsServiceFactory.createGcsService(RetryParams.getDefaultInstance());
		GcsFilename to_file = new GcsFilename(bucket, folder_path + filename);
		GcsFileOptions file_options = (new GcsFileOptions.Builder()).mimeType(filetype).build();
		storage_service.createOrReplace(to_file, file_options, contents);
	}
Posted on

Listing Files Within A Bucket Folder – Python

Here’s a short code example in Python to iterate through a folder’s ( thisisafolder ) contents within Google Cloud Storage (GCS). Each filename can be accessed through blobi.name – in the below code sample, we print it out and test whether it ends with .json.

Remember that folders don’t actually exist on GCS, but a folder-like structure can be created by prefixing filenames with the folder name and the forward slash character ( / ).

    client = storage.Client()
    bucket = client.get_bucket("example-bucket-name")
    blob_iterator = bucket.list_blobs(prefix="thisisafolder",client=client)
    #iterate through and print out blob filenames
    for blobi in blob_iterator:
        print(blobi.name)
        if blobi.name.endswith(".json"):
            #do something with blob that ends with ".json"