Tag: appengine

Posted on

Task Queue Fun: DeadlineExceeded

I always love pointing out fun errors – where I define “fun error” as an error that is intermittent/happens rarely in the context of regular operation in the application. Those errors are always the most fun to fix.

Today I was poking through some of my toy applications running on Google Cloud when I saw this:

And the text only:

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
._end_unary_response_blocking ( /layers/google.python.pip/pip/lib/python3.7/site-packages/grpc/_channel.py:1003 )	-	Jan 23, 2024	22 hours ago	-

DeadlineExceeded: 504 Deadline Exceeded
.error_remapped_callable ( /layers/google.python.pip/pip/lib/python3.7/site-packages/google/api_core/grpc_helpers.py:81 )	-	Jan 23, 2024	22 hours ago

Hmm – so an error occurred 22 hours ago, that last reoccurred almost 4 months ago (Jan 23, 2024). Doesn’t sound very important. But just for the laughs, let’s dig in.

Of the two errors, I know that the first one (InactiveRPCError) is most likely due to a connection being unable to complete. Not a giant problem, happens all the time in the cloud – servers get rebooted, VMs get shuffled off to another machine, etc. Not a serious problem. The deadline exceeded one concerns me though because I know this application connects to a bunch of different APIs and does a ton of time consuming operations, and I want to make sure that everything is back to normal.

So here’s the view of the error page:

So I know that the error is somewhere communicating with Google services since the error pops up in the google api core library. Let’s hop on over to logging and find the stack trace – I’ve redacted a line that doesn’t mean anything to the purpose of this post:

Traceback (most recent call last):
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 2070, in wsgi_app
    response = self.full_dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 1515, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 1513, in full_dispatch_request
    rv = self.dispatch_request()
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/flask/app.py", line 1499, in dispatch_request
    return self.ensure_sync(self.view_functions[rule.endpoint])(**req.view_args)
  [REDACTED]
  File "/srv/main.py", line 331, in launch_task
    task_creation_results = client.create_task(parent=queue_prop, task=task)
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/cloud/tasks_v2/services/cloud_tasks/client.py", line 2203, in create_task
    metadata=metadata,
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 131, in __call__
    return wrapped_func(*args, **kwargs)
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/api_core/timeout.py", line 120, in func_with_timeout
    return func(*args, **kwargs)
  File "/layers/google.python.pip/pip/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 81, in error_remapped_callable
    raise exceptions.from_grpc_error(exc) from exc
google.api_core.exceptions.DeadlineExceeded: 504 Deadline Exceeded

If you missed the culprit in the above text, let me help you out: the call to the Google Task Queue service on line 331 of my application ended up exceeding Google’s deadline, and threw up the exception from Google’s end. Perhaps there was a transient infrastructure issue, perhaps task queue was under maintenance, perhaps it was just bad luck. 

File "/srv/main.py", line 331, in launch_task
    task_creation_results = client.create_task(parent=queue_prop, task=task)

There’s really nothing to be done here, other than maybe catching the exception and trying again. I will point out that the task queue service is surprisingly resilient: out of tens/hundreds of thousands of task queue calls over the past 5 months that this application has performed, only 2 tasks (one in January 2024, one yesterday) have failed to enqueue. More importantly, my code is functioning as intended and I can mark this issue as Resolved or at least Muted.

Now honestly, this is a my bad sort of situation. If this was a real production app there should be something catching the exception. But since this is a toy application, I absolutely am fine tolerating the random and thankfully very rare failures in task queue.

Posted on

Using Google Protobuf Timestamp In PHP

This is more of a documentary post because I haven’t seen documentation on Google’s Timestamp class anywhere.

Google’s libraries – in particular, the GCP libraries for datastore/tasks/etc – use the Google/Protobuf/Timestamp class to represent time. Timestamp is a simple wrapper around the number of seconds since UNIX epoch, in the UTC timezone. For example here is how to create a Timestamp reflecting the current date and time, plus 2 minutes into the future (120 seconds):

use Google\Protobuf\Timestamp;

    $future_time_seconds = time() + 120;
    $future_timestamp = new Timestamp();
    $future_timestamp->setSeconds($future_time_seconds);
    $future_timestamp->setNanos(0);

There are equivalent classes and functions for Python/Java/Go/other languages that Google Cloud supports.

Using the Timestamp class – especially setting up future dates – is necessary for configuring my favorite Google Cloud service: Cloud Tasks. A Cloud Task can accept a future date to run at, thereby giving you a way to queue up and delay execution of an activity. For example, see the below screenshot: I’ve created 3 tasks 20 seconds ago, yet they’re set for a future execution 3 minutes 45 seconds from now:

Posted on

PHP Post To PubSub

Today is a rather large fragment demonstrating how to post to Google PubSub. While there are libraries to handle this, I prefer to understand the low-level process so debugging is easier.

Note that this fragment is designed to run on App Engine, as it relies on the App Identity service to pull the credentials required to publish to PubSub. You only need to set up 3 variables: $message_data, which should be a JSON-encodable object, NAMEOFGOOGLEPROJECT, which is the name of the Google project containing the pubsub funnel you want to publish to, and NAMEOFPUBSUB which is the pubsub funnel name.

It isn’t required, but it is good practice to customize the User-Agent header below. I have it set to Publisher, but a production service should have it set to an appropriate custom name.

use google\appengine\api\app_identity\AppIdentityService;

//Build JSON object to post to Pubsub

$message_data_string = base64_encode(json_encode($message_data));

$single_message_attributes = array ("key" => "iana.org/language_tag",
    "value" => "en",
);

$single_message = array ("attributes" => $single_message_attributes,
    "data" => $message_data_string,
);
$messages = array ("messages" => $single_message);

//Post to Pubsub

$url = 'https://pubsub.googleapis.com/v1/projects/NAMEOFGOOGLEPROJECT/topics/NAMEOFPUBSUB:publish';

$pubsub_data = json_encode($messages);

syslog(LOG_INFO, "Pubsub Message: " . $pubsub_data);

$access_token = AppIdentityService::getAccessToken('https://www.googleapis.com/auth/pubsub');

$headers = "accept: */*\r\n" .
    "Content-Type: text/json\r\n" .
    "User-Agent: Publisher\r\n" .
    "Authorization: OAuth " . $access_token['access_token'] . "\r\n" .
    "Custom-Header-Two: custom-value-2\r\n";

$context = [
    'http' => [
        'method' => 'POST',
        'header' => $headers,
        'content' => $pubsub_data,
    ]
];
$context = stream_context_create($context);
$result = file_get_contents($url, false, $context);

syslog(LOG_INFO, "Returning from PubSub: " . $result);

Posted on

DisallowedHost – Invalid HTTP_HOST Header “example.appspot.com”

When installing a new Django app on App Engine, I forgot to set the hosts header, and found this error:



The problem here is that I didn’t set the allowed hosts header in the settings file. To fix this, go to settings.py file and set ALLOWED_HOSTS to list a wildcard, like so:

ALLOWED_HOSTS = ['*']

You can see this replicated in Google’s own Django example: https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/appengine/standard/django/mysite/settings.py#L46 (see screenshot below).

A screenshot of allowed_hosts including a wildcard.

Warning: A wildcard is only acceptable because App Engine’s other security policies are protecting the application. If you move the Django site to another host, make sure you change out the wildcard to the appropriate domain/domains.

Posted on

Golang Users API Example

Here’s an example of how to use the Users API in Go. This example checks to see if the current user is logged in; if not, the user is given a link to log in. If the user is already logged in, then it creates and prints a link to log out.

//Creates an App Engine Context - required to access App Engine services.
c := appengine.NewContext(r)
//Acquire the current user
user := appengineuser.Current(c)
if user == nil {
    url, _ := appengineuser.LoginURL(c, "/")
    fmt.Fprintf(w, `<a href="%s">Sign in</a>`, url)
} else {
    url, _ := appengineuser.LogoutURL(c, "/")
    fmt.Fprintf(w, `Welcome, %s! (<a href="%s">sign out</a>)`, user, url)
}
Posted on

Searching For User Request Logs In App Engine

You can search for specific users by using the filter labels function of App Engine logging. Here’s a short example: the following filter searches for all users matching the substring inny :

The search finds the following request logs, where the user vinnyapp (my Gmail account) has logged in:

Posted on

Error Parsing YAML File: Mapping Values Are Not Allowed Here

An improperly configured YAML file may show the error Error Parsing YAML File: Mapping Values Are Not Allowed Here . This error is demonstrated below:

Here is an example YAML file that causes this error:

application:application-id
version:1
runtime:php
api_version:1
threadsafe:true
#Error happens at line 7 below, even though the incorrect lines are above.
handlers:
- url: /example

Even though the YAML parser reports the error at line 7, the actual incorrect lines are above that point: lines 1 – 5 are missing the space character between the colon and the value. If you encounter this error, make sure that the key: value pairs are separated by 1 colon and 1 space character, as shown below:

application: application-id
Posted on

Mail Service Error: Sender Is Not An Authorized Email Address

While using App Engine’s Mail API, some applications may encounter the following error:

This error means that the application attempted to send email with a non-whitelisted from address.

To send email from App Engine, applications must declare a sending address matching one of the following: a registered administrator of the application, the Google user account of the currently-logged-in user, or an email address of the form:

[any string]@[Application ID].appspotmail.com

For most purposes, using the appspotmail string as a from address is perfectly fine. To generate this sending address, you can use App Engine’s environment variables to collect the application ID. For example, here’s how to do it in Java:

String application_id = SystemProperty.applicationId.get();
String sender = "donotreply@" + application_id + ".appspotmail.com";

For applications that need to send email originating from their custom domain, register a Google Apps account with the address you want to use, then register it as an administrator of the application.

Posted on

IOException: tmpFile.renameTo Failed

On rare occasions, the Google App Engine dev server displays the following error:

The important part of the error is this text:

java.io.IOException: tmpFile.renameTo(classfile) failed

This exception crops up whenever an app file (in this case a JSP file) is currently being accessed by another program. If you see this exception, double check to ensure that the named file isn’t being accessed by another program.

If this error persists, close down and reopen Eclipse and the development app server – the file may have been left open from a previous run.

Posted on

Whitelisted PHP Extensions

Google App Engine permits only specific whitelisted extensions to be used within PHP applications. If you use a non-whitelisted extension, you’ll see the below error:

The [php_extension_name] extension is missing. 
Please check your PHP configuration.

If you need a certain extension for your PHP application, ensure that it’s enabled in GAE: check the official list at https://developers.google.com/appengine/docs/php/#PHP_Enabled_extensions . If your preferred extension is not listed, you can also try searching for a pure-PHP implementation (Pure PHP extensions can always be uploaded as part of an application; C based extensions must be whitelisted.)

If you need an extension not listed in the above link, you can request it via the App Engine issues tracker. For example, here’s a feature request for the ImageMagick extension: https://code.google.com/p/googleappengine/issues/detail?id=9424