How to group related request log entries GAE python 3.7 standard env

python logging google
google app engine
google app engine python 3
webapp2 logging
stackdriver logging tutorial
cloudlogginghandler
stackdriver custom logs
django stackdriver logging

I'm using Google App Engine python 3.7 standard and i'm trying to group related request log entries. According to the Writing Application Logs documentation, I should:

Set the trace identifier in the LogEntry trace field of your app log entries. The expected format is projects/[PROJECT_ID]/traces/[TRACE_ID]

Where/How should use LogEntry?

The Stackdriver Logging documentation doesn't show how it's possible. Am I missing something?

Code examples would be much appreciated.

[UPDATE] Following Duck Hunt Duo advice, I tried the following, without any success:

    trace_id = request.headers.get('X-Cloud-Trace-Context', 'no_trace_id').split('/')[0]
    client = logging.Client()
    logger = client.logger('appengine.googleapis.com%2Fstdout')  # Not shown
    # logger = client.logger('projects/{}/logs/stdout'.format(GOOGLE_CLOUD_PROJECT)) # error
    # logger = client.logger('projects/{}/logs/appengine.googleapis.com%2Fstdout'.format(GOOGLE_CLOUD_PROJECT)) # error

    logger.log_text('log_message', trace=trace_id)

The log doesn't appear in the GAE service log web console


You might want to take a look at an answer I provided here.

(This answer addresses how to add logging severity to Cloud Functions logs written into Stackdriver, but the basic workflow is the same)

Quoting it:

[...], you can still create logs with certain severity by using the Stackdriver Logging Client Libraries. Check this documentation in reference to the Python libraries, and this one for some usage-case examples.

Notice that in order to let the logs be under the correct resource, you will have to manually configure them, see this list for the supported resource types. As well, each resource type has some required labels that need to be present in the log structure.

Edit:

Updating the previous answer with an example for App Engine:

from google.cloud import logging
from google.cloud.logging.resource import Resource
from flask import Flask

app = Flask(__name__)

@app.route('/')
def logger():
    log_client = logging.Client()
    log_name = 'appengine.googleapis.com%2Fstdout'

    res = Resource( type='gae_app',
                    labels={
                        "project_id": "MY-PROJECT-ID",
                        "module_id": "MY-SERVICE-NAME"
                       })

    logger = log_client.logger(log_name)

    logger.log_struct({"message": "message string to log"}, resource=res, severity='ERROR') # As an example log message with a ERROR warning level

    return 'Wrote logs to {}.'.format(logger.name)

By using this code as example, and changing the resource type of the log to appengine.googleapis.com%2Fstdout should work, and change the Resource fields to be the same as in the gae_app labels described in here.

Writing and viewing logs, Standard Environment · Python 3 App Engine automatically creates entries in the request log. App log: log entries that you To write log entries, we recommend that you integrate the standard Python logging module with Cloud Logging. The app logs related to that request show up nested under the request log entry. When you are using the old Python 2.7 app engine standard, all logs are grouped by request. You can expand on a request log (in Log Viewer) and view all the corresponding request log. In Python 3.7 app engine standard, application logs still show up in Log Viewer but they are not group together/correlate with request logs (so it is very hard to tell the application logs belong to which HTTP request).


This is my basic solution:

    trace_id = request.headers.get('X-Cloud-Trace-Context', 'no_trace_id').split('/')[0]
    trace_str = "projects/{}/traces/{}".format(os.getenv('GOOGLE_CLOUD_PROJECT'), trace_id)
    log_client = logging.Client()

    # This is the resource type of the log
    log_name = 'stdout'

    # Inside the resource, nest the required labels specific to the resource type
    labels = {
        'module_id': os.getenv('GAE_SERVICE'),
        'project_id': os.getenv('GOOGLE_CLOUD_PROJECT'),
        'version_id': os.getenv('GAE_VERSION')
    }
    res = Resource(type="gae_app",
                   labels=labels,
                   )
    logger = log_client.logger(log_name)
    logger.log_text("MESSAGE_STRING_TO_LOG", resource=res, severity='ERROR', trace=trace_str)

After it was working, I wrapped it in a file so it would work similarly to Google's logger for python2.7 .

Here is my_gae_logging.py:

import logging as python_logging
import os

from flask import request
from google.cloud import logging as gcp_logging
from google.cloud.logging.resource import Resource

# From GCP logging lib for Python2.7
CRITICAL = 50
FATAL = CRITICAL
ERROR = 40
WARNING = 30
WARN = WARNING
INFO = 20
DEBUG = 10
NOTSET = 0

_levelNames = {
    CRITICAL: 'CRITICAL',
    ERROR: 'ERROR',
    WARNING: 'WARNING',
    INFO: 'INFO',
    DEBUG: 'DEBUG',
    NOTSET: 'NOTSET',
    'CRITICAL': CRITICAL,
    'ERROR': ERROR,
    'WARN': WARNING,
    'WARNING': WARNING,
    'INFO': INFO,
    'DEBUG': DEBUG,
    'NOTSET': NOTSET,
}


def get_trace_id():
    trace_str = None
    try:
        trace_id = request.headers.get('X-Cloud-Trace-Context', 'no_trace_id').split('/')[0]
        trace_str = "projects/{project_id}/traces/{trace_id}".format(
            project_id=os.getenv('GOOGLE_CLOUD_PROJECT'),
            trace_id=trace_id)
    except:
        pass
    return trace_str


class Logging:
def __init__(self):
    self._logger = None

@property
def logger(self):
    if self._logger is not None:
        return self._logger

    log_client = gcp_logging.Client()

    # This is the resource type of the log
    log_name = 'appengine.googleapis.com%2Fstdout'

    # Inside the resource, nest the required labels specific to the resource type

    self._logger = log_client.logger(log_name)
    return self._logger

@property
def resource(self):
    resource = Resource(
        type="gae_app",
        labels={
            'module_id': os.getenv('GAE_SERVICE'),
            'project_id': os.getenv('GOOGLE_CLOUD_PROJECT'),
            'version_id': os.getenv('GAE_VERSION')
        }
    )
    return resource

def log(self, text):
    text = str(text)
    self.logger.log_text(text, resource=self.resource, trace=get_trace_id())

def debug(self, text):
    text = str(text)
    self.logger.log_text(text, resource=self.resource, severity=_levelNames.get(DEBUG), trace=get_trace_id())

def info(self, text):
    text = str(text)
    self.logger.log_text(text, resource=self.resource, severity=_levelNames.get(INFO), trace=get_trace_id())

def warning(self, text):
    text = str(text)
    self.logger.log_text(text, resource=self.resource, severity=_levelNames.get(WARNING), trace=get_trace_id())

def warn(self, text):
    return self.warning(text)

def error(self, text):
    text = str(text)
    self.logger.log_text(text, resource=self.resource, severity=_levelNames.get(ERROR), trace=get_trace_id())

def critical(self, text):
    text = str(text)
    self.logger.log_text(text, resource=self.resource, severity=_levelNames.get(CRITICAL), trace=get_trace_id())


if os.getenv('GAE_VERSION'):  # check if running under gcp env
    logging = Logging()
else:
    # when not running under gcp env, use standard python_logging
    logging = python_logging

Usage:

from my_gae_logging import logging

logging.warn('this is my warning')

Python 3 release notes, The Python 3.7 runtime for the App Engine standard environment is now GA. When committing a cross-group transaction, version numbers returned for new or Previously headers for such requests contained no Content-Length entry. MVM application logs that have thread_id or request_id as a field in their log entry. Create a google cloud project with app engine. You shall be promoted to selected a Region (and this CANNOT be changed after selection). NOTE: A reference for latency across region using GCP. NOTE: If you intend to use some Beta products, I would recommend us-central. Create a virtualenv for your project. virtualenv -p python3 env source env/bin/activate NOTE: Though the official documentation


The Stackdriver Logging Client Library can be used to achieve this. The logger.log_text function sends a LogEntry object to the API. Example:

from google.cloud import logging

client = logging.Client()
logger = client.logger('appengine.googleapis.com%2Fstdout')
logger.log_text('log_message', trace=trace_id)

The trace_id should be retrieved from the request headers as the docs mention. The method of doing this will depend on how you're serving requests, but in Flask for example it would be simple as trace_id = request.headers['X-Cloud-Trace-Context'].split('/')[0]

App Engine Standard Python3.7 Correlate Application Log With , When you use Google App Engine Standard Python 3.7, one of the In Python 3.7 app engine standard, application logs still show up in Log Viewer but they are not group together/correlate with request logs (so it Set the trace identifier in the LogEntry trace field of your app log entries. Related entries:. Teams. Q&A for Work. Stack Overflow for Teams is a private, secure spot for you and your coworkers to find and share information.


Using the AppEngineHandler from Google Cloud Logging provides much of the infrastructure. This allows attaching to the python logging module, so that a standard logging import works.

Setting this up is straightforward enough:

    # Setup google  cloud logging.
    import logging
    import google.cloud.logging  # Don't conflict with standard logging
    from google.cloud.logging.handlers import AppEngineHandler, setup_logging

    client = google.cloud.logging.Client()
    handler = AppEngineHandler(client, name='stdout')
    logging.getLogger().setLevel(logging.INFO)
    setup_logging(handler)

The documentation at https://googleapis.dev/python/logging/latest/usage.html#cloud-logging-handler suggests very similar, but instead of using the AppEngineHandler uses the "CloudLoggingHandler". It also states that the "AppEngineHandler" is for the flexible environment, but this works in the standard python3 environment.

User Authentication with Identity-Aware Proxy, In addition, it can modify the request headers to include information about the How to write and deploy a simple App Engine app using Python 3.7; How to  Using Python Requests Module In Google App Engine January 16, 2018 Related entries: Setup Python 3.7 for Google App Engine Standard.


Combining correlated Log Lines in Google Stackdriver, The child log entry must have a different log name than the parent. is an app log entry in the app log, whereas the parent log entry is in the request log. The exception to this is spring on GAE Flex. Python Flask logging handler to group messages on Google Cloud Platform Setup go environment. Note that adding PYTHONUTF8=1 to the default environment variables will affect all Python 3.7+ applications on your system. If you have any Python 3.7+ applications which rely on the legacy system encoding, it is recommended to set the environment variable temporarily or use the -X utf8 command line option.


Django with postgresl on App Engine Standard, runtime: python env: flex manual_scaling: instances: 1 resources: You received this message because you are subscribed to the Google Groups "Google App Engine" group. 'django.template.context_processors.request', I deployed this test project with App Engine Standard + Python 3.7 + Postgres  In this step-by-step tutorial, you'll learn about the print() function in Python and discover some of its lesser-known features. Avoid common mistakes, take your "hello world" to the next level, and know when to use a better alternative.


App Engine's New Go 1.11 Runtime, If you're a current Go-on-App Engine customer, you should check out would support Go did not receive similar fanfare on golang.org. issue: https://groups.​google.com/forum/#!topic/google-api-go-announ. We're investigating a solution for GAE Standard Gen 2 (which go 1.11 is, along with python 3.7,  The Python 3.7 runtime for the App Engine standard environment is now in beta. A list of differences between Python 2.7 and Python 3.7 runtimes is available. July 12, 2018