Google App Engine Application Suddenly Stopped Working, returning 503

502 server error google app engine
google cloud 503 error
google app engine environment variables
google app engine 404
google app engine cache-control
google compute
google/api_core exceptions serviceunavailable: 503 deadline exceeded
the request failed because the instance could not start successfully

I have an application running on Google App Engine, I deployed the last version of latest version of the application on Sep 6, 2018 but today the application stopped serving request and it is returning error 503 with the following

<HTML>
  <HEAD>
    <TITLE>Service Unavailable</TITLE>
  </HEAD>
 <BODY BGCOLOR="#FFFFFF" TEXT="#000000">
<H1>Service Unavailable</H1>
 <H2>Error 503</H2>
  </BODY>
 </HTML>

I thought it was because i have not migrated to the deprecated Cloud Endpoint v1.1 to V2.0

But after I made the changes necessary for migration, when I try to deploy it still returns 503 while running the Maven command to deploy or generate Docs.

I am aware that Google stated that they are experiencing issues with some App Engine regions yesterday but I am not sure if it has anything to do with what I am experiencing.

My App engine application is using Us-central region.

Update: Looks like cloud endpoint V1.1 has been turned off and can no longer serve traffic so following the migration guide, I have been able to migrate to version 2.0 but unfortunately I am getting an error with the api-discovery doc

This is the error

500

API discovery response missing required fields.

I am trying to get around this

here is my pom.xml file

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0        http://maven.apache.org/xsd/maven-4.0.0.xsd">

<modelVersion>4.0.0</modelVersion>
<packaging>war</packaging>
<version>1.0-SNAPSHOT</version>

<groupId>group-id</groupId>
<artifactId>artifact-id</artifactId>

<properties>
    <app.id>project-id</app.id>
    <app.version>1</app.version>
    <appengine.version>1.9.64</appengine.version>
    <gcloud.plugin.version>2.0.9.121.v20160815</gcloud.plugin.version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
    <maven.compiler.source>1.7</maven.compiler.source>
    <maven.compiler.target>1.7</maven.compiler.target>
    <jackson.version>2.9.0</jackson.version>
    <archiveClasses>true</archiveClasses>
</properties>

<prerequisites>
    <maven>3.3.9</maven>
</prerequisites>

<dependencies>
    <!-- Compile/runtime dependencies -->
    <dependency>
        <groupId>com.google.endpoints</groupId>
        <artifactId>endpoints-framework</artifactId>
        <version>2.1.0</version>
    </dependency>

    <dependency>
        <groupId>javax.servlet</groupId>
        <artifactId>servlet-api</artifactId>
        <version>2.5</version>
        <scope>provided</scope>
    </dependency>
    <dependency>
        <groupId>javax.inject</groupId>
        <artifactId>javax.inject</artifactId>
        <version>1</version>
    </dependency>

    <!--  Objectify library ready -->
    <dependency>
        <groupId>com.googlecode.objectify</groupId>
        <artifactId>objectify</artifactId>
        <version>6.0</version>
    </dependency>

    <!--  Apache Commons Library -->
    <dependency>
        <groupId>org.apache.commons</groupId>
        <artifactId>commons-lang3</artifactId>
        <version>3.0</version>
    </dependency>

    <!-- Add support for joda money library  -->
    <dependency>
        <groupId>org.joda</groupId>
        <artifactId>joda-money</artifactId>
        <version>0.10.0</version>
    </dependency>

    <!--Added support for joda time library -->
    <dependency>
        <groupId>joda-time</groupId>
        <artifactId>joda-time</artifactId>
        <version>2.9.4</version>
    </dependency>
    <!-- end joda time api.-->

    <!-- Load scrypt library for password encryption and decryption -->
    <dependency>
        <groupId>com.lambdaworks</groupId>
        <artifactId>scrypt</artifactId>
        <version>1.4.0</version>
    </dependency>
    <!-- Objectify utils to help us with currency -->
    <dependency>
        <groupId>com.sappenin.objectify</groupId>
        <artifactId>objectify-utils</artifactId>
        <version>5.1.3</version>
    </dependency>
    <!-- google guava -->
    <dependency>
        <groupId>com.google.guava</groupId>
        <artifactId>guava</artifactId>
        <version>19.0</version>
    </dependency>
    <!-- Apache poi for excel file processing -->
    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi</artifactId>
        <version>3.15</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/org.apache.poi/poi-ooxml -->
    <dependency>
        <groupId>org.apache.poi</groupId>
        <artifactId>poi-ooxml</artifactId>
        <version>3.15</version>
    </dependency>


    <!-- jackson-dataformat-csv-->
    <dependency>
        <groupId>com.fasterxml.jackson.dataformat</groupId>
        <artifactId>jackson-dataformat-csv</artifactId>
        <version>2.7.0</version>
    </dependency>

    <!-- jackson data bind library for json -->
    <!-- Google cloud storage ends -->
    <!-- google cloud storage client -->
    <dependency>
        <groupId>com.google.appengine.tools</groupId>
        <artifactId>appengine-gcs-client</artifactId>
        <version>0.7</version>
        <exclusions>
            <exclusion>
                <groupId>com.google.api-client</groupId>
                <artifactId>google-api-client-appengine</artifactId>
            </exclusion>
            <exclusion>
                <groupId>com.google.http-client</groupId>
                <artifactId>google-http-client-jackson2</artifactId>
            </exclusion>
            <exclusion>
                <groupId>com.google.http-client</groupId>
                <artifactId>google-http-client</artifactId>
            </exclusion>
        </exclusions>
    </dependency>
    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud-storage</artifactId>
        <version>0.4.0</version>
    </dependency>
    <dependency>
        <groupId>com.google.api-client</groupId>
        <artifactId>google-api-client-appengine</artifactId>
        <version>1.23.0</version>
    </dependency>
    <dependency>
        <groupId>com.google.http-client</groupId>
        <artifactId>google-http-client-jackson2</artifactId>
        <version>1.23.0</version>
    </dependency>
    <dependency>
        <groupId>com.google.http-client</groupId>
        <artifactId>google-http-client</artifactId>
        <version>1.23.0</version>
    </dependency>
    <!-- Test Dependencies -->
    <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>org.mockito</groupId>
        <artifactId>mockito-all</artifactId>
        <version>2.0.2-beta</version>
        <scope>test</scope>
    </dependency>
    <dependency>
        <groupId>com.google.appengine</groupId>
        <artifactId>appengine-api-stubs</artifactId>
        <version>${appengine.version}</version>
        <scope>test</scope>
    </dependency>

    <!-- Mail Dependencies with MailGun -->
    <dependency>
        <groupId>com.sun.jersey</groupId>
        <artifactId>jersey-core</artifactId>
        <version>1.19.1</version>
    </dependency>
    <dependency>
        <groupId>com.sun.jersey</groupId>
        <artifactId>jersey-client</artifactId>
        <version>1.19.1</version>
    </dependency>
    <dependency>
        <groupId>com.sun.jersey.contribs</groupId>
        <artifactId>jersey-multipart</artifactId>
        <version>1.19.1</version>
    </dependency>
    <!-- End Mail Dependencies with MailGun -->
    <dependency>
        <groupId>org.freemarker</groupId>
        <artifactId>freemarker-gae</artifactId>
        <version>2.3.25-incubating</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-databind</artifactId>
        <version>${jackson.version}</version>
    </dependency>

    <dependency>
        <!-- Note: core-annotations version x.y.0 is generally compatible with
             (identical to) version x.y.1, x.y.2, etc. -->
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-annotations</artifactId>
        <version>${jackson.version}</version>
    </dependency>
    <dependency>
        <groupId>com.fasterxml.jackson.core</groupId>
        <artifactId>jackson-core</artifactId>
        <version>${jackson.version}</version>
    </dependency>

</dependencies>

<build>
    <!-- for hot reload of the web application-->
    <outputDirectory>${project.build.directory}/${project.build.finalName}/WEB-INF/classes</outputDirectory>
    <plugins>
        <plugin>
            <groupId>org.codehaus.mojo</groupId>
            <artifactId>versions-maven-plugin</artifactId>
            <version>2.1</version>
            <executions>
                <execution>
                    <phase>compile</phase>
                    <goals>
                        <goal>display-dependency-updates</goal>
                        <goal>display-plugin-updates</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>        
            <plugin>
                <groupId>com.google.cloud.tools</groupId>
                <artifactId>appengine-maven-plugin</artifactId>
                <version>1.3.2</version>
                <configuration>
                    <stage.enableJarClasses>true</stage.enableJarClasses>
                    <cloudSdkPath>/usr/local/Caskroom/google-cloud-sdk/latest/google-cloud-sdk</cloudSdkPath>
                </configuration>
            </plugin>
        <plugin>
            <groupId>com.google.cloud.tools</groupId>
            <artifactId>appengine-maven-plugin</artifactId>
            <version>1.3.2</version>
            <configuration>
                <!-- deploy configuration -->
            </configuration>

        </plugin>

        <plugin>
            <groupId>com.google.cloud.tools</groupId>
            <artifactId>endpoints-framework-maven-plugin</artifactId>
            <version>1.0.2</version>
            <configuration>
                <!-- plugin configuration -->
                <hostname>project-id.appspot.com</hostname>

            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>discoveryDocs</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>

Here is my web.xml file

 <?xml version="1.0" encoding="UTF-8"?>
<web-app version="2.5" xmlns="http://java.sun.com/xml/ns/javaee" xmlns:web="http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd">
    <!-- Endpoints web.xml -->
    <!-- [START web] -->
    <servlet>
        <display-name>Remote API Servlet</display-name>
        <servlet-name>RemoteApiServlet</servlet-name>
        <servlet-class>com.google.apphosting.utils.remoteapi.RemoteApiServlet</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>
    <servlet>
        <servlet-name>Grader</servlet-name>
        <servlet-class>com.softquest.apollo.servlets.Grader</servlet-class>
    </servlet>
    <servlet>
        <servlet-name>SheetHandler</servlet-name>
        <servlet-class>com.softquest.apollo.servlets.SheetHandler</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>RemoteApiServlet</servlet-name>
        <url-pattern>/remote_api</url-pattern>
    </servlet-mapping>
    <!-- Score Sheet Servlet -->
    <!-- Remote Api Servlet -->
    <servlet>
        <servlet-name>EndpointsServlet</servlet-name>
        <servlet-class>com.google.api.server.spi.EndpointsServlet</servlet-class>
        <init-param>
            <param-name>services</param-name>
            <param-value>com.softquest.apollo.services.UserServices</param-value>
        </init-param>
    </servlet>
    <!-- Payment Response Servlet -->
    <servlet>
        <servlet-name>PaymentResponseServlet</servlet-name>
        <servlet-class>com.softquest.apollo.servlets.PaymentResponseServlet</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>PaymentResponseServlet</servlet-name>
        <url-pattern>/PaymentResponse</url-pattern>
    </servlet-mapping>
    <!-- Payment Response Servlet-->
    <servlet-mapping>
        <servlet-name>EndpointsServlet</servlet-name>
        <url-pattern>/_ah/api/*</url-pattern>
    </servlet-mapping>
    <!-- Objectify filter registration -->
    <filter>
        <filter-name>ObjectifyFilter</filter-name>
        <filter-class>com.googlecode.objectify.ObjectifyFilter</filter-class>
    </filter>
    <filter-mapping>
        <filter-name>ObjectifyFilter</filter-name>
        <url-pattern>/*</url-pattern>
    </filter-mapping>
    <!-- Setup ServletContextListener to start up objectify entities on application -->
    <servlet>
        <servlet-name>BackEndStarter</servlet-name>
        <servlet-class>com.softquest.apollo.BackEndStarter</servlet-class>
        <load-on-startup>1</load-on-startup>
    </servlet>
    <servlet-mapping>
        <servlet-name>BackEndStarter</servlet-name>
        <url-pattern>/backend/starter</url-pattern>
    </servlet-mapping>
    <!-- Migration servlet -->
    <servlet>
        <servlet-name>MigrationServlet</servlet-name>
        <servlet-class>com.softquest.apollo.servlets.MigrationServlet</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>MigrationServlet</servlet-name>
        <url-pattern>/migration/*</url-pattern>
    </servlet-mapping>
    <!-- Migration Servlet-->
    <!-- Migration Handler Servlet -->
    <servlet>
        <servlet-name>MigrationHandler</servlet-name>
        <servlet-class>com.softquest.apollo.servlets.MigrationHandler</servlet-class>
    </servlet>
    <servlet-mapping>
        <servlet-name>MigrationHandler</servlet-name>
        <url-pattern>/migration-handler/*</url-pattern>
    </servlet-mapping>
    <!-- End of Migration handler Servlet-->
    <!-- Setup web.xml to start up context listener -->
    <servlet-mapping>
        <servlet-name>Grader</servlet-name>
        <url-pattern>/Grader</url-pattern>
    </servlet-mapping>
    <servlet-mapping>
        <servlet-name>SheetHandler</servlet-name>
        <url-pattern>/SheetHandler</url-pattern>
    </servlet-mapping>
    <welcome-file-list>
        <welcome-file>index.html</welcome-file>
    </welcome-file-list>
</web-app>

I get this error when trying to call the api discovery doc, I have traced it to a null pointer error on line 72 of the

com.google.api.server.spi.EndpointsServlet file

if (!dispatcher.dispatch(method, path, context)) {
        response.setStatus(HttpServletResponse.SC_NOT_FOUND);
        response.getWriter().append("Not Found");
      }

I am guessing the error is released to some third party library probably still using any of the older cloud endpoint maven dependencies or plugins. So I would remove all the dependencies from my pom.xml and try to deploy.

UPDATE: I was able to fix the API discovery error, it was caused by a reflection error that had to do with the Google Guava Library, while the migration Doc had stated that Version 19 would work, it didn't, upgrading to version 20 and above fixed that for me, now I am getting error 404 on all the endpoints. I would write an answer when I am able to fix this.

Im getting the same problem on an endpoint that hasn't been changed in months, nothing is appearing in the logs either. I assume its probably an app engine problem.


EDIT

I fixed our problem, though you mention the you've attempted to update to the new Cloud Endpoint v2.0 and you are still having the problem it turns out for our project following the migration steps from v1.1 to v2 fixed the problem https://cloud.google.com/endpoints/docs/frameworks/python/migrating.

The docs say v1.1 was to be turned off August 2nd, I guess they have just got round to it

Troubleshooting response errors, BAD_GATEWAY; API is not enabled for the project; Service control request failed like "API my-api.endpoints.example-project-12345.cloud.goog is not enabled for If you send a request and get back an HTTP 502 , 503 , or some other server return the expected result, try sending the request using another application. An App Engine application can easily cache data in an instance's global memory and in Memcache. In some cases, you will benefit from more aggressive caching strategies. For example, you can cache data on your mobile clients, which then update your application asynchronously.

Troubleshooting response errors, GitHub is home to over 50 million developers working together to host If my app engine application is running on localhost:8080 and my _request function in gcloud/datastore/connection.py which returned GoogleCloudPlatform/​google-cloud-datastore#21 There's not really anything we can act on. Figure 1: Tomoaki Suzuki, App Engine lead engineer at Applibot, Inc. Applibot is one of the major social application providers in Japan. The uniqueness of the company is their extensive knowledge and experience in building super-scalable social game services on Google App Engine, the Platform as a Service (PaaS) offering from Google. Leveraging

As saiyr stated in the comments, Google Cloud Endpoints v1.1 has been completely deprecated and has stopped serving traffic and that is the reason for the errors I was having. I solved this problem by migrating to cloud endpoints v2, the process is as easy as following the guide in the migration docs but here are some issues I had with Migration and how I fixed it.

  1. Google guava library has to be updated to a version greater than version 19, using version 19 or lower could result in the application throwing reflection errors when handling requests.

  2. If you have multiple service classes they have to be added to the web.xml file as service parameters while declaring the com.google.api.server.spi.EndpointsServlet Servlet. for some reasons listing only one service class worked for me in cloud endpoint 1.0 Api docs and routes were generated for all the service classes I had without declaring everyone of them explicitly under the com.google.api.server.spi.SystemServiceServlet Servlet, I think it has something to do with Objectify filters though. See the Docs for more info

  3. Since the Google App Engine Java 7 runtime is deprecated too, I decided to migrate to Java 8 but unfortunately the library I used to abstract the process of connecting to datastore (Objectify) appears not to work with the Java 8 runtime, I would look into this later, but reverting back to Java 7 runtime worked for me and the application is back up and running.

Update: The issue I faced migrating to Java 8 runtime had nothing to do with Objectify, the library I used for encrypting passwords (BCrypt) was the issue, it just did not work on the Google App Engine Java 8 runtime, I had to switch to the Encryption library in spring security.

"503 Backend Error" when connecting to App Engine local , 503 Error when trying anything #294 Is gcloud-node that is not working or the gcloud command line tool (the one you use to gcloud auth  Tips and tricks you didn't know you could do with Google for on the go, at work and having fun.

Issue #294 · googleapis/google-cloud-node, We tried to let the version self-heal, but it wasn't working. The version The Cloud Console Log Viewer does not show this HTTP request at all. Visit Google Play to find the latest apps on the market. Browse now. Get recommendations for new apps from our editors. Explore now.

App Engine Flexible -> Random 503 Errors, App Engine site returning 503 Over Quota when I am under quota Are there other sources of quota violations that are not listed on the Quota Details page? Try reuploading your application and clearing the application's 

App Engine site returning 503 Over Quota when I , When we use the App Engine webapp framework, it automatically knows how Depending on how you are running your App Engine, the log is either in a log for the application we just ran: apps csev$ dev_appserver.py ae-01-guess/ INFO 301 = Document has been moved (redirect), 404 = File not found, and 503 

Comments
  • Hello Ogbe, We are also struggling with the same and have been trying everything to make it working again since morning. Please add a comment if you have figured it out. This link claims that problems have been solved with Google app engine. Please have a look status.cloud.google.com/incident/appengine/18006
  • Endpoints v1 has been turned off. If you're getting issues after migrating when generating docs, you probably haven't migrated your endpoints Maven plugin yet. Please post your pom.xml if you're having trouble.
  • @saiyr I have posted my pom.xml file, I cant figure out why I have the error "API discovery response missing required fields."
  • Please include also your web.xml and when do you get this error?
  • @saiyr I have added the web.xml file
  • Do you get 503 and that HTML too?
  • Yes the exact same.