query in datastore without a model

datastore query by key
datastore query examples
datastore query limit
datastore get all entities
datastore query by id
datastore query golang
datastore query array
datastore projection query

I am trying to create an application in appengine that searches for a list of keys and then I use this list to delete these records from the datastore, this service has to be a generic service so I could not use a model just search by the name of kind, it is possible to do this through appengine features?

Below my code, but it requires that I have a model.

import httplib import logging from datetime import datetime, timedelta

import webapp2
from google.appengine.api import urlfetch
from google.appengine.ext import ndb

DEFAULT_PAGE_SIZE = 100000
DATE_PATTERN = "%Y-%m-%dT%H:%M:%S"


def get_date(amount):
    date = datetime.today() - timedelta(days=30 * amount)
    date = date.replace(hour=0, minute=0, second=0)
    return date


class Purge(webapp2.RequestHandler):

    def get(self):
        kind = self.request.get('kind')
        datefield = self.request.get('datefield')
        amount = self.request.get('amount', default_value=3)
        date = get_date(amount)

        logging.info('Executando purge para Entity {}, mantendo periodo de {} meses.'.format(kind, amount))

        # cria a query
        query = ndb.Query(kind=kind, namespace='development')

        logging.info('Setando o filtro [{} <= {}]'.format(datefield, date.strftime(DATE_PATTERN)))

        # cria um filtro
        query.filter(ndb.DateTimeProperty(datefield) <= date)

        query.fetch_page(DEFAULT_PAGE_SIZE)

        while True:
            # executa a consulta
            keys = query.fetch(keys_only=True)

            logging.info('Encontrados {} {} a serem exluidos'.format(len(keys), kind))

            # exclui usando as keys
            ndb.delete_multi(keys)

            if len(keys) < DEFAULT_PAGE_SIZE:
                logging.info('Nao existem mais registros a serem excluidos')
                break


app = webapp2.WSGIApplication(
    [
        ('/cloud-datastore-purge', Purge),
    ], debug=True)

Trace

Traceback (most recent call last):
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1535, in __call__
    rv = self.handle_exception(request, response, e)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1529, in __call__
    rv = self.router.dispatch(request, response)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1278, in default_dispatcher
    return route.handler_adapter(request, response)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 1102, in __call__
    return handler.dispatch()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 572, in dispatch
    return self.handle_exception(e, self.app.debug)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/third_party/webapp2-2.5.2/webapp2.py", line 570, in dispatch
    return method(*args, **kwargs)
  File "/base/data/home/apps/p~telefonica-dev-155211/cloud-datastore-purge-python:20180629t150020.410785498982375644/purge.py", line 38, in get
    query.fetch_page(_DEFAULT_PAGE_SIZE)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/ext/ndb/utils.py", line 160, in positional_wrapper
    return wrapped(*args, **kwds)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 1362, in fetch_page
    return self.fetch_page_async(page_size, **q_options).get_result()   
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 383, in get_result
    self.check_success()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along
    value = gen.throw(exc.__class__, exc, tb)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 1380, in _fetch_page_async
    while (yield it.has_next_async()):
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 427, in _help_tasklet_along
    value = gen.throw(exc.__class__, exc, tb)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 1793, in has_next_async
    yield self._fut
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/ext/ndb/context.py", line 890, in helper
    batch, i, ent = yield inq.getq()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/ext/ndb/query.py", line 969, in run_to_queue
    batch = yield rpc
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/ext/ndb/tasklets.py", line 513, in _on_rpc_completion
    result = rpc.get_result()
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/api/apiproxy_stub_map.py", line 613, in get_result
    return self.__get_result_hook(self)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2951, in __query_result_hook
    self.__results = self._process_results(query_result.result_list())
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/datastore/datastore_query.py", line 2984, in _process_results
    for result in results]
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/datastore/datastore_rpc.py", line 194, in pb_to_query_result
    return self.pb_to_entity(pb)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 690, in pb_to_entity
    modelclass = Model._lookup_model(kind, self.default_model)
  File "/base/alloc/tmpfs/dynamic_runtimes/python27g/7894e0c59273b2b7/python27/python27_lib/versions/1/google/appengine/ext/ndb/model.py", line 3101, in _lookup_model
    kind)
KindError: No model class found for kind 'Test'. Did you forget to import it?

The problem was found on the line where fetch_page is set.

Removing this line

query.fetch_page(DEFAULT_PAGE_SIZE)

for this

keys = query.fetch(limit=_DEFAULT_LIMIT, keys_only=True)

Datastore Queries | Cloud Datastore Documentation, Model. Base class for class hierarchies supporting polymorphic queries. Use this class to For example, Panther entities when stored to Cloud Datastore are of the kind Animal . Querying Set to 0 to try operation only once, with no retries. A query retrieves entities from Firestore in Datastore mode that meet a specified set of conditions.. The query operates on entities of a given kind; it can specify filters on the entities' property values, keys, and ancestors, and can return zero or more entities as results.

To run a datastore query without a model class available in the environment, you can use the google.appengine.api.datastore.Query class from the low-level datastore API.

See this question for other ideas.

Datastore Queries, Amplify DataStore provides a programming model for leveraging shared and distributed data locally to your device with DataStore, even without an AWS account. and operations defined by common verbs such as save , query and delete . Using datastores without a model Every modelin a Sails app is wired up to a particular datastore, so every time you call a built-in model method, the model communicates with its configured datastore implicitly. Even so, it's sometimes useful to be able to communicate with a datastore outsideof the context of any particular model.

Polymorphic Models and Queries — ndb documentation, Google Datastore with a relational data model Since there are no relations ( except for parent/child), managing data becomes On some occasions we choose to embed data in a single entity/table, to avoid multiple queries. Without further modification, the resulting query object will retrieve all existing entities of the kind specified by model_class. The instance methods filter() , ancestor() , and order() can then be used to customize the query with additional filter criteria, ancestor conditions, and sort orders.

DataStore - Getting started - Amplify Docs, Their role is to make queries faster. But it's You can't query the Datastore without indices. Let's say that we have the following User model: Describe the bug Two related issues related to documentation: creating the relationships and querying them using DataStore. The way the Amplify JS docs suggest creating one-to-many relationships seems a bit counterintuitive, it suggests

Google Datastore with a relational data model, You would query for all of the posts written by a particular user using the In Cloud Datastore, there are no tables, instead data entities have a� The model and topology objects are created, but the model contains no datastore yet. Reverse-engineering a Model To automatically populate datastores into the model you need to perform a reverse-engineering for this model.

Optimizing Indices In Google Cloud Datastore, A [GQL query](https://cloud.google.com/datastore/docs/apis/gql/gql_reference). GqlQueryParameter. A binding parameter for a GQL query. Key. A unique� Datastore is a schemaless database, which allows you to worry less about making changes to your underlying data structure as your application evolves. Datastore provides a powerful query engine that allows you to search for data across multiple properties and sort as needed.

Comments
  • Can you add the complete traceback? I'm having difficulty reproducing this.
  • @snakecharmerb added
  • Ok, so query.fetch_page(DEFAULT_PAGE_SIZE) is the problem - do you need this line? It seems to me the code works ok without it.
  • @snakecharmerb I added this to see stackoverflow in the pod, because it has kinds that have many records
  • @snakecharmerb thanks for your help this solved my problem