Read embedded entity from python ndb client

ndb-python 3
datastore delete entity
google cloud ndb
ndb properties
google cloud ndb python 3
google cloud datastore python
datastore create entity python
ndb sort

I am using the google cloud datastore python client to write an entity into the datastore which contains an embedded entity. An example entity might look like:

data_type: 1
raw_bytes: <unindexed blob>
values: <indexed embedded entity>

I checked the data from the console and the data is getting saved correctly and the values are present.

Next, I need to run a query from a python app engine application. I have represented the above as the following entity in my app engine code:

class DataValues(ndb.Model):
    param1 = ndb.BooleanProperty()
    param2 = ndb.IntegerProperty()
    param3 = ndb.IntegerProperty()

class MyEntity(ndb.Expando):
    data_type = ndb.IntegerProperty(required=True)
    raw_bytes = ndb.BlobProperty()
    values = ndb.StructuredProperty(DataValues)        

One of the filters in the query depends on a property in values. Sample query code is as below:

MyEntity.query().filter(MyEntity.data_type == 1).filter(MyEntity.values.param1 == True).get()

I have created the corresponding composite index in my index.yaml The query runs successfully but the resulting entity contains the embedded entity values as None. All other property values are present.

What can be the issue here ?

Add properties of DataValues entity as properties of the MyEntity.

The Python 2 NDB Client Library Overview, Platform for BI, data applications, and embedded analytics. The Google Datastore NDB Client Library allows App Engine Python apps to which typically gives fast and inexpensive reads via an in-context cache and Memcache​. Each entity is identified by a key, an identifier unique within the application's datastore. NDB invalidates its caches of the affected entity/entities. Thus, future reads will read from (and cache) the underlying Datastore instead of reading stale values from the cache. Finally, perhaps seconds later, the underlying Datastore applies the change. It makes the change visible to global queries and eventually- consistent reads.

This is a bit of a guess, but since datastore attributes are kind of keyed by both their name (in this case values) and the name of the "field type/class" (i.e. StructuredProperty), this might fix your problem:

class EmbeddedProperty(ndb.StructuredProperty):
    pass

class MyEntity(ndb.Expando):
    data_type = ndb.IntegerProperty(required=True)
    raw_bytes = ndb.BlobProperty()
    values = EmbeddedProperty(DataValues) 

Give it a shot and let me know if values starts coming back non-null.

Creating, Retrieving, Updating, and Deleting Entities, Apps that use these APIs will need to upgrade to Cloud NDB before migrating to the App Engine Python 3 runtime. Data objects in Datastore are known as entities​,  Great man!!, it works perfectly. I was missunderstanding a bit the keys with the ancestors, but I think now I understand it better. In my by_page method I was passing the chapter entity but searching by ancestor=chapter.key which is idiotic, because I only need the key.

I struggled with the same problem, wanting to convert the embedded entity into a Python dictionary. One possible solution, although not a very elegant one, is to use a GenericProperty:

class MyEntity(ndb.Model):
    data_type = ndb.IntegerProperty(required=True)
    raw_bytes = ndb.BlobProperty()
    values = ndb.GenericProperty()

values will then be read as an "Expando" object: Expando(param1=False,...). You can access the individual values with values.param1, values.param2 etc. I would prefer having a custom model class, but this should do the job.

Storing embedded entity, it's ignoring exclude_from_indexes from , I'm trying to have a embedded entity with a field that is larger than 1500bytes. record_key = client.key('Record', my_id) record_entity = datastore. Closed. dhermes added a commit to dhermes/google-cloud-python that Size/length limits for Datastore properties are enforced differently than NDB #1228. Apps that use these APIs will need to upgrade to Cloud NDB before migrating to the App Engine Python 3 runtime. Each entity is identified by a key that is unique within the application's Datastore instance, and consists of the following: kind.

python-ndb/model.py at master · googleapis/python-ndb · GitHub, context = context_module.Context(client).use() No validation happens when an entity is read from Cloud Datastore; however. property values read native support for embedded entities and NDB now uses that, by. # default. This handles  Python-NDB-ORM. NDB-ORM is a python3 compatible orm for google cloud datastore based on ndb.model (created by Guido van Rossum) and may be used outside of app engine standard environment. Plays nicely with datastore entities generated from app engine standard; Python3 compatible; Usable from outside of app engine standard; Based on the original ndb package

google.cloud.ndb.model, Context(client).use() context. Repeated properties are always represented as Python lists; if there is only one value, the list has No validation happens when an entity is read from Cloud Datastore; however property values Datastore now has # native support for embedded entities and NDB now uses that, by # default. Python-NDB-ORM. NDB-ORM is a python3 compatible orm for google cloud datastore based on ndb.model (created by Guido van Rossum) and may be used outside of app engine standard environment. Plays nicely with datastore entities generated from app engine standard; Python3 compatible; Usable from outside of app engine standard; Based on the original ndb package

Does usage of customer key id generates additional GAE datastore , Read embedded entity from python ndb client. I am using the google cloud datastore python client to write an entity into the datastore which contains an  * Plays nicely with datastore entities generated from app engine standard * Python3 compatible * Usable from outside of app engine standard * Based on the original ndb package * Protobuffer implementation was ported to google.cloud.proto.datastore.v1.entity_pb2 (which is also used by gcloud-datastore-python)

Comments
  • Let me give this a try. Will let you know. Thanks !
  • @sshntt can you confirm whether this solution solved your issue or not.
  • Sorry for the delay in responding back but as of now, I have not got around to implement the above solution. In the above case, for now, I removed the DataValues entity and added its properties as properties of the MyEntity. I will try to implement the above solution in another project and respond back here. If anyone gives it a go before me, please let me know here, so that I can accept the solution. Sorry for the delay again.
  • @sshntt could you write your solution as an answer?