Some issues and Error format heterogeneity

[sindbadz]

Hi every one
First of all I would like to thank you for this great project!

I am starting to develop a mircoservice in node.js to deal with GC datastore and gstore will help me a lot.

After a week trying to build my requests (create, update, delete or get entities) I 'm facing some behaviours that seem unclear to me.

I will try to describe these points:
-Enity.save (update mode) vs Model.update
-Error issue with Joi and Model.update
-Result/Error heterogeneity

1/ Enity.save (update mode) vs Model.update

I've tried both solutions to update my entities and discovered than when I use Model.update(), only the new values where updated whereas using Entity.save(null, { method: 'update' }), the all dataset stored in datastore was replace by the new one.
I might not perfectly understand the documentation, but I am not sure that the differences are explicitly described.

2/ Error issue with Joi and Model.update

I am using Joi to define my schemas and it works great in most of my tests. Neverthelss I found en little problem with Model.update error in the case where entity is not found (id not exising, or namespace not specified for example).

Indeed, that's what I get in these conditions with Entity.save(null, { method: 'update' }) :

{ Error: no entity to update: app: "..."
name_space: "..."
path <
  Element {
    type: "..."
    id: ...
  }
>
  code: 5,
  details: ...
 } }

and that's what I obtain with Model.update() (here with an unspecified namespace):

{ ValidationError: child "checkboxA1" fails because ["checkboxA1" is required]
  isJoi: true,
  name: 'ValidationError',
  details:
   [ { message: '"checkboxA1" is required',
       path: [Array],
       type: 'any.required',
       context: [Object] } ],
  _object:
   { [Symbol(KEY)]: Key { namespace: undefined, kind: '...', path: [Getter] } },
  annotate: [Function] }

I have absolutely no validation issue regardless to my joi schema, this error appears only when entity is not found.

3/ Result/Error heterogeneity

I have notices some differences in errors and results format between the diffrent types of request I have done with Gstore.
Before using gstore, I was creating "hand-made" request to datastore.
In order to bring more explicit comparisons, I will also present errors/results obtained with hand-made requests.

a) update informations in results (create, update, delete)

• Create

-with hand-made request(datastore.insert(entity)) :

[ { mutationResults:
     [ { key:
          { path: [ { kind: '...', id: '...', idType: 'id' } ],
            partitionId: { projectId: '...', namespaceId: '...' } },
         version: '1558688427677000',
         conflictDetected: false } ],
    indexUpdates: 41 } ]

-with Entity.save({method: insert}):
no informations

• Update

-with hand-made request (datastore.update(entity)):

[ { mutationResults:
     [ { key: null,
         version: '1558686100928000',
         conflictDetected: false } ],
    indexUpdates: 4 } ]

-with Entity.save(null, { method: 'update' }) :
no informations

-withModel.update() :
no informations

• Delete

-with hand-made request (datastore.delete()):

{ mutationResults:
   [ { key: null,
       version: '1558688022947000',
       conflictDetected: false } ],
  indexUpdates: 41 }

-with Model.delete():

{ mutationResults:
   [ { key: null,
       version: '1558687176725000',
       conflictDetected: false } ],
  indexUpdates: 41,
  key:
   Key {
     namespace: '...',
     id: 5668779698683904,
     kind: '...',
     path: [Getter] },
  success: true }

b) errors when no entity found (wrong id, missing namespace....)

• Update (wrong id, missing namespace)

-with hand-made request or with Entity.save(null, { method: 'update' }):

{ Error: no entity to update: app: "..."
name_space: "..."
path <
  Element {
    type: "..."
    id: ...
  }
>
  code: 5,
  details: ...
 } }

-with Model.update() : previously exposed Bug

{ ValidationError: child "checkboxA1" fails because ["checkboxA1" is required]
  isJoi: true,
  name: 'ValidationError',
  details:
   [ { message: '"checkboxA1" is required',
       path: [Array],
       type: 'any.required',
       context: [Object] } ],
  _object:
   { [Symbol(KEY)]: Key { namespace: undefined, kind: '...', path: [Getter] } },
  annotate: [Function] }

• Delete (wrong id, missing namespace)

-with hand-made request: no error, informations are in results

{ mutationResults:
   [ { key: null,
       version: '1558687176725000',
       conflictDetected: false } ],
  indexUpdates: 0 }

-with Model.delete(): no error, informations are in results

  { mutationResults: [ { key: null, version: '1', conflictDetected: false } ],
  indexUpdates: 0,
  key:
   Key { namespace: '...', id: ..., kind: '...', path: [Getter] },
  success: false }

• Get

-with hand-made get (datastore.get(key)) : no error, informations are in results (wrong id, missing namespace)
[ undefined ]

-with Model.get() (only one id): (wrong id, missing namespace)

{
 GstoreError: MyKind { myid } not found
 name: 'GstoreError',
 code: 'ERR_ENTITY_NOT_FOUND'
 }

-with hand-made get for array of ids (datastore.get(keys)) : no error, informations are in results (wrong id, missing namespace)
result[0] = []

-with Model.get (array of ids): no error, informations are in results (wrong id)

[]

-with Model.get (array of ids): (missing namespace)

{ Error: Key path element must not be incomplete: [MyKind: ]
  code: 3,
  details: 'Key path element must not be incomplete: [MyKind: ]',
  metadata: Metadata { internalRepr: Map {} },
  note:
   'Exception occurred in retry method that was not classified as transient' }

-with hand-made "get all entities" (datastore.runQuery(datastore .createQuery(namespace, kind)): no error, informations are in results (missing namespace)
result[0] = []

-with "get all entities" Model.query(namespace).run(): no error, informations are in results (missing namespace)

{ entities: [] }

I have propably missed some infos in documentation, but if I don't, it could be good to have more homogeneous error formats

4/ Operator issues on queries filters

I have tried to get entities filtering regardless to the existence of a property but I had some issues.
What I have done:

myModel.query()
	 .filter('myPropetry', '!=', null)
	.run()

or

myModel..findOne({ card: !null })

In both cases I get Error: a property filter must specify an operator
The errors appears also with the IN operator.
Theses two operators do not seem available for Node.js (https://cloud.google.com/nodejs/docs/reference/datastore/1.4.x/Query).
I do not known if there is another way to make similar filters?
It might be interesting to list the available operators in the documentation.

I hope my post is clear enough.
Good week-end to you.

[sebelga]

Hello,
Thanks for raising this and for your detailed analysis on the error/result. I am currently traveling so I can't answer each item separately as I'd like, but I will as soon as I am back (2 weeks aprox).

A quick note on the errors, there is indeed some differences like in the GET where the hand-made does not return anything when the entity is not found but where gstore does throw a 404 error in that case (there is, though, a global config setting to turn off that behaviour).

I thought that

{
 GstoreError: MyKind { myid } not found
 name: 'GstoreError',
 code: 'ERR_ENTITY_NOT_FOUND'
 }

was a better handled error for a entity that is not found error than... nothing. But that's opinated, I agree 😊

I will go through each of your use cases and comments. There is surely room for improvement.

As for the difference between the save() with { method: 'update' } vs Model.update(). The Model.update() does

1. Model.get()
2. Merge data
3. Model.save()

In a single transaction, to ease updating only a few properties so you don't need to do the above manually.
The { method: 'update' } means to only do the update if the entity already exists, but it will update (= replace) all the entity data.

Cheers

[sindbadz]

Hi,
Your wellcome :)

I agree that getting the :

{
 GstoreError: MyKind { myid } not found
 name: 'GstoreError',
 code: 'ERR_ENTITY_NOT_FOUND'
 }

is way better than getting nothing. I guess it could be even better if we get this error for all the different type of request when no entity is found.

Thanks a lot for your response.
Waiting for futher answers and solutions.

[sindbadz]

I have updated my first post adding point 4/ about query filter operators

[sindbadz]

Hi !
Is there any improvement about these points?

[sebelga]

Hello, I am hoping to take back my OSS work next week. I am not sure how much time I will be able to dedicate but at least get something moving forward 😊
Sorry for so much delay...

[sebelga]

1/ Enity.save (update mode) vs Model.update

Indeed both are different. The Entity.save(null, { method: 'update' }) maps to the google datastore update() method. The Model.update() is some sort of PATCH to update an entity with a new prop, without the need to provide the complete entity data. It does for you, under the hood, a GET + SAVE inside a transaction.

[EDIT]: I updated the docs to make this clearer. Thanks for the input! 👍

2/ Error issue with Joi and Model.update

Could you open a separate issue and copy the content there? I will try to reproduce but if I can't it will be better to talk about it in a separate thread.

3/ Result/Error heterogeneity

By design, there was no intention to return the same error that the underlying google datatstore lib. So differences between the two ("hand-made" and gstore) are expected. I will have a look and decide if it makes sense to add a datastoreResponse property to the gstore responses.

4/ Operator issues on queries filters

Indeed this 2 operators are not allowed. Here are the latest doc with the allowed operators:
https://googleapis.dev/nodejs/datastore/latest/Query.html#filter
I will link to them in the doc, thanks for the input!

[sebelga]

Point 2 was indeed a bug and I opened a PR to fix it (#181). Thanks for raising it! 👍

When the PR will be merged it will automatically close this issue. Please re-open new issues for the other points you'd like to be addressed.

The fix will be available in the next major version (v7.0.0)

[muhrezbas]

i want to ask why populate always null
PublicCredit.list(options).populate('user') .then((credits) => { res.status(200).json(credits); console.log(credits, "tes") }) .catch(next);``PublicCredit.list(options).populate('user') .then((credits) => { res.status(200).json(credits); console.log(credits, "tes") }) .catch(next);