filter include and filter fields don't work well together

[ywang-clarify]

I have 3 models A, B, C

A belongs to B
B belongs to C

I want to query A, include B, include C. But I only want a few fields from each model.
Using a filter like this without specifying any "fields" works fine and brings back the nested models.

{
  "include": {
    "relation": "modelB",
    "scope": {
      "include": {
        "relation": "modelC"
      }
    }
  }
}

But as soon as I add a "fields" filter property to any level of the query, all included relations from that level seem to be ignored. So if I queried with this filter:

{
  "fields": ["modelAField"],
  "include": {
    "relation": "modelB",
    "scope": {
      "include": {
        "relation": "modelC"
      }
    }
  }
}

I only get modelA instances with modelAField - as if I didn't ask for modelB to be included at all. If I omit "fields" at the top level and add it to the modelB scope, then I get all fields of modelA, the subset of modelB fields I ask for, but no inclusion of modelC.

Other than the differences in relationType (belongsTo instead of hasMany), I think this is similar to the example at Include Filter Examples

An unrelated issue, but I can't run unit tests successfully. I get:

... after a bunch of jshint grunt tasks ...
Running "mochaTest:unit" (mochaTest) task

..........................................................................................
..................................The configuration of MyTestModel is missing dataSource property.
Use null or false to mark models not attached to any data source.
........................................................
...............................................................Property base cannot be reconfigured for TestModel-26
...........................
...............................................The configuration of MyModel is missing dataSource property.
Use null or false to mark models not attached to any data source.
The configuration of SubMyModel is missing dataSource property.
Use null or false to mark models not attached to any data source.
The configuration of MyModel is missing dataSource property.
Use null or false to mark models not attached to any data source.
The configuration of SubMyModel is missing dataSource property.
Use null or false to mark models not attached to any data source.
...........................................
..........................................................................................
..........................................................................................
..........................................................................................
.....................................

653 passing (15s)
13 pending
1 failing

1. app app.dataSource adds data source name to error messages:
   AssertionError: expected [Function] to throw error matching /bad-ds.*throwing/ but got 'Cannot create data source "bad-ds": expected test error'
   at Context. (C:\src\loopback\test\app.test.js:743:18)
   at callFn (C:\src\loopback\node_modules\mocha\lib\runnable.js:315:21)
   at Test.Runnable.run (C:\src\loopback\node_modules\mocha\lib\runnable.js:308:7)
   at Runner.runTest (C:\src\loopback\node_modules\mocha\lib\runner.js:422:10)
   at C:\src\loopback\node_modules\mocha\lib\runner.js:533:12
   at next (C:\src\loopback\node_modules\mocha\lib\runner.js:342:14)
   at C:\src\loopback\node_modules\mocha\lib\runner.js:352:7
   at next (C:\src\loopback\node_modules\mocha\lib\runner.js:284:14)
   at Immediate._onImmediate (C:\src\loopback\node_modules\mocha\lib\runner.js:320:5)

Warning: Task "mochaTest:unit" failed. Use --force to continue.

Aborted due to warnings.

I tried adding a new unit test case to describe('Example Remote Method') in model.test.js:

    it('Call the findById with filter.include and filter.fields using HTTP / REST', function(done) {
      request(app)
        .post('/users')
        .send({first: 'x', last: 'y'})
        .expect('Content-Type', /json/)
        .expect(200)
        .end(function(err, res) {
          if (err) return done(err);
          var userId = res.body.id;
          assert(userId);
          request(app)
            .post('/users/' + userId + '/posts')
            .send({title: 'T1', content: 'C1'})
            .expect('Content-Type', /json/)
            .expect(200)
            .end(function(err, res) {
              if (err) return done(err);
              var post = res.body;
              request(app)
                .get('/users/' + userId + '?filter[include]=posts&filter[fields]=first')
                .expect('Content-Type', /json/)
                .expect(200)
                .end(function(err, res) {
                  if (err) return done(err);
                  assert.equal(res.body.first, 'x', 'first should be x');
                  assert(res.body.last === undefined, 'last should not be present');
                  assert.deepEqual(post, res.body.posts[0]);
                  done();
                });
            });
        });
    });

But it results in an exception instead of an assertion failure or success. I'm not sure why that is since filters of this format are working for me in my models A, B, C case. I also tried rephrasing the filter as JSON and got the same type exception.

2. Remote Methods Example Remote Method Call the findById with filter.include and filter.fields using HTTP / REST:
   Uncaught TypeError: Cannot read property 'toString' of undefined
   at KVMap.keyToString (C:\src\loopback\node_modules\loopback-datasource-juggler\lib\include_utils.js:64:46)
   at Object.KVMap.mapImpl.set (C:\src\loopback\node_modules\loopback-datasource-juggler\lib\include_utils.js:70:13)
   at Object.buildOneToOneIdentityMapWithOrigKeys (C:\src\loopback\node_modules\loopback-datasource-juggler\lib\include_utils.js:19:11)
   at includeHasManySimple (C:\src\loopback\node_modules\loopback-datasource-juggler\lib\include.js:498:36)
   at processIncludeItem (C:\src\loopback\node_modules\loopback-datasource-juggler\lib\include.js:275:16)
   at C:\src\loopback\node_modules\loopback-datasource-juggler\lib\include.js:173:5
   at C:\src\loopback\node_modules\loopback-datasource-juggler\node_modules\async\lib\async.js:157:13
   at _each (C:\src\loopback\node_modules\loopback-datasource-juggler\node_modules\async\lib\async.js:57:9)
   at Object.async.each (C:\src\loopback\node_modules\loopback-datasource-juggler\node_modules\async\lib\async.js:156:9)
   at Function.Inclusion.include (C:\src\loopback\node_modules\loopback-datasource-juggler\lib\include.js:172:9)
   at C:\src\loopback\node_modules\loopback-datasource-juggler\lib\connectors\memory.js:433:33
   at C:\src\loopback\node_modules\async-listener\glue.js:188:31

Any help would be appreciated. Thanks.

[ywang-clarify]

I walked through the loopback-connector code and it looks like what's happening is that because I didn't include the foreign key field in my "fields" array, the initial query didn't bring back enough information to correctly write the query to bring back the related objects.

The unit tests use the memory datasource, so this probably wouldn't be an issue. The problem is that the sql connector really should be automatically adding foreign key fields to the "fields" property if an include is going to need it. So I guess this issue belongs in loopback-connector.

[ywang-clarify]

So even though I know what's causing my initial problem. I'm still curious as to why I get these test failures.

[loay]

Hi @ywang-clarify
Is the issue still persisting?
Thanks

[ywang-clarify]

We're still on version 2.36.2 and I haven't tried it on the latest. But it should be easy to test. Basically, try to include any relation and at the same time, include a "fields" array, but do not include id or foreign key properties in the "fields" list. In version 2.36.2, it silently ignores the relation you tried to include because you've specified a fields list that doesn't include enough information to traverse the relationship.

[fleita9]

Had the same problem. This comment helped me