[RFC] Native UID support in search filter

[bpolaszek]

Q	A
Branch?	main
Tickets	api-platform/api-platform#1958 api-platform/api-platform#2134 #4313 #4656
License	MIT

Hello there,

Given:

• The wide and growing adoption of Ulids as primary keys
• The fact that using Ulids instead of integers as primary keys is encouraged for security and scalability concerns
• The usage of symfony/uid is natively supported in API-Platform as of Add Symfony Uid support #3715
• Docrtrine's SearchFilter currently doesn't work with Ulids as they're serialized as strings instead of binaries when used in DQL queries
• Current proposal to tackle this is to create a duplicate SearchFilter, specifically for handling Ulids, which is tedious and has to be done for every project, without any kind of standardization
• Fixing this seems at first glance as easy as this, may seem crappy/hacky at first sight but actually means "we support scalars or Uids, just like Symfony does"

I know there have been lots of dicussions, even PRs in the past regarding this, at a time when the UID component was more or less experimental and not a lot of people used it, but I think things have changed since then.
Is it something you could reconsider?

[soyuka]

Had to revert #3774 at #4134

You're patch looks way better as it doesn't change doctrine type system 👍 to merge!

[bpolaszek]

That's good news, but there's a final boss to fight: the CI! 😱
I don't even know where to start as it yells almost everywhere! 😅

[soyuka]

yes, the behat failure on abstract tests is related to symfony/symfony#50480 waiting for it to be released... The test on the Maker also needs a fix.

Just add your test and only test yours :)

[bpolaszek]

Sorry for acting as a Behat newbie (I am one), but I can't get the test to pass.

It looks like my UidBasedId API Resource is not properly registered as /uid-based-ids returns 404 - I actually don't get why.

  @createSchema
  Scenario: Get collection by ulid 01H2ZS93NBKJW5W4Y01S8TZ43M                                    # features/doctrine/search_filter.feature:721
    Given there is a UidBasedId resource with id "01H2ZS93NBKJW5W4Y01S8TZ43M"                    # ApiPlatform\Tests\Behat\DoctrineContext::thereIsAUidBasedIdResource()
    When I send a "GET" request to "/uid-based-ids?id=/uid-based-ids/01H2ZS93NBKJW5W4Y01S8TZ43M" # Behatch\Context\RestContext::iSendARequestTo()
    Then the response status code should be 200                                                  # Behat\MinkExtension\Context\MinkContext::assertResponseStatus()
      Current response status code is 404, but 200 expected.

[mrossard]

Shouldn't it just be

Suggested change

	When I send a "GET" request to "/uid-based-ids?id=/uid-based-ids/01H2ZS93NBKJW5W4Y01S8TZ43M"
	When I send a "GET" request to "/uid-based-ids?id=01H2ZS93NBKJW5W4Y01S8TZ43M"

?

[bpolaszek]

Passing just the id doesn't work on my side for some reason I have to figure out - anyway passing the IRI should work, or at least return an empty collection with a 200 status

[soyuka]

almost there

[soyuka]

Suggested change

	When I send a "GET" request to "/uid-based-ids?id=/uid-based-ids/01H2ZS93NBKJW5W4Y01S8TZ43M"
	When I send a "GET" request to "/uid_based_ids?id=/uid-based-ids/01H2ZS93NBKJW5W4Y01S8TZ43M"

[bpolaszek]

Thanks for spotting this! I've been using kebab-case for years, I didn't remember it was not the standard 🙃

[soyuka]

your commit could be: feat(doctrine): search filter uid support

[bpolaszek]

Well, should work better now.

Just 1 annoying thing: when using UIDs, you cannot use the SearchFilter with an ID, only an IRI.
This line returns the raw value (e.g. user input - 01H2ZS93NBKJW5W4Y01S8TZ43M for instance), which is a string and not an AbstractUid. Meaning DQL will keep it as a string instead of translating it to a binary, and it won't find anything.

This method is only aware of the raw value. Ideally it should know:

• Whether $metadata->hasAssociation($field) is true or false
• The class name of the association
• What property name its id is
• What type id is

Sounds relatively complex to set up - unless you know a better approach?

[soyuka]

Yes its a hard problem :/ especially because we don't have a query parameter sanitizer, it'll be better if you knew it was an IRI or not for starters :/. I wish we could have the identifier Type in that method, and we'd have to find out wheter its an IRI or not before getting its value.
Also, maybe that there should be an IriSearchFilter and a SearchFilter ^^.

[soyuka]

also https://github.com/api-platform/core/pull/5623/files#diff-9a51326378c8abdf577e784792a9245eeb219ffc426b09aa1a635bd20fcc51cdR127-R133

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[mrossard]

Forgot this PR...does this fix your issue? #5771

[bpolaszek]

Forgot this PR...does this fix your issue? #5771

Sorry still didn't take the time to review this - will have a look ASAP!

[bpolaszek]

Unfortunately it's not fixing it.
Consider an AccountMember resource with an Account relation having a public Ulid $id as id.

GET /api/account-members.jsonld?account=%2Fapi%2Faccounts%2F01H4K4R4SY577HK9A9XZG4KAGF

GET /api/account-members.jsonld?account=01H4K4R4SY577HK9A9XZG4KAGF

The $id gets casted as string instead of a binary, and the query returns 0 result.

[mrossard]

Unfortunately it's not fixing it. [{...]

Sorry, i misunderstood your use case - the UID is actually the orm id for you, which i didn't account for in my PR! I guees both are actually relevant, and I even might have to copy what you're doing here in some part of mine.

[mrossard]

One thing I'm not sure about : wouldn't using `type: 'symfony_uuid' in your doctrine column definition help? You're using 'ulid' in your test, but I remember having to use symfony_uuid in mine for this to work.

[rubobaquero]

Hello @bpolaszek @mrossard @soyuka ,

Do you have any updates about this PR? Facing this problem in a project that is migrating from ids to UUIDs.

[mrossard]

Hello @bpolaszek @mrossard @soyuka ,

Do you have any updates about this PR? Facing this problem in a project that is migrating from ids to UUIDs.

I haven't been able to work on this for the last few weeks, sorry.

[soyuka]

maybe that we should start working on a new Filter system to address this and #2400, stumble on #319 today which would quite simplify the code around these things...

[mrossard]

stumble on #319 today which would quite simplify the code around these things...

that could help make things clearer, but that would break a lot of things, wouldn't it?

[rubobaquero]

Thank you very much for your answers and your work @soyuka and @mrossard . Hope we can see those changes soon. Meanwhile, in case some other people end up here after a Google Search as I did, what is the suggested way to implement now a Search Filter on a foreign key that is stored as a binary UUID? Typical case of filtering a Book entity by Author UUID fr example.

Thanks in advance!

[bpolaszek]

Thank you very much for your answers and your work @soyuka and @mrossard . Hope we can see those changes soon. Meanwhile, in case some other people end up here after a Google Search as I did, what is the suggested way to implement now a Search Filter on a foreign key that is stored as a binary UUID? Typical case of filtering a Book entity by Author UUID fr example.

Thanks in advance!

Here you go:

<?php

declare(strict_types=1);

namespace App\State\Filter;

use ApiPlatform\Doctrine\Orm\Filter\AbstractFilter;
use ApiPlatform\Doctrine\Orm\Util\QueryNameGeneratorInterface;
use ApiPlatform\Metadata\Operation;
use Doctrine\ORM\QueryBuilder;
use Symfony\Component\HttpKernel\Exception\BadRequestHttpException;
use Symfony\Component\Uid\Ulid;
use Throwable;

use function array_map;
use function is_array;
use function sprintf;
use function str_ends_with;
use function Symfony\Component\String\u;

final class UlidFilter extends AbstractFilter
{
    /**
     * @param string|string[]|null $value
     */
    protected function filterProperty(
        string $property,
        mixed $value,
        QueryBuilder $queryBuilder,
        QueryNameGeneratorInterface $queryNameGenerator,
        string $resourceClass,
        Operation $operation = null,
        array $context = [],
    ): void {
        if (
            null === $value
            || !$this->isPropertyEnabled($property, $resourceClass)
            || !$this->isPropertyMapped($property, $resourceClass, true)
        ) {
            return;
        }

        $alias = $queryBuilder->getRootAliases()[0];
        $valueParameter = ':' . $queryNameGenerator->generateParameterName($property);
        $aliasedField = sprintf('%s.%s', $alias, $property);

        if (is_array($value)) {
            $values = array_map(
                fn (string $value) => $this->convertUlidFromStringToBinary($property, $value),
                $value
            );

            $queryBuilder
                ->andWhere($queryBuilder->expr()->in($aliasedField, $valueParameter))
                ->setParameter($valueParameter, $values);

            return;
        }

        $value = $this->convertUlidFromStringToBinary($property, $value);

        $queryBuilder
            ->andWhere($queryBuilder->expr()->eq($aliasedField, $valueParameter))
            ->setParameter($valueParameter, $value);
    }

    private function convertUlidFromStringToBinary(string $property, string $value): string
    {
        if ('/' === ($value[0] ?? '')) {
            $value = (string) u($value)->afterLast('/');
        }

        try {
            return Ulid::fromString($value)->toBinary();
        } catch (Throwable) {
            throw new BadRequestHttpException("`{$property}`: Unable to parse Ulid `{$value}`");
        }
    }

    /**
     * @return array<string, array<string, mixed>>
     *
     * @codeCoverageIgnore
     */
    public function getDescription(string $resourceClass): array
    {
        if (!$this->properties) {
            return [];
        }

        $description = [];

        foreach ($this->properties as $property => $strategy) {
            if (
                !$this->isPropertyEnabled($property, $resourceClass)
                || !$this->isPropertyMapped($property, $resourceClass, true)
            ) {
                continue;
            }

            /** @var string[] $filterParameterNames */
            $filterParameterNames = [$property, "{$property}[]"];

            foreach ($filterParameterNames as $filterParameterName) {
                $description[$filterParameterName] = [
                    'property' => $property,
                    'type' => 'string',
                    'required' => false,
                    'is_collection' => str_ends_with($filterParameterName, '[]'),
                ];
            }
        }

        return $description;
    }
}

Usage:

#[ApiFilter(UlidFilter::class, properties: ['author' => null])]

[soyuka]

This is awesome, I would like this to be part of API Platform but I'd also like for it not to be added to the Search filter... It's not possible right now I'll work on finding a way to do so.

[rubobaquero]

Thanks a lot @bpolaszek !

[soyuka]

@bpolaszek can you update this? I would like to merge that in the end.

[bpolaszek]

@bpolaszek can you update this? I would like to merge that in the end.

I won't have the time to tackle this until a week or two, I'll do my best.
Don't hesitate to ping me privately if I end up forgetting it

[soyuka]

I tried a quick rebase let's see.

[soyuka]

PostgreSQL and MongoDB have issues with the fix, if someone is willing to take a look we'll gladly merge this.