roadmap for django-filter

[nkryptic]

I think this project is great and extremely useful. But, I also think there are a few things to add, improve on, or remove which could make it even greater. I'm putting this together to determine what to focus on next and into the future.

I wanted to give an overview of all the items, so they can be thought through as a whole. I didn't want to just throw a bunch of individual issues up, since a few of the items relate to multiple existing issues and it makes it difficult to see the big picture.

With some feedback on what to pursue and a few design decisions, I'll be happy to start tackling them.

In no particular order:

overhaul and expand documentation

It's fairly spartan right now. A few things I'd like to see:

• add more customization examples
• add info on "when to declare or override filters"
• add info on 3rd party integration (pagination, floppyforms, crispyforms) and conflicts (django-sorting) - addresses issues Document how django-filter works with django-pagination #13 and django-tagging integration #17
• add info on contributing to django-filter
• add info on writing a custom filter

overhaul and expand test suite

I don't think there are any tests which cover ModelMultipleChoiceFilter there might be other large holes in the test coverage. A few things I'd like to see:

• basic coverage of all parts of the project
• add more tests for edge cases
• address tests that deal with fixtures where dates are used - addresses issue DateRangeFilter tests #32
• split tests/tests.py into multiple files to make adding tests to a relevant section easier
  • tests/filterset_tests.py
  • tests/filter_tests.py
  • tests/view_tests.py

replace RangeFilter and DateRangeFilter

These filters are limited.

• add NumericRangeFilter, TimeRangeFilter, DateRangeFilter and DateTimeRangeFilter
• deprecate RangeFilter and existing DateRangeFilter
  • existing DateRangeFilter has issues when dealing with timezones - how should you calculate 'last 7 days': user's local timezone?

addresses #48 and one of the commits from #41

make declaring 'Choice' filters easier

allow for easier overriding of certain attributes without having to restate everything.

• populate choices from model field on a declared ChoiceFilter and MultipleChoiceFilter when not provided
  • just like what is done in filter_for_field when the field has choices - addresses issue Error with ChoiceFiter and ManyToOneRel #30
• populate queryset on declared ModelChoiceFilter and ModelMultipleChoiceFilter when not provided
  • just like what is done in filter_for_field with the extras - similar to issue Error with ChoiceFiter and ManyToOneRel #30
  • to_field_name also for ModelChoiceFilter

improve ChoiceFilter, MultipleChoiceFilter and AllValuesFilter

• add any option to choices, which is definitely missing - addresses issues AllValuesFilter does not have an empty value #36, Bug in M2M Filter #41 and Add "Any" filter to ChoicesField #45
• optimize filter when model field is required
  • if all options are selected, then no need to filter - addresses issue Bug in M2M Filter #41

allow filtering of non-required fields for BLANKs

It currently almost impossible to filter for records that haven't been assigned a value

• for ChoiceField and MultipleChoiceField, add a blank option
  • e.g. find all Articles without any keywords
• for CharField, I don't know, maybe add a checkbox?
  • e.g. find all Articles without a description

possibly related to issue #41

allow for 'AND'ing a ModelMultipleChoiceFilter's selected values

e.g. find all Articles tagged 'django' AND 'filter' - addresses issue #24

easier overriding of widgets used

when using a non-declared filter, but you want a different widget - see issue #25

• could add a widgets attribute to FilterSet.Meta class?
• may also be a non-issue if declaring 'Choice' filters is made easier

improve order_by

• allow abstraction of ordering values used to avoid passing internal information in GET params (i.e. user__username)
• include ascending and descending values when order_by=True is used
• allow multiple ordering fields to be used at the same time
  • e.g. order by rating (descending) and then by price (ascending)

determine a future for LinkWidget

should it ever be used in a filterset with other filters that aren't using LinkWidget?

• difficult to use when you only want to show links that have results - see issue Hide filter items that produce zero results #29
• we could look to adapting LinkWidget and overall behavior to support filtering like django-easyfilters
• or we could drop the widget

add MultiFieldFilter (or just MultiFieldCharFilter)

single form field and the value is checked against multiple model fields (join with 'OR' )

• e.g. I enter 'django' and it will find all Articles with django in the 'title' or 'description' fields
• the behavior could mirror (or extend) the django admin's search functionality (see my comment below in response to @xordoquy)

I have created a gist to act as a proof of concept for this: https://gist.github.com/nkryptic/4727865

refactor filters to use Q objects

fairly substantial change, since it will change the Filter.filter method's call signature to no longer pass the queryset.

• see https://github.com/subsume/django-qfilters for background
  • it addresses fundamental problem with filtering a Model based on FK or M2M relationship attributes.

the only downside I can come up with is if the user is using a custom queryset with helper methods, it won't be possible to use them in a custom filter like you could when passing the queryset to each filter's filter method. I don't know how big an impact that would have.

[xordoquy]

Are there plans to add a search filter (similar to what the admin provides) ?

[nkryptic]

Thanks for the question @xordoquy.

Search in the admin, for User as an example, uses a single input field and performs an icontains query by default. Overriding the lookup type is easy for a filter in a FilterSet (by default it is exact). The more interesting part of the admin's search is that it searches against multiple fields, at the same time - which would be addressed by the addition of a MultiFieldFilter above (this could be done with a custom filter now). The final part, which might be something to think about adding, is how django admin's search allows overriding the individual lookup types for each field included in the single search. The relevant code looks like:

def construct_search(field_name):
    if field_name.startswith('^'):
        return "%s__istartswith" % field_name[1:]
    elif field_name.startswith('='):
        return "%s__iexact" % field_name[1:]
    elif field_name.startswith('@'):
        return "%s__search" % field_name[1:]
    else:
        return "%s__icontains" % field_name

The net result is a fairly comprehensive, yet flexible, search.

[nkryptic]

Here's an example for the Q object refactor thoughts above.

Given: an Article model which has a FK reporter to the User model.

# If you query a User for first_name and last_name
User.objects.filter(first_name='John', last_name='Smith')
# finds Users with first_name of John AND last_name of Smith

# If you chain the request (like django-filter would)
User.objects.filter(first_name='John').filter(last_name='Smith')
# finds Users with first_name of John AND last_name of Smith

# But, if you query an Article by the reporter's first_name and last_name
Article.objects.filter(reporter__first_name='John', reporter__last_name='Smith')
# finds Articles reported by a User with first_name of John AND last_name of Smith

# If you chain the request (like django-filter would)
Article.objects.filter(reporter__first_name='John').filter(reporter__last_name='Smith')
# finds Articles reported by a User with first_name of John OR last_name of Smith

# But, using Q objects and filtering the join of them will work for both
Article.objects.filter(Q(reporter__first_name='John'), Q(reporter__last_name='Smith'))
# finds Articles reported by a User with first_name of John AND last_name of Smith

[apollo13]

wow @nkryptic, looks like django-filter does have quite a future ahead of it :) Can't see anything where I'd object, keep the pull requests coming!

[nkryptic]

@apollo13,

django-easyfilters is shares a lot of core concepts with django-filter, but provides a much more functional version of the limited LinkWidget we have in this project. Would you have any issue with me starting a conversation with Luke Plant (maintainer of django-easyfilters) about the possibility of merging the two projects functionality?

[apollo13]

Sounds fine to me.

[olivergeorge]

I've just build a simple SearchFilter which works with django-filter. It uses the django admin query builder so should deliver the same functionality.

https://gist.github.com/olivergeorge/5088386

(One slight hack is I've set the filter name to point at 'pk' since this is effectively a multifield filter and the system doesn't seem to handle that well).

[nkryptic]

@olivergeorge,

Thanks for the comments and the gist. I worked out something similar:
https://gist.github.com/nkryptic/4727865

I was looking for extensibility, which declaring the construct_search within the method, like the admin does, makes difficult.

I've been toying with attempts to make it more generic, so that a MultiFieldFilter could be used as the base for a StringSearchFilter or even a DateSearchFilter (where different lookup types would make sense).

[olivergeorge]

Looks good.
On 6 Mar 2013 02:06, "Jacob Radford" notifications@github.com wrote:

@olivergeorge https://github.com/olivergeorge,

Thanks for the comments and the gist. I worked out something similar:
https://gist.github.com/nkryptic/4727865

I was looking for extensibility, which declaring the construct_searchwithin the method, like the admin does, makes difficult.

I've been toying with attempts to make it more generic, so that a
MultiFieldFilter could be used as the base for a StringSearchFilter or even
a DateSearchFilter (where different lookup types would make sense).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/75#issuecomment-14444273
.

[omerzimp]

@carltongibson Can you turn the original issue into a TODO list so we'll know what's implemented and what's not yet?

[carltongibson]

Hey @omerzimp (did you change your account?) 

TBH I'm not far off closing this ticket as stale/inactionable (take your pick). The reason I haven't done so already is that there are a number of ideas and I want to make sure I've grasped them before moving on.

As far as I'm concerned the Roadmap looks like:

1. Documentation, Documentation, Documentation — the docs are very poor.
2. Process the remaining Issues and Pull Requests (including this one.)

There is no 3.

[thedrow]

@carltongibson No I didn't change my account. I commented from my workplace's account by mistake :P
I think we should close this issue and start a new roadmap. What do you say?

[lampslave]

Is there MultiFieldFilter or similar functionality in django-filter for now?

[carltongibson]

@lampslave open a new issue, describing what you're looking for, and we'll discuss there

[lampslave]

@carltongibson MultiFieldFilter is already described and discussed on this page, but I couldn't see it in the docs and code :(