Team Swirl is excited to announce General Availability of Swirl 3.2!
This release includes the first version of the AutomaticPayloadMapperResultProcessor, RequireQueryStringInTitleResultProcessor, DropIrrelevantPostResultProcessor and NoModQueryProcessor, along with new SearchProviders for OpenSanctions.org and LittleSis.org. We've also added support for running OpenAI in Azure, storing query templates in JSON, plus updated Python, Django, and OpenAI API versions, and resolved issues found in previous releases. Finally, the Galaxy UI has been updated to ⭐ results above a particular relevancy score.
We're seeking anyone interested in testing new Connectors to Collibra, Thoughtspot and Pinecone DB. Please contact Swirl via email or Slack for access.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 New AutomaticPayloadMapperResultProcessor which profiles response data to find good strings for Swirl's title, body, and date_published fields. It is intended for SearchProviders that would otherwise have few (or no) good result_mappings options. It should be place after the MappingResultProcessor, and the result_mappings field should be blank.
"result_processors": [
"MappingResultProcessor",
"AutomaticPayloadMapperResultProcessor",
"CosineRelevancyResultProcessor"
],
...
"result_mappings": "",
🔹 New NoModQueryProcessor that only removes leading SearchProvider Tags and does not modify the query terms in any way. It is intended for repositories that allow non-search characters (such as brackets).
"query_processors": [
"NoModQueryProcessor"
],
🔹 New RequireQueryStringInTitleResultProcessor that drops results that do not contain the query_string_to_provider in the result title field. It should be added after the MappingResultProcessor and is now included by default in the "LinkedIn - Google PSE" SearchProvider.
"result_processors": [
"MappingResultProcessor",
"RequireQueryStringInTitleResultProcessor",
"DateFinderResultProcessor",
"CosineRelevancyResultProcessor"
],
🔹 New DropIrrelevantPostResultProcessor drops results with swirl_score < settings.MIN_SWIRL_SCORE (set to 500 by default) and results with no swirl_score.
🔹 Swirl now supports using Azure OpenAI models for RAG processing in addition to OpenAI's ChatGPT models.
2024-01-29 16:36:07,335 INFO get model AZUREAI gpt-35-turbo
2024-01-29 16:36:10,542 INFO HTTP Request: POST https://swirltest-openai.openai.azure.com/openai/deployments/gpt-35-turbo/chat/completions?api-version=2023-10-01-preview "HTTP/1.1 200 OK"
🔹 The Galaxy UI now highlghts results with a swirl_score above a configurable threshold with a star in the results list. The swirl_score configuration is available in theminimumSwirlScore entry of static/api/config/default, and the default is 100.
🔹 New SearchProvider.query_template_json field which stores the query_template as JSON. The MongoDB SearchProvider has been updated to use this feature.
"query_template_json": {
"$text": {
"$search": "{query_string}"
}
},
🔹 Support for lowercasing of URLs is now available via a new LC_URL option in SearchProvider result_mappings.
"result_mappings": "title=resource.name,body=resource.description, ... LC_URL,NO_PAYLOAD",
🔹 Added support for list-of-list responses from source APIs, where the first list element is the column names.
New SearchProviders
🔹 Search the LittleSis free database of "who-knows-who at the heights of business and government" with the new LittleSis.org SearchProvider.
🔹 Find sanctions targets and persons of interest with the new OpenSanctions.org SearchProvider.
🔑 Note: Additional SearchProviders contributed by the Swirl Community may be found in the SearchProviders/untested directory.
Improvements
🔹 Lists of SearchProviders can be added to the endpoint with copy/paste, and Swirl will load them all.
🔹 When signing in to Swirl via Microsoft OIDC, users are now automatically authorized to the M365 SearchProviders.
🔹 Swirl was validated on Python 3.12.1.
🔹 Swirl now runs on the new Django 5.0.x series Python packages.
🔹 Swirl now uses the latest 1.6.x OpenAI Python API.
🔹 The QueryProcessors base classes were updated.
🔹 The default AI Summary timeout value can now be overridden with a URL parameter in the Galaxy UI. For example: http://localhost:8000/galaxy/?q=gig%20economics&rag=true&rag_timeout=90000
🔹 The Snowflake SearchProvider query_template was revised.
🔹 The BigQuery SearchProvider no longer dedupes results.
🔹 Improved zero results feedback and logfile messages.
Resolved Issues
🔹 Removed duplicate static/api/config/default configuration file.
🔹 Removed unused timeout settings (SWIRL_Q_WAIT and SWIRL_RERUN_WAIT).
🔹 Fixed Elasticsearch authorization and addressed a certificate error.
🔹 Updated the index_email_elastic.py script and related documentation.
🔹 Stopwords are no longer highlighted in search results or AI Summaries.
🔹 Various small fixes to Galaxy UI behavior.
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app.
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation
Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference | AI Guide | Azure Marketplace Guide
Support
🔹 Join the Swirl Community on Slack!
🔹 Email support@swirl.today with issues, requests, questions, etc - we'd love to hear from you!