Releases: swirlai/swirl-search
Swirl 3.2.0
Team Swirl is excited to announce General Availability of Swirl 3.2!
This release includes the first version of the AutomaticPayloadMapperResultProcessor, RequireQueryStringInTitleResultProcessor, DropIrrelevantPostResultProcessor and NoModQueryProcessor, along with new SearchProviders for OpenSanctions.org and LittleSis.org. We've also added support for running OpenAI in Azure, storing query templates in JSON, plus updated Python, Django, and OpenAI API versions, and resolved issues found in previous releases. Finally, the Galaxy UI has been updated to ⭐ results above a particular relevancy score.
We're seeking anyone interested in testing new Connectors to Collibra, Thoughtspot and Pinecone DB. Please contact Swirl via email or Slack for access.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 New AutomaticPayloadMapperResultProcessor which profiles response data to find good strings for Swirl's title, body, and date_published fields. It is intended for SearchProviders that would otherwise have few (or no) good result_mappings options. It should be place after the MappingResultProcessor, and the result_mappings field should be blank.
"result_processors": [
"MappingResultProcessor",
"AutomaticPayloadMapperResultProcessor",
"CosineRelevancyResultProcessor"
],
...
"result_mappings": "",
🔹 New NoModQueryProcessor that only removes leading SearchProvider Tags and does not modify the query terms in any way. It is intended for repositories that allow non-search characters (such as brackets).
"query_processors": [
"NoModQueryProcessor"
],
🔹 New RequireQueryStringInTitleResultProcessor that drops results that do not contain the query_string_to_provider in the result title field. It should be added after the MappingResultProcessor and is now included by default in the "LinkedIn - Google PSE" SearchProvider.
"result_processors": [
"MappingResultProcessor",
"RequireQueryStringInTitleResultProcessor",
"DateFinderResultProcessor",
"CosineRelevancyResultProcessor"
],
🔹 New DropIrrelevantPostResultProcessor drops results with swirl_score < settings.MIN_SWIRL_SCORE (set to 500 by default) and results with no swirl_score.
🔹 Swirl now supports using Azure OpenAI models for RAG processing in addition to OpenAI's ChatGPT models.
2024-01-29 16:36:07,335 INFO get model AZUREAI gpt-35-turbo
2024-01-29 16:36:10,542 INFO HTTP Request: POST https://swirltest-openai.openai.azure.com/openai/deployments/gpt-35-turbo/chat/completions?api-version=2023-10-01-preview "HTTP/1.1 200 OK"
🔹 The Galaxy UI now highlghts results with a swirl_score above a configurable threshold with a star in the results list. The swirl_score configuration is available in theminimumSwirlScore entry of static/api/config/default, and the default is 100.
🔹 New SearchProvider.query_template_json field which stores the query_template as JSON. The MongoDB SearchProvider has been updated to use this feature.
"query_template_json": {
"$text": {
"$search": "{query_string}"
}
},
🔹 Support for lowercasing of URLs is now available via a new LC_URL option in SearchProvider result_mappings.
"result_mappings": "title=resource.name,body=resource.description, ... LC_URL,NO_PAYLOAD",
🔹 Added support for list-of-list responses from source APIs, where the first list element is the column names.
New SearchProviders
🔹 Search the LittleSis free database of "who-knows-who at the heights of business and government" with the new LittleSis.org SearchProvider.
🔹 Find sanctions targets and persons of interest with the new OpenSanctions.org SearchProvider.
🔑 Note: Additional SearchProviders contributed by the Swirl Community may be found in the SearchProviders/untested directory.
Improvements
🔹 Lists of SearchProviders can be added to the endpoint with copy/paste, and Swirl will load them all.
🔹 When signing in to Swirl via Microsoft OIDC, users are now automatically authorized to the M365 SearchProviders.
🔹 Swirl was validated on Python 3.12.1.
🔹 Swirl now runs on the new Django 5.0.x series Python packages.
🔹 Swirl now uses the latest 1.6.x OpenAI Python API.
🔹 The QueryProcessors base classes were updated.
🔹 The default AI Summary timeout value can now be overridden with a URL parameter in the Galaxy UI. For example: http://localhost:8000/galaxy/?q=gig%20economics&rag=true&rag_timeout=90000
🔹 The Snowflake SearchProvider query_template was revised.
🔹 The BigQuery SearchProvider no longer dedupes results.
🔹 Improved zero results feedback and logfile messages.
Resolved Issues
🔹 Removed duplicate static/api/config/default configuration file.
🔹 Removed unused timeout settings (SWIRL_Q_WAIT and SWIRL_RERUN_WAIT).
🔹 Fixed Elasticsearch authorization and addressed a certificate error.
🔹 Updated the index_email_elastic.py script and related documentation.
🔹 Stopwords are no longer highlighted in search results or AI Summaries.
🔹 Various small fixes to Galaxy UI behavior.
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app.
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation
Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference | AI Guide | Azure Marketplace Guide
Support
🔹 Join the Swirl Community on Slack!
🔹 Email support@swirl.today with issues, requests, questions, etc - we'd love to hear from you!
Swirl 3.1.0
Team Swirl is announcing General Availability of Swirl 3.1.0!
Swirl 3.1.0 includes new Connectors to MongoDB Atlas, Snowflake and Oracle, updates to Python and OpenAI API versions, and resolves numerous issues with previous versions.
Community Contributions
🏆 This version of Swirl includes additional contributions from the open source community. Many thanks to GitHub users Harsh Mahajan (HarshMN2345) and Harshil Khamar (Harshil0512) for their accepted, merged PRs; and also to Janvi Thakkar (Janvi-Thakkar) for inspiring us to add the Internet Archive as a source for Swirl!
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 New MongoDB Atlas Connector & SearchProvider for the sample_mflix collection, movies sample table
🔹 New Snowflake Connector & SearchProvider for the FreeCompanyResearch dataset
🔹 New Oracle Connector, tested against 23c Free (and presumably supporting earlier versions)
New SearchProviders
🔹 Search your Cards in Atlassian Trello
🔹 Find Tasks in Asana
🔹 Query the Internet Archive Library for almost anything
🔑 Note: Additional SearchProviders contributed by the Swirl Community may be found in the SearchProviders/untested directory.
Improvements
🔹 Swirl RAG is now available through a single API call, e.g. ?qs=metasearch&rag=true
🔹 Swirl now uses the latest 1.3.x OpenAI Python API.
🔹 Swirl 3.1.0 was validated on Python 3.11.6 (note that Python 3.12.0 support is awating one final dependency update).
🔹 The ServiceNow Knowledge SearchProvider was updated for the "Vancouver" release.
🔹 Pip requirements are now frozen on the main branch for releases (addresses Issue #912).
🔹 Default page-fetch configurations were added to the Enterprise Search, Mergers and Acquisition, Strategy Consulting, and Swirl Documentation Google PSE SearchProviders.
🔹 Configurations for a default timeout value (30 seconds) and the text to display when the timeout is exceeded were added to RAG processing.
🔹 Swirl's OpenAPI specification was updated: a new Swirl Postman collection and swagger.json are now available in the repo.
🔹 Matching query terms are now highlighted in the AI Response results.
🔹 Swirl now uses the latest 4.2.x Django packages.
Resolved Issues
🔹 Back button behavior with RAG processing was improved.
🔹 Addressed an issue in the version checker.
🔹 Resolved an issue with some RequestsPost searches returning 500 errors.
🔹 The Galaxy UI was updated to better handle an empty results set returned from some sources.
🔹 The Galaxy UI now displays ChatGPT SearchProvider results in the main results list.
🔹 The "Generate AI Response" option in the Galaxy UI is disabled until the full set of search results have returned.
🔹 Addressd a regression with search re-run functionality in the Galaxy UI.
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app.
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation
Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference | AI Guide
Support
🔹 Join the Swirl Community on Slack!
🔹 Email support@swirl.today with issues, requests, questions, etc - we'd love to hear from you!
Swirl 3.0.1
Team Swirl is announcing General Availability of Swirl 3.0.1!
This update addresses issues with two of our default GitHub SearchProviders, improves Swirl Unix timestamp handling, and pins the version of OpenAI's Python library in Swirl's requirements.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
Resolved Issues
🔹 The GitHub "Issues" and "Pull Requests" SearchProvider configurations were updated to realign them with the source's Search API. Please review the query_template value in SearchProviders/github.json for Issues and Pull Requests and update existing configurations to match.
🔹 Swirl's handling of Unix timestamps was enhanced to better handle millisecond values when present in the source's response.
🔹 OpenAI's Python library is now pinned to a known-good version for Swirl. To ensure Swirl is configured with this package version:
- Stop Swirl:
python swirl.py stop - Re-run
./install.shin your<swirl-home>directory - Start Swirl:
python swirl.py start
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
redis. You must now have redis installed and running before starting Swirl.
Documentation
Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference | AI Guide
Support
🔹 Join the Swirl Community on Slack!
🔹 Email: support@swirl.today with issues, requests, questions, etc - we'd love to hear from you!
Swirl 3.0.0
Team Swirl is announcing General Availability of Swirl 3.0!
Swirl 3.0 includes Real-Time Retrieval Augmented Generation (RAG). Harness the power of Generative AI like OpenAI ChatGPT with internal data by querying existing search engines, databases, applications and enterprise services. Search for anything; Swirl will use a built-in non-generative LLM to find the best results from all sources, and present them. Adjust the selections, then RAG in real-time to get human-directed, verifiable AI insight. Fetch the freshest available data from systems of record - without extracting, copying and pre-indexing anything.
Community Contributions
🏆 This version of Swirl includes numerous contributions from the open source community. A thousand thank-you's to the following GitHub users for their accepted, merged PRs: Yash Mathur (Yash-sudo-web), s-vamshi, Ikko Eltociear Ashimine (eltociear), Akash Kundu (Akash190104), Biplob Sutradhar (biplobsd), Harsh Mahajan (HarshMN2345), Varanasi Aditya Sai (fvaysh), Neel Shah (neelshah2409) and Chandan Mali (Chandanmali)
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 Swirl 3.0 supports Real-Time Retrieval Augmented Generation (RAG). No copying or indexing of data is required as Swirl queries existing search APIs. Once RAG is triggered, optionally using the updated Galaxy UI, the new Page Fetcher component will retrieve result URLs and prepare a prompt. Note that the community edition only supports fetching of pages that don't require authentication. Contact hello@swirl.today if you need to RAG using authenticated sources like M365, ServiceNow or Salesforce. Follow the new AI Guide to get started - the main requirement is to add an OpenAI ChatGPT API key. (Other Generative AI platforms will be supported soon.) The European PMC SearchProvider is pre-configured for RAG so insights can be generated in minutes.
🔹 The European PMC SearchProvider has been preloaded, set to active status, and configured to participate in RAG by default.
🔹 The new Retrieval Augmented Generation (RAG) Web Socket API is now available to developers. Refer to the Developer Guide for details.
New Swirl Documentation: https://docs.swirl.today/
Swirl's product documentation has a new home, a new template, and a handy new find-in-page option!
Check it out:
Swirl 3.0 includes a new Google PSE SearchProvider that targets the Swirl documentation website. This SearchProvider is enabled by default so you can search Swirl documentation with Swirl!
New SearchProviders
Query for a specific Bitcoin Address or Transaction Hash using the new Blockchain.com Bitcoin SearchProviders:
Search organizations in Crunchbase with our new Crunchbase Basic SearchProvider:
🔑 Note: SearchProviders contributed by the Swirl Community, including those for Algolia and Trello, may be found in the SearchProviders/untested directory. Some may include Connectors as well as SearchProviders. All will be moved into the production SearchProviders and pre-loaded in future releases.
Improvements
🔹 Repository Standards: Swirl has published a full set of GitHub Community Standards to support our growing community of contributors.
🔹 Microsoft OneDrive SearchProvider was updated to omit folders as results; it now only returns documents.
Resolved Issues
🔹 The Microsoft Authenticator toggle now turns green when active.
🔹 The updated RequestsGet connector now handles sources that return only a single result.
🔹 The legacy redis.conf file was removed.
🔹 The Galaxy UI login page now handles window resizing more gracefully.
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
redis. You must now have redis installed and running before starting Swirl.
Documentation
Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference | AI Guide
Support
🔹 Join the Swirl Community on Slack!
🔹 Email: support@swirl.today with issues, requests, questions, etc - we'd love to hear from you!
Swirl 2.6
Team Swirl is announcing General Availability of Swirl 2.6!
This release features new SearchProviders for ServiceNow, Google News, and a searchable Hacker News implementation! We've also validated Swirl on the latest stable Python version (3.11.5) and updated our Dockerfile image to the latest stable Debian release (Bookworm).
⚠️ As of Release 2.6, Swirl's start-up process no longer starts redis. You must now have redis installed and runnning before starting Swirl.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 Search the ServiceNow Knowledge and Service Catalog centers using new Basic Auth SearchProviders.
- For the
credentials, ServiceNow requries an admin level account for searching. - Both the
urlandresult_mappingsrequire the ServiceNow instance domain.
🔹 Search Google News posts with a dedicated SearchProvider for this source.
🔹 Search Stories and Comments from Hacker News with new SearchProviders for each.
🔹 Swirl now checks to see if you are running the latest version at startup.
You're using version 2.6 of Swirl, the current version.
Improvements
🔹 Swirl's installation script (install.sh) will now check for a running Redis instance and for the presence of jq on the system, and emit helpful messages if they are not found.
🔹 Swirl 2.6 has been verified on the latest stable Python release (3.11.5).
🔹 The Debian image in Swirl's Dockerfile has been updated from 3.11.4-slim-bullseye to 3.11.5-slim-bookworm, which addresses several known vulnerabilities.
🔹 Configuration for an OPENAI_API_KEY is now available in Swirl's .env file for use with the ChatGPT query_processor and query_mappings override options.
Resolved Issues
🔹 Removed an unecessary duplicate entry in requirements.txt
🔹 Addressed common non-date formatting issues in the DateFinderResultProcessor
🔹 Removed unnessesary parameter from the query url of the EuropePMC SearchProvider
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
⚠️ As of version 2.6, Swirl's start-up process no longer starts redis. You must now have redis installed and runnning before starting Swirl.
Documentation
Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference
(updated links on 13-October-2023)
Support
🔹 Join the Swirl Community on Slack!
🔹 Email: support@swirl.today with issues, requests, questions, etc - we'd love to hear from you!
Swirl Metasearch 2.5.1
Team Swirl is announcing General Availability of Swirl 2.5.1!
This release features three new SearchProviders (arXiv.org, EuropePMC.org, and LinkedIn profiles via Google PSE), enhanced result processing to handle XML responses from source APIs, plus updates to the ChatGPT Connector and QueryProcessor to support the latest GPT-3.5-Turbo and GPT-4 models.
🥇 We would also like to give a shout out to GitHub user Rajpratik71 for his contribution to this release which improved our Dockerfile configuration. Many thanks for contributing to Swirl!
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 Both the ChatGPT Connector and QueryProcessor were updated to use OpenAI's ChatCompletion method which supports the latest GPT models - including GPT-4 - and a much greater range of interactivity.
- Three new Tags and one new
query_mappingoption are available for the ChatGPT SearchProvider to help shape the Prompt or Default Role passed to ChatGPT along with the user's query. - The ChatGPT SearchProvder now queries the GPT-3.5-Turbo model by default.
🔹 Search LinkedIn profiles with the new LinkedIn Google PSE and SearchProvider.
🔹 Search the academic paper collections of arXiv.org and EuropePMC.org with new SearchProivders for each.
🔹 New result processing to handle XML responses from source APIs; see this in action with the arXiv.org SearchProvider.
🔹 Swirl's Dockerfile now uses the --no-cache-dir flag with pip to save space.
Thanks to GitHub user Rajpratik71 for this contribution!
Improvements
🔹 The Subscribe functionality was reworked to better handle Microsoft Authentication for subscribe runs that include M365 sources.
🔹 Some default SearchProviders have additional Tags: Internal or Develop. See the documentation for details about targeting specific groups of sources using SearchProvider Tags.
🔹 The install.sh script was further enhanced to synchronize spaCy model and code version updates.
Resolved Issues
🔹 Addressed an issue where Swirl could hang due to very long source repsonse times. Swirl's default timeout value in swirl_server/settings.py is now: SWIRL_TIMEOUT_DEFAULT = 10
🔹 Corrected a count inconsistency in the CosineRelevancyResultProcessor
🔹 Removed duplicate entries from urls.py
🔹 Fixed a regression where unmatched Tags were being dropped in searches
🔹 ChatGPT no longer returns a 500 error when enabled for Subscribed searches
🔹 After authenticating to Mircosoft in the Galaxy UI, the first search run now correctly includes M365 resutls
🔹 Addressed an issue with date-sorting in the M365 sources:
- The default date-sort for M365 sources is:
DEFAULT_DATESORT_X = "createdDateTime desc" - This can be overridden by setting
DATE_SORTin the SearchProviderquery_mapping - Owners of a Search object or Swirl's Subscribe feature can set the
sortattribute on a Search
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation
Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference
(updated links on 13-October-2023)
Support
🔹 Join the Swirl Metasearch Community on Slack!
🔹 Email: support@swirl.today with issues, requests, questions, etc - we'd love to hear from you!
Swirl Metasearch 2.5
Team Swirl is announcing General Availability of Swirl 2.5!
The theme for this release was performance. Configured with 12 SearchProviders, Swirl 2.5 supports up to 15 queries/second on a Standard F16s v2 server (16 vcpus, 32 GiB memory) with a median response time of 3 seconds.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
Performance Improvements
🔹 Swirl's asynchronous back-end was refactored to use the group method and replaced RabbitMQ with Redis
🔹 Relevancy processing was split into two stages to improve performance
- The revised
CosineRelevancyPostResultProcessormust be in theSearch.post_result_processorslist. - Please review the JSON in the
SearchProviders/directory and update existing configurations to match.
HubSpot SearchProviders
🔹 Search Company, Contact, and Deal records with new token-based HubSpot SearchProviders
- HubSpot requires creation of a "Private Apps" API token by the Super Admin with these scopes
- In
result_mappings, theurlmapping must contain the HubSpot Org ID
Improvements
🔹 The DateFindingResultProcessor was added to the Google PSE SearchProvider JSON. It finds a date in a large percentage of results that otherwise wouldn't have one, and copies the date to the date_published field. Existing PSE SearchProvider configurations should be updated as follows:
"result_processors": [
"MappingResultProcessor",
"DateFinderResultProcessor",
"CosineRelevancyResultProcessor"
],
🔹 Swirl's control script (python swirl.py) was updated
- A
defaultkey was added to the services list, includingdebug, inswirl/services.py - The
celery-beatsservice was set todefault: False swirl.pywas modified to load only services withdefault: True
To use the subscribe or expiration features, celery-beats must be started manually:
python swirl.py start celery-beats
Celery-beats must also be stopped separately.
🔹 Swirl now outputs a single log entry with each request at the default log level INFO
2023-08-02 10:49:09,466 INFO admin search 452 FULL_RESULTS_READY 32 2.2
Detailed logging is available in Debug mode: restart Swirl with the --debug flag to enable
🔹 The install.sh script was refactored to better handle spaCy upgrades
🔹 Swirl's Docker image was updated to use the latest, stable release of Python, version 3.11.4
🔹 Use of the unsafe eval() function was removed
Resolved Issues
🔹 Swirl's log files now persist and append new entries across restarts
🔹 The SearchProvider configurations for Elastic, OpenSearch, and Microsoft Outlook Messages were updated
🔹 A seesion-related issue in the connectors for Sqlite3, PostgreSQL, and Elastic was fixed
🔹 Errors in the DedupeByFieldResultProcessor were addressed
🔹 The SearchProvider results_per_query value is handled correctly now
🔹 An issue where some searches didn't appear to reach the Swirl backend was fixed
🔹 Unnecessary WARNING messages were removed from the logs
🔹 Result URLs on Search objects are now constructed correctly using the Request instead of loading from django settings
Known Issues
🔹 Clicking a Microsoft Teams result sometimes results in this error from the MS Teams app:
We cannot take you to that message because it's in a chat you're not in.
For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.
🔹 Subscribed searches are not returning results for Microsoft sources; this will be addressed in an upcoming release.
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
CosineRelevancyResultProcessor must be added last to the result_processors block of all SearchProvider configurations except ChatGPT. For example:
"result_processors": [
"MappingResultProcessor",
"LenLimitingResultProcessor",
"CosineRelevancyResultProcessor"
],
This following of Error message in the Swirl logs indicates that one or more SearchProviders have not been updated:
INFO search.py: invoking processor: CosineRelevancyPostResultProcessor
2023-07-31 16:31:39,268 ERROR CosineRelevancyPostResultProcessor_2051: Error: Dictionary of result lengths is empty. Was CosineRelevancyResultProcessor included in Search Providers Processor configuration?
- Elastic: Remove
hosts=andhttp-auth=(just that text, not the values that follow) - OpenSearch: Update the
query_templatesyntax to the following:
"query_template": "{\"query\":{\"query_string\":{\"query\":\"{query_string}\",\"default_field\":\"{default_field}\",\"default_operator\":\"and\"}}}",
- Microsoft Outlook Messages: Update both
result_grouping_fieldandresult_processorsto the following:
"result_grouping_field": "conversationId",
"result_processors": [
"MappingResultProcessor",
"DedupeByFieldResultProcessor",
"CosineRelevancyResultProcessor"
],
Documentation
Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference
(updated links on 13-October-2023)
Support
🔹 Join the Swirl Metasearch Community on Slack!
🔹 Email: support@swirl.today with issues, requests, questions, etc - we'd love to hear from you!
Swirl Metasearch 2.1.1
This update addresses an issue in the OpenSearch, Elasticsearch, and PostgreSQL connectors.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
Resolved Issues
🔹 The OpenSearch, Elasticsearch, and PostgreSQL connectors were updated to address an issue that resulted in a positional argument error. Please update your Swirl instance to v. 2.1.1 to correct this error.
Known Issues
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation Wiki
🔹 Quick Start
🔹 User Guide
🔹 Admin Guide
🔹 M365 Guide
🔹 Developer Guide
🔹 Developer Reference
Support
🔹 Join the Swirl Metasearch Community on Slack!
🔹 Create a GitHub Issue if something doesn't work, isn't clear, or should be documented
🔹 Email: support@swirl.today with issues, requests, questions, etc - we'd love to hear from you!
Swirl Metasearch 2.1
Team Swirl is announcing General Availability of Swirl Metasearch 2.1!
This version features the new Galaxy User Interface with Dark Mode, new GitHub SearchProviders, support for custom request headers, a new DateFinderResultProcessor that finds dates in the body field and copies them to date_published, plus new Result Processors for trimming and cleaning long responses. 2.1 also improves handling of recurring Microsoft 365 Calendar events and Outlook Message threads, showing only the most recent.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 Introducing the Galaxy UI:
🔹 Search public repositories with token-based GitHub SearchProviders for Code, Commits, Pull Requests, and Issues
🔹 Micrsoft 365 Calendar Events pre-configured SearchProvider with automatic hiding of recurring meetings
🔹 New date_published_display option available in SearchProvider result_mappings
You can now specify different values for date_published and date_published_display in SearchProvider configurations:
"result_mappings": ... date_published=foo.bar.date1,date_published_display=foo.bar.date2 ...
Swirl Result objects will reflect both date values:
"date_published": "2010-01-01 00:00:00",
"date_published_display": "c2010",
And, the new Galaxy UI displays date_published_display while date-sorting by date_published:
🔹 Pass custom HTTP Header values in SearchProviders
The optional http_request_headers field is available to all SearchProviders for passing custom HTTP Header values to the source alongside the user's query. The new GitHub SearchProviders use this feature to return a more accurate result snippet from GitHub that is then mapped to Swirl's body field:
"http_request_headers": {
"Accept": "application/vnd.github.text-match+json"
},
"result_mappings": "title=name,body=text_matches[*].fragment, ...
🔹 New Result Processor: LenLimitingResultProcessor
This processor checks to see if the title and body responses from a source exceed a configurable length, truncates anything after that value, and adds an elipsis ("..."). If the body field has been truncated, the processor reports the entire response in a new body_full field in the Payload. The default max_length value for this processor is set in swirl_server/settings.py: SWIRL_MAX_FIELD_LEN = 512 and can be overridden for a specific SearchProvider using a new Tag value (max_length:###):
"tags": [
"Articles",
"Wiki",
"YouTrack",
"max_length:256"
]
The processor reports on the number of items updated as expected. It should be considered for sources that consistently return lengthy title or body fields and should follow the MappingResultProcessor. The default SearchProvider configurations for Atlassian Jira and Confluence have been updated with this processor.
"result_processors": [
"MappingResultProcessor",
"LenLimitingResultProcessor"
],
🔹 New Result Processor: CleanTextResultProcessor
This processor removes non-alphanumeric characters from the source response. It should be considered for lengthy responses where URLs or other HTML or Markdown syntax appear in results. It should be installed before the LenLimitingResultProcessor when both are used. The default SearchProvider configurations for JetBrains YouTrack have been updated with these processors.
"result_processors": [
"MappingResultProcessor",
"CleanTextResultProcessor",
"LenLimitingResultProcessor"
],
🔹 New Result Processor: DateFinderResultProcessor
This processor looks for a date in any of the following forms in the body field of each result item. Should it find one, and the date_published for that item is 'unknown', it replaces date_published with the date extracted from the body, and notes this in the result.messages. This processor can detect the following date formats:
06/01/23
06/01/2023
06-01-23
06-01-2023
jun 1, 2023
june 1, 2023
Improvements
🔹 Updates to the genericRequestsPost connector and example SearchProvider JSON
🔹 Refined handling of threaded email messages in Outlook Messages results
🔹 Updated sorting of the search page Source dropdown and the result page Source facet in the Galaxy UI
🔹 Adjusted default Relevancy weight values to improve the relevancy ranking of shorter results
Resolved Issues
🔹 Re-running the previous search is now supported correctly in the Galaxy UI
🔹 An issue with Microsoft Authorization logout has been fixed
🔹 Clearing the search form now clears the previous Sort ("View By") selection correctly
🔹 Result Mixer choices are no long retained across searches
🔹 Result list numbering now displays correctly if Swirl's url field is not mapped
🔹 Various isssues with SearchProvider Tag searching were addressed
Known Issues
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation Wiki
🔹 Quick Start
🔹 User Guide
🔹 Admin Guide
🔹 M365 Guide
🔹 Developer Guide
🔹 Developer Reference
Support
🔹 Join the Swirl Metasearch Community on Slack!
🔹 Create a GitHub Issue if something doesn't work, isn't clear, or should be documented
🔹 Email: support@swirl.today with issues, requests, questions, etc - we'd love to hear from you!
Swirl Metasearch 2.0.3
Swirl Metasearch 2.0.3
This version adds a generic RequestsPost Connector and example SearchProvider.
PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search
New Features
🔹 A new RequestsPost Connector and example SearchProvider are available as of this release. See the Developer Reference Guide for more details, or Contact Support for help getting started.
Known Issues
🔹 Creating searches from a browser with q= can sometimes create two Search objects.
This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.
Please report any issues with this to support.
Upgrading
Documentation Wiki
🔹 Quick Start
🔹 User Guide
🔹 Admin Guide
🔹 M365 Guide
🔹 Developer Guide
🔹 Developer Reference
Support
🔹 Join the Swirl Metasearch Community on Slack!
🔹 Create a GitHub Issue if something doesn't work, isn't clear, or should be documented
🔹 Email: support@swirl.today with issues, requests, questions, etc - we'd love to hear from you!