SWIRL AI Connect 3.6.0.1 Community edition is now available!

This release includes improvements to LinkedIn searching and resolves an issue with the GenAI Connector, along with updates throughout to the new swirlaiconnect.com domain.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

Updates

🔹 The default LinkedIn SearchProvider has improved matching for person names and job titles.

🔹 An issue was resolved with the GenAI Connector.

🔹 Codebase references to SWIRL domains were updated to the new swirlaiconnect.com domain.

Known Issues

🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:

We cannot take you to that message because it's in a chat you're not in.

For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 3.6.0.1 does not requires database migration.

Documentation

Please visit our updated and expanded Documentation site: https://docs.swirlaiconnect.com/

Support

🔹 Join the SWIRL Community on Slack!

🔹 Email [email protected] with issues, requests, questions, etc. - we'd love to hear from you!

Team SWIRL is excited to announce General Availability of SWIRL AI Connect 3.6.0 Community edition!

The release includes a production QDrantDB Connector and SearchProvider, plus support for RAG processing based solely on search results content.
Thank you to Anush008 and others in the SWIRL community who contributed feedback addressed in this release.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

🔹 Production support for searching QDrantDB sources, including a new Connector and SearchProvider:

🔹 SWIRL's RAG processing will now fallback to using the search result content when pages are not available to be fetched.

🔹 SWIRL AI Connect has been validated on Python 3.12.4.

Known Issues

🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:

We cannot take you to that message because it's in a chat you're not in.

For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 3.6.0 requires database migration. See Upgrading SWIRL.

Documentation

Overview | Quick Start Guide | User Guide | Admin Guide | M365 Guide | Developer Guide | Tutorial: Extending Swirl | Developer Reference | AI Guide | Enterprise Guide

Support

🔹 Join the SWIRL Community on Slack!

🔹 Email [email protected] with issues, requests, questions, etc. - we'd love to hear from you!

Team SWIRL is excited to announce General Availability of SWIRL AI Connect 3.5 Community edition!\

The release includes a production Pinecone connector, support for running SWIRL in the VS Code debugger, plus GDPR and dataset visualization support in Galaxy.
Thank you to simsong, deg and others in the SWIRL community who contributed feedback addressed in this release.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

🔹 The Galaxy UI now renders a chart visualization for results from SearchProviders with the DATASET option specified in the result_mappings field.

🔹 Added GDPR cookie consent to the Galaxy UI.

🔹 Use the new PineconeDB Connector and SearchProvider to search vector database sources.

🔹 Developers can now run SWIRL in the VSCode debugger.

Improvements

🔹 The default SearchProvider configurations that ship with SWIRL have been updated; the multiple Google PSEs were consolidated into a new, simple provider definition, and many freely available sources are now enabled by default.

🔹 The Elastic and OpenSearch Connectors were refactored.

🔹 SWIRL's public API docuementation was updated and is now generated by the drf-spectacular package with support for the OpenAPI v.3 schema.

🔹 Scripts and other supporting utilities have been moved to the new DevUtils/ directory in the repository.

🔹 Logging has been refactored; detailed messaging can be selectively enabled using the new SWIRL_LOG_DEBUG entry in SWIRL's .env file. For example:

SWIRL_LOG_DEBUG='swirl.page_fetch.web_page, swirl.processors.rag'

🔹 CSRF support now available via the new CSRF_TRUSTED_ORIGINS entry in SWIRL's .env file. For example:

CSRF_TRUSTED_ORIGINS='http://localhost:8000'

🔹 SWIRL was validated on Python 3.12.2.

Resolved Issues

🔹 The Galaxy UI now reports the total number of results returned for a search instead of the total number of results configured in the active SearchProviders.

🔹 The URL presented with each Galaxy results is now clickable, just like the result title.

🔹 The .install.sh script was adjusted to keep pip and other essential packages properly updated.

Known Issues

🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:

We cannot take you to that message because it's in a chat you're not in.

For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 3.5.0 requires database migration. See Upgrading SWIRL.

Documentation

Overview | Quick Start Guide | User Guide | Admin Guide | M365 Guide | Developer Guide | Tutorial: Extending Swirl | Developer Reference | AI Guide | Enterprise Guide

Support

🔹 Join the SWIRL Community on Slack!

🔹 Email [email protected] with issues, requests, questions, etc. - we'd love to hear from you!

Team Swirl is excited to announce General Availability of Swirl 3.2!

This release includes the first version of the AutomaticPayloadMapperResultProcessor, RequireQueryStringInTitleResultProcessor, DropIrrelevantPostResultProcessor and NoModQueryProcessor, along with new SearchProviders for OpenSanctions.org and LittleSis.org. We've also added support for running OpenAI in Azure, storing query templates in JSON, plus updated Python, Django, and OpenAI API versions, and resolved issues found in previous releases. Finally, the Galaxy UI has been updated to ⭐ results above a particular relevancy score.

We're seeking anyone interested in testing new Connectors to Collibra, Thoughtspot and Pinecone DB. Please contact Swirl via email or Slack for access.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

🔹 New AutomaticPayloadMapperResultProcessor which profiles response data to find good strings for Swirl's title, body, and date_published fields. It is intended for SearchProviders that would otherwise have few (or no) good result_mappings options. It should be place after the MappingResultProcessor, and the result_mappings field should be blank.

 "result_processors": [
    "MappingResultProcessor",
    "AutomaticPayloadMapperResultProcessor",
    "CosineRelevancyResultProcessor"
],
...
"result_mappings": "",

🔹 New NoModQueryProcessor that only removes leading SearchProvider Tags and does not modify the query terms in any way. It is intended for repositories that allow non-search characters (such as brackets).

"query_processors": [
    "NoModQueryProcessor"
],

🔹 New RequireQueryStringInTitleResultProcessor that drops results that do not contain the query_string_to_provider in the result title field. It should be added after the MappingResultProcessor and is now included by default in the "LinkedIn - Google PSE" SearchProvider.

"result_processors": [
    "MappingResultProcessor",
    "RequireQueryStringInTitleResultProcessor",
    "DateFinderResultProcessor",
    "CosineRelevancyResultProcessor"
],

🔹 New DropIrrelevantPostResultProcessor drops results with swirl_score < settings.MIN_SWIRL_SCORE (set to 500 by default) and results with no swirl_score.
⚠️ The Galaxy UI will not display the correct number of results if this ResultProcessor is deployed. This will be addressed in a future release.

🔹 Swirl now supports using Azure OpenAI models for RAG processing in addition to OpenAI's ChatGPT models.

2024-01-29 16:36:07,335 INFO     get model AZUREAI gpt-35-turbo
2024-01-29 16:36:10,542 INFO     HTTP Request: POST https://swirltest-openai.openai.azure.com/openai/deployments/gpt-35-turbo/chat/completions?api-version=2023-10-01-preview "HTTP/1.1 200 OK"

🔹 The Galaxy UI now highlghts results with a swirl_score above a configurable threshold with a star in the results list. The swirl_score configuration is available in theminimumSwirlScore entry of static/api/config/default, and the default is 100.

🔹 New SearchProvider.query_template_json field which stores the query_template as JSON. The MongoDB SearchProvider has been updated to use this feature.

"query_template_json": {
    "$text": {
        "$search": "{query_string}"
    }
},

🔹 Support for lowercasing of URLs is now available via a new LC_URL option in SearchProvider result_mappings.

"result_mappings": "title=resource.name,body=resource.description, ... LC_URL,NO_PAYLOAD",

🔹 Added support for list-of-list responses from source APIs, where the first list element is the column names.

New SearchProviders

🔹 Search the LittleSis free database of "who-knows-who at the heights of business and government" with the new LittleSis.org SearchProvider.

🔹 Find sanctions targets and persons of interest with the new OpenSanctions.org SearchProvider.

🔑 Note: Additional SearchProviders contributed by the Swirl Community may be found in the SearchProviders/untested directory.

Improvements

🔹 Lists of SearchProviders can be added to the endpoint with copy/paste, and Swirl will load them all.

🔹 When signing in to Swirl via Microsoft OIDC, users are now automatically authorized to the M365 SearchProviders.

🔹 Swirl was validated on Python 3.12.1.

🔹 Swirl now runs on the new Django 5.0.x series Python packages.

🔹 Swirl now uses the latest 1.6.x OpenAI Python API.

🔹 The QueryProcessors base classes were updated.

🔹 The default AI Summary timeout value can now be overridden with a URL parameter in the Galaxy UI. For example: http://localhost:8000/galaxy/?q=gig%20economics&rag=true&rag_timeout=90000

🔹 The Snowflake SearchProvider query_template was revised.

🔹 The BigQuery SearchProvider no longer dedupes results.

🔹 Improved zero results feedback and logfile messages.

Resolved Issues

🔹 Removed duplicate static/api/config/default configuration file.

🔹 Removed unused timeout settings (SWIRL_Q_WAIT and SWIRL_RERUN_WAIT).

🔹 Fixed Elasticsearch authorization and addressed a certificate error.

🔹 Updated the index_email_elastic.py script and related documentation.

🔹 Stopwords are no longer highlighted in search results or AI Summaries.

🔹 Various small fixes to Galaxy UI behavior.

Known Issues

🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app.

We cannot take you to that message because it's in a chat you're not in.

For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 3.2.0 requires database migration. See Upgrading Swirl.

Documentation

Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference | AI Guide | Azure Marketplace Guide

Support

🔹 Join the Swirl Community on Slack!

🔹 Email [email protected] with issues, requests, questions, etc - we'd love to hear from you!

Team Swirl is announcing General Availability of Swirl 3.1.0!

Swirl 3.1.0 includes new Connectors to MongoDB Atlas, Snowflake and Oracle, updates to Python and OpenAI API versions, and resolves numerous issues with previous versions.

Community Contributions

🏆 This version of Swirl includes additional contributions from the open source community. Many thanks to GitHub users Harsh Mahajan (HarshMN2345) and Harshil Khamar (Harshil0512) for their accepted, merged PRs; and also to Janvi Thakkar (Janvi-Thakkar) for inspiring us to add the Internet Archive as a source for Swirl!

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

🔹 New MongoDB Atlas Connector & SearchProvider for the sample_mflix collection, movies sample table

🔹 New Snowflake Connector & SearchProvider for the FreeCompanyResearch dataset

🔹 New Oracle Connector, tested against 23c Free (and presumably supporting earlier versions)

New SearchProviders

🔹 Search your Cards in Atlassian Trello

🔹 Find Tasks in Asana

🔹 Query the Internet Archive Library for almost anything

🔑 Note: Additional SearchProviders contributed by the Swirl Community may be found in the SearchProviders/untested directory.

Improvements

🔹 Swirl RAG is now available through a single API call, e.g. ?qs=metasearch&rag=true

🔹 Swirl now uses the latest 1.3.x OpenAI Python API.

🔹 Swirl 3.1.0 was validated on Python 3.11.6 (note that Python 3.12.0 support is awating one final dependency update).

🔹 The ServiceNow Knowledge SearchProvider was updated for the "Vancouver" release.

🔹 Pip requirements are now frozen on the main branch for releases (addresses Issue #912).

🔹 Default page-fetch configurations were added to the Enterprise Search, Mergers and Acquisition, Strategy Consulting, and Swirl Documentation Google PSE SearchProviders.

🔹 Configurations for a default timeout value (30 seconds) and the text to display when the timeout is exceeded were added to RAG processing.

🔹 Swirl's OpenAPI specification was updated: a new Swirl Postman collection and swagger.json are now available in the repo.

🔹 Matching query terms are now highlighted in the AI Response results.

🔹 Swirl now uses the latest 4.2.x Django packages.

Resolved Issues

🔹 Back button behavior with RAG processing was improved.

🔹 Addressed an issue in the version checker.

🔹 Resolved an issue with some RequestsPost searches returning 500 errors.

🔹 The Galaxy UI was updated to better handle an empty results set returned from some sources.

🔹 The Galaxy UI now displays ChatGPT SearchProvider results in the main results list.

🔹 The "Generate AI Response" option in the Galaxy UI is disabled until the full set of search results have returned.

🔹 Addressd a regression with search re-run functionality in the Galaxy UI.

Known Issues

🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app.

We cannot take you to that message because it's in a chat you're not in.

For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 3.1.0 requires database migration. See Upgrading Swirl.

Documentation

Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference | AI Guide

Support

🔹 Join the Swirl Community on Slack!

🔹 Email [email protected] with issues, requests, questions, etc - we'd love to hear from you!

Team Swirl is announcing General Availability of Swirl 3.0.1!

This update addresses issues with two of our default GitHub SearchProviders, improves Swirl Unix timestamp handling, and pins the version of OpenAI's Python library in Swirl's requirements.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

Resolved Issues

🔹 The GitHub "Issues" and "Pull Requests" SearchProvider configurations were updated to realign them with the source's Search API. Please review the query_template value in SearchProviders/github.json for Issues and Pull Requests and update existing configurations to match.

🔹 Swirl's handling of Unix timestamps was enhanced to better handle millisecond values when present in the source's response.

🔹 OpenAI's Python library is now pinned to a known-good version for Swirl. To ensure Swirl is configured with this package version:

• Stop Swirl: python swirl.py stop
• Re-run ./install.sh in your <swirl-home> directory
• Start Swirl: python swirl.py start

Known Issues

🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:

We cannot take you to that message because it's in a chat you're not in.

For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 3.0.1 does not requires database migration.

⚠️ As of version 2.6, Swirl's start-up process no longer starts redis. You must now have redis installed and running before starting Swirl.

Documentation

Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference | AI Guide

Support

🔹 Join the Swirl Community on Slack!

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!

Team Swirl is announcing General Availability of Swirl 3.0!

Swirl 3.0 includes Real-Time Retrieval Augmented Generation (RAG). Harness the power of Generative AI like OpenAI ChatGPT with internal data by querying existing search engines, databases, applications and enterprise services. Search for anything; Swirl will use a built-in non-generative LLM to find the best results from all sources, and present them. Adjust the selections, then RAG in real-time to get human-directed, verifiable AI insight. Fetch the freshest available data from systems of record - without extracting, copying and pre-indexing anything.

Community Contributions

🏆 This version of Swirl includes numerous contributions from the open source community. A thousand thank-you's to the following GitHub users for their accepted, merged PRs: Yash Mathur (Yash-sudo-web), s-vamshi, Ikko Eltociear Ashimine (eltociear), Akash Kundu (Akash190104), Biplob Sutradhar (biplobsd), Harsh Mahajan (HarshMN2345), Varanasi Aditya Sai (fvaysh), Neel Shah (neelshah2409) and Chandan Mali (Chandanmali)

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

🔹 Swirl 3.0 supports Real-Time Retrieval Augmented Generation (RAG). No copying or indexing of data is required as Swirl queries existing search APIs. Once RAG is triggered, optionally using the updated Galaxy UI, the new Page Fetcher component will retrieve result URLs and prepare a prompt. Note that the community edition only supports fetching of pages that don't require authentication. Contact [email protected] if you need to RAG using authenticated sources like M365, ServiceNow or Salesforce. Follow the new AI Guide to get started - the main requirement is to add an OpenAI ChatGPT API key. (Other Generative AI platforms will be supported soon.) The European PMC SearchProvider is pre-configured for RAG so insights can be generated in minutes.

🔹 The European PMC SearchProvider has been preloaded, set to active status, and configured to participate in RAG by default.

🔹 The new Retrieval Augmented Generation (RAG) Web Socket API is now available to developers. Refer to the Developer Guide for details.

New Swirl Documentation: https://docs.swirl.today/

Swirl's product documentation has a new home, a new template, and a handy new find-in-page option!

Check it out:

Swirl 3.0 includes a new Google PSE SearchProvider that targets the Swirl documentation website. This SearchProvider is enabled by default so you can search Swirl documentation with Swirl!

New SearchProviders

Query for a specific Bitcoin Address or Transaction Hash using the new Blockchain.com Bitcoin SearchProviders:

Search organizations in Crunchbase with our new Crunchbase Basic SearchProvider:

🔑 Note: SearchProviders contributed by the Swirl Community, including those for Algolia and Trello, may be found in the SearchProviders/untested directory. Some may include Connectors as well as SearchProviders. All will be moved into the production SearchProviders and pre-loaded in future releases.

Improvements

🔹 Repository Standards: Swirl has published a full set of GitHub Community Standards to support our growing community of contributors.

🔹 Microsoft OneDrive SearchProvider was updated to omit folders as results; it now only returns documents.

Resolved Issues

🔹 The Microsoft Authenticator toggle now turns green when active.

🔹 The updated RequestsGet connector now handles sources that return only a single result.

🔹 The legacy redis.conf file was removed.

🔹 The Galaxy UI login page now handles window resizing more gracefully.

Known Issues

🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:

We cannot take you to that message because it's in a chat you're not in.

For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 3.0.0 requires database migration. See Upgrading Swirl.

⚠️ As of version 2.6, Swirl's start-up process no longer starts redis. You must now have redis installed and running before starting Swirl.

Documentation

Overview | Quick Start | User Guide | Admin Guide | M365 Guide | Developer Guide | Developer Reference | AI Guide

Support

🔹 Join the Swirl Community on Slack!

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!

Team Swirl is announcing General Availability of Swirl 2.6!

This release features new SearchProviders for Service Now, Google News and a searchable Hacker News implementation! We've also tested Swirl on the latest stable Python version (3.11.5) and updated our Dockerfile image to the latest stable Debian release (Bookworm).

⚠️ As of Release 2.6, Swirl's start-up process no longer starts redis. You must now have redis installed and runnning before starting Swirl.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

🔹 Service Now SearchProviders...

🔹 Search Google News articles and both Stories and Coments from Hacker News with new SearchProviders for each source.

🔹 Swirl 2.6 has been thoroughly tested and validated to run on the latest stable Python release (3.11.5).

🔹 Swirl's Dockerfile Debian image has been updated from 3.11.4-slim-bullseye to 3.11.5-slim-bookworm, which addresses several vulnerabilities.

Improvements

🔹 Swirl's installation script (install.sh) will now check for a running Redis instance and for the presence of jq on the system, and emit helpful messages if they are not found.

🔹

🔹

Resolved Issues

🔹

🔹

🔹

🔹

Known Issues

🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:

We cannot take you to that message because it's in a chat you're not in.

For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 2.6 requires database migration. See Upgrading Swirl.

⚠️ As of version 2.6, Swirl's start-up process no longer starts redis. You must now have redis installed and runnning before starting Swirl.

Documentation

🔹 Quick Start 🔹 User Guide 🔹 Admin Guide 🔹 M365 Guide 🔹 Developer Guide 🔹 Developer Reference

Support

🔹 Join the Swirl Community on Slack!

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!

Team Swirl is announcing General Availability of Swirl 2.5.1!

This release features three new SearchProviders (arXiv.org, EuropePMC.org, and LinkedIn profiles via Google PSE), enhanced result processing to handle XML responses from source APIs, plus updates to the ChatGPT Connector and QueryProcessor to support the latest GPT-3.5-Turbo and GPT-4 models.

🥇 We would also like to give a shout out to GitHub user Rajpratik71 for his contribution to this release which improved our Dockerfile configuration. Many thanks for contributing to Swirl!

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

🔹 Both the ChatGPT Connector and QueryProcessor were updated to use OpenAI's ChatCompletion method which supports the latest GPT models - including GPT-4 - and a much greater range of interactivity.

• Three new Tags and one new query_mapping option are available for the ChatGPT SearchProvider to help shape the Prompt or Default Role passed to ChatGPT along with the user's query.
• The ChatGPT SearchProvder now queries the GPT-3.5-Turbo model by default.

🔹 Search LinkedIn profiles with the new LinkedIn Google PSE and SearchProvider.

🔹 Search the academic paper collections of arXiv.org and EuropePMC.org with new SearchProivders for each.

🔹 New result processing to handle XML responses from source APIs; see this in action with the arXiv.org SearchProvider.

🔹 Swirl's Dockerfile now uses the --no-cache-dir flag with pip to save space.
Thanks to GitHub user Rajpratik71 for this contribution!

Improvements

🔹 The Subscribe functionality was reworked to better handle Microsoft Authentication for subscribe runs that include M365 sources.

🔹 Some default SearchProviders have additional Tags: Internal or Develop. See the documentation for details about targeting specific groups of sources using SearchProvider Tags.

🔹 The install.sh script was further enhanced to synchronize spaCy model and code version updates.

Resolved Issues

🔹 Addressed an issue where Swirl could hang due to very long source repsonse times. Swirl's default timeout value in swirl_server/settings.py is now: SWIRL_TIMEOUT_DEFAULT = 10

🔹 Corrected a count inconsistency in the CosineRelevancyResultProcessor

🔹 Removed duplicate entries from urls.py

🔹 Fixed a regression where unmatched Tags were being dropped in searches

🔹 ChatGPT no longer returns a 500 error when enabled for Subscribed searches

🔹 After authenticating to Mircosoft in the Galaxy UI, the first search run now correctly includes M365 resutls

🔹 Addressed an issue with date-sorting in the M365 sources:

• The default date-sort for M365 sources is: DEFAULT_DATESORT_X = "createdDateTime desc"
• This can be overridden by setting DATE_SORT in the SearchProvider query_mapping
• Owners of a Search object or Swirl's Subscribe feature can set the sort attribute on a Search

Known Issues

🔹 Clicking a Microsoft Teams result sometimes results in this error from the Teams app:

We cannot take you to that message because it's in a chat you're not in.

For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 2.5.1 requires database migration. See Upgrading Swirl.

⚠️ Version 2.5.0 replaced RabbitMQ with Redis. If you are upgrading from any Swirl 1.x version to Release 2.5.1, please see the Quick Start - Local Installation section for details on how to install Redis with Swirl on MacOS or Linux.

Documentation

🔹 Quick Start 🔹 User Guide 🔹 Admin Guide 🔹 M365 Guide 🔹 Developer Guide 🔹 Developer Reference

Support

🔹 Join the Swirl Metasearch Community on Slack!

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!

Team Swirl is announcing General Availability of Swirl 2.5!

The theme for this release was performance. Configured with 12 SearchProviders, Swirl 2.5 supports up to 15 queries/second on a Standard F16s v2 server (16 vcpus, 32 GiB memory) with a median response time of 3 seconds.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

Performance Improvements

🔹 Swirl's asynchronous back-end was refactored to use the group method and replaced RabbitMQ with Redis

🔹 Relevancy processing was split into two stages to improve performance

• The revised CosineRelevancyPostResultProcessor must be in the Search.post_result_processors list.
• Please review the JSON in the SearchProviders/ directory and update existing configurations to match.

HubSpot SearchProviders

🔹 Search Company, Contact, and Deal records with new token-based HubSpot SearchProviders

• HubSpot requires creation of a "Private Apps" API token by the Super Admin with these scopes
• In result_mappings, the url mapping must contain the HubSpot Org ID

Improvements

🔹 The DateFindingResultProcessor was added to the Google PSE SearchProvider JSON. It finds a date in a large percentage of results that otherwise wouldn't have one, and copies the date to the date_published field. Existing PSE SearchProvider configurations should be updated as follows:

"result_processors": [
            "MappingResultProcessor",
            "DateFinderResultProcessor",
            "CosineRelevancyResultProcessor"
        ],

🔹 Swirl's control script (python swirl.py) was updated

• A default key was added to the services list, including debug, in swirl/services.py
• The celery-beats service was set to default: False
• swirl.py was modified to load only services with default: True

To use the subscribe or expiration features, celery-beats must be started manually:
python swirl.py start celery-beats

Celery-beats must also be stopped separately.

🔹 Swirl now outputs a single log entry with each request at the default log level INFO

2023-08-02 10:49:09,466 INFO     admin search 452 FULL_RESULTS_READY 32 2.2

Detailed logging is available in Debug mode: restart Swirl with the --debug flag to enable

🔹 The install.sh script was refactored to better handle spaCy upgrades

🔹 Swirl's Docker image was updated to use the latest, stable release of Python, version 3.11.4

🔹 Use of the unsafe eval() function was removed

Resolved Issues

🔹 Swirl's log files now persist and append new entries across restarts

🔹 The SearchProvider configurations for Elastic, OpenSearch, and Microsoft Outlook Messages were updated

🔹 A seesion-related issue in the connectors for Sqlite3, PostgreSQL, and Elastic was fixed

🔹 Errors in the DedupeByFieldResultProcessor were addressed

🔹 The SearchProvider results_per_query value is handled correctly now

🔹 An issue where some searches didn't appear to reach the Swirl backend was fixed

🔹 Unnecessary WARNING messages were removed from the logs

🔹 Result URLs on Search objects are now constructed correctly using the Request instead of loading from django settings

Known Issues

🔹 Clicking a Microsoft Teams result sometimes results in this error from the MS Teams app:

We cannot take you to that message because it's in a chat you're not in.

For Teams links to resolve correctly, you must have the Microsoft Teams app open before clicking the result link.

🔹 Subscribed searches are not returning results for Microsoft sources; this will be addressed in an upcoming release.

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 2.5 requires database migration. See Upgrading Swirl.

⚠️ Version 2.5 replaces RabbitMQ with Redis. Swirl's latest Docker image already contains this change. See the Quick Start - Local Installation section for details on how to install Redis with Swirl on MacOS or Linux.

⚠️ The revised CosineRelevancyResultProcessor must be added last to the result_processors block of all SearchProvider configurations except ChatGPT. For example:

"result_processors": [
            "MappingResultProcessor",
            "LenLimitingResultProcessor",
            "CosineRelevancyResultProcessor"
        ],

This following of Error message in the Swirl logs indicates that one or more SearchProviders have not been updated:

INFO     search.py: invoking processor: CosineRelevancyPostResultProcessor
2023-07-31 16:31:39,268 ERROR    CosineRelevancyPostResultProcessor_2051: Error: Dictionary of result lengths is empty. Was CosineRelevancyResultProcessor included in Search Providers Processor configuration?

⚠️ Exisitng SearchProvider configurations for Elastic, OpenSearch, and Microsoft Outlook Messages shoudl be updated:

• Elastic: Remove hosts= and http-auth= (just that text, not the values that follow)
• OpenSearch: Update the query_template syntax to the following:

"query_template": "{\"query\":{\"query_string\":{\"query\":\"{query_string}\",\"default_field\":\"{default_field}\",\"default_operator\":\"and\"}}}",

• Microsoft Outlook Messages: Update both result_grouping_field and result_processors to the following:

"result_grouping_field": "conversationId",
"result_processors": [
            "MappingResultProcessor",
            "DedupeByFieldResultProcessor",
	    "CosineRelevancyResultProcessor"
],

Documentation

🔹 Quick Start 🔹 User Guide 🔹 Admin Guide 🔹 M365 Guide 🔹 Developer Guide 🔹 Developer Reference

Support

🔹 Join the Swirl Metasearch Community on Slack!

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!

This update addresses an issue in the OpenSearch, Elasticsearch, and PostgreSQL connectors.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

Resolved Issues

🔹 The OpenSearch, Elasticsearch, and PostgreSQL connectors were updated to address an issue that resulted in a positional argument error. Please update your Swirl instance to v. 2.1.1 to correct this error.

Known Issues

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 2.1.1 does not require a database migration.

Documentation Wiki

🔹 Quick Start
🔹 User Guide
🔹 Admin Guide
🔹 M365 Guide
🔹 Developer Guide
🔹 Developer Reference

Support

🔹 Join the Swirl Metasearch Community on Slack!

🔹 Create a GitHub Issue if something doesn't work, isn't clear, or should be documented

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!

Team Swirl is announcing General Availability of Swirl Metasearch 2.1!

This version features the new Galaxy User Interface with Dark Mode, new GitHub SearchProviders, support for custom request headers, a new DateFinderResultProcessor that finds dates in the body field and copies them to date_published, plus new Result Processors for trimming and cleaning long responses. 2.1 also improves handling of recurring Microsoft 365 Calendar events and Outlook Message threads, showing only the most recent.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

🔹 Introducing the Galaxy UI:

🔹 Search public repositories with token-based GitHub SearchProviders for Code, Commits, Pull Requests, and Issues

🔹 Micrsoft 365 Calendar Events pre-configured SearchProvider with automatic hiding of recurring meetings

🔹 New date_published_display option available in SearchProvider result_mappings
You can now specify different values for date_published and date_published_display in SearchProvider configurations:

"result_mappings":  ... date_published=foo.bar.date1,date_published_display=foo.bar.date2 ...

Swirl Result objects will reflect both date values:

"date_published": "2010-01-01 00:00:00",
"date_published_display": "c2010",

And, the new Galaxy UI displays date_published_display while date-sorting by date_published:

🔹 Pass custom HTTP Header values in SearchProviders
The optional http_request_headers field is available to all SearchProviders for passing custom HTTP Header values to the source alongside the user's query. The new GitHub SearchProviders use this feature to return a more accurate result snippet from GitHub that is then mapped to Swirl's body field:

"http_request_headers": {
            "Accept": "application/vnd.github.text-match+json"
        },

"result_mappings": "title=name,body=text_matches[*].fragment, ...

🔹 New Result Processor: LenLimitingResultProcessor
This processor checks to see if the title and body responses from a source exceed a configurable length, truncates anything after that value, and adds an elipsis ("..."). If the body field has been truncated, the processor reports the entire response in a new body_full field in the Payload. The default max_length value for this processor is set in swirl_server/settings.py: SWIRL_MAX_FIELD_LEN = 512 and can be overridden for a specific SearchProvider using a new Tag value (max_length:###):

"tags": [
        "Articles",
        "Wiki",
        "YouTrack",
        "max_length:256"
    ]

The processor reports on the number of items updated as expected. It should be considered for sources that consistently return lengthy title or body fields and should follow the MappingResultProcessor. The default SearchProvider configurations for Atlassian Jira and Confluence have been updated with this processor.

"result_processors": [
            "MappingResultProcessor",
            "LenLimitingResultProcessor"
        ],

🔹 New Result Processor: CleanTextResultProcessor
This processor removes non-alphanumeric characters from the source response. It should be considered for lengthy responses where URLs or other HTML or Markdown syntax appear in results. It should be installed before the LenLimitingResultProcessor when both are used. The default SearchProvider configurations for JetBrains YouTrack have been updated with these processors.

"result_processors": [
            "MappingResultProcessor",
            "CleanTextResultProcessor",
            "LenLimitingResultProcessor"
        ],

🔹 New Result Processor: DateFinderResultProcessor
This processor looks for a date in any of the following forms in the body field of each result item. Should it find one, and the date_published for that item is 'unknown', it replaces date_published with the date extracted from the body, and notes this in the result.messages. This processor can detect the following date formats:

06/01/23
06/01/2023
06-01-23
06-01-2023
jun 1, 2023
june 1, 2023

Improvements

🔹 Updates to the genericRequestsPost connector and example SearchProvider JSON

🔹 Refined handling of threaded email messages in Outlook Messages results

🔹 Updated sorting of the search page Source dropdown and the result page Source facet in the Galaxy UI

🔹 Adjusted default Relevancy weight values to improve the relevancy ranking of shorter results

Resolved Issues

🔹 Re-running the previous search is now supported correctly in the Galaxy UI

🔹 An issue with Microsoft Authorization logout has been fixed

🔹 Clearing the search form now clears the previous Sort ("View By") selection correctly

🔹 Result Mixer choices are no long retained across searches

🔹 Result list numbering now displays correctly if Swirl's url field is not mapped

🔹 Various isssues with SearchProvider Tag searching were addressed

Known Issues

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 2.1 requires database migration. See Upgrading Swirl.

Documentation Wiki

🔹 Quick Start
🔹 User Guide
🔹 Admin Guide
🔹 M365 Guide
🔹 Developer Guide
🔹 Developer Reference

Support

🔹 Join the Swirl Metasearch Community on Slack!

🔹 Create a GitHub Issue if something doesn't work, isn't clear, or should be documented

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!

Swirl Metasearch 2.0.3

This version adds a generic RequestsPost Connector and example SearchProvider.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

🔹 A new RequestsPost Connector and example SearchProvider are available as of this release. See the Developer Reference Guide for more details, or Contact Support for help getting started.

Known Issues

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 2.0.3 does not require a database migration.

Documentation Wiki

🔹 Quick Start
🔹 User Guide
🔹 Admin Guide
🔹 M365 Guide
🔹 Developer Guide
🔹 Developer Reference

Support

🔹 Join the Swirl Metasearch Community on Slack!

🔹 Create a GitHub Issue if something doesn't work, isn't clear, or should be documented

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!

Swirl Metasearch 2.0.2

Swirl Metasearch 2.0.2 adds support for the X-Api-Key authentication method to SearchProviders.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

🔹 The X-Api-Key authentication method is now supported in RequestsGet and RequestsPost SearchProviders.

Known Issues

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 2.0.2 does not require a database migration.

Documentation Wiki

🔹 Quick Start
🔹 User Guide
🔹 Admin Guide
🔹 M365 Guide
🔹 Developer Guide
🔹 Developer Reference

Support

🔹 Join the Swirl Metasearch Community on Slack!

🔹 Create a GitHub Issue if something doesn't work, isn't clear, or should be documented

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!

Swirl Metasearch 2.0.1

Swirl Metasearch 2.0.1 improves default relevancy ranking, especially for some M365 SearchProvider results.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

Improvements

🔹 Default relevancy weights and similarity thresholds were adjusted to improve overall relevancy ranking, especially for M365 sources.

Known Issues

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 2.0.1 does not require a database migration.

Documentation Wiki

🔹 Quick Start
🔹 User Guide
🔹 Admin Guide
🔹 M365 Guide
🔹 Developer Guide
🔹 Developer Reference

Support

🔹 Join the Swirl Metasearch Community on Slack!

🔹 Create a GitHub Issue if something doesn't work, isn't clear, or should be documented

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!

SWIRL SEARCH 1.10.1

This version adds SearchProviders for Atlassian's Confluence and Jira Cloud services.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

🔹 New Atlassian SearchProviders allow searching through Confluence and Jira content.

Here's the new Confluence provider JSON:

    {
        "name": "Wiki (web/Atlassian.com)",
        "active": false,
        "default": false,
        "connector": "RequestsGet",
        "url": "https://<your-domain>.atlassian.net/wiki/rest/api/content/search?expand=history,space,metadata.labels.results,children.attachment,children.page,children.comment,ancestors,body.view",
        "query_template": "{url}&cql=text~'{query_string}'",
        "query_processor": "",
        "query_processors": [
            "AdaptiveQueryProcessor"
        ],
        "query_mappings": "",
        "result_processor": "",
        "result_processors": [
            "MappingResultProcessor"
        ],
        "response_mappings": "FOUND=size,RESULTS=results",
        "result_mappings": "title=title,body=body[*].view[*].value,author=history[*].createdBy[*].publicName,date_published=history[*].createdDate,url='https://<your-domain>.atlassian.net/wiki{_links[*].webui}',space.name,metadata[*].labels[*].results[*].name,children[*].attachment[*].results[*].title,children[*].page[*].results[*].title,ancestors[*].title,NO_PAYLOAD",
        "results_per_query": 10,
        "credentials": "HTTPBasicAuth('<your-username>','<your-atlassian-token>')",
        "tags": [
            "Confluence",
            "Articles",
            "Wiki",
            "Atlassian"
        ]
    }

Here is an example result in JSON:

{
     "swirl_rank": 3,
     "swirl_score": 86.49796345374605,
     "searchprovider": "Wiki (web/Atlassian.com)",
     "searchprovider_rank": 2,
     "title": "Testing Space",
     "url": "https://testingbw.atlassian.net/wiki/spaces/TS/overview",
     "body": "Welcome to your team space! We've added some suggestions and placeholders. Everything is customizable. Get started with page templates: Template - Project plan Template - <em>Meeting</em> <em>notes</em> Template - Weekly status report Check out Get the most out of your team space for more tips. About Chat GPT Confluence: Confluence is a collaboration tool that enables teams to create, organize, and share their knowledge. It provides features such as document collaboration, team communication, and content management. Mission and vision What is your team's mission? What is your vision? <em>Meet</em> the team Add team members to your space. Team member Role Responsibility Team member Role Responsibility Team member Role Responsibility Contact us How can someone reach out to your team? [email protected] Tickets Jira board #channel Important Pages List them here Onboarding FAQs Add resources for new hires <em>Meeting</em> <em>notes</em> Add links to <em>meeting</em> <em>notes</em> Team goals List them here Team news Create a blog post to share team news. It will automatically appear here once it's published. Blog stream Create a blog post to share news and announcements with your team and company. Create blog post",
     "date_published": "2023-03-28 16:54:28.874000+00:00",
     "date_retrieved": "2023-04-14 11:13:38.740897",
     "author": "Jason Temple",
     "title_hit_highlights": [],
     "body_hit_highlights": [],
     "payload": {
         "space.name": "Testing Space",
         "children[*].attachment[*].results[*].title": [
             "astronaut.svg",
             "ledger.svg",
             "raised_hand.svg",
             "claudia.svg",
             "gael.svg",
             "angie.svg",
             "header_team_space.svg"
         ],
         "children[*].page[*].results[*].title": [
             "Template - Project plan",
             "Template - Meeting notes",
             "Template - Weekly status report",
             "Get the most out of your team space"
         ]
     }
 }

Here's another result in the Spyglass UI:

Resolved Issues

🔹 Result mappings now support underscore values within curly brace notation

🔹 Swirl results now aggregate JSON lists into strings correctly

🔹 Subscribe now correctly handles the relocated Results hits clause

Known Issues

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch

Please report any issues with this to support.

Upgrading

⚠️ Version 1.10.1 does not require a database migration.

Documentation Wiki

🔹 Quick Start
🔹 User Guide
🔹 Developer Guide
🔹 Admin Guide

Support

🔹 Create an Issue if something doesn't work, isn't clear, or should be documented

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!

SWIRL SEARCH 1.10

This version adds Querqy integration, improved and configurable hit highlighting, plus YouTrack SearchProviders and support for the open source Spyglass UI!

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

🔹 Swirl now incorporates feedback from Querqy including synonyms and other rewrites.

The terms are highlighted and used in relevancy.

🔹 Highlighting has been improved and supports tags as well as character markings. The default, specified in swirl_server/settings.py, is:

SWIRL_HIGHLIGHT_START_CHAR = '<em>'
SWIRL_HIGHLIGHT_END_CHAR = '</em>'

🔹 New YouTrack SearchProviders allows searching through Issues and Articles.

Here's the Issues provider JSON:

{
    "name": "Issues (web/YouTrack.cloud)",
    "connector": "RequestsGet",
    "url": "https://<your-domain>.youtrack.cloud/api/issues?fields=project(name),idReadable,tags(name),summary,description,reporter(fullName),created,votes,commentsCount,comments(text),subtasks(issues(idReadable)),customFields(name,value(name))",
    "query_template": "{url}&query={query_string}",
    "query_processors": [
        "AdaptiveQueryProcessor"
    ],
    "result_processors": [
        "MappingResultProcessor"
    ],
    "result_mappings": "title=summary,body=description,date_published=created,author=reporter.fullName,url='https://<your-domain>.youtrack.cloud/issue/{idReadable}',project.name,customFields[3].value[*].name,customFields[0].value[*].name,customFields[2].value[*].name,customFields[6].value[*].name,votes,tags[*].name,commentsCount,comments[*].text,subtasks[*].issues[*].idReadable,NO_PAYLOAD",
    "results_per_query": 10,
    "credentials": "bearer=your-youtrack-permanent-token",
    "tags": [
        "Issues",
        "YouTrack"
    ]
}

Here is an example result in JSON:

{
      "swirl_rank": 5,
      "searchprovider": "Issues (web/YouTrack.cloud)",
      "searchprovider_rank": 3,
      "title": "Bad <em>Relevancy</em> link in User Guide",
      "url": "https://swirl.youtrack.cloud/issue/SP-1",
      "body": "User Guide in '<em>Relevancy</em>' section shows link to: for 'Developers Guide, <em>Relevancy</em> Section' but it does not go anywhere.\n\n\n\n![](image.png){width=70%}",
      "date_published": "2023-03-22 11:35:30.956000",
      "date_retrieved": "2023-03-28 17:56:41.043101",
      "author": "Lee Eichelberger",
      "title_hit_highlights": [],
      "body_hit_highlights": [],
      "payload": {
          "project.name": "Swirl - Product",
          "customFields[3].value[*].name": "Sid Probstein",
          "customFields[0].value[*].name": "Normal",
          "customFields[2].value[*].name": "Fixed",
          "commentsCount": 1,
          "comments[*].text": "Fixed by Sid.  \n\nCommit: 1ffe987771e76643308ed2f97e1c0544be510ed8"
      }
      ...etc...

Resolved Issues

🔹 BigQuery connector error with single result_mapping that is type date #146

🔹 Review parsing of credentials and tokens everywhere #159

🔹 Hit highlighting: no highlighting of title terms with adjacent punctuation #77

🔹 Swirl now handles Unix-style dates in SearchProvider responses

🔹 RequestsGet no longer requires a RESULTS entry in the response_mapping

Known Issues

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch

Please report any issues with this to support.

Upgrading

⚠️ Version 1.10 requires database migration. See Upgrading Swirl.

Documentation Wiki

🔹 Quick Start
🔹 User Guide
🔹 Developer Guide
🔹 Admin Guide

Support

🔹 Create an Issue if something doesn't work, isn't clear, or should be documented

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!

Swirl Metasearch 2.0

We are proud to annouce Swirl Metaseearch 2.0 - a major new version featuring enterprise-grade OAUTH2 support plus pre-configured connectors to Microsoft 365 Outlook, OneDrive, Teams and Sharepoint! Using the updated Spyglass UI, knowledge workers can systematically review the best results from their enterprise services, including search engines like Elastic, Solr, AWS OpenSearch, databases like PostgreSQL and Google BigQuery, plus online services like Northern Light Research and Miro.com.

PLEASE STAR OUR REPO: https://github.com/swirlai/swirl-search

New Features

🔹 Microsoft 365 Integration

This update of Spyglass includes a clean new layout, a live source facet, and the option to view results by relevancy, date or the top pick from each source. Swirl 2.0 can also be quickly and securely integrated with M365 as described in the new M365 Guide.

🔹 Query Transformations

Swirl now includes built in Query Transformations, including re-writing and replacement of matching terms. Full details are in the updated Developer's Guide.

🔹 Separate Blocks for Specific SearchProvider Results

ChatGPT's commentary on each query can now be presented separately on the results page.

🔹 New Options for SearchProvider result_mappings

Swirl now supports the FILE_SYSTEM directive, which causes the CosineRelevancyPostResultProcessor to boost matches in body when there are no matches in title. This is the default for Microsoft OneDrive.

Swirl also supports the BLOCK=some-block-name directive. This is configured by default for ChatGPT, as noted above.

Improvements

🔹 Simplified PAYLOAD field display - Swirl now displays just the right-most JSON key for maximum readability

🔹 Improved handling of Unix timestamp variations from sources

🔹 Refined handling of nested source response in JSON

Resolved Issues

🔹 Fixed issue with NOT searches being incorrectly handled in some cases

🔹 Addressed issues in Subscribe functionality

🔹 Hit highlighting: SWIRL adds a space before matching terms next to a comma #119

🔹 Removed single-processor properties

Known Issues

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch.

Please report any issues with this to support.

Upgrading

⚠️ Version 2.0 requires database migration. See Upgrading Swirl.

Documentation Wiki

🔹 Quick Start
🔹 User Guide
🔹 Admin Guide
🔹 M365 Guide
🔹 Developer Guide
🔹 Developer Reference

Support

🔹 Join the Swirl Metasearch Community on Slack!

🔹 Create a GitHub Issue if something doesn't work, isn't clear, or should be documented

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!

Swirl Metasearch 1.10:

• Integrates Querqy feedback from Elastic, Solr, OpenSearch into relevancy and highlighting
• Improves hit highlighting, and now supports configurable highlighting character(s) or tags with default term
• Resolves issues with hit highlighting, BigQuery, providers that don't specify RESULTS and more
• Adds YouTrack SearchProviders for Issues and Articles
• Adds Optional Spyglass UI integration

SWIRL SEARCH 1.9

This version adds a ChatGPT connector, SearchProvider for Miro.com and more!

PLEASE STAR OUR REPO: http://swirl.today/

New Features

🔹 OpenAI ChatGPT Connector & SearchProvider:

The Connector rewrites keyword queries into question form, and answers them. It doesn't alter queries that are already questions, as shown above. ChatGPT responses are usually relevant, regardless of accuracy.

🔹 A set of experimental ChatGPT Processors can be used to rewrite queries as questions, booleans and more.

• ChatGPTQueryImproverProcessor - writes a better query
• ChatGPTQueryExpanderProcessor - writes a broader query using top 5 concepts
• ChatGPTQueryBooleanProcessor - writes a boolean + phrases; 3+ terms works best

🔹 New Miro SearchProvider allows searching drawings.

{
    "name": "Boards (web/Miro.com)",
    "connector": "RequestsGet",
    "url": "https://api.miro.com/v2/boards",
    "query_template": "{url}?query={query_string}",
    "query_processors": [
        "AdaptiveQueryProcessor"
    ],
    "query_mappings": "",
    "result_processors": [
        "MappingResultProcessor"
    ],
    "response_mappings": "FOUND=total,RESULTS=data",
    "result_mappings": "title=name,body=description,url=viewLink,date_published=modifiedAt,owner.name,picture.imageURL,NO_PAYLOAD",
    "credentials": "bearer=your-miro-api-token",
    "tags": [
        "Drawings"
    ]
}

Changes

🔹 Starting queries with a SearchProvider tag now limits the search to just providers with that tag.

For example:

company:tesla

With the funding dataset installed, this query will return records like:

{
        "swirl_rank": 1,
        "swirl_score": 438.9910228828939,
        "searchprovider": "Company Funding Records (cloud/BigQuery)",
        "searchprovider_rank": 1,
        "title": "*Tesla* Motors",
        "url": "tesla-motors",
        "body": "*Tesla* Motors raised $40000000 series c on 2006-05-01. The company is located in San Carlos CA and has 270 employees.",
        "date_published": "2006-05-01 00:00:00",
        "date_retrieved": "2023-02-13 23:34:52.306501",
        "author": "",
        "payload": {},
        "explain": {
            "stems": "tesla",
            "title": {
                "tesla_*": 0.8357298742623626,
                "Tesla_0": 0.8357298742623626,
                "result_length_adjust": 1.0,
                "query_length_adjust": 1.0
            },
            "body": {
                "Tesla_0": 0.7187157993182859,
                "result_length_adjust": 1.0,
                "query_length_adjust": 1.0
            }
        }
    }

🔹 RequestsGet now supports Bearer Tokens

This is used by the Miro SearchProvider:

    "credentials": "bearer=your-api-token-here",

Resolved Issues

🔹 BigQuery connector/SP returns "Error: list index out of range" on zero results #96

Known Issues

🔹 Creating searches from a browser with q= can sometimes create two Search objects.

This is because of browser prefetch AKA predictive service. Turn off Chrome prediction service. Turn off Safari prefetch

Please report any issues with this to support.

Upgrading

⚠️ Version 1.9 requires database migration.

Documentation Wiki

🔹 Quick Start
🔹 User Guide
🔹 Developer Guide
🔹 Admin Guide

Support

🔹 Create an Issue if something doesn't work, isn't clear, or should be documented

🔹 Email: [email protected] with issues, requests, questions, etc - we'd love to hear from you!