A TYPO3 extension that integrates the Apache Solr search server with TYPO3 CMS. dkd Internet Service GmbH is developing the extension. Community contributions are welcome. See CONTRIBUTING.md for details.
GPL-3.0 License
Published by timohund about 7 years ago
This release is a bugfix only release. The release contains the following bugfixes
The sorting counts have been caluclated wrong. This patch fixes the calculation.
See also:
The TypoScript pathes to add the plugin just with TypoScript have been removed. This patch adds these pathes again.
See also:
The default fallback path was missing in the typoscript (plugin.tx_solr.view.templateRootPaths/partialRootPaths/layoutRootPaths).0
These pathes have been added to the default configuration.
See also:
The parameter "group.field" was filled with the wrong data. This was fixed.
See also:
The output of the TranslateViewHelper and AbstractWidgetViewHelper was not escaped. If you use user input (e.g. the querystring) in this context, the output was not properly escaped (not the case in the default partials).
Impact:
Check the partial "Resources/Private/Templates/ViewHelpers/Widget/ResultPaginate/Index.html"
Avoid to pass special characters to the translate view helper, e.g. as before:
<s:translate key="paginate_previous">«</s:translate>
Use a translation or don't use the translate ViewHelper:
<f:if condition="{s:translate(key: 'paginate_next')}">
<f:then><s:translate key="paginate_next" /></f:then>
<f:else>»</f:else>
</f:if>
See also:
Thanks to everyone who has supported this release.
Like always this release would not have been possible without the help from our
awesome community. Here are the contributors for this release.
(patches, comments, bug reports, reviews, ... in alphabetical order)
Also a big thanks to our partners that have joined the EB2017 program:
Thanks to everyone who helped in creating this release!
There are many ways to get involved with Apache Solr for TYPO3:
Support us in 2017 by becoming an EB partner:
http://www.typo3-solr.com/en/contact/
or call:
+49 (0)69 - 2475218 0
Published by timohund about 7 years ago
This release is a bugfix only release.
To prevent the html tag from rendering of the final output, the attribute data-namespace-typo3-fluid="true" was added to the template.
https://github.com/TYPO3-Solr/ext-solr/pull/1548
Like always this release would not have been possible without the help from our
awesome community. Here are the contributors for this release.
(patches, comments, bug reports, reviews, ... in alphabetical order)
Also a big thanks to our partners that have joined the EB2017 program:
Thanks to everyone who helped in creating this release!
There are many ways to get involved with Apache Solr for TYPO3:
Support us in 2017 by becoming an EB partner:
http://www.typo3-solr.com/en/contact/
or call:
+49 (0)69 - 2475218 0
Published by timohund about 7 years ago
We are happy to release EXT:solr 7.0.0. This release brings several smaller and some bigger changes
One and a half years ago we started to implement FLUID templating for EXT:solr. This project was initially started as the addon solrfluid. Solrfuid was only available for our partners.
With EXT:solr 7.0.0 the new templating is the default templating in EXT:solr. A lot of code was added and several old stuff was removed. Since some things are conceptional different in FLUID and you also have a lot of possibilities we also dropped some parts, that can be build with fluid itself or do not make sence to do them before rendering the result in the view.
Most of the things just work like before but in the following parts we made conceptional changes (for good reasons):
The following typoscript settings have been removed because they can be implemented with FLUID:
plugin.tx_solr.search.faceting.facetLinkATagParams You can add them in your project partials. If you need it just for one facet, please overwrite the render partial with facet.partialName and render the attributes different there
plugin.tx_solr.search.faceting.[facetName].facetLinkATagParams You can add them in your project partials. If you need it just for one facet, please overwrite the render partial with facet.partialName and render the attributes different there
plugin.tx_solr.search.faceting.facets.[facetName].selectingSelectedFacetOptionRemovesFilter
This can be implemented with FLUID logic. Please check the example "Search - (Example) Options with on/off toggle" that implements that (by using the partial Facets/OptionsToggle.html)
plugin.tx_solr.search.results.fieldRenderingInstructions Please use custom ViewHelpers or the cObject ViewHelper for that.
plugin.tx_solr.search.results.fieldProcessingInstructions Please use custom ViewHelpers or the cObject ViewHelper for that.
Important: The support of fluid templating would not have been possible without the financial support of all partners! If you want to support us with the implementation of features like this, please think about to join the EB 2017 or 2018. Special thanks also to Frans Saris and beech.it for working on solrfluid together!
In EXT:solr 7.0.0 the backend modules are structured into multiple backend modules. This makes the user experience in the TYPO3 backend more consistent and allows you, to give different permissions on each module.
When you login into the backend, you now have the following modules available:
Info: Gives information of your Solr system, index fields and search usage.
Core Optimization: This module can be used to maintain the synonyms and stopwords in the Apache Solr server.
Index Queue: Gives an overview on indexed records and can be used to requeue records for indexing.
Index Administration: This module can be used for administrator tasks on your solr system (clear index, index queue or reload a core)
If you want to exclude an optionValue from the facets when they get retrieved, you can configure this now:
plugin.tx_solr.search.faceting.facets.colors_stringM.excludeValues = red
The example below will exclude the option "red" from the results when it is in the response.
In previous EXT:solr versions it was possible to set a custom entry templating using:
plugin.tx_solr.templateFiles.search = EXT:solr/Resources/Templates/PiSearch/search.htm
This configuration could be overwritten with a text value in the flexform.
With the move to FLUID we improved this part and made it more editor friendly:
The following example shows, how you can load your own partials and provide different entry templates for the editor:
plugin.tx_solr {
view {
templateRootPaths.100 = EXT:your_config_extension/Resources/Private/Templates/
partialRootPaths.100 = EXT:your_config_extension/Resources/Private/Partials/
layoutRootPaths.100 = EXT:your_config_extension/Resources/Private/Layouts/
templateFiles {
results = Results
results.availableTemplates {
default {
label = Default Searchresults Template
file = Results
}
products {
label = Products Template
file = ProductResults
}
}
}
}
}
With the prevision configuration the editor can switch from "Default Searchresults Template" to "Products Template".
The Query class is one of the biggest classes in EXT:solr and grown over time. This class has now been splitted into several classes.
Along with that a concept of "ParameterBuilder" has been introduced. A ParameterBuilder is responsible to build a parameter part of the query.
E.g. the Grouping ParameterBuilder is responsible to build all parameters of the solr query for the grouping.
In Solrfluid there was one folder for each facet, that contains the facet class and a parser that parsers the solr response into the facet object.
The opposite part(parse the url, build the solr query) was previously done in EXT:solr, with a FilterEncoder that was registered in the FacetRendererFactory.
Now because solrfluid and solr have been merged, this logic can also be streamlined. Every facet is now structured in a FacetPackage.
A FacetPackage describes:
You can also implement custom facet types by registering an on FacetPackage with the FacetRegistry.
Migration:
When you have implemented an own FacetParser for solrfluid, you should add a FacetPackage, that references a UrlDecoder and QueryBuilder.
If you have used a custom FacetParser without registring a custom facet type in EXT:solr (ApacheSolrForTypo3\Solr\Facet\FacetRendererFactory::registerFacetType) you can just reference DefaultUrlDecorder and DefaultFacerQueryBuilder in your FacetPackage.
Before solrfluid was merged there were several parts in EXT:solr where the data was read using GeneralUtility::_GET. The drawback of this approach is that the structure of the urls is hard to change and it is not possible to have custom namespaces for each instance of a plugin.
With solrfuid a SearchRequest object was introduced. This object holds all data from the user request. Now this object is used, whenever data from the user action is read. This allows us to make the request namespace changeable. You can now add your custom plugin namespace to a search plugin instance.
As an ongoing task, we started with the migration of database queries to doctrine. Since the database is used in many parts of the extension there are still many parts open.
If you want to work on that, your help is very welcome.
If you want or need to limit the initialization of solr connections to a special rootpage, you can now do this by adding the argument --rootpageid.
This setting was not evaluated in EXT:solrfluid before and is now available also with FLUID rendering.
This setting was not evaluated in EXT:solrfluid before and is now available also with the FLUID rendering.
This setting was not evaluated in EXT:solrfluid before and is now available also with the FLUID rendering.
By now EXT:solr had the dependency on an existing domain record. This can be a problem, when you domain is dynamic or
you need to be able to generate it.
Now you can configure a domain by the rootPageId in the TYPO3_CONF_VARS, the domain record is still used, when nothing is configured here.
$GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['solr']['sites'][###rootPageId###]['domains'] = ['mydomain.com'];
Note:
There might be an approach to support this in TYPO3 Version 9 by the core and we will adopt this then.
During the implementation of this the logic to retrieve the SiteHash and get the SolrConfiguration was moved to the SiteRepository,
this requires an update of the scheduler instances because the scheduler saves a serialized task. Please run the shipped migration to
update scheduler tasks created with 6.1.x.
Several things that will be removed with 9 have been changed:
The following code has been removed since it is not used anymore:
Classes:
Methods:
Methods:
Method Arguments:
Hooks:
Like always this release would not have been possible without the help from our
awesome community. Here are the contributors for this release.
(patches, comments, bug reports, reviews, ... in alphabetical order)
Also a big thanks to our partners that have joined the EB2017 program:
Thanks to everyone who helped in creating this release!
In the next release we want to focus on the user experience in the backend and in the frontend. As preparation we collected several tasks.
The goal of some of them (e.g. bootstrap templating, checkbox facets, filterable options partial) is to make more things possible out of the box and make the extension more user friendly:
https://github.com/TYPO3-Solr/ext-solr/issues?q=is%3Aissue+is%3Aopen+label%3AUX
If you have allready implemented one this (or something else), that you want to share or make available outofthebox feel free to contanct us!
There are many ways to get involved with Apache Solr for TYPO3:
Support us in 2017 by becoming an EB partner:
http://www.typo3-solr.com/en/contact/
or call:
+49 (0)69 - 2475218 0
Published by timohund about 7 years ago
This release is a bugfix only release.
This change:
https://github.com/TYPO3-Solr/ext-solr/pull/1527
Applies htmlspecialchars on the output of the ScoreCalculation.
https://github.com/TYPO3-Solr/ext-solr/pull/1528
Applies htmlspecialchars on the output of the SiteSelector.
https://github.com/TYPO3-Solr/ext-solr/pull/1533
The relation handler did not handle page overlays correctly before.
This pr provides a fix to handle the translation of pages correctly.
https://github.com/TYPO3-Solr/ext-solr/pull/1535
Like always this release would not have been possible without the help from our
awesome community. Here are the contributors for this release.
(patches, comments, bug reports, reviews, ... in alphabetical order)
Also a big thanks to our partners that have joined the EB2017 program:
Thanks to everyone who helped in creating this release!
There are many ways to get involved with Apache Solr for TYPO3:
Support us in 2017 by becoming an EB partner:
http://www.typo3-solr.com/en/contact/
or call:
+49 (0)69 - 2475218 0
Published by timohund over 7 years ago
6.1.2 - Maintenance release
This release is a bugfix only release.
This missing parameter type leads to a warning in PHP7.
https://github.com/TYPO3-Solr/ext-solr/pull/1386
Creates the datafolder before giving the permissions and allows to use the Dockerfile without further configuration.
https://github.com/TYPO3-Solr/ext-solr/issues/1278
If the root page is set to 'hide default translation of page' this causes a 404 in TypoScriptFrontendController with initializing the TSFE. To avoid this a language uid has to be thrown to the initialize function.
https://github.com/TYPO3-Solr/ext-solr/pull/1414
A cache in getFullItemRecord avoids the correct initialization of the TSFE. Since Util::initializeTsfe is cached anyways, this cache could be removed and makes sure that a proper TSFE is initialized.
Like always this release would not have been possible without the help from our
awesome community. Here are the contributors for this release.
(patches, comments, bug reports, reviews, ... in alphabetical order)
Also a big thanks to our partners that have joined the EB2017 program:
Thanks to everyone who helped in creating this release!
There are many ways to get involved with Apache Solr for TYPO3:
Support us in 2017 by becoming an EB partner:
http://www.typo3-solr.com/en/contact/
or call:
+49 (0)69 - 2475218 0
Published by timohund over 7 years ago
This release is a bugfix only release.
When the rootPage resolver was called with $pageId = -1 it tried to check if this is really a rootPage.
Since we know that this is a workspace page, the RootPageResolver returns false now.
To be able to initialize only a few connections the option --rootpageid was added.
The Indexer uses the item record page to resolve the configuration. This can lead to problems when the records is outside a siteroot because the configuration is not present there.
This fix uses the configuration of the siteroot when no configuration is present as fallback to have an indexing configuration available.
The following scenario is given:
Site a references record from site b in additionalPageIds. When a change is done in the record we expect that the index queue item for both sites is created, since it is referenced in both sides. By now the tracking of the record outside the siteroot is only working when the referenced record does not belong to another site.
Like always this release would not have been possible without the help from our
awesome community. Here are the contributors for this release.
(patches, comments, bug reports, reviews, ... in alphabetical order)
Also a big thanks to our partners that have joined the EB2017 program:
Thanks to everyone who helped in creating this release!
There are many ways to get involved with Apache Solr for TYPO3:
Support us in 2017 by becoming an EB partner:
http://www.typo3-solr.com/en/contact/
or call:
+49 (0)69 - 2475218 0
Published by timohund over 7 years ago
We are ready for TYPO3 8 LTS - EXT:solr 6.1 has been released
Close to the release of TYPO3 8 LTS we are happy to announce EXT:solr 6.1 that is compatible with TYPO3 8 LTS.
In this release we've merged over 160 pull requests! With these pull requests several new features and bugfixes were added.
EXT:solr 6.1 is ready for TYPO3 8 LTS and php 7.1 while keeping the compatibility to TYPO3 7.6 LTS and PHP 5.5 and 5.6.
During the development of version 6.1 there have been a lot of smaller and bigger performance improvements.
The following changes might be interesting.
When a record is saved in the backend by now the whole typoscript configuration is evaluated for the page where the record is located. In many setups it is enough to just use the closest template in the rootline to parse the configuration.
Since there are cases, where this method does not work (e.g. when you use conditions based on page ids) you need to switch this behaviour explicitly on, by enable "useConfigurationFromClosestTemplate".
To optimize the performance the RecordMonitor and GarbageCollector class have been splitted into several components. These components use the TYPO3 caching framework to cache result that do not need to be retrieved multiple times.
If a record is relevant can only be decided in the context of a solr site because the typoscript configuration defines the indexing configuration.
When you want to tweek the backend performance you can define a global list of monitoredTables. Other tables will then be ignored and also the parsing of the typoscript configuration is not needed then.
Since extjs will be removed more and more from the TYPO3 core we decided to migrate the index inspector to fluid.
You can use the index inspector as usual from "Web > Info > Search Index Inspector" to analyze which documents are in the
solr server for a specific page.
Related Issues:
When the property "additionalPageIds" is configured for an indexing configuration this configuration is now taken into account in the record monitor to re-queue these elements when a change in the backend is done.
Since this is an expensive operation, you can disable this feature by configuring the extension setting (useConfigurationTrackRecordsOutsideSiteroot) in the extension manager.
Related Issues:
By now the devLog from TYPO3 was used to log messages in the extension. Since several years there is a more flexible logging framework available in the TYPO3 core.
In EXT:solr 6.1 we make use of this and use the TYPO3 logging framework to write the logs.
Related Issues:
The minimum php version (PHP 5.5) allows us to benefit from PHP language features that make the code more readable:
[]
can be used instead of array()
::class
all places where classes get instanciated with GeneralUtility::makeInstance can be simplyfiedRelated Issues:
For some search use cases it makes sence to support partial matches e.g. when you search for "boch" and want to mach a field with the content "bochum".
EXT:solr 6.1.0 shippes new dynamic field types that make "N-Gram" and "Edge N-Gram" processed fields available:
\*_textEdgeNgramS
Edge Ngram (hello => hello, hell..)\*_textEdgeNgramM
Edge Ngram (hello => hello, hell..)\*_textNgramS
Ngram (hello => he,ll,lo,hel,llo)\*_textNgramM
Ngram (hello => he,ll,lo,hel,llo)See also:
https://cuiborails.wordpress.com/2012/07/13/apache-solr-ngramedgengram/
Related Issues:
The siteHash is unique for a solr site in the TYPO3 system. When solr does a query the allowedSites setting can be used to control the set of documents that are queried in a solr core.
This is useful when you want to split the content from multiple sites in a single solr core. In some cases it is useful to disable this limitation.
E.g. when:
Before the extension "solr_disablesitehash" was required to turn this sitehash check off. With solr 6.1.0 we've changed the meaning of the allowedSites:
Migration: When you are using * for query.allowedSites change the setting to __all.
Related Issues:
In this release we have updated the shipped jQuery and jQueryUi components to the following versions:
Related Issues:
A lot of issues (formatting, small bugs, documentation,...) reported by scrutinizer have been resolved in this release.
Related Issues:
The solr access filter plugin has been optimized to use a solr post filter. By using a post filter the performance of this plugin is much better, because less documents need to be evaluated.
In this release we ship this new version 2.0.0 of the access filter with the default configSet and in our docker container.
Related Issues:
In the next release we will drop the support of TYPO3 7 LTS and only support 8 LTS. Since in the TYPO3 core the database access was migrated to Doctrine, we will use Doctrine as well.
To simplify this, we have centralized a lot of database queries in this release and also dropped duplicate queries to reduce to amount of work that needs to be done for this in the next releaese.
Related Issues:
Not every project can use officially signed SSL certificates for all stages. Because of that we allowed for the page indexing to index a host, with a self signed certificate.
You can enable this feature by configuring the extension setting (allowSelfSignedCertificates) in the extension manager to true.
Related Issues:
When you use EXT:solr in a deployment scenario (e.g. platform.sh) you maybe want to define the solr endpoints by environment variables or from variables in TYPO3_CONF_VARS. Both approaches are supported by the typoscript TEXT object.
Therefore it makes sence for the solr endpoint settings in plugin.tx_solr.solr
to support the usage of cObjects there. This allows you to define connections like this:
Addition to AdditionalConfiguration.php:
$GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['solr']['host'] = 'mysolrserver.de';
Usage in Typoscript:
plugin.tx_solr.solr {
host = TEXT
host {
value = localhost
override.data = global:TYPO3_CONF_VARS|EXTCONF|solr|host
}
}
Related Issues:
For the continues improvement of the codestructure and preparation for upcoming tasks, the following methods have been marked as deprecated and will be removed in EXT:solr 7.0 (next release):
Please check your deprecation log and replace the usages in your code.
In this release some breaking changes have been required. Please follow the notes below when you migrate from 6.0 to 6.1
The default variantId was table/uid
before since this id is not unique across multiple TYPO3 system, a system hash was added before.
The scheme of the new variantId is systemhash/table/uid
and allows to use grouping when data from multiple TYPO3 systems get indexed and searched.
Migration:
Reindex the solr data to get the new variantId.
Related Issues:
Migration:
When you use plugin.tx_solr.search.query.allowedSites = *
you should now use plugin.tx_solr.search.query.allowedSites = __all
Related Issues:
Because the devLog was replaced by the logging framework, the setting plugin.tx_solr.logging.debugDevlogOutput
does not make sence anymore.
Migration:
When you want to see the log output in the backend or frontend please use plugin.tx_solr.logging.debugOutput
now.
Related Issues:
Along we the removal of the reference GeneralUtility::getUserObj
was replaced with GeneralUtility::makeInstance
, because the usage with ":" is deprecated since TYPO3 8 and will be removed.
Migration:
When you reference custom indexer (in $GLOBALS['TYPO3_CONF_VARS']['EXTCONF']['solr']['IndexQueueIndexer']['indexItemAddDocuments']
) you should reference only an autoloadable classname.
The following steps are required:
Related Issues:
StatisticRepository::getTopKeyWordsWithOrWithoutHits $limit, $withoutHits have no default values
https://github.com/TYPO3-Solr/ext-solr/issues/1143
Wrong TS path in frontendDataHelper documentation
https://github.com/TYPO3-Solr/ext-solr/issues/964
Invalid include paths after scheduler run
https://github.com/TYPO3-Solr/ext-solr/issues/921
Backend Summary not working
https://github.com/TYPO3-Solr/ext-solr/issues/731
Queue initialization returns wrong (duplicate) results for second site root in multisite environment
https://github.com/TYPO3-Solr/ext-solr/issues/488
solrconfig.xml More Like This Handler configuration is broken
https://github.com/TYPO3-Solr/ext-solr/issues/765
Fix position of default stopwords
https://github.com/TYPO3-Solr/ext-solr/issues/578
Bug/unwanted feature in method getPages in Site.php
https://github.com/TYPO3-Solr/ext-solr/issues/652
Custom field is not indexed for custom page queue configuration
https://github.com/TYPO3-Solr/ext-solr/issues/842
GarbageCollector fails to check endtime correctly
https://github.com/TYPO3-Solr/ext-solr/issues/1212
"Only variables should be assigned by reference" in IndexQueue\Indexer::preAddModifyDocuments()
https://github.com/TYPO3-Solr/ext-solr/issues/1183
Use urlencode for deletion of synonyms and stopwords
https://github.com/TYPO3-Solr/ext-solr/issues/1205
https://github.com/TYPO3-Solr/ext-solr/issues/1206
This is the last release for TYPO3 7.6 LTS and the last release with marker based templating. The next release (7.0.0) will drop the marked based templating and will require TYPO3 8 LTS. If you want to start with fluid templating now we suggest to use solrfluid (available as EB partner)
Thanks to everyone who has supported the release of EXT:solr 6.1.0.
Like always this release would not have been possible without the help from our
awesome community. Here are the contributors for this release.
(patches, comments, bug reports, reviews, ... in alphabetical order)
Also a big thanks to our partners that have joined the EB2017 program:
Thanks to everyone who helped in creating this release!
There are many ways to get involved with Apache Solr for TYPO3:
Support us in 2017 by becoming an EB partner:
http://www.typo3-solr.com/en/contact/
or call:
+49 (0)69 - 2475218 0
Published by timohund over 7 years ago
This release is a bugfix only release.
When no username of password was configured an authentication with empty credentials was done.
#1182
When the solr path has more then two segments (e.g. /solr/core) a core reload failed.
#1211
Published by timohund almost 8 years ago
We're happy to announce the release of Apache Solr for TYPO3 (EXT:solr) version 6.0.0.
With this release we ship support for Apache Solr 6.3 and provide a Docker file, that
can be used to spin up a Solr server configured and ready to run within a few steps.
Besides that the release contains many other features and bugfixes.
In this release we've merged over 120 pull requests!
With these pull requests several new features and bugfixes were added.
The focus of this release was adding support for the latest version of Apache Solr
and making setup easier. Configuration of Apache Solr is now provided via configSets.
A configSet contains everything you need - solrconfig.xml, schema.xml & custom
access filter jar files - to configure and run a Solr server for use with TYPO3.
Related PRs:
The next release of EXT:solrfal will allow merging of duplicates into one document.
The implementation of this feature requires that the permissions are stored in a multi-value
field because different references to a file can have different permissions.
Thus the access field is now a multi-value field and the access filter plugin
is able to evaluate multiple access values.
Related PRs:
https://github.com/TYPO3-Solr/ext-solr/pull/675
By using the Fluid standalone views we have adapted the status reports
to use these views and create better output.
Related PRs:
https://github.com/TYPO3-Solr/ext-solr/pull/740
With the new Apache Solr version we have added languages Irish, Serbian, and Latvian.
You can now use them to index sites in these languages by creating a Solr core using the
configuration provided by the extension.
Related PRs:
https://github.com/TYPO3-Solr/ext-solr/pull/779
The old cli commands have been replaced with an Extbase command controller.
Please use the following commands to update Solr connections through cli:
php ./typo3/cli_dispatch.phpsh extbase solr:updateConnections
Related PRs:
https://github.com/TYPO3-Solr/ext-solr/pull/694
Our goal is to be able to support TYPO3 CMS v8 as soon as version 8 LTS will be released.
Many patches from the community have been merged to improve the compatibility for version 8 already.
Thanks to everyone who worked on that and provided contributions!
Related PRs:
Avoid schema retrieval:
Until now the Solr schema was fetched in the frontend even when it was not needed.
Now we only fetch the schema in the backend when it's really needed (when updating stopswords or synonyms).
Reduce ping requests:
The amount of ping requests has been reduced to only do one ping when the plugin is rendered.
Improved configuration caching:
Configuration object, are now cached in an in memory cache. This gives an improvement when pages in the backend are copied.
Related PRs:
Until now you could only limit the facet options shown when they get rendered in the frontend.
Sometimes however you want to reduce the available facet options at an earlier time when
retrieving them from Solr.
The following setting now allows you to configure this limit:
plugin.tx_solr.search.faceting.facetLimit = 50
Related PRs:
To ensure changes don't reduce the quality of the code we have introduced Scrutinizer-CI to
provide feedback on code quality. After adding Scrutinizer-CI we also started to resolve
initial issues reported step by step and to increase the coverage of automated tests.
You can have a look at the results of the inspections and see the impact there:
https://scrutinizer-ci.com/g/TYPO3-Solr/ext-solr/statistics/
A couple pull requests have been merged to introduce Scrutinizer and to fix several issues:
Related PRs:
Our goal is to improve the code step by step and don't get worse.
To avoid an error in Apache Solr with "too many open files", it is possible
to use the compound index format. Apache Solr is then writting to only one
index file instead of many. The downside however, is that the performance may
be slightly reduced but should not be significant for most projects.
We decided to change the default configuration here to use a more robust setting.
If you have very high performance requirements you can still change this behavior
by setting <useCompoundFile>false</useCompoundFile>
in your solrconfig.xml.
Related PRs:
We've added a new data type "textExact" to the solr schema. Beside that copyFields have been added for the following fields:
Beside that this type is also available as dynamic fields with the following suffixes:
Related PRs:
Username and password can now be configured for the solr connection:
plugin.tx_solr.solr.username = username
plugin.tx_solr.solr.password = password
Related PRs:
A timeout for the solr connection can now be configured:
plugin.tx_solr.solr.timeout = 20
Related PRs:
The following rendering instruction can be used, when you want to format a date as option facet and store it as date or timestamp.
plugin.tx_solr.search.faceting.facets.created {
field = created
label = Created
sortBy = alpha
reverseOrder = 1
renderingInstruction = TEXT
renderingInstruction {
field = optionValue
postUserFunc = ApacheSolrForTypo3\Solr\Domain\Search\ResultSet\Facets\RenderingInstructions\FormatDate->format
}
}
Related PRs:
Updating from version 5.1.1 or earlier there are some breaking changes that
your need to keep in mind when updating.
Since version 6 Apache Solr comes with an embedded Jetty server which completely
changes the setup compared to Apache Solr 4.10. It is now required to install a
new instance of Apache Solr Server. This can be done with our install script for
development or you can use our Docker image to start a new Solr server as a container.
Both approaches are described in our documentation. The embedded Solr Jetty server also uses a new default port (8983) to serve connections.
We are now using a command controller to update the Solr server connections.
Please use the following command to update your connections:
php ./typo3/cli_dispatch.phpsh extbase solr:updateConnections
The following bugs have been fixed in this release.
In the next release (6.1) our focus will be to prepare support for TYPO3 CMS
version 8 LTS as well as possible. The next release (6.1) will also be the
last release for TYPO3 CMS version 7.6 LTS.
Like always this release would not have been possible without the help from our
awesome community. Here are the contributors for this release.
(patches, comments, bug reports, reviews, ... in alphabetical order)
Also a big thanks to our partners that have joined the EB2016 program:
Thanks also to our partners who already singed up for a 2017 partnership (EB2017):
Thanks to everyone who helped in creating this release!
There are many ways to get involved with Apache Solr for TYPO3:
Support us in 2017 by becoming an EB partner:
http://www.typo3-solr.com/en/contact/
or call:
+49 (0)69 - 2475218 0
Published by timohund almost 8 years ago
This release is a bugfix only release.
By using the SOLR_RELATION option enableRecursiveValueResolution
relations can be resolved recursive if the given foreignLabelField
is a relation too. The bugfix resolves this issue and adapts the integration test to check if this feature is working.
Details: #690
Published by timohund about 8 years ago
We're happy to announce the release of Apache Solr for TYPO3 (EXT:solr) version 5.1.0. This release ships a few bugfixes and smaller features
The focus of this release was to provide the new features from the codesprint and bugfixes from the issue tracker. Beside that, we've added a variants feature that could be used together with the upcomming solrfal release as one approach to avoid duplicated files in the search frontend.
This feature adds a new optional field variantId that is filled with type/uid and can be used for collapsing by default.
The following example shows the required typoscript configuration that is needed:
plugin.tx_solr.search {
variants = 0
variants {
expand = 1
variantField = variantId
limit = 10
}
}
The collapsing can be used to collapse on numeric and string fields, text fields are not supported for collapsing by Solr.
Note: You need to update your solrconfig and schema to use the new field variantId. When you include the TS Template "Everything on", variants are active. You need to reindex in this case to make sure that the field is filled correctly.
Details: https://github.com/TYPO3-Solr/ext-solr/pull/550
During the transition from forge to github, we've moved the rest of the documentation to our rst documentation.
Details: https://github.com/TYPO3-Solr/ext-solr/pull/547
During the codesprint in Frankfurt a new statistics module was added for that backend that allows you to get some usefull insights
of your search installation.
During the codesprint several UX improvements in the backend module have been done:
Replace content element wizard icon with svg icon:
Detail: https://github.com/TYPO3-Solr/ext-solr/pull/581
Make init connections icon use SVG:
Details: https://github.com/TYPO3-Solr/ext-solr/pull/566
Backend module layout fixes
Details: https://github.com/TYPO3-Solr/ext-solr/pull/575
When you are using absRefPrefix auto and index in cli mode and you webroot differs from PATH_site you have the possibility to configure a
webroot in the scheduler task to make absRefPrefix=auto work in this case
Details: https://github.com/TYPO3-Solr/ext-solr/pull/495
When a site in the scheduler task was selected the selection could get lost.
Details: https://github.com/TYPO3-Solr/ext-solr/pull/557
In the next release (6.0) we will focus on the integration of the latest Apache Solr Version (currently 6.1.0)
Like always this release would not have been possible without the help from our
awesome community. These are the contributors for this release.
(patches, comments, bug reports, review, ... in alphabetical order)
Also a big thanks to our partners that have joined the EB2016 program:
Thanks to everyone who helped in creating this release!
There are many ways to get involved with Apache Solr for TYPO3:
Published by timohund over 8 years ago
We're happy to announce the release of Apache Solr for TYPO3 (EXT:solr) version 5.0.0. With this release we ship the requirements to use our new addon EXT:solrfluid, that allows you to render your search results with the fluid templating engine.
Beside the adaptions for solrfluid we mainly did bugfixes and cleanup in the code base. If you want to use solrfluid in your projects, you should signup a partnership with dkd (visit typo3-solr.com or call dkd +49 (0)69 - 247 52 18-0 for more information).
As announced in Version 4.0 the following methods have been removed:
Since EXT:solr 5.0 is only running on TYPO3 7.6 LTS and classmap files for backwards compatibility have been dropped. This also gives am small performance improvement for the current version.
Because we want to use the labels in EXT:solr and EXT:solrfluid, and extbase requires to use one xliff file per language, we take the chance to streamline the language files into one file per language.
Because TypoScriptConfiguration is used quite frequently, we did some performance improvements by optimizing the access to the underling array structure.
You can now add an additionalWhereClause for SOLR_RELATION items.
Example:
plugin.tx_solr.index.queue {
record = 1
record {
table = tx_extension_domain_model_record
fields {
title = title
category_stringM = SOLR_RELATION
category_stringM {
localField = category
multiValue = 1
additionalWhereClause = pid=2
}
}
}
}
See: https://github.com/TYPO3-Solr/ext-solr/issues/426
You can now show the written log messages as output of the debug console in the backend or debug messages in the frontent, when you enable the following setting:
plugin.tx_solr.logging.debugDevlogOutput = 1
NOTE: $GLOBALS['TYPO3_CONF_VARS']['SYS']['devIPmask'] needs to be configured and match.
We've ported the typoscript reference from the forge wiki into an rst format. At the same time we've enhanced the documentation with chapters about:
If you want to contribute and bring the documentation one step further feel free to proof-read or provide additional parts as a pull request.
In the next release (6.0) we will focus on our codesprint (11. to 13. august in frankfurt). Beside that we will focus
on issues in the issue tracker and general improvements of the codebase and documentation.
Like always this release would not have been possible without the help from our
awesome community. These are the contributors for this release.
(patches, comments, bug reports, review, ... in alphabetical order)
Also a big thanks to our partners that have joined the EB2016 program:
Thanks to everyone who helped in creating this release!
There are many ways to get involved with Apache Solr for TYPO3:
Published by timohund over 8 years ago
This is a bugfix and maintenance release that contains the following fixes:
Like always this release would not have been possible without the help from our
awesome community. These are the contributors for this release.
(patches, comments, bug reports, review, ... in alphabetical order)
Published by timohund over 8 years ago
We're happy to announce the release of Apache Solr for TYPO3 (EXT:solr)
version 4.0.0. With this release we now support TYPO3 CMS versions 7.6LTS
together with Apache Solr version 4.10.
At the same time we also started making the extension work with TYPO3 8.
To stay up-to-date with the TYPO3 core we decided to drop the support of 6.2 LTS
and support 7.6 LTS only. We also made sure that EXT:solr works
with TYPO3 8.0, this will be an ongoing effort until the LTS release of TYPO3 8.
Until then you may experience occasional bugs as we try to keep up with the
core's development.
During the preparation for the upcoming fluid integration we used the chance to
refactor the frontend part of the extension.
What we did here:
TypoScript configuration is used in many places and accessing it with an object
has some advantages:
isset()
/is_array()
/if..else
logic to read and evaluateDuring the preparation of the current release we implemented this
TypoScriptConfiguration object and replaced all parts in the scope of EXT:solr
Migration:
For backwards compatibility the TypoScriptConfiguration object can be used with
array access operators. If you want to read the configuration you can use the
speaking method name for a configuration path or use the methods
getValueByPath()
or getObjectByPath()
.
During the development we made sure that the code of EXT:solr works with PHP 7.0.
Beside the major changes that have been mentioned before, there are a few minor
changes that might be interesting:
You can now use a new TypoScript setting plugin.tx_solr.index.queue.pages.excludeContentByClass
.
e.g.
plugin.tx_solr.index.queue.pages.excludeContentByClass = removeme
Since solrconfig.xml contained a default value for the mm
(MinimumMatch) and
q.op
parameters and it does not make sense to have both at the same time for
eDismax we removed the q.op
parameter from the default configuration to avoid
confusion.
To avoid switching from the backend module to scheduler and back, we added a
button to do a simple indexing run with just 10 documents.
The SOLR_RELATION
object is now taking an overlay into account when relations
are resolved. This allows you to resolve different relations from a translated
entity for an original entity.
The goal of the search is to keep the visitor on the site and provide the best
results for them with the smallest possible interaction.
There was a feature in EXT:solr (searchUsingSpellCheckerSuggestion) that could
be used to automatically trigger a search using the spellchecker's suggestions
when a search with the original term did not retrieve any results. To simplify
the integration we added the needed markup to the default templates so that you
now just need to enable the feature and use it with the shipped default template.
How to use it?
plugin.tx_solr.search.spellchecking.searchUsingSpellCheckerSuggestion = 1
Use the shipped default templates or add the following snippet to your template (between
<!-- ###SOLR_SEARCH_NO_RESULTS### begin -->
and <!-- ###SOLR_SEARCH_NO_RESULTS### end -->
:
<div class="tx-solr-search-suggestion_results">
<!-- ###IF:###NO_RESULTS.HAS_SUGGESTION_RESULTS###|==|1### begin -->
###NO_RESULTS.SUGGESTION_RESULTS###
<!-- ###IF:###NO_RESULTS.HAS_SUGGESTION_RESULTS###|==|1### end -->
</div>
For spellchecking we had no dynamic field type before. Since 4.0.0 we now ship
the dynamic field types *_textSpellS
and *_textSpellM
to allow you to
easily define fields for spell checking.
To keep the styling of the backend up-to-date and use the new JavaScript modals
we updated the backend module to use these new features.
Highlighting can be slow for large text documents - f.e. created when
indexing pdf documents - and slow down search queries. To help prevent this
issue we added support for Solr's FastVectorHighlighter
The FastVectorHighlighter will now be used by default if fragmentSize
is set
to at least 18 (this is required by the FastVectorHighlighter to work).
Configuration is retrieved and evaluated many times during indexing. Thus it
made sense to use the TYPO3 caching framework to cache these expensive
operations and improve their performance.
Translation files have been migrated to the xliff format. The EXT:solr team
would be happy if you contribute missing translations.
The Record Monitor is now able to evaluate the "hidden" flag on subpages when
the extendToSubpages
page property is set or unset.
You can now register for a new signal "afterIndexItems".
Beside the improvements mentioned before many smaller bugs have been fixed in
this release as well.
This version of EXT:solr is based on the same Apache Solr Version (4.10) as the
last version so an update for the Solr server itself is not needed.
However, a few modifications to the schema and Solr configuration where made.
Please make sure to use the latest solrconfig.xml and schema versions.
You can always check TYPO3's system status report to see whether your setup
meets the requirements and is up-to-date to be used with EXT:solr.
The following functions and methods have been marked as deprecated and will be
removed in version 5.0
The next release (5.0) will focus on the needed refactorings to allow implementing
fluid template rendering. At the same time we will ship the first version of our
new extension "solrfluid" to our EAP partners.
Like always this release would not have been possible without the help from our
awesome community. These are the contributors for this release.
(patches, comments, bug reports, review, ... in alphabetical order)
Thanks to everyone who helped in creating this release!
There are many ways to get involved with Apache Solr for TYPO3:
Published by timohund over 8 years ago
This release is a maintenance release for the 3.1 branch. Since EXT:solr 4.0.0 will require TYPO3 7.6 LTS or higher this release provides bugfixes for users on TYPO3 6.2 LTS
https://github.com/TYPO3-Solr/ext-solr/issues/128
https://github.com/TYPO3-Solr/ext-solr/issues/241
https://github.com/TYPO3-Solr/ext-solr/issues/264
https://github.com/TYPO3-Solr/ext-solr/issues/276
https://github.com/TYPO3-Solr/ext-solr/issues/231
https://github.com/TYPO3-Solr/ext-solr/issues/307
Thx to the following contributors for patches, reviews and backports:
Published by irnnr almost 9 years ago
Fixes an issue related to composer when installing an extension that depends on EXT:solr
Published by irnnr almost 9 years ago
We're happy to announce the release of Apache Solr for TYPO3 (EXT:solr) version 3.1.0. With this release we now support TYPO3 CMS versions 6.2LTS and 7.6LTS together with Apache Solr version 4.10.
We added support for TYPO3 CMS 7.6LTS while still supporting version 6.2LTS to allow an easy upgrade. Support for TYPO3 version 4.5LTS has been dropped with this release. The next release will likely require at least TYPO3 7.6LTS.
On the code side we now use PSR-2 coding style like in TYPO3 7.6LTS. This way it is easy for contributors and developers to stick with a common coding style. For contributions PSR-2 style is also enforced through automatic checks now.
Overall we resolved over 40 issues and added almost 700 commits.
The extension comes with an installer that installs Apache Tomcat 8.0.29 and Apache Solr 4.10.4. Please make sure to update to Apache Solr 4.10.4 and the new Solr schema and configuration provided by the extension to make sure the extension works as intended.
With EXT:solr 3.0 we introduced an all new backend module to manage the Index Queue, check index data, and manage synonyms. With this release we're adding a new sub module to allow editing stopwords.
Stopwords are usually used to discard common words when indexing. This can increase relevance of documents.
Our complete codebase is now namespaced and uses root namespace "\ApacheSolrForTypo3\Solr" for all classes. We also added a class map to provide backwards compatibility. Nevertheless we suggest updating your extensions to use the namespaced class names if you use them.
For a long time we already had a small unit test suite. However, it was not kept up-to-date, tests weren't executed automatically and generally in a state that left room for improvement. Starting with this release we will focus on increasing test coverage and keeping them up-to-date.
The unit test suite is now executed automatically for each Pull Request through Travis CI (https://travis-ci.org/TYPO3-Solr/ext-solr). Currently we run tests against TYPO3 version 6.2 LTS, 7.6 LTS, and dev-master with the same PHP versions as TYPO3 CMS, PHP 5.3-5.6.
Additionally Travis CI also checks for PSR-2 compliance and will fail a PR in case the code does not fit that coding standard.
In case a Pull Request is rejected by Travis CI you can follow the link to the build and inspect its output to see what needs to be fixed.
The data ist now esapced right after the retrieval from Solr. In rare cases when you need to store HTML in Solr documents you can use the following configuration to mark these fields as trusted fields and skip the escaping:
plugin.tx_solr.search.trustedFields = my_first_html_field,my_second_html_field
Synonyms are now handled at query and index time. The advantage is, that when editing synonyms from the backend module those changes become effective immediatly.
As we were preparing to be compatible with TYPO3 7.6LTS we found that there is no compatible version of EXT:pagebrowse. We used that extension to provide the page browser in results listings. To become compatible with TYPO3 7.6LTS we now have integrated the page browser into EXT:solr itself. So if you are not using EXT:pagebrowse for anything else you can safely remove the extension from your installation.
To install EXT:solr version 3.1.0 you will need either TYPO3 6.2LTS or TYPO3 7.6LTS. On the Solr side we require Apache Solr 4.10.4, which requires Java 7.
When updating an existing installation to EXT:solr 3.1.0 please make sure to update to Apache Solr 4.10.4 including the new schema and configuration files. The extension comes with a script that will install a working setup for you automatically. The script can be found in EXT:solr/Resources/Install/.
In rare cases you may have to re-initialize the Solr connections. You can do so from TYPO3's clear cache menu in the top toolbar.
Because of the changes made to the schema you will need to re-index your site.
Make sure to check the system status report for any warnings or errors.
During the migration to namespace we also had to change the TypoScript paths:
tx_solr_pi_results => tx_solr_PiResults_Results
tx_solr_pi_search => tx_solr_PiSearch_Search
tx_solr_pi_frequentsearches => tx_solr_PiFrequentSearches_FrequentSearches
If you're building custom TypoScript content objects referencing solr plugin configurations you should make to update these as well.
With this release out the door we will focus on updating the add-on extensions like EXT:solrgrouping, EXT:tika, and EXT:solrfal for file indexing, these should be available soon, too.
The next release within the 3.x series will likely require at least TYPO3 7.6LTS and might as well arrive soon as there are no bigger changes planned yet besides dropping support for TYPO3 6.2LTS.
After that we will start working on the long-awaited move to Fluid as the template engine for version 4.0.
Like always this release would not have been possible without the help from our awesome community. These are the contributors for this release.
(patches, comments, bug reports, review, ... in alphabetical order)
Thanks to everyone who helped in creating this release!
Last but not least we would like to introduce Timo Schmidt. Timo recently joined dkd to work on Apache Solr for TYPO3 in support and development. If you've been active in our Slack channel or followed our GitHub commits you may have been in contact with Timo already. We're looking forward to working with Timo.
There are many ways to get involved with Apache Solr for TYPO3:
Published by irnnr about 9 years ago
In this release we are providing 10 bugfixes including one security related fix.
A non-persistent XSS issue is being fixed with this release. It is a low severity issue since the keepExistingParametersForNewSearches configuration option must be explicitly set to 1, it is 0 per default. Thanks to Hendrik Nadler for discovering the issue.
Since this is a patch-level release you can simply update the extension. No further changes are necessary.
Thanks to all the contributors without whom this release would not have been possible. Thank you all! (no particular order)
Alexander Stehlik
Andreas Allacher
Romain Canon
Markus Kobligk
Published by irnnr over 9 years ago
In this release we are delivering about 75 changes including 55 bug fixes. The next release will be version 3.1 and will focus on optimized TYPO3 6.2+ support and drop support for TYPO3 4.5.
We try to avoid making changes to the Solr schema and solrconfig files for patch level release like this. However, this is not always possible. We had to make changes to the schema this time. Please check the status report to see whether you have to update your Solr configuration.
Thanks to all the contributors without whom this release would not have been possible. Thank you all! (no particular order)
Andreas Allacher
Sascha Egerer
Witali Rott
Alexander Stehlik
Andreas Wolf
Hans Höchtl
Jigal van Hemert
Peter Kraume
Sascha Nowak
Stefan Neufeind
Maxime Lafontaine
Steffen Müller
Phuong Doan
Frank Nägler
Steffen Ritter
Heiko Hardt
Stefan Galinski
Marc Bastian Heinrichs
Markus Friedrich