Giter Club home page Giter Club logo

searchstax-cloud-connector-for-sitecore's Introduction

SearchStax Cloud Connector for Sitecore

Introduction

This script is used to connect a Sitcore installation to a SearchStax' Solr instance. It does following:

  • Upload the config files to Solr
  • Create collections in Solr
  • Configure sitecore files

Supported Sitecore Versions

Currently the script only supports following sitecore XP versions and their XConnect:

  • 9.0 Update-2 (9.0.2)
  • 9.1 Update-1 (9.1.1)
  • 9.2 Initial Update (9.2.0)
  • 9.3 Initial Update (9.3.0)
  • 10.0 Initial Update (10.0.0)
  • 10.0 Update-1 (10.0.1)
  • 10.0 Update-2 (10.0.2)
  • 10.0 Update-3 (10.0.3)
  • 10.1 Initial Update (10.1.0)
  • 10.1 Update-1 (10.1.1)
  • 10.1 Update-2 (10.1.2)
  • 10.2 Initial Update (10.2.0)
  • 10.3 Initial Update (10.3.0)

Requirements

  • Powershell v6 or above
  • Powershell-yaml module

Installing

Powershell v6

Powershell v6 can be installed by running following command via Powershell Windows.

iex "& { $(irm https://aka.ms/install-powershell.ps1) } -UseMSI"

Powershell Yaml Module

Install-Module powershell-yaml

In order to run the script, first you have to update the config file

Config File

Config file is located at .\config.yml

It contains following fields:

Name Description Example
accountName Name of the SearchStax account ABCInternational
deploymentUid UID of the SearchStax deployment to connect to ss123456
sitecorePrefix Prefix of the sitecore installation sitecore
pathToWWWRoot Path to wwwroot folder in inetpub, i.e. your %IIS_SITE_HOME% variable C:\inetpub\wwwroot
solrUsername Solr username (Optional)
solrPassword Solr password (Optional)
sitecoreVersion Version of sitecore from the above list 10.1.1
isUniqueConfigs "true" will create a separate config file for each collection, "false" will create only 1 config which will be used by all the collections. (Note: This defaults to true for Sitecore v9.0.2) true/false
configurationMode Select the part of Sitecore being configured - "XP", "XCONNECT" XP|XCONNECT
isSxa "True" will add two additional collections to Solr for SXA support true/false
switchOnRebuild Extended configuration for switch on rebuild.
switchOnRebuild.enableForPlatformIndexes When "true", will add additional collections and aliases for Sitecore Platform Indexes (core, master, web, etc.). true/flase
switchOnRebuild.enableForMarketingIndexes When "true", will add additional collections and aliases for Sitecore Platform Marketing Indexes (content testing, fxm, etc). Generally, this will always be false unless a specific use case calls for this configuration. true/flase
switchOnRebuild.enableForSXA When "true", will add additional collections and aliases for Sitecore SXA Indexes. true/flase
switchOnRebuild.sufix The suffix of the rebuild collection name. Default: "_rebuild" string value
switchOnRebuild.MainAlias The suffix of the Main Alias name. Default: "_MainAlias" string value
switchOnRebuild.rebuildAlias The suffix of the Rebuild Alias name. Default: "_RebuildAlias" string value
switchOnRebuild.sitecorePrefix The prefix of the Rebuild collctions and Aliases. Default: "sitecore" string value
customIndexes Allows of creation of custom indexes. Can be repeated for multiple custom indexes
customIndexes.core Collection name of the custom index string value
customIndexes.isSwitchOnRebuild When "true" will create additional rebuild collection and aliases true/false

Instructions

  1. Configure the config.yml file.
  2. Start Powershell v6 as Administrator.
  3. Change the execution policy to skip checking.
Set-ExecutionPolicy -Scope Process -ExecutionPolicy Bypass
  1. Execute following command
.\sitecore-searchstax-connector.ps1
  1. Go to your sitecore page > Control Panel > Populate Solr Managed Schema > Select All > Populate
  2. On the same page, Indexing Manager > Select All > Rebuild

Best Practice/Recommendation:

Use your CI/CD deployment to set Configuration and ConnectionString values using secure environment unique tokens/variables.

Instructions for PaaS/Scaled/Custom Environments

Every Sitecore Web Instance (CD/CM) needs these updates:

  1. Web.config: <AppSettings><add key="search:define" value="Solr"/> - set to Solr
  2. App_Config/ConnectionStrings.config: <add name="solr.search" connectionString=

Every XConnect Instance needs these updates:

  1. App_Config/ConnectionStrings.config: <add name="solrCore" connectionString=
  2. App_Data\jobs\continuous\IndexWorker\App_Config\ConnectionStrings.config: <add name="solrCore" connectionString=
  3. Use the SearchStax API and upload the Schema - App_Data\solrcommands\schema.json to both XDB Collections. - reference: https://lucene.apache.org/solr/guide/8_6/schema-api.html

How can I get help with SearchStax Cloud Connector for Sitecore?

You can use GitHub to submit bug reports or feature requests for Searchstax-Cloud-Connector-for-Sitecore. Please do not submit usage questions via GitHub.

FAQ

Recommended Performance Optimization Settings by Sitecore

Sitecore recommends updating some settings for performance optimization when working with Solr. You can find more details here: Performance optimizations when using Solr

Sitecore SXA

This plugin currently does not support Sitecore SXA configuration.

IP Filtering

If you have enabled IP filtering on your Solr instance, then make sure that you add the IP/CIDR block of your network or machine to the IP Filtering page. For more instructions on how to set up IP filtering, please follow our guide here - How To Set-up IP Filtering

Sitecore v9.0 Update-2+

The plugin will automatically default to creating a separate config directory for every collection when being used to setup Sitecore v9.0.2.

Have additional questions?

Check SearchStax Help Center

searchstax-cloud-connector-for-sitecore's People

Contributors

cheeto-bandito avatar dilascio avatar dkapoor avatar hugolima88 avatar hugosantos-konabos avatar karanjeetsingh avatar kpatil-hi avatar maggon avatar radical-dave avatar sitecoredave avatar vapok avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

searchstax-cloud-connector-for-sitecore's Issues

Aliases are not created for SwitchOnRebuildSolrCloudSearchIndex

As part of #2 , I have discovered that the aliases are required when using SwitchOnRebuildSolrCloudSearchIndex. The preference is to set the setting ContentSearch.Solr.EnforceAliasCreation to false in Sitecore, so that everything that is required from Solr is created using this plugin.

Improve ReadMe files ease of use

Finding account names, deployment IDs and instructing users to user their SearchStax passwords (Or 2FA) rather than deployment passwords will help developers become more self sufficient with the connector

Remove Commerce Support

This is an un-used feature which requires additional maintainability for the connector

Criteria:
When the user specifies Configuration Mode, COMMERCE should not be an accepted option.

XDB collections and aliases don't contain prefix

The aliases and collections used for the XDB rebuild collections are not prefixed by default. This means that you cannot have multiple sites running on the same instance without alterations.

Cloud Connector 2.0

Requirements:

  • Powershell interface
  • Support for SearchStax 2FA/MFA authentication if it's enabled for the SearchStax user account
  • All configuration files created include 300000 maxTime for autoCommit and 120000 maxTime for autoSoftCommit for all Solr config.xml files
  • Easier Maintainability
    • Keep xConnect schema file in the same zip directory as the same way other Solr configs are. (#48)
    • Remove On-Premise configuration changes to Sitecore and assume it is Azure PaaS (#47)
    • Remove Sitecore Commerce Support (#46)

User should be able to specify:

  • Sitecore version (This exists already)

    • Compatible version of Sitecore configs is uploaded to target deployment
  • Configuration Mode (XM, XP, XCONNECT) (This exists already)

    • Default Sitecore indexes are created for the configuration mode selected
  • If SXA is required (This exists already)

    • If yes, then prefix_sxa_web_index and prefix_sxa_master_index are created
  • If XConnect is required (This exists already)

    • If yes, Rebuild index for xDB and aliases are configured for SwitchOnRebuild support
  • If SwitchOnRebuild is enabled for Sitecore indexes (New)

    • If yes, then Rebuild indexes and corresponding mainAliases and rebuildAliases are created for each Index by default using the values below provided by the user
    • The rebuildSuffix (README File can guide them on whether "_" preceding rebuildSuffix is provided by the user or added automatically)
    • The mainAliasSuffix
    • The rebuildAliasSuffix
  • Custom Index name (New)

    • User can pass an list of custom index names
    • User can also specify for each custom index, if it will have SwitchOnRebuild
      - If yes, then use the values provided in the SwitchOnRebuild section above for rebuildSuffix, mainAliasSuffix and rebuildAliasSuffix

Add Private Network Setting for Sitecore 10.0 and above

As per https://doc.sitecore.com/en/developers/100/platform-administration-and-architecture/walkthrough--setting-up-solrcloud.html,

"If you only access nodes of SolrCloud within a private network, set the ContentSearch.IndexingManager.DisplayShortStatistic setting to true. This eliminates direct requests to the nodes used to display additional index statistics in the Indexing Manager Wizard."

Enhance the script to set this to true for SItecore versions 10.0 and above

XDB Alias creation fails for XP1

ZD Tkt - #5119

Creating XDB Aliases ...
xdb_internal
Invoke-WebRequest: C:\pkgs\searchstax-sitecore-plugin-1.15\searchstax-sitecore-plugin-1.15\src\searchstax-sitecore-xconnect.ps1:58
Line |
58 | Invoke-WebRequest -Uri $url
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~
| { "responseHeader":{ "status":400, "QTime":28}, "Operation createalias caused
| exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Can't create
| collection alias for collections='xdb_internal', 'xdb_internal' is not an existing collection or
| alias", "exception":{ "msg":"Can't create collection alias for collections='xdb_internal',
| 'xdb_internal' is not an existing collection or alias", "rspCode":400}, "error":{
| "metadata":[ "error-class","org.apache.solr.common.SolrException",
| "root-error-class","org.apache.solr.common.SolrException"], "msg":"Can't create collection alias
| for collections='xdb_internal', 'xdb_internal' is not an existing collection or alias",
| "code":400}}

xdb_rebuild_internal
Invoke-WebRequest: C:\pkgs\searchstax-sitecore-plugin-1.15\searchstax-sitecore-plugin-1.15\src\searchstax-sitecore-xconnect.ps1:58
Line |
58 | Invoke-WebRequest -Uri $url
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~
| { "responseHeader":{ "status":400, "QTime":10}, "Operation createalias caused
| exception:":"org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Can't create
| collection alias for collections='xdb_rebuild_internal', 'xdb_rebuild_internal' is not an existing
| collection or alias", "exception":{ "msg":"Can't create collection alias for
| collections='xdb_rebuild_internal', 'xdb_rebuild_internal' is not an existing collection or alias",
| "rspCode":400}, "error":{ "metadata":[
| "error-class","org.apache.solr.common.SolrException",
| "root-error-class","org.apache.solr.common.SolrException"], "msg":"Can't create collection alias
| for collections='xdb_rebuild_internal', 'xdb_rebuild_internal' is not an existing collection or
| alias", "code":400}}

Updating XConnect ConnectionStrings in 'D:\sitecore[sitecore.cm](http://sitecore.cm/)\sitecore.xconnect\App_Config\ConnectionStrings.config' file
Write-Error: C:\pkgs\searchstax-sitecore-plugin-1.15\searchstax-sitecore-plugin-1.15\src\searchstax-sitecore-xconnect.ps1:74
Line |
74 | Update-XML $path $xpath $attributeKey $attributeValue
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Could not find D:\sitecore[sitecore.cm](http://sitecore.cm/)\sitecore.xconnect\App_Config\ConnectionStrings.config File

Updating XConnect ConnectionStrings in 'D:\sitecore[sitecore.cm](http://sitecore.cm/)\sitecore.xconnect\App_Data\jobs\continuous\IndexWorker\App_Config\ConnectionStrings.config' file
Write-Error: C:\pkgs\searchstax-sitecore-plugin-1.15\searchstax-sitecore-plugin-1.15\src\searchstax-sitecore-xconnect.ps1:74
Line |
74 | Update-XML $path $xpath $attributeKey $attributeValue
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Could not find
| D:\sitecore[sitecore.cm](http://sitecore.cm/)\sitecore.xconnect\App_Data\jobs\continuous\IndexWorker\App_Config\ConnectionStrings.config File

Get-Content: C:\pkgs\searchstax-sitecore-plugin-1.15\searchstax-sitecore-plugin-1.15\src\searchstax-sitecore-xconnect.ps1:86
Line |
86 | $json = Get-Content -Raw -Path $path
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Cannot find path 'D:\sitecore[sitecore.cm](http://sitecore.cm/)\sitecore.xconnect\App_Data\solrcommands\schema.json' because
| it does not exist.

https://ss856956-knf6sz36-us-west-2-aws.searchstax.com/solr/
Updating XDB Schema ...
xdb_internal
Invoke-RestMethod: C:\pkgs\searchstax-sitecore-plugin-1.15\searchstax-sitecore-plugin-1.15\src\searchstax-sitecore-xconnect.ps1:102
Line |
102 | Invoke-RestMethod -Uri $url -ContentType 'application/jso …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Error 404 Not Found HTTP ERROR 404 Problem accessing /solr/xdb_internal/schema. Reason: Not
| Found

xdb_rebuild_internal
Invoke-RestMethod: C:\pkgs\searchstax-sitecore-plugin-1.15\searchstax-sitecore-plugin-1.15\src\searchstax-sitecore-xconnect.ps1:102
Line |
102 | Invoke-RestMethod -Uri $url -ContentType 'application/jso …
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
| Error 404 Not Found HTTP ERROR 404 Problem accessing /solr/xdb_rebuild_internal/schema. Reason:
| Not Found

Restarting IIS
NOTE: If you have UAC enabled, then this step might fail with 'Access Denied' error.
Please either disable UAC, or restart IIS manually if the error occurs.

Provide Commit Settings based on SearchStax recommendations

It could either be

Two options being presented in the Yaml module and pros/cons listed in the plugin documentation
Option a) Use standard solr configs (-1 autoSoftCommit, 15000ms autoCommit)
Option b) Use SearchStax recommended configs (120000ms autoSoftCommit, 300000ms autoCommit)

Or simply using SearchStax recommended configs by default. This would provide a more sensible onboarding experience and reduce post-onboarding support tickets for performance issues and backup failures.

SearchStax recommendations are here for reference - https://www.searchstax.com/blog/5-ways-to-optimize-your-solr-search-performance-for-sitecore/

Remove On-Premise Sitecore config changes

  • Remove PathtoRoot (pathToWWWRoot) field from YAML file since this change will not be made in Sitecore.
  • Remove isAzurePaaS field from YAML file and always assume TRUE for this

Currently this can be run in On-Premise or Remotely modes. New implementation will only be for Remotely mode.

Criteria:
If a user runs this script with on-premise Sitecore, the changes to Sitecore configuration files should not happen.

Plugin assumes too much about file paths

This plugin assumes too much about how a Sitecore site might be set up. For instance, if my prefix is mylittlepony and root is C:\inetpub\wwwroot it assumes the connection strings are located in C:\inetpub\wwwroot\mylittlepony.xconnect \App_Config\ConnectionStrings.config. This is quite often not the case as many of the Sitecore SIF installation steps are customizable.

Plugin has error when trying to overwrite existing configset

Prospect ran Sitecore Plugin, got an error saying that it could not overwrite an existing configset. (Our Zookeeper API "create" will not overwrite a configset.) Prospect was running the plugin in XCONNECT mode, and an existing xDB configset broke it.

Suggest either (1) fix the Zookeeper API "create" method to overwrite like zkcli does, or
(2) fix Plugin to delete existing configsets before writing new ones. BDC

Feature Request

I propose the scripts to be re-organised, so that it no longer needs an external yaml file, rather all the parameters can be passed in via script arguments instead. This will allow a user to run this on their developer PC, as well as on a CI / CD tool like Azure DevOps. Here is how I will break it down.

  1. Do not zip the Solr config in the repo, rather fully expand them so that all users can see all the content.
  2. Allow independent naming of Solr Configuration that will be uploaded, and the Index Cores collection prefixes.
  3. Add a Powershell module to zip the Solr configurations temporarily when Posting to the Solr API
  4. Add more index collection support for XDB, SXA and Sitecore Commerce.
  5. Provide documentation inside Powershell scripts, and Readme file, so that users have a good idea on how to use the scripts and add their own custom index collections.
  6. Add script to add and remove IP filtering

I will progressively work through this.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.