Skip to content

Instantly share code, notes, and snippets.

@StephenBrown2

StephenBrown2/install-solr4-and-create-instances.md

Last active December 31, 2015 18:59
Show Gist options
  • Save StephenBrown2/8030743 to your computer and use it in GitHub Desktop.
Save StephenBrown2/8030743 to your computer and use it in GitHub Desktop.
Download ZIP
This allows one to download and configure solr 4 (4.6 at the time of this writing) on a CentOS machine and create multiple cores via a scripted, repeatable manner. Supported module configurations are from apachesolr and search_api_solr
Raw
install-solr4-and-create-instances.md

Source

How to automatically install apache solr multicore locally for drupal

This article will give you the tools to setup fully automated an apache solr multicore locally for drupal.

The advantages: When you have new projects you will only have to create a new apache solr core based on previous configuration. This will save you valuable time.

How does it work? Whatever we have done manually we can script using bash. This is exactly what we have done to install the apache solr multicore locally.

The script depends on drush and curl. The script will detect these dependencies and will try to download them.

Here is the script to install apachesolr and setup the multicore. We will use a second script to setup our cores later.

Install apache solr as a multicore

#!/bin/bash

#Validate if the executing user is root
if [ "$(id -u)" != "0" ]; then
   echo "This script must be run as root" 1>&2
   exit 1
fi

# ********* #
# Functions #
# ********* #

#Install drush
function install_drush () {
  
  version=$1
  
  # Set tag to latest of each version
  drush5_tag='5.10.0'
  drush6_tag='6.2.0'
  drush7_tag='master' # drush 7 doesn't have a stable release tag yet - 12/01/2013

  case "$version" in
    5)  
      if [[ `php -r "echo version_compare(PHP_VERSION,'5.2.0','>=') ? '1' : '0';"` == '1' ]]; 
      then
        drush_tag=$drush5_tag
      else
        echo -e "\e[00;31m ERROR:You cannot install drush 5 with PHP less than 5.2.0. Please update PHP.  \e[00m"
        return 1
      fi  
    ;;  
    6)  
      if [[ `php -r "echo version_compare(PHP_VERSION,'5.3.0','>=') ? '1' : '0';"` == '1' ]]; 
      then
        drush_tag=$drush6_tag
      else
        echo -e "\e[00;31m ERROR:You cannot install drush 6 with PHP less than 5.3.0. Please update PHP.  \e[00m"
        return 1
      fi  
    ;;  
    7)  
      if [[ `php -r "echo version_compare(PHP_VERSION,'5.3.0','>=') ? '1' : '0';"` == '1' ]]; 
      then
        drush_tag=$drush7_tag
      else
        echo -e "\e[00;31m ERROR:You cannot install drush 7 with PHP less than 5.3.0. Please update PHP.  \e[00m"
        return 1
      fi  
    ;;  
  esac
  
  # Set the file to be downloaded
  drush_zip="${drush_tag}.zip"
  
  # Set the URL for the file to be downloaded
  drush_url="http://github.com/drush-ops/drush/archive/${drush_zip}"
  
  # Set the directory for the files to be extracted to
  drush_dir="/opt/drush${version}"
  
  # Download the files to /tmp
  wget -nc $drush_url -O /tmp/drush-${drush_zip}

  # Extract the files to their proper place
  unzip /tmp/drush-${drush_zip} -d ${drush_dir}/
  
  # Link them so they are usable
  drush_bin="/usr/local/bin/drush${version}"
  ln -s ${drush_dir}/drush $drush_bin
  
  # Run drush once to make sure all libraries are downloaded
  $drush_bin > /dev/null &2>1
  
  echo -e "\e[00;33m NOTICE:Drush $version installed \e[00m"
  return 0
}

function set_primary_drush () {
  version=$1
  if [ -f /usr/local/bin/drush${version} ]
  then
    ln -fs /usr/local/bin/drush${version} /usr/local/bin/drush
  else
    echo "/usr/local/bin/drush${version} does not exist! Have you installed drush ${version} yet?"
    exit
  fi
}

#Test the connection to the solr installation
function solr_install_test_connection {
  response=$(curl --write-out %{http_code} --silent --output /dev/null http://127.0.0.1:8983/solr/)
  if [ $response != 200 ]; then
    echo -e "\e[00;31m ERROR:Solr installation failed manual intervention needed. Solr reponds at http://127.0.0.1:8983/solr/ with http code: $reponse  \e[00m"
  else
    echo -e "\e[00;32m NOTICE:Solr reponds at http://127.0.0.1:8983/solr/ with http code:$response  \e[00m"
  fi
}

# ****** #
# Script #
# ****** #

#Install the needed sources if needed: drush, curl and java
DRUSH=`command -v drush`
if [ -z "$DRUSH" ]; then
  install_drush 5
  install_drush 6
  install_drush 7
  set_primary_drush 6
else
  echo -e "\e[00;33m WARNING:Drush already installed at '$DRUSH' \e[00m"
fi

CURL=`command -v curl`
if [ -z "$CURL" ]; then
  yum install curl
  echo -e "\e[00;33m NOTICE:Curl installed \e[00m"
else
  echo -e "\e[00;33m WARNING:Curl already installed at '$CURL' \e[00m"
fi

JAVA=`command -v java`
if [ -z "$JAVA" ]; then
  yum install java-1.6.0-openjdk
  echo -e "\e[00;32m NOTICE:Java installed \e[00m"
else
  echo -e "\e[00;33m WARNING:Java already installed at '$JAVA' \e[00m"
fi

SOLR_VERSION='4.6.0'

# Download Solr
SOURCE=solr-${SOLR_VERSION}.tgz
FOLDER=solr-${SOLR_VERSION}
FILE=/opt/$SOURCE
if [ ! -e $FILE ]; then
  mkdir -p /opt
  cd /opt
  wget -nc http://apache.tradebit.com/pub/lucene/solr/${SOLR_VERSION}/${SOURCE}
  echo -e "\e[00;32m NOTICE:Sources downloaded  \e[00m"
else
  echo -e "\e[00;33m WARNING:Source already downloaded. skipping \e[00m"
fi

# Extract Solr
SOLR_HOME=/opt/$FOLDER
if [ ! -d $SOLR_HOME ]; then
  echo -e "\e[00;32m NOTICE:Extracting sources  \e[00m"
  cd /opt
  tar xzf solr-4.6.0.tgz
  echo -e "\e[00;32m NOTICE:Apachesolr dir created  \e[00m"
else
  echo -e "\e[00;33m WARNING:Apachesolr dir already exists, skipping \e[00m"
fi

# Create the solr multicore structure, and base core to copy later
SOLR_HOME_DRUPAL=${SOLR_HOME}/drupal
if [ ! -d $SOLR_HOME_DRUPAL ]; then
  mkdir -p ${SOLR_HOME_DRUPAL}/cores/core0/conf

  # Copy the necessary files and folders
  cp -rf ${SOLR_HOME}/example/{contexts,etc,lib,resources,start.jar,webapps} ${SOLR_HOME_DRUPAL}/
  cp -rf ${SOLR_HOME}/example/solr/collection1/conf/* ${SOLR_HOME_DRUPAL}/cores/core0/conf
  echo -e "\e[00;32m NOTICE:Drupalsolr dir created  \e[00m"
else
  echo -e "\e[00;33m WARNING:Drupalsolr dir already exists, skipping \e[00m"
fi

# Create the discovery solr.xml file
FILE=${SOLR_HOME_DRUPAL}/cores/solr.xml
if [ -e $FILE ]; then
  NEW=`grep core0 ${FILE} 2>dev/null 1>&2`
  if [ -z "$NEW" ]; then
    rm ${FILE}
  fi
fi

if [ ! -e $FILE ]; then
  echo '<solr></solr>' > ${FILE}
  echo -e "\e[00;32m NOTICE:Solr.xml multicore setup complete  \e[00m"
else
  echo -e "\e[00;33m WARNING:Solr.xml already installed, skipping \e[00m"
fi


## APACHESOLR Drupal module setup
cd ${SOLR_HOME_DRUPAL}
DIR=${SOLR_HOME_DRUPAL}/apachesolr

if [ ! -d $DIR ]; then
  # Download drupal module so we can use the correct configuration
  drush dl apachesolr
  cd $DIR

  # Create the core conf directory
  apachesolr_version=$(grep version ${DIR}/apachesolr.info | sed -e 's#version = "\([^"]\+\)"#\1#g')
  apachesolr_solrdir='base_apachesolr-'${apachesolr_version}'_conf'
  if [ ! -d ${SOLR_HOME_DRUPAL}/cores/${apachesolr_solrdir}/conf ]; then
    mkdir -p ${SOLR_HOME_DRUPAL}/cores/${apachesolr_solrdir}/conf

    # We'll need to patch the solrconfig.xml first because of changes in solr 4.5+
    wget -nc http://drupal.org/files/2107417-2.patch
    patch -p1 < 2107417-2.patch

    # And copy the necessary stock configuration files
    cp -r ${SOLR_HOME}/example/solr/collection1/conf/{elevate.xml,mapping-ISOLatin1Accent.txt,stopwords.txt,synonyms.txt} ${SOLR_HOME_DRUPAL}/cores/${apachesolr_solrdir}/conf/

    # Then copy the module's config files over
    cp -rfb ${DIR}/solr-conf/solr-4.x/* ${SOLR_HOME_DRUPAL}/cores/${apachesolr_solrdir}/conf/

    # Rename the directory with the version, for posterity
    mv ${DIR} ${DIR}-${apachesolr_version}
  
    echo -e "\e[00;32m NOTICE:Apachesolr ${apachesolr_version} settings installed  \e[00m"
  else
    echo -e "\e[00;33m WARNING:Apachesolr settings already there, skipping \e[00m"
  fi
else
  rm -rf $DIR
fi

# Do the same for search_api_solr
## SEARCH_API_SOLR Drupal module setup
cd ${SOLR_HOME_DRUPAL}
DIR=${SOLR_HOME_DRUPAL}/search_api_solr

if [ ! -d $DIR ]; then
  # Download drupal module so we can use the correct configuration
  drush dl search_api_solr
  cd $DIR

  # Create the core conf directory
  search_api_solr_version=$(grep version ${DIR}/search_api_solr.info | sed -e 's#version = "\([^"]\+\)"#\1#g')
  search_api_solr_solrdir='base_search_api_solr-'${search_api_solr_version}'_conf'
  if [ ! -d ${SOLR_HOME_DRUPAL}/cores/${search_api_solr_solrdir}/conf ]; then
    mkdir -p ${SOLR_HOME_DRUPAL}/cores/${search_api_solr_solrdir}/conf

    # We'll need to patch the solrconfig.xml first because of changes in solr 4.5+
    wget -nc http://drupal.org/files/issues/2122391-2.patch
    patch -p1 < 2122391-2.patch

    # Then copy the module's config files over
    cp -rfb ${DIR}/solr-conf/4.x/* ${SOLR_HOME_DRUPAL}/cores/${search_api_solr_solrdir}/conf/

    # Rename the directory with the version, for posterity
    mv ${DIR} ${DIR}-${search_api_solr_version}
  
    echo -e "\e[00;32m NOTICE:Search API Solr ${search_api_solr_version} settings installed  \e[00m"
  else
    echo -e "\e[00;33m WARNING:Search API Solr settings already there, skipping \e[00m"
  fi
else
  rm -rf $DIR
fi

#Create the start|stop|restart script /etc/init.d/solr4
INIT_FILE=/etc/init.d/solr4
if [ ! -e $INIT_FILE ]; then
echo '#!/bin/sh -e
# Starts, stops, and restarts Apache Solr.
#
# chkconfig: 2345 64 36
# description: Starts and stops Apache Solr

SOLR_DIR="'${SOLR_HOME_DRUPAL}'"
JAVA_BIN="/usr/bin/java"
JAVA_HEAP="-Xmx1024m"
JETTY_STOP_OPTIONS="-DSTOP.PORT=8079 -DSTOP.KEY=stopkey"
JETTY_PORT="-Djetty.port=8983"
SOLR_HOME="-Dsolr.solr.home=cores"
JAVA_OPTIONS="$JAVA_HEAP $JETTY_STOP_OPTIONS $JETTY_PORT $SOLR_HOME"
LOG_FILE="/var/log/solr/solr.log"
COMMAND="$JAVA_BIN $JAVA_OPTIONS -jar start.jar"

case $1 in
    start)
        echo "Starting Solr '${SOLR_VERSION}'"
        cd $SOLR_DIR
        $COMMAND 1>&2 2> $LOG_FILE &
        ;;
    stop)
        echo "Stopping Solr '${SOLR_VERSION}'"
        cd $SOLR_DIR
        $COMMAND --stop
        ;;
    restart)
        $0 stop
        sleep 1
        $0 start
        ;;
    *)
        echo "Usage: $0 {start|stop|restart}" >&2
        exit 1
        ;;
esac' > ${INIT_FILE}

  #Set permissions
  chmod a+rx ${INIT_FILE}
  chkconfig --add $(basename ${INIT_FILE})
  ${INIT_FILE} start
  sleep 5

  #verify the connection
  solr_install_test_connection

  echo -e "\e[00;32m NOTICE:Solr installation completed  \e[00m"
else
  echo -e "\e[00;33m WARNING:Solr already installed, skipping \e[00m"
fi

#Exit output
echo -e '\e[00;32m Use the solr-instance script to install cores for your project. Execute sudo "./create-solr-instance.sh". \e[00m'

How to use it?

Create a file called solr-install.sh and paste the above code in it. Then execute as root:

If everything goes as planned you will see this:

Install independent apache solr multicore

Create solr multicore instances with this script

#!/bin/bash

#Validate if executing user is root
if [ "$(id -u)" != "0" ]; then
   echo "This script must be run as root" 1>&2
   exit 1
fi

PROJECT=$1
SOLR_MODULE=${2:-apachesolr}
SOLR_MODULE_VERSION=${3:-'7.x-1.6'}
SOLR_VERSION='4.6.0'
FOLDER=solr-${SOLR_VERSION}
SOLR_HOME=/opt/$FOLDER
SOLR_HOME_DRUPAL=${SOLR_HOME}/drupal

if [[ -z $PROJECT ]]; then
  echo -e "\e[00;31m please provide a projectname as 1st variable \e[00m"
  echo -e "\e[00;31m please provide an optional 2nd var to indicate to use the apachesolr or search_api_solr drupal module configuration, and the drupal module version as 3rd variable \e[00m"
  echo ' '
  exit
else
  echo ' '
  echo '*************Solr core creation started**************'
  echo ' '
fi

# ******** #
# Functions #
# ******** #

function apachesolr_config {
## APACHESOLR Drupal module setup
cd ${SOLR_HOME_DRUPAL}
DIR=${SOLR_HOME_DRUPAL}/apachesolr

if [ ! -d $DIR ]; then
  # Download drupal module so we can use the correct configuration
  drush dl apachesolr-${SOLR_MODULE_VERSION}
  mv apachesolr apachesolr-${SOLR_MODULE_VERSION}
  DIR=${DIR}-${SOLR_MODULE_VERSION}
  cd $DIR

  # Create the core conf directory
  apachesolr_version=$(grep version ${DIR}/apachesolr.info | sed -e 's#version = "\([^"]\+\)"#\1#g')
  apachesolr_solrdir='base_apachesolr-'${apachesolr_version}'_conf'
  if [ ! -d ${SOLR_HOME_DRUPAL}/cores/${apachesolr_solrdir}/conf ]; then
    mkdir -p ${SOLR_HOME_DRUPAL}/cores/${apachesolr_solrdir}/conf

    # We'll need to patch the solrconfig.xml first because of changes in solr 4.5+
    wget -nc http://drupal.org/files/2107417-2.patch
    patch -p1 < 2107417-2.patch

    # And copy the necessary stock configuration files
    cp -r ${SOLR_HOME}/example/solr/collection1/conf/{elevate.xml,mapping-ISOLatin1Accent.txt,stopwords.txt,synonyms.txt} ${SOLR_HOME_DRUPAL}/cores/${apachesolr_solrdir}/conf/

    # Then copy the module's config files over
    cp -rfb ${DIR}/solr-conf/solr-4.x/* ${SOLR_HOME_DRUPAL}/cores/${apachesolr_solrdir}/conf/
  
    echo -e "\e[00;32m NOTICE:Apachesolr ${apachesolr_version} settings installed  \e[00m"
  else
    echo -e "\e[00;33m WARNING:Apachesolr settings already there, skipping \e[00m"
  fi
else
  rm -rf $DIR
fi
}

function search_api_solr_config {
## SEARCH_API_SOLR Drupal module setup
cd ${SOLR_HOME_DRUPAL}
DIR=${SOLR_HOME_DRUPAL}/search_api_solr

if [ ! -d $DIR ]; then
  # Download drupal module so we can use the correct configuration
  drush dl search_api_solr-${SOLR_MODULE_VERSION}
  mv search_api_solr search_api_solr-${SOLR_MODULE_VERSION}
  DIR=${DIR}-${SOLR_MODULE_VERSION}
  cd $DIR

  # Create the core conf directory
  search_api_solr_version=$(grep version ${DIR}/search_api_solr.info | sed -e 's#version = "\([^"]\+\)"#\1#g')
  search_api_solr_solrdir='base_search_api_solr-'${search_api_solr_version}'_conf'
  if [ ! -d ${SOLR_HOME_DRUPAL}/cores/${search_api_solr_solrdir}/conf ]; then
    mkdir -p ${SOLR_HOME_DRUPAL}/cores/${search_api_solr_solrdir}/conf

    # We'll need to patch the solrconfig.xml first because of changes in solr 4.5+
    wget -nc http://drupal.org/files/issues/2122391-2.patch
    patch -p1 < 2122391-2.patch

    # Then copy the module's config files over
    cp -rfb ${DIR}/solr-conf/4.x/* ${SOLR_HOME_DRUPAL}/cores/${search_api_solr_solrdir}/conf/
  
    echo -e "\e[00;32m NOTICE:Search API Solr ${search_api_solr_version} settings installed  \e[00m"
  else
    echo -e "\e[00;33m WARNING:Search API Solr settings already there, skipping \e[00m"
  fi
else
  rm -rf $DIR
fi
}
#Test connection to apache solr
function solr_install_test_connection {
  response=$(curl --write-out %{http_code} --silent --output /dev/null http://127.0.0.1:8983/solr/)
  if [ $response != 200 ]; then
    rm /etc/init.d/solr
    echo -e "\e[00;31m ERROR:Solr installation failed manual intervention needed. Solr reponds at http://127.0.0.1:8983/solr/ with http code: $reponse  \e[00m"
  else
    echo -e "\e[00;32m NOTICE:Solr reponds at http://127.0.0.1:8983/solr/ with http code:$response  \e[00m"
  fi
}

# ****** #
# Script #
# ****** #

#Variables
SOLR_CORES=${SOLR_HOME_DRUPAL}/cores

#Copy a preconfigured core to use as a basis for our (see solr-install.sh)
DIR="${SOLR_CORES}/${PROJECT}"
if [ -d $DIR ]; then
  echo -e "\e[00;33m $DIR exists, skipping SOLR core creation\e[00m"
else

  if [ $SOLR_MODULE == apachesolr ]; then
    apachesolr_config
  elif [ $SOLR_MODULE == search_api_solr ]; then
    search_api_solr_config
  else
    echo -e "\e[00;32m NOTICE:Please enter only 'apachesolr' or 'search_api_solr'. You entered '$SOLR_MODULE'  \e[00m"
    exit
  fi

  cp -r ${SOLR_CORES}/base_${SOLR_MODULE}-${SOLR_MODULE_VERSION}_conf ${SOLR_CORES}/${PROJECT}
  echo -e "\e[00;32m Solr core $PROJECT created in ${SOLR_CORES}/${PROJECT}  \e[00m"
fi

# CREATE the core via CoreAdmin API
curl 'http://localhost:8983/solr/admin/cores?action=CREATE&name='${PROJECT}
sleep 5
solr_install_test_connection

#Exit output
echo -e "\e[00;32m Now you can use 'http://127.0.0.1:8983/solr/#/"$PROJECT"'.
 Check out all your cores at http://127.0.0.1:8983/solr/#/~cores \e[00m"

How to use it?

  ./create-solr-instance.sh [projectname] [optional: apachesolr or search_api_solr] [required with module: module_version]

Use the second optional parameter if you want to install apache solr multicore for the module (suffix with d6 or d7). By default the settings from the module will be installed. So dont use the second param if you are using those modules.

  ./create-solr-instance.sh drupalsolrsite

You should get a result like this:

Review

Go through this script piece by piece so you can learn some bash and learn to use this to your advantage. As drupal developers we are more or less php developers but learning some bash can be a great time saver. For example this script saved me countless hours. Everytime I had to install a multicore on a vm or on somebodies installation I m coaching, we are winning valueable time. You could even use this to control your production installation too.

Read the comments carefully and understand what every step does. We can be lazy by letting the automated scripts do the work for us but we can't allow ourselves to become stupid. You still need to understand what happens. You will need to know how it works because automated stuff always breaks sooner or later.

Other advantages

As already told this script just saves time by automating a repetitive task. But there are some other advantages.

I also use this as an instrument to teach junior drupal developers about apache solr without having them loose to much time on the details. They can set up a solr instance quickly and continue to be productive. In their training time and/or spare time they can pick apart the script like we will do now and learn about the details.

Another advantage is: You know by using the script to install an apache solr multicore locally that everyone has the same configuration. There are so much tutorials on how to set up solr on the web it becomes very possible that everyone in the team has a differten version/setup.

Conclusion

Using scripts to automate these tasks like installing apache solr is highly recommended. But you should always know what you are executing. Always be carefully to review things so you know what to do if the automations break down .

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment