Use the power of Solr search for your website with our search solution SolrHQ, a hosted Solr search solution that is easy to set up and scales with your growth. 
We offer plans to meet the needs of any sized organization. We offer free tech support and free set up when you sign up for SolrHQ. Our new service SolrHQ is an expandable cloud hosted Apache Solr solution, making Enterprise Search possible for everybody. Read more…
Download the presentation and see the video.
Michael McIntosh, Vice President of Enterprise Search Technologies at TNR, spoke at the Lucene Revolution conference in Boston, MA October 7-8, 2010. Michael reviewed the migration from Fast ESP to Lucene/Solr open source search. He discussed approaches to identifying core content areas of HTML documents such as Text-To-Tag Ratio Heuristics and Page Stereotype/Site Template Analysis, and reviewed specific use cases that we have encountered as search integration experts and discuss available tools.
TNR Global was a sponsor of Lucene Revolution. The conference gathered over 400 professionals from the enterprise search industry. We were happy to see so much interest in Lucene/Solr open source search, and get to know and learn from the folks who have done large scale implementations, including Twitter, LinkedIn, and eHarmony. Not surprisingly, there was a lot of interest about migration from proprietory search systems to Solr, especially from FAST ESP due to Microsoft’s discontinuing FAST ESP support for Linux. If you would like to learn more about how a migration from FAST ESP to Lucene Solr can benefit your company, contact us for a free consultation.
So, you’ve installed a fresh copy of Apache Solr. You have tested it out running the examples from the Solr tutorial. And now you are ready to start indexing some of your own data. Just one problem. The fields for your data are not recognized by the default Solr instance. You notice in the schema.xml file that the default fields have names like cat, weight, subject, includes, author, title, payloads, popularity, price, etc. These fields are defined for the purpose of being used with the sample data provided with Solr. Most of their names are likely not relevant to your dataset, and even if you can manage to make things “fit” with misnamed fields even just for the purpose of experimenting, you also face the problem that their set properties may not be what you would expect them to be.
Of course you can modify the schema.xml file and apply strong data-typing to each field that you plan to use to fit the exact needs of your project, reload Solr, and then start to index your data. But if you are just getting started with Solr, or starting a new project and experimenting with adding to your dataset, you may not know exactly what fields you need to define or what properties to define for them. Or you might be interested updating an existing index with some additional fields, but do not want to explicitly add them to the schema.
Fortunately, Solr gives the option to define dynamic fields. Further, there are pre-defined dynamic fields for many of the common data-types in the default schema. Here are the some of the dynamic fields that are found in the default schema.xml:
<dynamicField name="*_i" type="int" indexed="true" stored="true"/> <dynamicField name="*_s" type="string" indexed="true" stored="true"/> <dynamicField name="*_l" type="long" indexed="true" stored="true"/> <dynamicField name="*_t" type="text" indexed="true" stored="true"/> <dynamicField name="*_b" type="boolean" indexed="true" stored="true"/> <dynamicField name="*_f" type="float" indexed="true" stored="true"/> <dynamicField name="*_d" type="double" indexed="true" stored="true"/> <dynamicField name="*_dt" type="date" indexed="true" stored="true"/>
The field names are defined with a glob-like pattern that is either at the beginning or end of the name. With the above dynamic fields, you can index data with field names that begin with any valid string and end in one of the suffixes in the name attributes (i.e. article_title_s, article_content_t, posted_date_dt, etc.) and Solr will dynamically create any dynamic field of the particular type with the name that you give it.
<add> <doc> <field name="article_title_s">My Article</field> <field name="article_content_t">Lorem Ipsum...</field> <field name="posted_date_dt">1995-12-31T23:59:59Z</field> </doc> </add>
After you’ve indexed some data, you can actually view the dynamic field names in the schema viewer, located at http://YOUR-INSTANCE/admin/schema.jsp
Using dynamic fields is a great way to get started at using Apache Solr with minimal setup.
If you are looking for a fast and easy way to populate a Solr instance using Python, read on.
The script provided here is a basic starting point to building the Solr index for any website with a sitemap, within minutes. Simply modify the script to use your Solr instance and run with a path to your valid XML sitemap and it will begin populating your Solr index.
While you certainly can modify this script to fit your specific needs, you may even find that this script satisfies your Solr indexing requirements as-is.
To start, you need to be running Python 2.6 and have the following modules installed:
You can install these using easy_install or manually.
You will also require an Apache Solr instance. (If you are looking for fully managed solution for hosting your Solr search application with a wide range of services, feel free to contact us.)
Ideally you will use this script on your own sitemap. For detailed information on how to construct your sitemap click here: http://www.sitemaps.org/protocol.php. You can search the web for other scripts that will automatically make sitemaps out of common CMS’s like WordPress and Joomla. There are also sitemap generators available. You can also find a valid sitemap for testing here: http://www.google.com/sitemap.xml (~4Mb). We will assume that you have have a valid sitemap.
We will also assume that you have the default Solr schema.xml installed.
Write the following python script sitemap-indexer.py, replacing the value for solrUrl with the location of your own instance:
#! /usr/bin/env python
""" Index links from a sitemap to a SOLR instance"""
import sys
from BeautifulSoup import BeautifulSoup
import solr
import hashlib
import urllib2
from xml.etree.ElementTree import parse
# How many iterations max? Enter 0 for no limit.
limit = 0
# The URL of the solr instance
solrUrl = 'http://localhost:8080/sitemap-indexer-test'
# The xmlns for the sitemap schema
sitemaps_ns = 'http://www.sitemaps.org/schemas/sitemap/0.9'
if len(sys.argv) != 2:
print 'Usage: ./sitemap-indexer.py path'
sys.exit(1)
sitemapTree = parse(sys.argv[1])
solrInstance = solr.SolrConnection(solrUrl) # Solr Connection object
counter = 0
numAdded = 0
# Find all of the URLs in the form <url>...<loc>URL</loc>...</url>
for urlElem in sitemapTree.findall('{%s}url/{%s}loc'%(sitemaps_ns,sitemaps_ns)):
counter = counter + 1 # Increment counter
if limit > 0 and counter > limit:
# For testing, if the limit is reached, break
break;
url = urlElem.text # Get the url text from the element
try: # Try to get the page at url
response = urllib2.urlopen(url)
except:
print "Error: Cannot get content from URL: "+url
continue # Cannot get HTML. Skip.
try: # Try to parse the HTML of the page
soup = BeautifulSoup(response.read())
except:
print "Error: Cannot parse HTML from URL: "+url
continue # Cannot parse HTML. Skip.
if soup.html == None: # Check if there is an <html> tag
print "Error: No HTML tag found at URL: "+url
continue #No <html> tag. Skip.
try: # Try to set the title
title = soup.html.head.title.string.decode("utf-8")
except:
print "Error: Could not parse title tag found at URL: "+url
continue #Could not parse <title> tag. Skip.
try: # Try to set the body
body = str(soup.html.body).decode("utf-8")
except:
print "Error: Could not parse body tag found at URL: "+url
continue #Could not parse <body> tag. Skip.
# Get an md5 hash of the url for the unique id
url_md5 = hashlib.md5(url).hexdigest()
try: # Add to the Solr instance
solrInstance.add(id=url_md5,url_s=url,text=body,title=body)
except Exception as inst:
print "Error adding URL: "+url
print "\tWith Message: "+str(inst)
else:
print "Added Page \""+title+"\" with URL "+url
numAdded = numAdded + 1
try: # Try to commit the additions
solrInstance.commit()
except:
print "Could not Commit Changes to Solr Instance - check logs"
else:
print "Success. "+str(numAdded)+" documents added to index"
Make the script executable and run it:
./sitemap-indexer.py /path/to/sitemap.xml
It will start to go through the sitemap, parsing the content of each URL and if no errors found will add it to the Solr index. This process can take several minutes. There may be errors parsing many of the documents. They will simply be skipped, you may have to fine-tune the parser to fit your specific needs.
Once finished, it will output the number of documents that were committed to the Solr index.
You should be able to access your Solr Instance and run queries. There are numerous resources on the web to help you form query strings. There is also a query form in your Solr web admin interface that allows setting the various request parameters.
If you experience Solr Exceptions, check your Solr logs. If you modified your schema, be sure to reload your Solr instance as this may be the cause of Unrecognized Field Exceptions. You can find the default Solr schema in the example/solr/ directory of a new install of Solr.
If you would like to parse the documents for more specific tags than simply taking the entire body element (as this script does), refer to this documentation:
http://www.crummy.com/software/BeautifulSoup/documentation.html.
If you are looking for a fast and easy way to populate a Solr instance using Python, read on.
The script provided here is a basic starting point to building the Solr index for any website with a sitemap, within minutes. Simply modify the script to use your Solr instance and run with a path to your valid XML sitemap and it will begin populating your Solr index.
While you certainly can modify this script to fit your specific needs, you may even find that this script satisfies your Solr indexing requirements as-is.
To start, you need to be running Python 2.6 and have the following modules installed:
You can install these using easy_install or manually.
You will also require an Apache Solr instance. (If you are looking for fully managed solution for hosting your Solr search application with a wide range of services, feel free to contact us.)
Ideally you will use this script on your own sitemap. For detailed information on how to construct your sitemap click here: http://www.sitemaps.org/protocol.php. You can search the web for other scripts that will automatically make sitemaps out of common CMS’s like WordPress and Joomla. There are also sitemap generators available. You can also find a valid sitemap for testing here: http://www.google.com/sitemap.xml (~4Mb). We will assume that you have have a valid sitemap.
We will also assume that you have the default Solr schema.xml installed.
Write the following python script sitemap-indexer.py, replacing the value for solrUrl with the location of your own instance:
#! /usr/bin/env python26
""" Index links from a sitemap to a SOLR instance"""
import sys
from BeautifulSoup import BeautifulSoup
import solr
import hashlib
import urllib2
from xml.etree.ElementTree import parse
limit = 0 # How many iterations max? Enter 0 for no limit.
solrUrl = 'http://localhost:8080/sitemap-indexer-test' # The URL of the solr instance
sitemaps_ns = 'http://www.sitemaps.org/schemas/sitemap/0.9' # The xmlns for the sitemap schema
if len(sys.argv) != 2:
print 'Usage: ./sitemap-indexer.py path'
sys.exit(1)
sitemapTree = parse(sys.argv[1])
solrInstance = solr.SolrConnection(solrUrl) # Solr Connection object
counter = 0
numAdded = 0
# Find all of the URLs in the form <url>...<loc>URL</loc>...</url>
for urlElem in sitemapTree.findall('{%s}url/{%s}loc'%(sitemaps_ns,sitemaps_ns)):
counter = counter + 1 # Increment counter
if limit > 0 and counter > limit:
break; # For testing, you can set a limit to how many pages of the sitemap to consider
url = urlElem.text # Get the url text from the element
try:
response = urllib2.urlopen(url) # Try to get the page at url
except:
print "Error: Cannot get content from URL: "+url
continue # Cannot get HTML. Skip.
try:
soup = BeautifulSoup(response.read()) # Try to parse the HTML of the page
except:
print "Error: Cannot parse HTML from URL: "+url
continue # Cannot parse HTML. Skip.
if soup.html == None: # Check if there is an <html> tag
print "Error: No HTML tag found at URL: "+url
continue #No <html> tag. Skip.
try:
title = soup.html.head.title.string.decode("utf-8") # Try to set the title
except:
print "Error: Could not parse title tag found at URL: "+url
continue #Could not parse <title> tag. Skip.
try:
body = str(soup.html.body).decode("utf-8") # Try to set the body
except:
print "Error: Could not parse body tag found at URL: "+url
continue #Could not parse <body> tag. Skip.
# Note, decode("utf-8") is used to avoid non-ascii characters in the solrInstance.add below
# Get an md5 hash of the url for the unique id
url_md5 = hashlib.md5(url).hexdigest()
try:
# Add to the Solr instance
solrInstance.add(id=url_md5,url_s=url,text=body,title=body)
except Exception as inst:
print "Error adding URL: "+url
print "\tWith Message: "+str(inst)
else:
print "Added Page \""+title+"\" with URL "+url
numAdded = numAdded + 1
try:
solrInstance.commit() # Commit the additions
except:
print "Could not Commit Changes to SOLR Instance - Check SOLR logs for more info"
else:
print "Success. "+str(numAdded)+" documents added to index"
Make the script executable and run it:
./sitemap-indexer.py /path/to/sitemap.xml
It will start to go through the sitemap, parsing the content of each URL and if no errors found will add it to the Solr index. This process can take several minutes. There may be errors parsing many of the documents. They will simply be skipped, you may have to fine-tune the parser to fit your specific needs.
Once finished, it will output the number of documents that were committed to the Solr index.
You should be able to access your Solr Instance and run queries. There are numerous resources on the web to help you form query strings. There is also a query form in your Solr web admin interface that allows setting the various request parameters.
If you experience Solr Exceptions, check your Solr logs. If you modified your schema, be sure to reload your Solr instance as this may be the cause of Unrecognized Field Exceptions. You can find the default Solr schema in the example/solr/ directory of a new install of Solr.
If you would like to parse the documents for more specific tags than simply taking the entire body element (as this script does), refer to this documentation:
http://www.crummy.com/software/BeautifulSoup/documentation.html.
If you’re using server-side features of MongoDB that have a minimum version requirement (like pushing a unique value to a list), it is a good idea to make sure you have the required version running on the server. To check the version of the MongoDB server using PyMongo, you can use something like this:
import pymongo
connection = pymongo.Connection()
serverVersion = tuple(connection.server_info()['version'].split('.'))
requiredVersion = tuple("1.3.3".split("."))
if serverVersion < requiredVersion:
# handle the error
return 1
...
It’s important to note that you must connect to the admin database to determine the version number. Otherwise, you will probably run into something like this:
pymongo.errors.OperationFailure: command SON([('buildinfo', 1)]) failed: access denied
If you need to check the version of the server from the interactive prompt, run the following from the mongo prompt:
> db.version() 1.4.2
In a production (or even stable) ESP environment, it is difficult to make a change to the Document Processing Pipeline and test it without wiping out the existing collection (not to mention the time it takes to perform a full re-crawl if the collection is even moderately large). In this case, the best option is to use postprocess to feed existing documents to a new (empty) collection.
Making a duplicate collection provides several benefits:
$FASTSEARCH/bin/nctrl stop crawler
$FASTSEARCH/bin/postprocess -R origcollection -k default:newcollection
Notes about this command:
$FASTSEARCH/bin/nctrl start crawler
Just wanted to take this error message off of the, “Hey, we’ve seen this before… now how did we resolve this..?” pile. This is the full text of the error:
WARNING Could not send batch to ESP content distributor, will retry automatically.
Reason given: process() failed: exception (no_resources) no doc procs registered to
process a batch with priority 0
At first glance, it looks pretty clear that you just need to [re]start your document processor(s). However, this won’t necessarily solve the problem. Turns out that the a likely reason for this to pop up is a bad Document Processing Pipeline (DPP) Stage. The docprocs fire up, hit the bad stage (e.g. python errors etc.) and don’t recover.
To debug your DPP Stage, take a look at the logs for the document processor(s). They’re usually located in $FASTSEARCH/var/log/procserver and, in our experience, there’s probably an uncaught python exception lurking somewhere in there.
TNR Global is a systems design and integration company focused on providing customers effective search and cloud computing solutions. We develop scalable web-based search solutions focusing on news sites, publishing, web directories and catalogs, information portals, education, manufacturing and distribution, customer service, and life sciences.
Below are links to our mini white papers, addressing questions about enterprise search and more.
Enterprise Search Basics (pdf:157,797) Enterprise Search and Government (pdf:112,083) Enterprise Search for Law Firms (pdf:110,350) Enterprise Search and E-Discovery (pdf:108,855)Please contact us for additional information.
Migration from FAST ESP to Lucene Solr, by Michael McIntosh, VP, Enterprise Search Technologies, TNR Global
Video from the Lucene Revolution conference in Boston, MA.
