solr – Page 2 – TNR Global

For Many Companies, Migration to a New Search Engine is Inevitable

“It’s basically a road map for companies looking at options for migration, and we outline Solr as a very good option”

HADLEY, MA– March 12, 2012

In the world of Enterprise Search, everything is changing. Companies who have been using Microsoft’s internal search engine, FAST Enterprise Search Platform, will be forced to make a change as Microsoft discontinues support for the search platform for companies using Linux as their operating system. Anticipating the need for a solution, local technology consultants TNR Global is pleased to announce the release of a White Paper for migrating off FAST ESP to a new search engine, Solr. The paper is titled Bridging the Gap: A Migration Path from Fast ESP to Apache Solr.

This effort began last October when TNR Global presented on the subject of migration from FAST to Solr at the open source conference, Apache Lucene Eurocon in Barcelona, Spain. The paper contains a case study with architecture overview, loading millions of documents into Solr indexes, evaluation and recommendation of tools to bridge the feature gap, migrating custom pipeline code, and the vastly improved ROI after implementation. “It’s basically a road map for companies looking at options for migration, and we outline Solr as a very good option” said Karen E. Lynn, Director of Business Development.

“We have spent over 9 years working with the FAST ESP product and we understand the nuances of what customers have come to expect from the technology. We’ve identified Solr as a top choice for migrating off FAST as support for the product drops off” said Michael McIntosh, VP of Search Technologies and lead author of the paper. “Solr is an open source technology that has matured and is certainly stable enough for commercial use” said Chris Miles, Senior Software Engineer and contributor to the paper. “We’re excited about this migration option for our customers, and we believe over the long run, it will save them a lot of money and give them greater control over their search engine.”

This heavily anticipated paper will assist companies and organizations in planning their own FAST ESP to Apache Solr migrations and alert them to tools and techniques that can help them achieve a relatively painless process. Several large blue chip companies have expressed interest in the paper. “We’ve had a healthy response to the paper” said Lynn.

Internal search engines differ from public search engines like Google or Bing, in that an internal search engine only searches for content inside the company’s firewall. Google cannot access internal content, therefore companies use search technology to make their content ‘findable.’ “Companies want to keep internal information safe and private. But they still need to find it” explained Lynn. “That’s why they need search technology integrated into their organization’s system.”

For more information on search engines, product search, web portals and search engine migration, visit TNR’s main website. To receive a free copy of the white paper, click here.

TNR Global www.tnrglobal.com, is a systems design and integration company focused on enterprise search and cloud computing solutions for publishing companies, news sites, web directories, academia, enterprise, and SaaS companies. TNR’s past clients include the University of Massachusetts Amherst, Mass Art & Culture, InterNano, Innovara, and the Allegis Group. TNR Global is located at 245 Russell Street, Suite 10 in Hadley, Massachusetts. TNR Global serves clients throughout New England, nationally, and world-wide. Its offices are in Hadley and Greenfield, Massachusetts.

UK Software Company TwigKit Partners with TNR Global to Deliver Search Solutions for FAST and Solr

Hadley, MA–November 28, 2011–TNR Global announced today that they have entered into a strategic partnership with London, UK software company TwigKit.

“Our companies have a number of qualities in common that allow us to combine forces and service clients with a very complete solution” says Karen Lynn, TNR Global’s Director of Business Development. “TwigKit has a very appealing user interface for users across several platforms, and TNR’s strength is on creating a powerful back end search application. Combined, it’s a powerful solution for companies needing a strong search function with an easy to use interface.”

Representatives from the two companies have been in friendly talks for over a year now, meeting periodically at industry conferences. Both companies were in attendance at the Apache Lucene EuroCon conference in Barcelona last October, where the partnership was formalized. Both presentations from TwigKit and TNR can be viewed here.

“TNR’s focus on implementing and servicing enterprise search solutions across a number of platforms is an excellent fit for TwigKit,” says Stefan Olafsson, TwigKit’s co-founder and chief architect. “Our software enables polished user interfaces for search-based applications, provides a rapid development framework, and works across a number of enterprise search platforms including Microsoft FAST and Apache Solr. We’re excited about working with TNR to produce search solutions that boast both a superb user experience and an outstanding technical implementation.”

TwigKit powers enterprise search applications in government and blue-chip organizations. Encapsulating search best practices into configurable components, TwigKit establishes a platform-independent standard compatible with most search technologies including Microsoft FAST, Google Search Appliance, and Apache Solr. Started in London in 2009, TwigKit’s founders organize the 350-member Enterprise Search London meetup, regularly speak at conferences, and write about search and user experience for publications including A List Apart, Boxes & Arrows, and UX Magazine.

TNR Global (TNR) is a systems design and integration company focused on enterprise search and cloud computing solutions. TNR develops scalable web-based search solutions built on the open source LAMP stack. TNR has over 10 years of hands-on experience in web systems and enterprise search implementations, both proprietary and open source search technologies, specializing in FAST and Lucene Solr search applications. Specifically TNR works with content intensive websites for companies and organizations in the following industries: News Sites, Publishing, Web Directories, Information Portals, Web Catalogs, Education, Manufacturing and Distribution, Customer Service, and Life Sciences. TNR Global has offices in western Massachusetts.

We’re at Apache Lucene EuroCon in Barcelona 2011

“We’re certain that the urgency to migrate off FAST ESP will be ramping up significantly.”

We’re very excited to be in attendance at the Apache Lucene EuroCon in Barcelona October 17, 18, 19, and 20th, 2011. Our own Michael McIntosh, VP of Search Technologies will be presenting a talk on October 19th, Enterprise Search: FAST ESP to Lucene Solr. The good folks at Lucid Imagination are presenting the conference and will be video recording his talk for future broadcast.

After the conference, Michael will author a White Paper on migrating from FAST ESP to Lucene Solr, expected in November 2011. For a free copy of the White Paper, email us expressing your interest at fast2solr@tnrglobal.com. We believe that those businesses operating on a Linux system will be seeking out the power of Lucene Solr as their licenses expire and support for FAST ESP dries up. We’ve worked with FAST ESP for 7 years and understand it’s strengths and weaknesses. We know businesses who are used to the power of FAST ESP will need something just as powerful, and Lucene Solr is a very nice fit. “It’s a robust platform, capable of a lot that FAST ESP covers,” said Michael. “We’re certain that the urgency to migrate off FAST ESP will be ramping up significantly.”

FAST ESP to Lucene Solr Presentation: Open Call for Questions

To pre-load the discussion on Michael’s Enterprise Search: FAST ESP to Lucene Solr talk, send your questions to: fast2solr@tnrglobal.com We want to hear from you!

TNR Global is excited to be participating in the Apache Lucene EuroCon conference in Barcelona. Our own Michael McIntosh is scheduled to present: “Enterprise Search: FAST ESP to Lucene Solr” Here is your chance to pre-load the discussion. Before Michael puts the final touches on his talk, he wants to know what issues or questions you may be have. In the following video, he touches on some of the highlights of his upcoming talk, and asks for your input.

Enterprise Search: FAST ESP to Lucene Solr pre-conferece video - Click to Watch — Enterprise Search: FAST ESP to Lucene Solr pre-conf video

To participate in advance, send you questions or comments to: fast2solr@tnrglobal.com. While Michael cannot promise he will include your question or commentary in his actual talk, he will work to address them in an upcoming White Paper, to be released after the conference in November 2011. We look forward to hearing from you!

Migration Still Looms Large on the Horizon for FAST ESP Customers

“Designing a non-trivial search solution to fully meet your needs from scratch is hard enough on its own. If you are migrating an existing solution, it is very unlikely that you will find a one to one mapping of all of the features in a new search engine that you have come to depend upon with your existing implementation.” –Michael McIntosh, VP of Search Technologies, TNR Global, LLC

Microsoft acquired FAST all the way back in 2008 and then in early 2010 disclosed it’s plans to stop updating the FAST product on a Linux operating system after 2010, making FAST ESP 5.3 the latest and greatest, and very last update Linux users will see involving any improvements to the proprietary search platform. It was clear to anyone on Linux that a migration would need to occur, and as content grows, depending upon the size of your organization, that migration should probably happen sooner than later.

Buzz about migration ensued–an inevitable certainty for many companies, especially ones with huge amounts of data. But how many companies have jumped in with both feet? I had the opportunity to speak with an open source search engine expert who, along with the industry, believed that the move from Microsoft was a windfall for anyone in the business of enterprise search design and implementation. However, she admitted “we haven’t seen as large a response as we expected.”

This isn’t exactly surprising to everyone. “It’s coming” says our VP of Search Technologies, Michael McIntosh. “Corporations have an enormous investment in FAST ESP and it makes sense that they would be reluctant to move to something new until they absolutely have to.” That means, when their licenses expire.

“They will likely weigh the performance and support, or lack thereof, for the FAST ESP technical team with the timing of renewing a license and wait until they absolutely have to change to something else,” says McIntosh.

The purchase of Autonomy and the shift of HP from hardware to software could signal a recognition from Goliath HP the kind of growth opportunity enterprise search software offers, and that the “great shift” from FAST ESP to another search platform is very much on the horizon.

But as the clock continues to tick, companies using FAST ESP should be strategizing for migration now. “It’s an enormous undertaking to migrate an entire search solution from FAST to another platform. Designing a non-trivial search solution to fully meet your needs from scratch is hard enough on its own. If you are migrating an existing solution, it is very unlikely that you will find a one to one mapping of all of the features in a new search engine that you have come to depend upon with your existing implementation. Solving challenging issues like that requires both creativity and expertise to address your needs.” says McIntosh. If a need for migration is eminent, there will be a real need for expertise in the field of enterprise search on both proprietary and open source platforms, depending upon several factors like size, in house talent, and growth expectations.

How is your company preparing for the discontinuation of support of FAST ESP? Need guidance? Contact us for pointers, analysis, or architecture for a full migration.

TNR Global to present at Apache Lucene Eurocon 2011 in Barcelona

We are happy to announce that TNR Global’s own Michael McIntosh will be presenting at the Apache Lucene Eurocon 2011 in Barcelona this October. Michael’s talk is titled “Enterprise Search: FAST ESP to Lucene Solr.” His presentation will discuss migration from the FAST ESP platform to a Lucene Solr search platform. There are many reasons an IT department with a large scale search installation would want to move from a proprietary platform to Lucene Solr. In the case of FAST Search, the company’s purchase by Microsoft and discontinuation of the Linux platform has created an urgency for FAST users. Illustrated through actual case studies, the presentation will include challenges and concerns, present solutions and work-arounds to overcome migration issues.

Michael has more than 16 years of experience in large scale systems design and operation, online consumer product development, high volume transaction processing and engineering management. He has extensive experience developing, integrating and maintaining search technology solutions for companies such as FAST Search and Lycos.

We’re excited that Michael will be presenting in Barcelona this fall. Please introduce yourself if you’re able to go!

Dynamic Fields in Apache Solr

So, you’ve installed a fresh copy of <a taget=”_blank” href=”http://lucene.apache.org/solr/”>Apache Solr</a>. You have tested it out running the examples from the <a href=”http://lucene.apache.org/solr/tutorial.html”>Solr tutorial</a>. And now you are ready to start indexing some of your own data. Just one problem. The fields for your own data are not recognized by your Solr instance. You notice in the schema.xml file that the default fields have names like cat, weight, subject, includes, author, title, payloads, popularity, price, etc. These fields are all defined for the purpose of being used with the sample data provided with Solr. Most of their names are likely not relevant to your search project, and even if you are willing to put up with misnamed fields at least for experimenting with your instance, you also face the problem that their set properties may not be what you would expect them to be.

Of course you can modify the schema.xml file and apply strong data-typing to each field that you plan to use to fit the exact needs of your project, reload Solr, and then start to index your data. But if you are just getting started with Solr, or starting a new project and experimenting with adding your dataset, you may not know exactly what fields you need to define or what properties to define for them. Or you might be interested updating an existing index with some additional fields, but do not want to add to explicitly add them to the schema.

Fortunately, Solr gives the option to define dynamic fields – fields that are defined in the schema with a glob-like pattern that is either at the beginning or end of the name. Further, there are pre-defined dynamic fields for most of the common data-types that you may use, in the default schema. Here are the some of the dynamic fields that are defined in the default schema.xml:

<dynamicField name="*_i"  type="int"    indexed="true"  stored="true"/>
<dynamicField name="*_s"  type="string"  indexed="true"  stored="true"/>
<dynamicField name="*_l"  type="long"   indexed="true"  stored="true"/>
<dynamicField name="*_t"  type="text"    indexed="true"  stored="true"/>
<dynamicField name="*_b"  type="boolean" indexed="true"  stored="true"/>
<dynamicField name="*_f"  type="float"  indexed="true"  stored="true"/>
<dynamicField name="*_d"  type="double" indexed="true"  stored="true"/>
<dynamicField name="*_dt" type="date"    indexed="true"  stored="true"/>

The field names are defined with a glob-like pattern that is either at the beginning or end of the name. With the above dynamic fields, you can index data with field names that begin with any valid string and end in one of the suffixes in the name attributes (i.e. article_title_s, article_content_t, posted_date_dt, etc.) and Solr will dynamically create any dynamic field of the particular type with the name that you give it. After you’ve indexed some data, you can actually view this dynamically created field in the schema viewer for your instance, located at http;//YOUR-INSTANCE/admin/schema.jsp

<add>
<doc>
<field name="article_title_s">My Article</field>
<field name="article_content_t">Lorem Ipsum...</field>
<field name="posted_date_dt">1995-12-31T23:59:59Z</field>
</doc>
</add>

SolrHQ, Hosted Solr Search

Use the power of Solr search for your website with our search solution SolrHQ, a hosted Solr search solution that is easy to set up and scales with your growth.

We offer plans to meet the needs of any sized organization. We offer free tech support and free set up when you sign up for SolrHQ. Our new service SolrHQ is an expandable cloud hosted Apache Solr solution, making Enterprise Search possible for everybody. Read more…

Migration from FAST ESP to Lucene Solr

Download the presentation and see the video.

Michael McIntosh, Vice President of Enterprise Search Technologies at TNR, spoke at the Lucene Revolution conference in Boston, MA October 7-8, 2010. Michael reviewed the migration from Fast ESP to Lucene/Solr open source search. He discussed approaches to identifying core content areas of HTML documents such as Text-To-Tag Ratio Heuristics and Page Stereotype/Site Template Analysis, and reviewed specific use cases that we have encountered as search integration experts and discuss available tools.

TNR Global was a sponsor of Lucene Revolution. The conference gathered over 400 professionals from the enterprise search industry. We were happy to see so much interest in Lucene/Solr open source search, and get to know and learn from the folks who have done large scale implementations, including Twitter, LinkedIn, and eHarmony. Not surprisingly, there was a lot of interest about migration from proprietory search systems to Solr, especially from FAST ESP due to Microsoft’s discontinuing FAST ESP support for Linux. If you would like to learn more about how a migration from FAST ESP to Lucene Solr can benefit your company, contact us for a free consultation.

Migration from Microsoft FAST to Apache Lucene Solr

Is your company using Microsoft FAST ESP on a Linux platform? Unfortunately, Microsoft announced in 2010 they will cease technical support for FAST ESP 5.3 after it’s 5 year life cycle for anyone using Linux as their operation system. Migration to another search platform will be a priority, and business leaders and technology professionals are looking closely at Lucene Solr as a solution.

We can assist your organization in any stage of a migration. We can perform an evaluation of your current architecture, draft a plan for migration, work with your internal team on the migration or just consult as needed. Whatever your specific needs are, we can help you achieve your goals. Read our White Paper released February 2012 that presents a Case Study on migration. The paper discusses:

Loading millions of documents into Solr indexes
Evaluation and recommendations for tools to bridge the features gap
Migrating custom pipeline code to Pypes with minimal changes
Proven ROI after a complete migration

Additionally, we have presented on the subject of FAST ESP to Lucene Solr migrations for the Lucene Revolution Conference in Boston, MA (2010 Slides: Migration from FAST ESP to Lucene Solr (PDF) (pdf:4,067,091) ) and at the Apache Lucene Eurocon (web site dead) Barcelona (October 2011). Watch our VP of Search Technology Michael McIntosh’s presentation on FAST to Lucene Solr Migration below. If you like what you see, contact us to explore a Solr migration solution.

Slide presentation