Elasticsearch Evaluation White Paper Released: Promising for Big Data

We believe that Elasticsearch is a product that everyone working in the field of big data will want to take a look at.

There are many new technologies emerging around search, and we’ve been investigating several of them for our clients. Search has never been “easy” but Elasticsearch attempts to make it at least easier. Elasticsearch is billed to be “built for the cloud,” and with so many companies moving into the cloud, it seems like a natural that search would move there too.  This paper is designed to show you just how Elasticsearch works by setting up a cluster and feeding it data.  We also let you know what tools we use so you can test out the technology and we include a rough sketch of code as well. Finally, we make conclusions about how Elasticsearch can help with problems like Big Data and other search related uses.

Elasticsearch is an open source technology developed by one developer, Shay Bannon. This paper is simply a first look at elasticsearch and is not associated with an additonal product or variation of elaticsearch. The appeal for big data is due to elasticsearch’s wonderful ability to scale with growing content, which has largely been associated with the “big data problem” we all keep hearing about. It’s very easy to add new nodes and it handles the balancing of your data across the available nodes. It handles the failure of nodes in a graceful way that is important in a cloud environment. And lastly, we simply evaluate and test the technology. We really don’t believe there is a one size fits all technology in the realm of enterprise search, it is really highly dependent upon your systems, how many documents you have, how much unstructured data you have, and how you want your site to function. But that said– in terms of storing big data, it is as capable as any Lucene based product; it can handle a much larger load that the current Solr release as the notion of breaking the index up into smaller chunks is “baked in” to the product.

Here is an except from the paper:

“Products like Elasticsearch that lack a document processing component entirely become more attractive. In fact, most projects that involve a data set large enough to qualify as “big data”³² are building their own document processing stages anyway as part of their ETL cycle.”

If you are interested in downloading this free White Paper, sign up with us here.

If you would like help using Elasticsearch with your search project, contact us.

For Many Companies, Migration to a New Search Engine is Inevitable

“It’s basically a road map for companies looking at options for migration, and we outline Solr as a very good option”

HADLEY, MA– March 12, 2012

In the world of Enterprise Search, everything is changing.  Companies who have been using Microsoft’s internal search engine, FAST Enterprise Search Platform, will be forced to make a change as Microsoft discontinues support for the search platform for companies using Linux as their operating system.  Anticipating the need for a solution, local technology consultants TNR Global is pleased to announce the release of a White Paper for migrating off FAST ESP to a new search engine, Solr.  The paper is titled Bridging the Gap: A Migration Path from Fast ESP to Apache Solr.

This effort began last October when TNR Global presented on the subject of migration from FAST to Solr at the open source conference, Apache Lucene Eurocon in Barcelona, Spain. The paper contains a case study with architecture overview, loading millions of documents into Solr indexes, evaluation and recommendation of tools to bridge the feature gap, migrating custom pipeline code, and the vastly improved ROI after implementation.  “It’s basically a road map for companies looking at options for migration, and we outline Solr as a very good option” said Karen E. Lynn, Director of Business Development.

“We have spent over 9 years working with the FAST ESP product and we understand the nuances of what customers have come to expect from the technology. We’ve identified Solr as a top choice for migrating off FAST as support for the product drops off” said Michael McIntosh, VP of Search Technologies and lead author of the paper. “Solr is an open source technology that has matured and is certainly stable enough for commercial use” said Chris Miles, Senior Software Engineer and contributor to the paper. “We’re excited about this migration option for our customers, and we believe over the long run, it will save them a lot of money and give them greater control over their search engine.”

This heavily anticipated paper will assist companies and organizations in planning their own FAST ESP to Apache Solr migrations and alert them to tools and techniques that can help them achieve a relatively painless process.  Several large blue chip companies have expressed interest in the paper.  “We’ve had a healthy response to the paper” said Lynn.

Internal search engines differ from public search engines like Google or Bing, in that an internal search engine only searches for content inside the company’s firewall.  Google cannot access internal content, therefore companies use search technology to make their content ‘findable.’ “Companies want to keep internal information safe and private.  But they still need to find it” explained Lynn.  “That’s why they need search technology integrated into their organization’s system.”

For more information on search engines, product search, web portals and search engine migration, visit TNR’s main website.  To receive a free copy of the white paper, click here.

TNR Global www.tnrglobal.com, is a systems design and integration company focused on enterprise search and cloud computing solutions for publishing companies, news sites, web directories, academia, enterprise, and SaaS companies. TNR’s past clients include the University of Massachusetts Amherst, Mass Art & Culture, InterNano, Innovara, and the Allegis Group. TNR Global is located at 245 Russell Street, Suite 10 in Hadley, Massachusetts. TNR Global serves clients throughout New England, nationally, and world-wide. Its offices are in Hadley and Greenfield, Massachusetts.

Mobile Search: Good UX means fewer touches, simple design

Mobile search must be designed for a minimum number of touches before users arrive at the end result. If it takes more than 2-3 touches, the user will look elsewhere for answers.

Mobile phones are rapidly taking over the scene of web development, significantly impacting commerce, advertising, gaming, entertainment, banking and news. 77% of the world’s population or 5.3 billion people are mobile subscribers. China and India lead the way in overall mobile growth. Virtually every measurable metric concerning mobile phone growth indicates entire economies being influenced by mobile technology. It’s not surprising that search technology is powering mobile growth just as it has it’s larger cousin the desktop.

Mobile search used to be clunky and a pain to use. Until recently, the answer was to miniaturize the website. For a time, people thought mobile search would never be as good as the desktop search. But, as people use their phones for more and more, it has forced designers to consider how to make search, as well as all mobile apps, simple and powerful and built for end users.

The Mobile Only World

Outside the US, countries like India, South Africa and Egypt are  leaders in mobile only--meaning users do not or infrequently use a desktop or laptop to access the Internet–making mobile search their primary mechanism for accessing queried information. Since these are also the countries sporting the most mobile growth, they are driving the need for quality relevant search for the mobile market.

Young and free

Another driving metric in the mobile game are young people. The under 25 crowd use a cell phone as their primary mode of accessing the Internet. Mobile phones, smart phones in particular, are used to do nearly everything. Younger people are more open to conducting transactions online via phone than any other demographic. Shopping, banking, GPS, social media, gaming–mobile access allows mobile subscribers to do everything they need to without restricting the user to an office.

Key differences for UX Impact

Key difference between mobile search and desktop search seem obvious. On a cell phone, the screen is much, much smaller. Users are on the go and may access the Internet between tasks or meetings, instead of being in one area. Access needs to be quick and simple. Mobile search must be designed for a minimum number of touches before users arrive at the end result. If it takes more than 2-3 touches, the user will look elsewhere for answers. Fewer touches mean a simpler design, engineered for the user without a lot of fanfare or complication.

Huge Growth

Google reports that 1 in 7 searches are now done via mobile vs. desktop. Mobile searches have increased fourfold in just the last year. Businesses need a mobile application to ensure they are reaching the inbound web traffic looking for their services and products. Mobile applications need a strong search technology to ensure the consumer can connect with the products or information they are looking for. The companies that build their web solution for the mobile market are the companies who will gain more market share and capture that 14% of customers searching for their products on the mobile web.

For the enterprise, accessing important information inside and outside the firewall is vitally important as more content is built within businesses and accessed digitally. With the mounting demand placed on mobile phones and devices, the performance we’ve come to expect from out desktop needs to be scaled to a smaller screen by simplifying wireframes with sophistication and well thought out design.

Our View
TNR Global’s expertise lies in deep back end knowledge using powerful search technologies to give users fast, relevant search results for enterprise sites and large web portals. Recognizing the need for search to work as powerfully for a mobile application as well as a web application, we have teamed up with talented UX designers specifically in the field of search application design for web and mobile. Whether you are looking for a customized UX front end for your search solution or an out of the box answer for mobile search, TNR can connect you with a total solution to answer your web based and mobile search needs. For a free consultation, contact us.

UK Software Company TwigKit Partners with TNR Global to Deliver Search Solutions for FAST and Solr

“TNR’s focus on implementing and servicing enterprise search solutions across a number of platforms is an excellent fit for TwigKit,” says Stefan Olafsson, TwigKit’s co-founder and chief architect.

Hadley, MA–November 28, 2011–TNR Global announced today that they have entered into a strategic partnership with London, UK software company TwigKit.

“Our companies have a number of qualities in common that allow us to combine forces and service clients with a very complete solution” says Karen Lynn, TNR Global’s Director of Business Development. “TwigKit has a very appealing user interface for users across several platforms, and TNR’s strength is on creating a powerful back end search application. Combined, it’s a powerful solution for companies needing a strong search function with an easy to use interface.”

logo-twigkit-light
Representatives from the two companies have been in friendly talks for over a year now, meeting periodically at industry conferences. Both companies were in attendance at the Apache Lucene EuroCon conference in Barcelona last October, where the partnership was formalized.  Both presentations from TwigKit and TNR can be viewed here.

“TNR’s focus on implementing and servicing enterprise search solutions across a number of platforms is an excellent fit for TwigKit,” says Stefan Olafsson, TwigKit’s co-founder and chief architect. “Our software enables polished user interfaces for search-based applications, provides a rapid development framework, and works across a number of enterprise search platforms including Microsoft FAST and Apache Solr. We’re excited about working with TNR to produce search solutions that boast both a superb user experience and an outstanding technical implementation.”

TwigKit powers enterprise search applications in government and blue-chip organizations. Encapsulating search best practices into configurable components, TwigKit establishes a platform-independent standard compatible with most search technologies including Microsoft FAST, Google Search Appliance, and Apache Solr. Started in London in 2009, TwigKit’s founders organize the 350-member Enterprise Search London meetup, regularly speak at conferences, and write about search and user experience for publications including A List Apart, Boxes & Arrows, and UX Magazine.

TNR LOGO

TNR Global (TNR) is a systems design and integration company focused on enterprise search and cloud computing solutions. TNR develops scalable web-based search solutions built on the open source LAMP stack. TNR has over 10 years of hands-on experience in web systems and enterprise search implementations, both proprietary and open source search technologies, specializing in FAST and Lucene Solr search applications. Specifically TNR works with content intensive websites for companies and organizations in the following industries: News Sites, Publishing, Web Directories, Information Portals, Web Catalogs, Education, Manufacturing and Distribution, Customer Service, and Life Sciences. TNR Global has offices in western Massachusetts.

Search and Steel Girders

“Search by itself may look like a simple box, but behind the box is a foundry of girders, cross beams, and structural support that allows you to find what you need.”

“Search ties people together…”

This was one of the many themes at the Enterprise Search Summit in Washington, DC last week. It seems like a fairly obvious statement, but it quickly becomes part of the landscape, taken for granted even though the landscape couldn’t function without it. I have compared search function to the steel girders of a skyscraper. When you walk into the building, you aren’t thinking about the beams holding the building up or connecting floors, but without them, you wouldn’t have a building at all (you couldn’t even find the lobby). Other metaphors overheard include oxygen (invisible yet essential), sunlight (lest we remain in the dark) and electricity (everything stops without it).

Attendees of the conference know how important search is to companies, but increasingly, companies are taking search for granted. There is a fundamental gap in communicating the importance and difficulty of implementing a good search platform.

Companies who need search to run on their website or intranet, expect search to work as it does on the Internet, but this is an apples and oranges scenario.

Here are the main disconnects:

  1. Search is easy
  2. Search is cheap
  3. It never has to be touched again

People expect search inside the firewall to function much like Google does outside the firewall. Google exists for end users and is really, really incredible. It Geo-locates, it auto-completes. It uses your browsing history to provide more relevant results. And you had no financial investment in using this really lovely, elegant, useful tool that doesn’t just assist your Internet experience, but facilitates it. But behind the firewall, things are different. Let me explain.

  • Your business content isn’t publicly available or known. I mean, that would be bad, right? It’s behind the firewall for a reason. So keeping it there yet allowing your staff to access certain levels of information takes some architecture and planning.
  • Google has thousands of developers working on this beautiful, incredible technology every day. They finance this by ad content. How many people do you have on your search team? And how much of their day do they really spend on search? What department is being billed for it? Business leaders need to embrace this as a necessary cost of doing business and budget accordingly, or face the crippling result of staff and customers not being able to find the information they need.

  • 80% of your content is unstructured. Meaning, search engines can’t really read it until some love and care is put into cleaning the data. This is a vital, yet time intense process. Our VP of Search Technologies Michael McIntosh says “We spend about 90% of our time on the document processing pipeline, conditioning data to be fed into the engine.” Moreover, unstructured data isn’t a set number. It’s being creating faster than you can blink by your entire enterprise. Processing it is never a done deal.


So if search connects us, hopefully this finds you thinking about search in more realistic terms. Search by itself may look like a simple box, but behind the box is a foundry of girders, cross beams, and structural support that allows you to find what you need to “make money outside the firewall or save money inside the firewall.”

TNR Global Attends KMWorld’s Enterprise Search Summit Fall 2011

A proof of concept and rapid integration are essential for search customers–they cannot visualize what a search solution will look like without some help from the search professional.

ESSFallLogo

Last week TNR Global attended the Enterprise Search Summit organized by KMWorld in Washington, DC.  VP of Search Technologies Michael McIntosh and Director of Business Development Karen E. Lynn attended the three day conference and Enterprise Solutions Showcase at the Marriott Wardman Park.  Several companies were in attendance, and some common themes emerged.  Among these were designing for users, dealing with unstrcutured content, the need for better search and content analytics to facilitate better search results, as well as tagging content as part of a best practice in workflow.  Also discussed was the need for search vendors to demonstrate to search customers was “right looks like” in a search solution.  A proof of concept and rapid integration are essential for search customers–they cannot visualize what a search solution will look like without some help from the search professional.

An unexpected surprise came when the speaker on open source search was unable to attend at the last moment, our own Michael McIntosh was asked to step in and present on the subject.  Fortunately, he was fresh from his presentation at Apache Lucene EuroCon and already had his presentation loaded on his machine.  Michael discussed Solr and made general points on migrating from a commercial search engine like FAST ESP to a open source platform like Lucene Solr.

Overall it was a great conference with lots of informative talks and friendly search professionals.  We’re looking forward to the next Enterprise Search Summit in Spring, 2012.

Search Fuels Business Intelligence for Decision Making

“The jungle is dark, but full of diamonds.” said Arthur Miller. Can your search technology find the gems buried inside your own business?

“The jungle is dark, but full of diamonds.” said Arthur Miller. The same can be said about the invaluable data inside your business. It’s there, ready to be mined. But unless you have the right tools, you’ll never get to those diamonds.


Content is expanding at an exponential rate. I don’t know anyone in any business who can keep up with the pace of content growth, without the use of powerful search engines to find and extract relevant information. Business analysts expect content to grow 800% over the next 5 years. Business intelligence requires extraction of the right information, and most enterprises have both structured and unstructured data. Structured data is easy for most search engines to search. The rub is in unstructured content–of which there is abundance. Unstructured content is said to account for 70-80% of data in all organizations. This type of content is often in the form of documents, email messages, health records, HTML pages, books, metadata, audio, video, and various other files. All these files have to be “cleaned up” before feeding them through a search engine in order to get results with any kind of value or relevance.


Mining this data is going to be essential for not just the success, but the survival of many businesses. James Kobielus, an analyst at Forrester Research, reports in an interview with ComputerWorld that businesses will increasingly turn to a self-service BI throughout 2011 and beyond. “Increasingly, enterprises will adopt new Web-based interactive querying and reporting tools that are designed to put more data analytics capabilities into the hands of end users,” he said. A good search engine that can find data quickly and easily can “take the burden off IT and speed up the development of reports to a considerable degree,” Kobielus said. The information mined by a search engine tuned to the specific business needs facilities better decision making for people a every job function within the enterprise. “Because every business is a little different, and so many organizations house so much unstructured content, most search engines can’t cover everything that is needed without some customization” said Michael McIntosh, our VP of Search Technologies at TNR Global. “Data conditioning is vital to unstructured content. Without someone paying attention to filtering out the garbage in unstructured content, you’re not going to get a good search result. The last thing a business needs is it’s search results working against them.”


“The jungle is dark, but full of diamonds.” said Arthur Miller. Can your search technology find the gems buried inside your own business?


For more information on how data mining and a customized search engine can move your business forward, contact us for a free consultation.


We’re at Apache Lucene EuroCon in Barcelona 2011

“We’re certain that the urgency to migrate off FAST ESP will be ramping up significantly.”

We’re very excited to be in attendance at the Apache Lucene EuroCon in Barcelona October 17, 18, 19, and 20th, 2011.  Our own Michael McIntosh, VP of Search Technologies will be presenting a talk on October 19th, Enterprise Search: FAST ESP to Lucene Solr.  The good folks at Lucid Imagination are presenting the conference and will be video recording his talk for future broadcast.

Barcelona_Logo_Shaded

After the conference, Michael will author a White Paper on migrating from FAST ESP to Lucene Solr, expected in November 2011.  For a free copy of the White Paper, email us expressing your interest at fast2solr@tnrglobal.com. We believe that those businesses operating on a Linux system will be seeking out the power of Lucene Solr as their licenses expire and support for FAST ESP dries up. We’ve worked with FAST ESP for 7 years and understand it’s strengths and weaknesses.  We know businesses who are used to the power of FAST ESP will need something just as powerful, and Lucene Solr is a very nice fit.  “It’s a robust platform, capable of a lot that FAST ESP covers,” said Michael.  “We’re certain that the urgency to migrate off FAST ESP will be ramping up significantly.”

Living with Bad Enterprise Search: The Costs of Not Finding What Your Business Needs

Search technology is critical to the mission of any business. It facilitates cash flow, revenue, Business Intelligence (BI), productivity and employee satisfaction.

Do you remember TV Guide? There was a time when TV Guide sat on nearly every coffee table in every living room in America. If you didn’t have a subscription, you would grab it in the checkout line at the grocery store every week. If you wanted to plan out your evening in front of the tube, you would pick it up, thumb through it, read the synopsis of the show, and make an informed decision about watching Dallas or Falcon Crest that evening.


Then everything changed. Not overnight, but let’s fast forward to today. If you are 20, you don’t know what TV Guide is. Most cable packages have a guide built in so you can plan your viewing, record shows you will miss, or call up ones you want to watch, even from last season. Schedules for networks are posted online. And it’s a good thing, because back when TV Guide sat on our coffee tables, there were three networks. How many are there now? Imagine how thick that TV Guide would be.


The explosion of content is not exclusive to television. Businesses have had an estimated 60% growth in digital content per year, and it shows no signs of stopping. Unfortunately, a lot of businesses haven’t upgraded their cable box, so to speak. They are looking for crucial documents and data on a manual dial. The truth is, companies have been living with bad search for a long time. And they’ve been paying for it.


The IDC estimates that 2.5 hours a day per employee are wasted looking for information they need to perform their job, or recreating that information altogether. Additionally, making sound decisions depends strongly on having valid information to make those decisions. Without access to information, bad business decisions are made, and bad business decisions are deadly to the enterprise. Business intelligence efforts can fall short without the right search platform powering fast relevant results. Worst of all, if your customers cannot find the product or service they need on your system, they will go somewhere else for it.


Content Management Systems are gaining in popularity, but what’s powering the search? How well does it deal with unstructured content? Does it give results with the relevance you need to make the best decision? Can your employees find what to need to execute their tasks? Can customers find your products?


Search technology is critical to the mission of any business. It facilitates cash flow, revenue, Business Intelligence (BI), productivity and employee satisfaction. It has an immediate impact of the bottom line of the business. It is an essential ingredient to the successful enterprise on so many levels, to run a business with inadequate search technology is like using an old copy of TV Guide to try and find and decide what to watch.

If you are assessing your search platform and it’s bottom line impact on your business, contact us.  We can analyze your systems and provide a free consultation on the best enterprise search solution for your company.

FAST ESP to Lucene Solr Presentation: Open Call for Questions

To pre-load the discussion on Michael’s Enterprise Search: FAST ESP to Lucene Solr talk, send your questions to: fast2solr@tnrglobal.com We want to hear from you!

TNR Global is excited to be participating in the Apache Lucene EuroCon conference in Barcelona.  Our own Michael McIntosh is scheduled to present:  “Enterprise Search: FAST ESP to Lucene Solr” Here is your chance to pre-load the discussion. Before Michael puts the final touches on his talk, he wants to know what issues or questions you may be have.  In the following video, he touches on some of the highlights of his upcoming talk, and asks for your input.

Enterprise Search: FAST ESP to Lucene Solr pre-conferece video - Click to Watch
Enterprise Search: FAST ESP to Lucene Solr pre-conf video

To participate in advance, send you questions or comments to:  fast2solr@tnrglobal.com.  While Michael cannot promise he will include your question or commentary in his actual talk, he will work to address them in an upcoming White Paper, to be released after the conference in November 2011. We look forward to hearing from you!