Elasticsearch Evaluation

by Karen Lynn

There has been a lot of talk about Big Data and “The Big Data Problem” as of late, and several tools are emerging to manage massive data sets more effectively.

One of our largest clients has a similar data set. We have been handling multiple terabytes of data for some time, and as with most “big data problems,” the data just keeps growing. In managing their data, conditioning the content and indexing it to produce relevant search results, we have developed the procedural and administrative expertise to build and scale systems that can handle many terabytes of data.

But we see the content explosion with our client, and with many businesses across several industries.  Effectively managing big data for our clients is a top priority.

We’ve conducted an evaluation of several tools, among them elasticsearch. We plan to release a White Paper on our evalution of elasticsearch in the near future.  Here is an except from the paper we plan to release:

As many people move away from the monolithic document processing model of Solr, products like elasticsearch that lack a document processing component entirely become more attractive.  In fact, most projects that involve a data set large enough to qualify as “big data” are building their own document processing stages anyway as part of their ETL cycle.  Even if elasticsearch is only evaluated and another product is chosen, elasticsearch easily sets the bar for cloud-aware Lucene indexing.

We’re excited about the emerging tools for big data and search. Businesses will need to embrace big data strategies to unlock the valuable information stored inside the enterprise to stay competitive within their own marketplace.

Posted in Big Data, Database, ElasticSearch, business intelligence, enterprise search, search | No Comments »

#SearchFacts: The Facts of Search

by Karen Lynn


Business leaders, are you listening? If someone gave you information that you could act on before the rest of the marketplace could–information that gave your business a competitive advantage, that could affect your profitability and market share, wouldn’t you want that piece of knowledge? Of course you would! Now, what if I said that nugget of vital information was buried in the stack of papers on your desk? because it probably is.


This is the irony of the Enterprise Search market.


In evaluating the marketplace, I have observed the huge gap in efforts to promote enterprise search technology and a lack of understanding on behalf of business leaders on the importance, the benefits, and intricacy of search engines within the technology infrastructure. Business leaders often need extensive convincing that investing in a solid yet nimble search architecture is an investment that will pay off, even when your business is dependent on your users getting good search results. Business leaders are often nervous about walking out on a ledge to invest money in something that many people have come to think of as a technology that has matured to the point of being easy. It’s true that search has become commoditized to some extent. There are some great out of the box solutions that work reasonably well. But in real life, one size never fits all. Because of this wrinkle, this is the point where decision makers start to hedge a bit. There is seldom a plug in feature to index and search all of your content and give relevant search results. Most content lacks structure. Only structured content can be properly read by a search engine. We can help explain what all this means and moreover, we can help your organization fix these problems to get the search results that enable information to flow inside your business, no matter what search solution you are using.


This disconnect between enterprise search companies and between business leaders of small, mid and large sized enterprises needs a bridge, not a ledge. Over the next few weeks TNR Global will write, explain, and promote a series of facts about search technology for organizations and the enterprise called #SearchFacts. The hashtag represents it’s presence on Twitter, and we’ll be actively promoting this effort on our blog, Twitter and LinkedIn. The biggest thing we’d like to accomplish by doing this is taking some of uncertainty out of what enterprise search is and how search technology can help a business grow. Business leaders shouldn’t feel like investing in search technology is like walking out on a scary ledge, rather they should understand it’s more like crossing over to a stronger, smarter business infrastructure, which is the safest ledge to stand on.

Tags: , , , , , ,
Posted in business intelligence, enterprise search, search | Comments Off

Search System Audit

by Karen Lynn

Selecting a search platform is a big decision. You may already have a solution in place that is ineffective and you need a change. But what’s not working, and why? Which search engine is best for my needs? We perform an audit and full evaluation of your current search architecture and report our recommendations for optimizing what you already have, and list attributes of alternative technologies like Solr, LucidWorks Enterprise, ElasticSearch, SearchBlox or other solutions. Our full evaluation includes:

  1. Audit of existing enterprise search solution and evaluation of available options for improvement.
  2. Report on current performance, areas of targeted improvement, and recommendations
  3. Migration Plan. At the conclusion of the audit engagement, we provide a step by step plan on what needs to be done to migrate to a search solutions that supports your business goals and needs

Our audits are conducted by one of our senior software developers and takes place over a 3 day period. We will interviewing relevant staff members to gain valuable insights to pain points on performance. Our developer will access your back-end systems and review the following areas:

  • Review all source code, if available and applicable
  • Crawling web resources: pages and documents, forums, blogs Content processing and conversion
  • Content enhancement and extraction
  • PDF search by page
  • Search and database integration architecture design, scaling, performance, tuning, hardware requirements, and related expertise
  • Related searches, database indexing, vertical search engines, custom scoring/ranking, etc.
  • Performance troubleshooting (slow queries, high memory usage)
  • Database and file-system indexing, web crawling and searching
  • Document parsing and information extraction
  • Large-scale indexing and searching, distributed search
  • Crawling web resources: CMS based sites, blogs, forums, static html, PDFs, docs
  • Content processing and conversion
  • Content enhancement and extraction
  • PDF search results by page, mutli-page indexing, and multi-page image caching
  • Search and database integration

After this extensive review, we can better advise on which search platform to move your systems. Your business needs are weighed heavily in this evaluation, because after all, your search system is a core technology that enables knowledge management and data extraction for business decision making. We conduct search architecture audits for web portals, online directories, online databases, for the enterprise, and for virtually any web based application. Audits are the first step in making the right decision about a search platform. For more information on how a search audit can assist you in making a smarter technology decision for your organization, contact us.

Tags: , , , , , , ,
Posted in Database, ElasticSearch, LucidWorks Enterprise, SearchBlox, Solr, business intelligence, enterprise search | Comments Off

Migration from FAST to Solr White Paper to be Released

by Karen Lynn

TNR Global plans to release a White Paper on FAST to Solr Migration by the end of this week on Friday the 13th, 2012 Wednesday, January 18, 2012. The paper will build upon Michael McIntosh’s presentation at Apache Lucene EuroCon.  Newer information will be included including different tools used for implementation.

The paper will be free and available to those who sign up here.

With the dwindling support for FAST customers on a Linux operating system, TNR believes that migrating off FAST ESP to another search engine like Lucene Solr will be mission critical for many companies over the next few years.  TNR has worked with the FAST search engine for many years now and understands the qualities that many companies have grown to love and depend upon for the search needs.  We have turned to Solr to take the best features of FAST and enhance it with some of Solr’s own strengths.  It is a highly reliable, scalable, lightweight technology with fast searching and indexing speeds. Solr has powerful result relevance and ranking, faceting.  And because Solr is open source, it’s a lot easier to make adjustments to the engine–something that was always a challenge with any proprietary search engine–including FAST.

Contact us if you want to talk to us about how we can help your team migrate off FAST to Solr or another search engine.  We’re happy to assist with a short consultation, a partial, or a full implementation.

Posted in FAST ESP, Solr, enterprise search, search | Comments Off

Mobile Search: Good UX means fewer touches, simple design

by Karen Lynn

Mobile phones are rapidly taking over the scene of web development, significantly impacting commerce, advertising, gaming, entertainment, banking and news. 77% of the world’s population or 5.3 billion people are mobile subscribers. China and India lead the way in overall mobile growth. Virtually every measurable metric concerning mobile phone growth indicates entire economies being influenced by mobile technology. It’s not surprising that search technology is powering mobile growth just as it has it’s larger cousin the desktop.

Mobile search used to be clunky and a pain to use. Until recently, the answer was to miniaturize the website. For a time, people thought mobile search would never be as good as the desktop search. But, as people use their phones for more and more, it has forced designers to consider how to make search, as well as all mobile apps, simple and powerful and built for end users.

The Mobile Only World

Outside the US, countries like India, South Africa and Egypt are  leaders in mobile only--meaning users do not or infrequently use a desktop or laptop to access the Internet–making mobile search their primary mechanism for accessing queried information. Since these are also the countries sporting the most mobile growth, they are driving the need for quality relevant search for the mobile market.

Young and free

Another driving metric in the mobile game are young people. The under 25 crowd use a cell phone as their primary mode of accessing the Internet. Mobile phones, smart phones in particular, are used to do nearly everything. Younger people are more open to conducting transactions online via phone than any other demographic. Shopping, banking, GPS, social media, gaming–mobile access allows mobile subscribers to do everything they need to without restricting the user to an office.

Key differences for UX Impact

Key difference between mobile search and desktop search seem obvious. On a cell phone, the screen is much, much smaller. Users are on the go and may access the Internet between tasks or meetings, instead of being in one area. Access needs to be quick and simple. Mobile search must be designed for a minimum number of touches before users arrive at the end result. If it takes more than 2-3 touches, the user will look elsewhere for answers. Fewer touches mean a simpler design, engineered for the user without a lot of fanfare or complication.

Huge Growth

Google reports that 1 in 7 searches are now done via mobile vs. desktop. Mobile searches have increased fourfold in just the last year. Businesses need a mobile application to ensure they are reaching the inbound web traffic looking for their services and products. Mobile applications need a strong search technology to ensure the consumer can connect with the products or information they are looking for. The companies that build their web solution for the mobile market are the companies who will gain more market share and capture that 14% of customers searching for their products on the mobile web.

For the enterprise, accessing important information inside and outside the firewall is vitally important as more content is built within businesses and accessed digitally. With the mounting demand placed on mobile phones and devices, the performance we’ve come to expect from out desktop needs to be scaled to a smaller screen by simplifying wireframes with sophistication and well thought out design.

Our View
TNR Global’s expertise lies in deep back end knowledge using powerful search technologies to give users fast, relevant search results for enterprise sites and large web portals. Recognizing the need for search to work as powerfully for a mobile application as well as a web application, we have teamed up with talented UX designers specifically in the field of search application design for web and mobile. Whether you are looking for a customized UX front end for your search solution or an out of the box answer for mobile search, TNR can connect you with a total solution to answer your web based and mobile search needs. For a free consultation, contact us.

Tags: , , , , , ,
Posted in enterprise search, search | Comments Off

Selling Search Internally–Part 2–How to get buy in from the staff

by Karen Lynn

You’ve convinced the powers that be that a search solution is a necessary strategy for success and competitive advantage. Congratulations! Nice work. Think your job is done? Not by a long shot.


Ask your staff–what would a good solution look like to them? After you’ve decided to move forward with a search solution, it’s important, no–it’s crucial that you consider strongly the end user. If you have a web portal that you manage, it’s worth polling your typical customer to gather vital data on how they want their experience to be. If you are looking at an enterprise search solution, you need to spend time exploring what your staff wants and needs out of a solution, and ensure your search solution addresses design for them….not a boilerplate solution that only meets some of your needs. Search is an expensive endeavor, if you’re spending the money, you might as well get exactly what you want.


The truth is that if your end user of the solution doesn’t like the solution, they won’t use it. So getting the end user involved in the planning stage of the search project is vital to it’s overall success. If they have input to it’s overall features and design, they will be more invested in using it. Involving users manufactures all kinds of good-will collateral that can help develop better morale and a positive workplace. Doing this early in the process also introduces change more slowly to users–and people rarely react well to lots of radical change.  Making them a part of the process and doing it early with lots of prepping for change can affect overall satisfaction rates with the search implementation after it’s complete.


Once the implementation actually goes live, you’ll need to ensure a training plan is in place and executed to ensure ongoing success.  A successful search solution isn’t just done once it’s implemented.  You need to work to include your whole team in the training process, and allow them to see for themselves how the solution is going to help them in their day to day tasks. If you included your staff in the planning of the design from the beginning, you’ll be much more successful once the solution is deployed, because they were part of the solution all along.

Tags: , , , ,
Posted in enterprise search | Comments Off

Selling Search Internally–Part 1–How to get buy-in from your boss

by Karen Lynn

You are a <insert your profession here>  (Department Head, IT Leader, Operations Regional Manager, HR Manager, End User) in your business or organization. You have a problem. You can’t find information. Your staff is spending time tracking down that invoice from a few years ago, looking for the part number that a customer needs, searching for that great resume. It’s somewhere….but where? You see your staff is frustrated, disenchanted, defeated. You see that time is being wasted, and customers are grouchy because they can’t access products or information you have online. Maybe you’re losing customers. It’s hard to tell because you just got another 20 emails since you checked an hour ago and there are 10 reports on your desk awaiting your review. You are awash in information–drowning, and you are supposed to be in charge of keeping all this organized. Sound familiar?

Search solves this. Search, discovery, sharing information…it all leads to faster service to customers, less staff frustration, and higher productivity. It has been said that a good search solution either saves you money inside the firewall or makes you money outside the firewall. Either way, your organization is more competitive with a search solution that delivers the right information at the right time.

But how do you convince the boss that your company would benefit in an investment in search technology? This is the tricky part for many managers inside organizations. Search is hard, and often expensive. Here are the main points you should make with your boss.

  • Search isn’t a box, it’s an engine
  • Search makes money / saves money = bottom line results
  • Sharing information promotes better decision making, faster response time
  • Search will give your organization a competitive edge in a cutthroat marketplace

This isn’t a single conversation. The most successful campaigns for better search technology involve many voices, not just yours. An organized vocal group inside your organization who can present business leaders with a solution that will effect the bottom line is hard to ignore. And the basics in selling any idea to your boss should also be minded, such as:

  • If you come with a problem, come with a solution
  • Give real examples to back up your suggestions for improvement
  • Be diplomatic: even the best organizations can be political
  • Maintain relationships (don’t throw anyone under the bus)
  • Who will benefit and how?  Break it down for consideration

Change is tough, but I find that this saying by B.C Forbes sums up why it’s important to push the issue “If you don’t drive your business, you will be driven out of business.”

Posted in System Administration | Comments Off

Search and Steel Girders

by Karen Lynn

“Search ties people together…”
This was one of the many themes at the Enterprise Search Summit in Washington, DC last week. It seems like a fairly obvious statement, but it quickly becomes part of the landscape, taken for granted even though the landscape couldn’t function without it. I have compared search function to the steel girders of a skyscraper. When you walk into the building, you aren’t thinking about the beams holding the building up or connecting floors, but without them, you wouldn’t have a building at all (you couldn’t even find the lobby). Other metaphors overheard include oxygen (invisible yet essential), sunlight (lest we remain in the dark) and electricity (everything stops without it).
Attendees of the conference know how important search is to companies, but increasingly, companies are taking search for granted. There is a fundamental gap in communicating the importance and difficulty of implementing a good search platform.
Companies who need search to run on their website or intranet, expect search to work as it does on the Internet, but this is an apples and oranges scenario.
Here are the main disconnects:

  1. Search is easy
  2. Search is cheap
  3. It never has to be touched again

People expect search inside the firewall to function much like Google does outside the firewall. Google exists for end users and is really, really incredible. It Geo-locates, it auto-completes. It uses your browsing history to provide more relevant results. And you had no financial investment in using this really lovely, elegant, useful tool that doesn’t just assist your Internet experience, but facilitates it. But behind the firewall, things are different. Let me explain.

  • Your business content isn’t publicly available or known. I mean, that would be bad, right? It’s behind the firewall for a reason. So keeping it there yet allowing your staff to access certain levels of information takes some architecture and planning.
  • Google has thousands of developers working on this beautiful, incredible technology every day. They finance this by ad content. How many people do you have on your search team? And how much of their day do they really spend on search? What department is being billed for it? Business leaders need to embrace this as a necessary cost of doing business and budget accordingly, or face the crippling result of staff and customers not being able to find the information they need.
  • 80% of your content is unstructured. Meaning, search engines can’t really read it until some love and care is put into cleaning the data. This is a vital, yet time intense process. Our VP of Search Technologies Michael McIntosh says “We spend about 90% of our time on the document processing pipeline, conditioning data to be fed into the engine.” Moreover, unstructured data isn’t a set number. It’s being creating faster than you can blink by your entire enterprise. Processing it is never a done deal.

So if search connects us, hopefully this finds you thinking about search in more realistic terms. Search by itself may look like a simple box, but behind the box is a foundry of girders, cross beams, and structural support that allows you to find what you need to “make money outside the firewall or save money inside the firewall.”

Tags: , , , , ,
Posted in enterprise search, search | Comments Off

Search Fuels Business Intelligence for Decision Making

by Karen Lynn

“The jungle is dark, but full of diamonds.” said Arthur Miller. The same can be said about the invaluable data inside your business. It’s there, ready to be mined. But unless you have the right tools, you’ll never get to those diamonds.

Content is expanding at an exponential rate. I don’t know anyone in any business who can keep up with the pace of content growth, without the use of powerful search engines to find and extract relevant information. Business analysts expect content to grow 800% over the next 5 years. Business intelligence requires extraction of the right information, and most enterprises have both structured and unstructured data. Structured data is easy for most search engines to search. The rub is in unstructured content–of which there is abundance. Unstructured content is said to account for 70-80% of data in all organizations. This type of content is often in the form of documents, email messages, health records, HTML pages, books, metadata, audio, video, and various other files. All these files have to be “cleaned up” before feeding them through a search engine in order to get results with any kind of value or relevance.

Mining this data is going to be essential for not just the success, but the survival of many businesses. James Kobielus, an analyst at Forrester Research, reports in an interview with ComputerWorld that businesses will increasingly turn to a self-service BI throughout 2011 and beyond. “Increasingly, enterprises will adopt new Web-based interactive querying and reporting tools that are designed to put more data analytics capabilities into the hands of end users,” he said. A good search engine that can find data quickly and easily can “take the burden off IT and speed up the development of reports to a considerable degree,” Kobielus said. The information mined by a search engine tuned to the specific business needs facilities better decision making for people a every job function within the enterprise. “Because every business is a little different, and so many organizations house so much unstructured content, most search engines can’t cover everything that is needed without some customization” said Michael McIntosh, our VP of Search Technologies at TNR Global. “Data conditioning is vital to unstructured content. Without someone paying attention to filtering out the garbage in unstructured content, you’re not going to get a good search result. The last thing a business needs is it’s search results working against them.”

“The jungle is dark, but full of diamonds.” said Arthur Miller. Can your search technology find the gems buried inside your own business?

For more information on how data mining and a customized search engine can move your business forward, contact us for a free consultation.

Tags: , , , , , , , , , , , ,
Posted in business intelligence, enterprise search, search | Comments Off

Living with Bad Enterprise Search: The Costs of Not Finding What Your Business Needs

by Karen Lynn

“The jungle is dark, but full of diamonds.” said Arthur Miller. The same can be said about the invaluable data inside your business. It’s there, ready to be mined. But unless you have the right tools, you’ll never get to those diamonds.

Content is expanding at an exponential rate. I don’t know anyone in any business who can keep up with the pace of content growth, without the use of powerful search engines to find and extract relevant information. Business analysts expect content to grow 800% over the next 5 years. Business intelligence requires extraction of the right information, and most enterprises have both structured and unstructured data. Structured data is easy for most search engines to search. The rub is in unstructured content–of which there is abundance. Unstructured content is said to account for 70-80% of data in all organizations. This type of content is often in the form of documents, email messages, health records, HTML pages, books, metadata, audio, video, and various other files. All these files have to be “cleaned up” before feeding them through a search engine in order to get results with any kind of value or relevance.

Mining this data is going to be essential for not just the success, but the survival of many businesses. James Kobielus, an analyst at Forrester Research, reports in an interview with ComputerWorld that businesses will increasingly turn to a self-service BI throughout 2011 and beyond. “Increasingly, enterprises will adopt new Web-based interactive querying and reporting tools that are designed to put more data analytics capabilities into the hands of end users,” he said. A good search engine that can find data quickly and easily can “take the burden off IT and speed up the development of reports to a considerable degree,” Kobielus said. The information mined by a search engine tuned to the specific business needs facilities better decision making for people a every job function within the enterprise. “Because every business is a little different, and so many organizations house so much unstructured content, most search engines can’t cover everything that is needed without some customization” said Michael McIntosh, our VP of Search Technologies at TNR Global. “Data conditioning is vital to unstructured content. Without someone paying attention to filtering out the garbage in unstructured content, you’re not going to get a good search result. The last thing a business needs is it’s search results working against them.”

“The jungle is dark, but full of diamonds.” said Arthur Miller. Can your search technology find the gems buried inside your own business?

For more information on how data mining and a customized search engine can move your business forward, contact us for a free consultation.

Tags: , , , , , , , ,
Posted in System Administration, enterprise search, search | Comments Off

« Older Entries