Mountain Lion Roars….and Leaves Some Blood

One issue I ran into when installing Virtualbox revolved around Apple’s software installation security in Mountain Lion

Feeling adventurous yesterday, I decided to upgrade my Mac AIR to Mountain Lion. Other than a fairly long download time, which is to be expected on the release day, the upgrade went fairly smoothly.
Until…
I opened iTerm. I use virtualenv & virtualenvwrapper to manage multiple Amazon accounts and python environments. The following error was thrown:
ImportError: No module named virtualenvwrapper.hook_loader
Luckily, this was easy. To fix, just do the following:
sudo easy_install pip
sudo pip install virtualenv virtualenvwrapper
Once done, open another iTerm window. You may see several directories created, such as “virtualenvwrapper.user_scripts creating /Users/mklatsky/.virtualenvs/premkproject”.
So far so good.
But- where is java?
“java -version” throws the error:
No Java runtime present, requesting install.
Again- easy. Just the act of issuing the “java -version” command launches a dialog box asking if I’d like to install or upgrade java. Five minutes later and I am back in business.
And then…
I use Virtualbox (actually Vagrant and Veewee). Attempting to launch one of my boxes resulted in a kernel panic, which rebooted my computer. Really? A kernel panic? At any rate- a little Googling, and an install of the latest Virtualbox (4.1.18) and I was all set.
One issue I ran into when installing Virtualbox revolved around Apple’s software installation security in Mountain Lion. There are now 3 levels of security for installable software: Mac App Store, Mac App Store and identified Developers and Anywhere.
Unfortunately, Oracle is not an “identified Developer”, so installation is impossible under the default settings. You’ll need to go to System Preferences->Security & Privacy ->General to change this. Then you can install Virtualbox. I’d recommend changing this setting back to the default after you are done.
I’ll conclude by stating that so far, Mountain Lion seems to be performing nicely, with few issues other than the above noted.
For further reading, check out the articles below:
http://arstechnica.com/apple/2012/07/os-x-10-8/
http://www.jongales.com/blog/2012/07/25/fixing-virtualenv-after-installing-mountain-lion/
https://www.virtualbox.org/ticket/10267

Our VP of Systems Administration Michael Klatsky discusses the latest OS upgrade from Apple–Mountain Lion.  The following is a repost from his blog which can be viewed here along with other articles.

_____________________________________________________________________________________________________

Feeling adventurous yesterday, I decided to upgrade my Mac AIR to Mountain Lion. Other than a fairly long download time, which is to be expected on the release day, the upgrade went fairly smoothly.

Until…

I opened iTerm. I use virtualenv & virtualenvwrapper to manage multiple Amazon accounts and python environments. The following error was thrown:

ImportError: No module named virtualenvwrapper.hook_loader

Luckily, this was easy. To fix, just do the following:

sudo easy_install pip

sudo pip install virtualenv virtualenvwrapper

Once done, open another iTerm window. You may see several directories created, such as “virtualenvwrapper.user_scripts creating /Users/mklatsky/.virtualenvs/premkproject”.

So far so good.

But- where is java?

“java -version” throws the error:

No Java runtime present, requesting install.

Again- easy. Just the act of issuing the “java -version” command launches a dialog box asking if I’d like to install or upgrade java. Five minutes later and I am back in business.

And then…

I use Virtualbox (actually Vagrant and Veewee). Attempting to launch one of my boxes resulted in a kernel panic, which rebooted my computer. Really? A kernel panic? At any rate- a little Googling, and an install of the latest Virtualbox (4.1.18) and I was all set.

One issue I ran into when installing Virtualbox revolved around Apple’s software installation security in Mountain Lion. There are now 3 levels of security for installable software: Mac App Store, Mac App Store and identified Developers and Anywhere.

Unfortunately, Oracle is not an “identified Developer”, so installation is impossible under the default settings. You’ll need to go to System Preferences->Security & Privacy ->General to change this. Then you can install Virtualbox. I’d recommend changing this setting back to the default after you are done.

I’ll conclude by stating that so far, Mountain Lion seems to be performing nicely, with few issues other than the above noted.

For further reading, check out the articles below:

http://arstechnica.com/apple/2012/07/os-x-10-8/

http://www.jongales.com/blog/2012/07/25/fixing-virtualenv-after-installing-mountain-lion/

https://www.virtualbox.org/ticket/10267

Bishop: Makes Your Web Service Shiny

The idea is to provide a relatively small library that will make your life easier and hopefully more pleasant by making it straightforward to provide a consistent web service API that obeys HTTP semantics.

Christopher Miles, one of our Senior Software Developers here at TNR, wrote this post on Bishop.  It all started when he was asked the question:

“What happens if I send it something that’s not JSON?”…….“I don’t know, but I bet it logs a really big stack trace!”

The question begged an answer, and Chris give an extremely thorough answer in his own blog.  Here’s a small except that gives a taste of his analysis:

After taking a closer look at HTTP and it’s specification, it was clear that it could do a lot more than I had thought. Looking back on past projects, it’s painfully obvious that I’ve been taking what is really an application protocol and ignoring all of the interesting bits, instead using it as little more than a pipe to push documents through. I’ve been using either the requested URL or parameters or maybe even neither and simply examined the body content, thus eliminating any of the real advantages of using HTTP in the first place.

And there are advantages. The protocol is already thinking about caching your data where it makes the most sense. There’s already an algorithm for taking the list of content types that the client wants and the content types the server provides and picking the best match. It can manage safe updating of resources as well as notifying the client of conflicts. And so on, by ignoring what the HTTP protocol has to offer I was making more work for myself.

So he decided to take matters into his own hands by creating a library. More from Chris’ blog:

The idea is to provide a relatively small library that will make your life easier and hopefully more pleasant by making it straightforward to provide a consistent web service API that obeys HTTP semantics. It will make the lives of those around you easier as well, clients can expect your service to respond to the common HTTP request methods with reasonable responses. Placing caches around your service will also be much simpler and you’ll have some level of control over how your service’s data is cached.

Since creating this library, other developers have responded positively and are watching the project.  If you would like to take a look at how our approach to solving this problem, take a look for yourself here.

If you’d like to talk to us on how we can solve some of your enterprise search, cloud, or scalability issues, contact us.

New Systems and DevOps Blog

has lots of new approaches to discuss in terms of systems, cloud computing, DevOps, System Architecture, and how developers and systems staff need to communicate well and work together

Our VP of Systems Administration, Michael Klatsky has started a blog specifically discussing Systems.  Fresh from the AWS Summit 2012 in NYC, Michael has lots of new approaches to discuss in terms of systems, cloud computing, DevOps, System Architecture, and how developers and systems staff need to communicate well and work together for the best results in web development.  The blog is his own but we feel it’s a great technical resource for our colleagues in systems and web development.  You can take a look at his blog here. Michael welcomes commentary and discussion, and hopes to provide some shortcuts for fellow System Administrators.

Selling Search Internally–Part 1–How to get buy-in from your boss

“If you don’t drive your business, you will be driven out of business.” –BC Forbes

You are a <insert your profession here>  (Department Head, IT Leader, Operations Regional Manager, HR Manager, End User) in your business or organization. You have a problem. You can’t find information. Your staff is spending time tracking down that invoice from a few years ago, looking for the part number that a customer needs, searching for that great resume. It’s somewhere….but where? You see your staff is frustrated, disenchanted, defeated. You see that time is being wasted, and customers are grouchy because they can’t access products or information you have online. Maybe you’re losing customers. It’s hard to tell because you just got another 20 emails since you checked an hour ago and there are 10 reports on your desk awaiting your review. You are awash in information–drowning, and you are supposed to be in charge of keeping all this organized. Sound familiar?

Search solves this. Search, discovery, sharing information…it all leads to faster service to customers, less staff frustration, and higher productivity. It has been said that a good search solution either saves you money inside the firewall or makes you money outside the firewall. Either way, your organization is more competitive with a search solution that delivers the right information at the right time.

But how do you convince the boss that your company would benefit in an investment in search technology? This is the tricky part for many managers inside organizations. Search is hard, and often expensive. Here are the main points you should make with your boss.

  • Search isn’t a box, it’s an engine
  • Search makes money / saves money = bottom line results
  • Sharing information promotes better decision making, faster response time
  • Search will give your organization a competitive edge in a cutthroat marketplace

This isn’t a single conversation. The most successful campaigns for better search technology involve many voices, not just yours. An organized vocal group inside your organization who can present business leaders with a solution that will effect the bottom line is hard to ignore. And the basics in selling any idea to your boss should also be minded, such as:

  • If you come with a problem, come with a solution
  • Give real examples to back up your suggestions for improvement
  • Be diplomatic: even the best organizations can be political
  • Maintain relationships (don’t throw anyone under the bus)
  • Who will benefit and how?  Break it down for consideration

Change is tough, but I find that this saying by B.C Forbes sums up why it’s important to push the issue If you don’t drive your business, you will be driven out of business.”

Living with Bad Enterprise Search: The Costs of Not Finding What Your Business Needs

Search technology is critical to the mission of any business. It facilitates cash flow, revenue, Business Intelligence (BI), productivity and employee satisfaction.

Do you remember TV Guide? There was a time when TV Guide sat on nearly every coffee table in every living room in America. If you didn’t have a subscription, you would grab it in the checkout line at the grocery store every week. If you wanted to plan out your evening in front of the tube, you would pick it up, thumb through it, read the synopsis of the show, and make an informed decision about watching Dallas or Falcon Crest that evening.


Then everything changed. Not overnight, but let’s fast forward to today. If you are 20, you don’t know what TV Guide is. Most cable packages have a guide built in so you can plan your viewing, record shows you will miss, or call up ones you want to watch, even from last season. Schedules for networks are posted online. And it’s a good thing, because back when TV Guide sat on our coffee tables, there were three networks. How many are there now? Imagine how thick that TV Guide would be.


The explosion of content is not exclusive to television. Businesses have had an estimated 60% growth in digital content per year, and it shows no signs of stopping. Unfortunately, a lot of businesses haven’t upgraded their cable box, so to speak. They are looking for crucial documents and data on a manual dial. The truth is, companies have been living with bad search for a long time. And they’ve been paying for it.


The IDC estimates that 2.5 hours a day per employee are wasted looking for information they need to perform their job, or recreating that information altogether. Additionally, making sound decisions depends strongly on having valid information to make those decisions. Without access to information, bad business decisions are made, and bad business decisions are deadly to the enterprise. Business intelligence efforts can fall short without the right search platform powering fast relevant results. Worst of all, if your customers cannot find the product or service they need on your system, they will go somewhere else for it.


Content Management Systems are gaining in popularity, but what’s powering the search? How well does it deal with unstructured content? Does it give results with the relevance you need to make the best decision? Can your employees find what to need to execute their tasks? Can customers find your products?


Search technology is critical to the mission of any business. It facilitates cash flow, revenue, Business Intelligence (BI), productivity and employee satisfaction. It has an immediate impact of the bottom line of the business. It is an essential ingredient to the successful enterprise on so many levels, to run a business with inadequate search technology is like using an old copy of TV Guide to try and find and decide what to watch.

If you are assessing your search platform and it’s bottom line impact on your business, contact us.  We can analyze your systems and provide a free consultation on the best enterprise search solution for your company.

Building for Enterprise Search: A Systems View, Part 2

“It’s important to incorporate expected behaviors into modeling and monitoring on both applications and systems sides and how they interact with one another.”

When we left off, Michael Klatsky, VP of Systems Administration was telling me how important communication between the systems side and search side of is to developing an enterprise search solution. The process of building, testing, monitoring, adjusting, more testing, and more monitoring ensures systems function that way they are intended to function. Let’s resume our conversation where Michael discusses the tools he uses to ensure the system he’s building works the way the client wants it to. This is the second portion of a two part blog post.
*********************************************************************************************************************
Tools for BDD: Part 2

Karen: It’s sounding like the Search Team and Sys Admin Team need to have a good relationship and communicate often to ensure the system will accommodate the work the search team does.

Michael: Yes, search sometimes has to construct their scripts to conforms to systems. Testing is run on both sides, but small changes can affect others down the line, so it’s important to incorporate expected behaviors into modeling and monitoring on both applications and systems sides and how they interact with one another.

Karen: How do you make sure that happens?

Michael: We’re exploring some tools to help us make sure the machine will act just as we expect it to, like cucumber and cucumber nagios We’re using certain tools to facilitate the systems behaves in the way that we expect it to. We’re exploring cucumber for basic modeling and for testing. Cucumber is cool for testing because it returns values to you in colors. Red, meaning it failed, yellow meaning there’s problem, and green meaning its good. According to their docs, they instruct you to “keep running it until it’s a cucumber.”

Karen: Ah, I get it.

Michael: Right. And what cucumber nagios does is it takes cucumber and allows you to create a nagios monitoring check script. So if you pass, great, if you god red, nagios will throw an alert to the systems administrator so we have an opportunity to fix it before more is built.

Karen: Sounds like it’s an attentive way to build a system.

Michael: The only way to scale is to have machines do things for themselves. That’s the way to do it.

Karen: To automate.

Michael: Yes. Automation. Not to just set things up to automatically do configuration management beforehand, but to test afterwards to determine that your machine is behaving just as you (and your client) envisioned it.

For more information on how you can plan your enterprise search in cooperation with your systems administration team, contact us for a free consultation.

Building for Enterprise Search: A Systems View, Part 1

We need to determine what right looks like, and have the system behave that way.

I sat down with our VP of Systems Administration, Michael Klatsky to discuss some of his thoughts on how Systems Administration needs to work in concert with the Search Team to implement search technologies for clients. This is the first portion of a two part blog post.

**********************************************************************************************************************
Karen: You wanted to discuss how your approaching the systems side of search, and using a Behavior Driven Development (BDD) approach. Tell me about that.

Michael: Well, one of the problems we run into when systems brings up machines for enterprise search clusters is the search software (FAST ESP for example) is very particular about it’s environment- more so than many of the more common applications such as the Apache webserver. Properly configured DNS, specific environment variables, specific library versions have to be present. There are ownership and permissions that need to be in place, and performance metrics that must adhere to a given baseline. There can be slow disks can affect performance. There has to be the right amount of memory, and different classifications of systems roles. Currently, we have homegrown scripts that bring up systems, then we have other scripts we run to detect issues. These scripts will tell us if the system is ready for what we need it to do. We also monitor the systems for standard items such as diskspace, memory usage, as well as basic search functionality. For example we’ll run a quick search on say paper clips, and if comes back with results we know it’s running.

That’s what we’ve done historically. But now, we need to bring up larger numbers of machines,and have confidence that they will perform exactly as we expect. Additionally, we have a set of functional tasks that must be available without fail As we bring up clusters of larger numbers of machines, and as we need to be more nimble, how can we ensure that it will respond the way we expect it to?

Karen: This is where Behavior Driven Development comes in, right?

Michael: Right. There is a lot of discussion out there on Behavior Driven Development which would include behavior driven modelling, behavior driven monitoring, behavior driven architecture and infrastructure. So not only does a machine come up and is listening on these ports, but I can bring a machine up, I can go to that machine and I’m able to log in, install certain software, and peform tasks. I can go to another machine and perform a task. So, the question is, how do you model that? How do we ensure the system will behaves as it should?

Karen: So you’re looking at replicating the behavior of these systems so that every time we deploy something it will be the same way.

Michael: Right. And if a change is made, even a small change, we’ll see it right away because a system or service will fail and be able to fix it. Sometimes a service will fail silently. But we test and monitor constantly to ensure the system will do exactly what we expect it to do. It’s all a part of the build process.

Karen: Sounds like a smart approach.

Michael: Yes. And if we make a change, we’ll find out how that change will affect the rest of the system. For instance, we run tests and if something is wrong it should give you an error. For example if you change the location of your SSH keys. You may still be able to get into the machine by SSH, but one little change could make it impossible to SSH from one machine to another in the cluster. So rather than find that out after you begin your manual work on that, we make it part of the build process by constantly monitoring and testing the system as we build it.

Karen: It sounds like building a house and then realizing you have bricks out of place after it’s built.

Michael: Worse, it’s like building a house and realizing you forgot to build a door! At the very least while you are building, you can test, and let me know, “Hey! I don’t have a door to my house!” So that I can fix it before you move in.

There are certain things the search team needs to do to ensure their work will function in the system, like SSHing around the machines in the cluster–they need to be able to do that. There are certain ports that system need to be listening on, there are certain services that need to return a normal range of results. We need to define what a proper operation looks like. We can’t necessarily say that if we search for gold plated paperclips for example, that the search result should show 1000 results every time, that may or may not be the case–we don’t necessarily know if this is a proper result every time, but we should determine if the result returned is within a proper range of normal.

We’re defining what a proper operation looks like and ensure it functions that way. Part of the behavior driven model which is what I’m really interested in, we can set up a natural language looking config file. This config file should describe the actions or behaviors I expect. For example, when I go to ABC.com website and search for gold plated paperclips, I expect to see results. One result should be X. There should be more than Y results. When I return that result, I should be able to click on one result and go to that products feature list. Basically I’m describing how the customer will interact with the search, what I expect the customer to do, and design the system to respond with the customer’s actions in mind.

Karen: So your engineering it with the customer’s behaviors in mind.

Michael: That’s exactly what we’re doing. Then that if I look for a certain item, I get that result, describe the behavior of what the customer should do and make the system behave in cooperation with the customer behavior. We need to determine what right looks like, and have the system behave that way.

Karen: And what right looks like is really different for each client.

Micheal: Yes. You can write in somewhat natural English what that looks like. It’s not magic, but you still have to come up with specification of what right looks like. But you can do a lot of sophisticated things in this manner because you will know you’ll have a website that’s going to perform the way it’s suppose to perform. The bottom line is: Define what your systems should “look” like, deploy those systems using those definitions, and after deployment, test to ensure that those systems “look” like your definition.

For more information on how you can plan your enterprise search in cooperation with your systems administration team, contact us for a free consultation.

Open Source Search: Isn’t It Expensive?

You’ve heard the debate on open source search vs. proprietary search. One question that constantly comes up for prospective clients is “What’s all this going to cost me?”

In these times, it’s a good question. Because proprietary has neatly packaged, practically shrink wrapped plans, it’s much easier to discern how much you will spend on a solution. But how much will it cost? That’s an entirely different question.

I see you cocking your head sideways.

Proprietary search has hidden costs. What if the software doesn’t perform the way you need it to? Does the software understand the nuances of your business? How adaptable is it? How much will it cost to adapt that software to get it to perform the way my business needs it to? Questions like this need to be asked, and answered. Eventually you will ask yourself….why am I paying for all of this? And your developer will ask, “why can’t I access the source code?”

What I’m getting at is this: It is a reassuring feeling for a customer to see what a package costs, to understand what services you will get with a solution, and to anticipate what the licensing fee will cost on an annual basis. If it’s your job to research a solution and present findings to your executive team to make a decision, then proprietary search, on the surface, seems a more secure choice. But rarely, if ever, are these solutions a perfect fit for the customer. It’s like buying a Ferrari, with all the brand recognition and polish a Ferrari offers, and not ever driving it past second gear, or cutting the wheel more than 15 degrees, or getting a chance to have your trusted mechanic look under the hood. This is why open source is such a good solution for businesses who want their IT to move quickly.

We’re hearing more buzz about companies waking up to the agility of an open source solution. Most recently, with the acquisition of Autonomy by HP, the industry is telling stories of ex Autonomy customers migrating to Solr (open source search) with only the annual licencing budget to finance the migration. Without an annual expenditure of cash for licensing, and the freedom of not being under a licensing agreement, companies quickly recoup the initial expenditure of a migration.

What kind of car does your company drive?

If you are examining the different choices for implementing search technology in your organization, contact us.  We’re happy to talk to you about the best solution for your business.


Setting JAVA_HOME on OSX lion

I recently upgraded to OSX 10.6.8 (Lion) and found that my JAVA_HOME was no longer set correctly. I found this out when my command line ec2 tools failed.

If you currently have your JAVA_HOME set to something like “/Library/Java/Home”,
under OSX Lion, you’ll want to change that to $(/usr/libexec/java_home), thus:

export JAVA_HOME=$(/usr/libexec/java_home)

The script /usr/libexec/java_home outputs the true location of JAVA_HOME, so unless that script goes away, this should prove upgrade safe.

I found this at http://steveswinsburg.wordpress.com/2011/07/22/java_home-on-os-x-lion/

Cloud Platforms: The Promise vs. The Reality

Recently our VP of Search, Michael McIntosh sat down and talked to me about his thoughts on cloud computing and what businesses should be aware of when investing in the cloud.


Karen: So, how does enterprise search and cloud computing fit together?  What’s good about it for companies?

Michael: The advent of cloud computing makes it a lot easier for companies to get into search without investing a huge sum of money up front. Some of the pay-as-you-go computing approaches make it possible to do things that in the past wouldn’t have been financially viable such as natural language processing on content.  Something that could have taken days, weeks, or even months can now take much less time by throwing more hardware at a problem for a shorter time span.

For example, you could throw 20 machines at a problem for 12 hours and do a bunch of computations in a massively parallel way, and then stop it as soon as it’s done….versus the old model where you have to buy all the hardware, or rent it, and make sure it’s not underutilized so you make your investment back.

But if you need a lot of processing power for a short amount of time, it’s really quite amazing what we can do now with an approach like this.

Karen: Is this a new technology for TNR?

Michael: TNR has been using cloud computing platforms for several years now—3 or 4 years.  Cloud computing in itself is sort of a buzz word, because distributed processing and hosting has been around for a while, but the pay-as-you-go computing model is relatively new. So we have a great deal of experience with the reality of cloud computing platforms vs. the promise of cloud computing platforms.

Karen: So, what is the difference between the “promise” and the “reality” of cloud computing platforms?

Michael: Well, A lot of people think of cloud computing as this magical thing; all their problems will be solved and it will be super dependable because there are very large businesses like Amazon running the underlying infrastructure and you don’t have to worry about it.

But, as the physical infrastructure becomes easier to deploy, other critical factors come into play. You won’t have to worry about the physical logistics of getting hardware in place. But, you will have to manage multiple instances, you have to make sure that when you provision temporary processing resources, you have to remember to retire it when it’s no longer needed. Otherwise you’ll be paying more than you need to. Since virtualization uses physical hardware you do not control or maintain—there are fewer warning signs to a potential systemic failure. Now Amazon, which is the one we use the most, does a good job of backing up instances and making things available to you even when there are failures. But we’ve had problems where we’ve lost entire zones. Even if we’ve had multiple machines configured with fault tolerance, Amazon has experienced outages that have taken entire regions offline despite every conservative effort to ensure continuous up time. So we’ve had our entire service clusters go down because of problems Amazon was having. It becomes critically important for companies to develop and maintain a disaster and recovery plan. Companies need to make sure things that are critical are backed up in multiple locations. Now historically, this has been hard to do because companies typically buy enough equipment for production needs, but not enough equipment for development and staging environments.

Karen: That sounds like a costly mistake.

Michael: It can be very costly because people often develop disaster recovery plans without ongoing testing to confirm the approach continues to work. If the approach is flawed, when you do suffer an outage, you can be offline for hours, days or weeks. Even worse, you may not be able to recover your data at all.

Karen: That sounds extremely costly.

Michael: Yes, it’s no fun at all.

There are upsides though. Some pluses are that cloud computing forces you to be more formal about how you manage your technical infrastructure. For example, for training purposes; with a new developer, we can just give them a copy of a production system, and have them go to town on it, make modifications, whatever without risking the actual production servers. And if they make a mistake, which is human (you have to factor in human error), you can reprovision a brand new one, and retire the one that is fouled up. Instead of having to spend hours and hours trying to fix the problem on the machine they were working on.

Karen: This sounds like it’s a lot more flexible and time efficient, with a layer of safety built in.

Michael: Yes. Cloud computing also comes in handy if you ever have a security breach. If a hacker gets into the system and the system is compromised–if this happens, system administrators can go in and try to correct the problem. But hackers can often install backdoors to get in and out. So a cloud platform with a good disaster contingency and backup can allow system administrators to bring a whole instance down and do the patch on a whole new machine without the security breaches and patches in place. This is pretty easy to do with a cloud platform.

Karen: So TNR can help their clients do all these things?

Michael: Yes, we’ve worked with large customers over many years and we’ve seen a wide variety of things that can possibly go wrong, and we’ve been through several physical service outages both with Amazon Web Services and with Rackspace.

Cloud computing in itself is no panacea, but if you have the technical and organization proficiency to effectively leverage the platform, it can be a powerful tool used to accelerate your company’s rate of innovation.

If you are assessing the cloud as a solution in your business, contact us.  There are a variety of options for hosting that can save your company money and minimize outages. Let us show you the option that is the best fit for your organization.