Amazon Web Services: My EBS is stuck!

Many of us were affected by the Amazon EBS issues at the end of October 2012. If you had EC2 instances in us-east-1, you were likely affected by the issues.

Many of us were affected by the Amazon EBS issues at the end of October 2012. If you had EC2 instances in us-east-1, you were likely affected by the issues. EBS volumes appeared “stuck”, snapshots would not complete, etc.
While the issues have been resolved (although we required some Amazon Support intervention for a few volumes), we have recently noticed what appear to be some vestigial issues related to the EBS outage.
The symptoms are, simply, that EC2 instances appear to be extremely slow. I/O is almost non-existent. Luckily, the fix is simple: perform a stop/start on the instance (not a restart). Your instance will be provisioned to new hardware, and you’ll have to ensure you account for a different IP address, but other than that, you’ll be back in business.
Of course- for next time- make sure that your instances are in multiple Availability Zones and Regions.
Until next time….
Many of us were affected by the Amazon EBS issues at the end of October 2012. If you had EC2 instances in us-east-1, you were likely affected by the issues. EBS volumes appeared “stuck,” snapshots would not complete, etc.
While the issues have been resolved (although we required some Amazon Support intervention for a few volumes), we have recently noticed what appear to be some vestigial issues related to the EBS outage.
The symptoms are, simply, that EC2 instances appear to be extremely slow. I/O is almost non-existent. Luckily, the fix is simple: perform a stop/start on the instance (not a restart). Your instance will be provisioned to new hardware, and you’ll have to ensure you account for a different IP address, but other than that, you’ll be back in business.
Of course- for next time- make sure that your instances are in multiple Availability Zones and Regions.
-Michael Klatsky, VP of Systems Administration

Setting JAVA_HOME on OSX lion

I recently upgraded to OSX 10.6.8 (Lion) and found that my JAVA_HOME was no longer set correctly. I found this out when my command line ec2 tools failed.

If you currently have your JAVA_HOME set to something like “/Library/Java/Home”,
under OSX Lion, you’ll want to change that to $(/usr/libexec/java_home), thus:

export JAVA_HOME=$(/usr/libexec/java_home)

The script /usr/libexec/java_home outputs the true location of JAVA_HOME, so unless that script goes away, this should prove upgrade safe.

I found this at http://steveswinsburg.wordpress.com/2011/07/22/java_home-on-os-x-lion/

Postfix SMTP AUTH w/TLS

The sysadmins at TnR Global, LLC enable email to be successfully delivered from EC2 instances, instead of being caught by Spamhaus and others.

One of the widely discussed issues with Amazon EC2 instances is the inability to reliably send email from the instances. In all too many cases, email from EC2  instances is automatically categorized as spam by the various relay databases, and by many ISP’s and carriers. There are several solutions, with the most common being a smarthost setup using either an external smarthost smtp service, such as http://authsmtp.com, or using an existing smtp server within our infrastructure. Continue reading “Postfix SMTP AUTH w/TLS”

Cloud Enabled Personalized Genomics

Focus on expertise, available, on-demand resources, and the agility to experiment with big ideas will continue to draw some personal genomics researchers to public cloud computing.

Personalized medicine is a goal of the Department of Health and Human Services. It is a driver of genomic research. It is one version of the future of medicine, using our unique genetic code toward the prevention of disease and the use of more effective or safer tailored drug therapies. Cloud computing enables access to the computational resources needed, on demand, for the data analysis needed to lay the groundwork for revolution in health care. Continue reading “Cloud Enabled Personalized Genomics”

Will Amazon AWS save me money?

One of this first questions we asked when deciding to sue Amazon’s AWS services was: Will we save money?

At first, your EC2 servers may be simply architected, perhaps small instances. Once a more serious commitment is made for a more robust architecture at Amazon, there will be additional costs introduced. For example, a small instance, running a 2 disk 100GB EBS volume w/RAID0 plus monitoring and a static IP, and the cost goes from the current ~$68 monthly per system to ~$88 monthly per system. Bump the instances to large instances (likely needed for any server running mysql, or any other equally intensive application) and that cost goes to ~$265 per instance. Add in the costs of the additional services like bandwidth, Static IP, Cloudwatch etc and the costs can quickly escalate. Of course, upfront payments for Reserved Instances can drastically reduce the costs further.

However, the savings in development and deployment costs I think far outweighs a narrower gap in the savings between physical and AWS servers, and the real MRC on the AWS servers will likely be lower for a given amount of computing resources.

So- can you save money? Yes. In some cases, it will be a direct apples to oranges savings of hard dollars. In other cases, the agility gained will provide the greatest savings. In most cases- a combination of both will drive your cost savings.

To learn more about how operating in the cloud can save your company money, contact us for a free consultation.

AMIs for Bioinformatics on AWS

Bio-Linux and other bioinformatics tools available for EC2, Amazon’s Elastic Compute Cloud, were recently highlighted on the Amazon Web Services (AWS) blog. Customized Amazon machine images (AMIs) allow for the packaging and rapid, web based deployment of the data sets and tools needed for these specialized tasks. Because AMIs can be saved, reproducing past results is simplified and because these can also be shared, the computation environment of a particular analysis can be easily replicated both from within and outside your organization.

Continue reading “AMIs for Bioinformatics on AWS”

Amazon EC2 system restore

Recently, one of our small EC2 instances failed. While we had Nagios monitoring it, Nagios only provides alerts when services fail, or when the host goes down. In this case, the failure was on Amazon’s side- the hardware where our instance resided was failing.

Recently, one of our small EC2 instances failed.  While we had Nagios monitoring it, Nagios only provides alerts when services fail, or when the host goes down. In this case, the failure was on Amazon’s side- the hardware where our instance resided was failing.

Continue reading “Amazon EC2 system restore”

Amazon EC2 ’steals’ from you

As we implement more systems in the EC2 architecture, we are noticing a not so insignificant amount cpu cycles ’stolen’. What is a ’steal’ time? It is CPU time that is taken by the Xen hypervisor for something else other than your processes- from what I have read, other people’s processes. What we need to understand is how this affects performance. Does it truly matter? We have one virtual system that consistently has steal time of between 6-12%. That would mean that 6-12% of the CPU time we pay for is being used for instances other than our own. We will have to research this more to see what the true impact is on our systems, and if there is a way to mitigate it.