Category: Blogs

Transparent MySQL migration using MySQL proxy

How can we transparently migrate MySQL from one server to another when we don’t want to disrupt end users? That was the question posed as we come to the final phase of decommissioning a server. We have transitioned almost all services away from the older server (CHIMAY)- but there is one external cron that is not under our control that we can see in the logs which generates several MySQL queries. Therefore- we need to transparently move MySQL through another server (TECATE). Here’s the scenario: Continue reading “Transparent MySQL migration using MySQL proxy”

Identifying bottlenecks on your server

As heavy users of the LAMP stack for our applications, we of course find that various systems are not performing as expected. We have one webserver (part of an application cluster) that often spikes loads that seem to be unrelated to the actual traffic on the machine. For example, we may have 80 httpd requests, yet the load on the machine is 8 or 9. So- how do we begin to identify where the bottleneck exists? Typically, those bottlenecks can be narrowed to two places- CPU and I/O (disk). We can check our system with a couple of tools to identify where the problem is: vmstat and iostat. Continue reading “Identifying bottlenecks on your server”

Amazon EC2 ’steals’ from you

As we implement more systems in the EC2 architecture, we are noticing a not so insignificant amount cpu cycles ’stolen’. What is a ’steal’ time? It is CPU time that is taken by the Xen hypervisor for something else other than your processes- from what I have read, other people’s processes. What we need to understand is how this affects performance. Does it truly matter? We have one virtual system that consistently has steal time of between 6-12%. That would mean that 6-12% of the CPU time we pay for is being used for instances other than our own. We will have to research this more to see what the true impact is on our systems, and if there is a way to mitigate it.

FAST ESP Overview

We use FAST ESP to power a large industrial search engine listing over 1 million companies and over 3 million indexed documents and receiving millions of visitors every month. I have been working with ESP since 2003 (then known as FDS 3.2).

FAST ESP is extremely flexible and can deal with indexing many document types (html, pdf, word, etc). It has a very robust crawler for web documents and you can use their intermediary FastXML format to load custom document formats into the system or use their Content APIs. Continue reading “FAST ESP Overview”

Working with missing Makefile.PL scripts in CPAN modules

Some versions of CPAN (notably, the one that ships with Red Hat / CentOS 4.6, v1.7601) will automatically attempt to create a ‘Makefile.PL’ script if a module you’re trying to install through CPAN is missing one. However, sometimes this may lead to an error during the module installation, usually something like: “Too early to specify a build action ‘Build’.  Do ‘Build Build’ instead.”. This appears to be caused by a missing argument to ExtUtils::MakeMaker’s WriteMakefile subroutine. If the ‘PL_FILES’ argument doesn’t exist, MakeMaker will incorrectly attempt to use the Build.PL file included with the module. Continue reading “Working with missing Makefile.PL scripts in CPAN modules”

Per-directory access restrictions with Trac and Subversion (mod_svn)

The below assumes that you are requiring a login to access Subversion over http(s) and Trac, and that the credentials users use for each service are the same.

Your access control file:

Both Trac and mod_svn use the same access control file format, so you should only need one file for both. The file format is described in some detail here, but I’ll go over some basics. Continue reading “Per-directory access restrictions with Trac and Subversion (mod_svn)”

Busy holidays, and back to blogging

It’s been a long, busy holiday- and now time to resume blogging. My latest interest is testing our mysql servers using mysqlslap. While a great tool, it is unfortunately only distributed with MySQL 5.1.4 and above. However, many of our servers are in the 5.0 release, with some in the 4.x release. What I did was grab the latest 5.1.x version of mysql, compile it statically and test it out on a MYSQL 5.0.45 version, and it worked just fine. I have yet to test it on another machine, or against a version other than 5.0.45. I’ll update when I do.

Here are the steps I took:

1) wget “http://dev.mysql.com/get/Downloads/MySQL-5.1/mysql-5.1.30.tar.gz/from/http://mysql.llarian.net/”

2) tar -xf mysql-5.1.30.tar.gz&&cd mysql-5.1.30

3) ./configure –without-server –disable-shared

4) make&&cp -i ./client/mysqlslap /usr/local/bin/

Securely specify mysql credentials in automated scripts

Often, you may want to run a script that uses a username and password to access data in a MySQL database. Securely running a script like this manually is easy – simply use the ‘-p’ option for the MySQL client, and it will prompt you for the password. However, this is not an option if you want to automate the script.

There are several ways to provide the password in a way that can be used with automated scripts, but only one that is both flexible and secure. You can specify the password on the command line itself (with ‘mysql -p ‘); however, this allows the password be seen by other users who run commands like ‘ps’. Another option is setting the environment variable “MYSQL_PWD” to the password, but this can also be seen by other users. Continue reading “Securely specify mysql credentials in automated scripts”

Using the Data::Dumper module to debug your Perl scripts

If you’re ever writing a Perl script that contains fairly complicated data structures (multi-dimensional arrays, hashes of arrays, etc), or you’re debugging code that contains complicated objects, it may be helpful to view the current state of those data structures in their entirety.

This is easy to do with Data::Dumper. Although the features the module provides are fairly complex, it’s easy to use it to just print out the contents of something. Do do this, you’ll first need to import the module with ‘use Data::Dumper;’. Then, to dump the data in an array(for example), simply do ‘print Dumper(@array)’. Continue reading “Using the Data::Dumper module to debug your Perl scripts”