Today I though I would post about how cool moxi is. If you don't know, moxi is a memcached proxy (http://labs.northscale.com/moxi/) which allows you to move any complexity of having to set up the list of memcached servers you are using. Also, moxi has some great features such as:
* De-duplication of requests for popular keys
* A front cache, L1 cache to avoid network hops
* Fire and forget SET (Like an async SET) this means 'set a value, but don't wait to know if it was successful'
* Time-outs for setting the maximum time for operations
This post will show you how moxi can be used with the memcached UDFs for MySQL (https://launchpad.net/memcached-udfs). It's really quite amazingly easy. I'll even thrown in compilation of moxi and setup of moxi.
First, obtain the source for moxi which can be found at:
git clone git://github.com/northscale/moxi.git
To build moxi, you will need both libevent and libmemcached. I'll leave it to you to obtain both of those. For libevent, just use the package for either centos or ubuntu. For libmemcached, use the source because many packages available for it are somewhat stale, and libmemcached is actively developed by Trond et al!
Now enter the moxi source directory and do the following:
patrick-galbraiths-macbook-pro:moxi patg$ sh autogen.sh
libtoolize...
aclocal...
autoheader...
automake...
autoconf...
./configure
make
sudo make install
At this point, moxi is ready to use. Now, depending where you have memcached running, this will determine which servers you proxy to. In this demonstration, I'm using 4 different memcached servers:
* Two locally within my network
* Two Northscale memcached Amazon EC2 images (http://labs.northscale.com/memcached-ami/, ami ami-bf4cadd6). These are great-- you just launch them and then *use* them!
So, to launch moxi (I changed the hostnames to protect the innocent)
moxi -z 33133=127.0.0.1:22122,192.168.1.118:11211,ec2-00-00-00-yy.compute1.amazonaws.com:11211,ec2-00-00-00-xx.compute-1.amazonaws.com:11211 -vvv
So, the syntax is quite simple:
moxi -z 'port I want to answer to'='host:port server list I am proxying for'
In this example, just as you can also specify with memcached itself, I'm running with the -vvv flag for verbosity because I'm a geek and this is a great way to see what moxi is doing. Also, I'm running so I'm using port 33133 to answer to.
So, I installed my UDFs on the latest version of MariaDB, which I have installed in /usr/local/maria:
bzr branch lp:memcached-udfs
cd memcached-udfs
sh config/bootstrap
./configure --with-mysql=/usr/local/maria/bin/mysql_config --libdir=/usr/local/maria/lib/mysql/plugin/
make
sudo make install
mysql -u root test < sql/install_functions.sql
At this point, the UDFs are ready to use!
The convenient thing here is that moxi makes it so you don't have to specify a long server list in the memc_servers_set() UDF:
mysql> select memc_servers_set('127.0.0.1:33133');
+-------------------------------------+
| memc_servers_set('127.0.0.1:33133') |
+-------------------------------------+
| 0 |
+-------------------------------------+
Now, I was able to start storing and retrieving data via moxi to my 4 memcached instances:
mysql> select memc_set('key1', 'test value');
+--------------------------------+
| memc_set('key1', 'test value') |
+--------------------------------+
| 1 |
+--------------------------------+
1 row in set (0.07 sec)
mysql> select memc_get('key1');
+------------------+
| memc_get('key1') |
+------------------+
| test value |
+------------------+
1 row in set (0.08 sec)
And of interest, observe the output of moxi:
cproxy_create on port 33133, downstream 127.0.0.1:22122,192.168.1.118:11211,ec2-72-44-62-67.compute-1.amazonaws.com:11211,ec2-67-202-38-195.compute-1.amazonaws.com:11211
cproxy_listen on port 33133, downstream 127.0.0.1:22122,192.168.1.118:11211,ec2-72-44-62-67.compute-1.amazonaws.com:11211,ec2-67-202-38-195.compute-1.amazonaws.com:11211
<40 server listening (proxy-upstream-ascii)
<40 initialized conn_funcs to default
<41 server listening (proxy-upstream-ascii)
<41 initialized conn_funcs to default
<41 cproxy listening on port 33133
<40 cproxy listening on port 33133
moxi listening on 33133 with 2 conns
<42 new proxy-upstream-ascii client connection
42: going from conn_new_cmd to conn_waiting
42: going from conn_waiting to conn_read
42: going from conn_read to conn_parse_cmd
<42 cproxy_process_upstream_ascii version
>42 VERSION 0.9.6
42: going from conn_parse_cmd to conn_write
42: going from conn_write to conn_new_cmd
42: going from conn_new_cmd to conn_waiting
42: going from conn_waiting to conn_read
<43 new proxy-upstream-ascii client connection
43: going from conn_new_cmd to conn_waiting
43: going from conn_waiting to conn_read
43: going from conn_read to conn_parse_cmd
<43 cproxy_process_upstream_ascii set key1 0 0 10
43: going from conn_parse_cmd to conn_nread
43 pause_upstream_for_downstream
43: going from conn_nread to conn_pause
assign_downstream
cproxy_add_downstream 0 4
cproxy_create_downstream: 127.0.0.1:22122,192.168.1.118:11211,ec2-72-44-62-67.compute-1.amazonaws.com:11211,ec2-67-202-38-195.compute-1.amazonaws.com:11211, 0, 7
release_downstream
check_downstream_config 1
check_downstream_config 1
assign_downstream, matched to upstream 43
<44 new proxy-downstream-ascii client connection
<45 new proxy-downstream-ascii client connection
<46 new proxy-downstream-ascii client connection
<47 new proxy-downstream-ascii client connection
46: going from conn_pause to conn_mwrite
assign_downstream, done
46: going from conn_mwrite to conn_new_cmd
46: going from conn_new_cmd to conn_waiting
46: going from conn_waiting to conn_read
46: going from conn_read to conn_parse_cmd
<46 cproxy_process_a2a_downstream STORED
46: going from conn_parse_cmd to conn_pause
>43 STORED
43: going from conn_pause to conn_write
<46 cproxy_on_pause_downstream_conn
46 release_downstream_conn, downstream_used 1 1
release_downstream
check_downstream_config 1
assign_downstream
assign_downstream, done
43: going from conn_write to conn_new_cmd
43: going from conn_new_cmd to conn_waiting
43: going from conn_waiting to conn_read
43: going from conn_read to conn_parse_cmd
<43 cproxy_process_upstream_ascii quit
43: going from conn_parse_cmd to conn_closing
<43 cproxy_on_close_upstream_conn
<43 connection closed.
<43 new proxy-upstream-ascii client connection
43: going from conn_new_cmd to conn_waiting
43: going from conn_waiting to conn_read
43: going from conn_read to conn_parse_cmd
<43 cproxy_process_upstream_ascii get key1
43 pause_upstream_for_downstream
43: going from conn_parse_cmd to conn_pause
assign_downstream
cproxy_add_downstream 0 4
cproxy_create_downstream: 127.0.0.1:22122,192.168.1.118:11211,ec2-72-44-62-67.compute-1.amazonaws.com:11211,ec2-67-202-38-195.compute-1.amazonaws.com:11211, 0, 7
release_downstream
check_downstream_config 1
check_downstream_config 1
assign_downstream, matched to upstream 43
<48 new proxy-downstream-ascii client connection
<49 new proxy-downstream-ascii client connection
<50 new proxy-downstream-ascii client connection
<51 new proxy-downstream-ascii client connection
forward multiget get key1 (3 0)
50: going from conn_pause to conn_mwrite
forward multiget nwrite 1 out of 4
assign_downstream, done
50: going from conn_mwrite to conn_new_cmd
50: going from conn_new_cmd to conn_waiting
50: going from conn_waiting to conn_read
50: going from conn_read to conn_parse_cmd
<50 cproxy_process_a2a_downstream VALUE key1 0 10
50: going from conn_parse_cmd to conn_nread
<50 cproxy_process_a2a_downstream_nread 0 200
50: going from conn_nread to conn_new_cmd
<43 cproxy ascii item response success
50: going from conn_new_cmd to conn_parse_cmd
<50 cproxy_process_a2a_downstream END
50: going from conn_parse_cmd to conn_pause
<50 cproxy_on_pause_downstream_conn
50 release_downstream_conn, downstream_used 1 1
release_downstream
43: going from conn_pause to conn_mwrite
check_downstream_config 1
assign_downstream
You can see (yes, this a lot of output) how moxi is talking to the 4 different servers, a simple set and get of the key 'key1'.. Also you see this concept of 'downstream' and 'upstream' -- to where the data is actually being stored versus where it is passed to moxi from, for instance a PHP or Perl client, or even another moxi.
So, as you can see, using moxi with the memcached Functions for MySQL (memcached UDFs), you gain even more convenience and ability to store data to memcached. Also, you've seen mention of Northscale's Amazon EC2 Machine Images (AMIs) and how you can easily add more memcached instances for your application's needs.
PlanetMySQL Voting:
Vote UP /
Vote DOWN"
Posts
No comments:
Post a Comment