sábado, 16 de marzo de 2013

Installing Apache-Cassandra and encrypting its data directories with zNcrypt on ubuntu

You can Install apache-cassandra or datastax-cassandra, we will just cover apache-cassandra. 

For datastax-cassandra installation you can go to: http://www.datastax.com/docs/1.2/install/install_deb

apache-cassandra installation can be also found at: 

This blog is divided in 3 sections: Installing apache-cassandra and protecting apache-cassandra with zNcrypt and troubleshooting.

Installing apache-cassandra can also be found at: http://wiki.apache.org/cassandra/GettingStarted

Ec2 Instance: ami-3fec7956 
Instance type: m1.xlarge - This already contains kernel headers to build zNcrypt module. 
user: ubuntu 
Ubuntu 12.04
zNcrypt 3.1.2 

Installing apache-cassandra

1. Download cassandra tar file from http://cassandra.apache.org/download/ and untar it to '/'

root@ip-10-140-26-2:/home/ubuntu# wget http://apache.webxcreen.org/cassandra/1.2.2/apache-cassandra-1.2.2-bin.tar.gz--2013-03-13 22:18:46--  http://apache.webxcreen.org/cassandra/1.2.2/apache-cassandra-1.2.2-bin.tar.gzResolving apache.webxcreen.org (apache.webxcreen.org)... to apache.webxcreen.org (apache.webxcreen.org)||:80... connected.HTTP request sent, awaiting response... 200 OKLength: 16263315 (16M) [application/x-gzip]Saving to: `apache-cassandra-1.2.2-bin.tar.gz'3,547,464   1.55M/s              p100%[=====>] 16,263,315  3.54M/s   in 5.7s    2013-03-13 22:18:52 (2.72 MB/s) - `apache-cassandra-1.2.2-bin.tar.gz' saved [16263315/16263315]
root@ip-10-140-26-2:/home/ubuntu# file apache-cassandra-1.2.2-bin.tar.gz apache-cassandra-1.2.2-bin.tar.gz: gzip compressed data, from FAT filesystem (MS-DOS, OS/2, NT) 
root@ip-10-140-26-2:/home/ubuntu# tar -xvf apache-cassandra-1.2.2-bin.tar.gz  
root@ip-10-140-26-2:/home/ubuntu# mv apache-cassandra-1.2.2 / 
root@ip-10-140-26-2:/home/ubuntu# ls /apache-cassandra-1.2.2/bin  CHANGES.txt  conf  interface  javadoc  lib  LICENSE.txt  NEWS.txt  NOTICE.txt  pylib  README.txt  tools
2. Configure your  /apache-cassandra-1.2.2/conf/cassandra.yaml

NOTE: If you have installed Cassandra from a deb or rpm package, the configuration files will be located in /etc/cassandra

a) choose a cluster_name like 'Blog Ring'
cluster_name: 'Blog Ring'

b) choose your data directory, in this example we use /mnt because it has enough storage space, /mnt/var/lib/cassandra should exist :


    - /mnt/var/lib/cassandra/data

c) choose your commit log directory:

commitlog_directory: /var/lib/cassandra/commitlog

d) choose your cache directory:

saved_caches_directory: /mnt/var/lib/cassandra/saved_caches

e) Choose your seed IP, 
       In this case we are installing a single node, so it is the current node IP. In a multiple nodes ring, it is recommended to have multiple seeds.
 - seeds: ""

f) IMPORTANT: choose the listen address, this is always your current node IP

g) save your file.

3. It is recommended to disable swap on your nodes as stated in: http://www.datastax.com/docs/1.2/troubleshooting/index#freeze

root@ip-10-140-26-2:/home/ubuntu# sudo swapoff --all

4. Tune /etc/security/limits.conf as stated in http://www.datastax.com/docs/1.2/troubleshooting/index#ulimit-settings to:

* soft nofile 32768    
* hard nofile 32768    
* soft memlock unlimited    
* hard memlock unlimited

5. Install JNA as recommended in http://www.datastax.com/docs/1.2/install/install_jre#install-jna; create a symlink at <apache-installacion>/lib to activate the jna.jar jar 

root@ip-10-140-26-2:~$ apt-get install libjna-java...

For Centos it is: yum install jna

root@ip-10-140-26-2:/home/ubuntu# find / -name "*jna*" | grep jna.jar/usr/share/java/jna.jar

root@ip-10-140-26-2:/home/ubuntu# ln -s /usr/share/java/jna.jar  /apache-cassandra-1.2.2/lib/
root@ip-10-140-26-2:/home/ubuntu# ls /apache-cassandra-1.2.2/lib/*jna*/apache-cassandra-1.2.2/lib/jna.jar 

6. Install JDK 6 or JRE 6 as recommended on: http://www.datastax.com/docs/1.2/install/install_jre#install-jre-deb.

root@ip-10-140-26-2:/home/ubuntu# sudo add-apt-repository ppa:webupd8team/java....
root@ip-10-140-26-2:/home/ubuntu# sudo apt-get update....
root@ip-10-140-26-2:/home/ubuntu# sudo apt-get install oracle-java6-installer....

7. Start cassandra; At start time some messages are displayed, the vbolded are the important:

root@ip-10-140-26-2:/home/ubuntu#  INFO 23:46:07,428 Logging initialized INFO 23:46:07,447 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_43 INFO 23:46:07,448 Heap size: 3884974080/3886022656 INFO 23:46:07,448 Classpath: /apache-cassandra-1.2.2/bin/../conf:/apache-cassandra-...................... INFO 23:46:10,811 JNA mlockall successful INFO 23:46:10,825 Loading settings from file:/apache-cassandra-1.2.2/conf/cassandra.yaml INFO 23:46:11,225 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap... INFO 23:46:12,729 Replaying /mnt/var/lib/cassandra/commitlog/CommitLog-2-1363217926982.log, /mnt/var/lib/cassandra/commitlog/CommitLog-2-1363217926983.log.... INFO 23:46:12,878 Completed flushing ... INFO 23:46:13,171 Cassandra version: 1.2.2 INFO 23:46:13,171 Thrift API version: 19.35.0 INFO 23:46:13,172 CQL supported versions: 2.0.0,3.0.1 (default: 3.0.1) INFO 23:46:13,214 Loading persisted ring state INFO 23:46:13,217 Starting up server gossip... INFO 23:46:13,432 Node / state jump to normal INFO 23:46:13,438 Startup completed! Now serving reads. INFO 23:46:13,455 Compacted 4 sstables to [/mnt/var/lib/cassandra/data/system/local/system-local-ib-10,].  960 bytes to 502 (~52% of original) in 51ms = 0.009387MB/s.  4 total rows, 1 unique.  Row merge counts were {1:0, 2:0, 3:0, 4:1, }

Installing zNcrypt and protecting Cassandra directories

1. To install zNcrypt it is needed to add Gazang's repo, download its GPG keys and install it like:

root@ip-10-140-26-2:/home/ubuntu# add-apt-repository https://archive.gazzang.com/stable
root@ip-10-140-26-2:/home/ubuntu# gpg --keyserver pgpkeys.mit.edu --recv-key  D2B6F0B7FADF302Fgpg: directory `/root/.gnupg' createdgpg: new configuration file `/root/.gnupg/gpg.conf' createdgpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this rungpg: keyring `/root/.gnupg/secring.gpg' createdgpg: keyring `/root/.gnupg/pubring.gpg' createdgpg: requesting key FADF302F from hkp server pgpkeys.mit.edugpg: /root/.gnupg/trustdb.gpg: trustdb createdgpg: key FADF302F: public key "Gazzang Jenkins (Build Server) <sysops@gazzang.com>" importedgpg: Total number processed: 1gpg:               imported: 1  (RSA: 1)
root@ip-10-140-26-2:/home/ubuntu# gpg -a --export D2B6F0B7FADF302F | sudo apt-key add -OK 
root@ip-10-140-26-2:/home/ubuntu# apt-get update....
root@ip-10-140-26-2:/home/ubuntu# apt-get install zncrypt....

2. register zNcrypt by defining a MASTER key and create/prepare a mounting directory 

root@ip-10-140-26-2:/home/ubuntu# zncrypt register

3. Start cassandra ang get its pid to create its zNcrypt profile, save the profile in a file.

root@ip-10-140-26-2:/home/ubuntu# ps aux | grep cassandraroot     15109 22.4 27.0 6347932 4143708 pts/0 SLl  00:22   0:08 java -ea -javaagent:/apache-......
root@ip-10-140-26-2:/home/ubuntu# zncrypt-profile --pid=15109 > cassandra.profileroot@ip-10-140-26-2:/home/ubuntu# cat cassandra.profile {"uid":"0","comm":"java","cmdline":"java -ea -javaagent:/apache-cassandra-1.2.2/bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms3745M -Xmx3745M -Xmn400M -.....cassandra-1.2.2/bin/../lib/snappy-java- org.apache.cassandra.service.CassandraDaemon"}

4. Create zNcrypt java process Rule based on Cassandra profile:

root@ip-10-140-26-2:/home/ubuntu# zncrypt acl --add --rule="ALLOW @cassandra * /usr/lib/jvm/java-6-oracle/jre/bin/java" --profile=cassandra.profile Type MASTER passphrase: 1 rule(s) were addedroot@ip-10-140-26-2:/home/ubuntu#  
root@ip-10-140-26-2:/home/ubuntu# zncrypt acl --listType MASTER passphrase: # -  Type     Category       Path                     Profile  Process             1    ALLOW    @cassandra     *                        YES      /usr/lib/jvm/java-6-oracle/jre/bin/javaroot@ip-10-140-26-2:/home/ubuntu# 

5. Make sure that cassandra is not running and encrypt /mnt/var/lib/cassandra directory like:

root@ip-10-140-26-2:/home/ubuntu# zncrypt-move encrypt @cassandra /mnt/var/lib/cassandra/ /mnt/encrypted/Type MASTER passphrase: 
Size to encrypt: 676 KbMoving from: '/mnt/var/lib/cassandra'Moving to:   '/mnt/encrypted/cassandra/mnt/var/lib/cassandra'  [=====================================================================>] 100%

6. start Cassandra. this should start without problems. If cassandra doesn't start the ACL rule might be wrong, perhaps the java process location. Take a look to dmesg to see which java process tried to access encrypted data and correct the rule

7. How do I know that the data is encrypted?

Take a look to /mnt/var/lib/cassandra, it has a symlink to 

lrwxrwxrwx 1 root root 46 Mar 14 00:29 /mnt/var/lib/cassandra -> /mnt/encrypted/cassandra/mnt/var/lib/cassandra/

Try to take a look to /mnt/encrypted/cassandra/mnt/var/lib/cassandra/

# ll /mnt/encrypted/cassandra/mnt/var/lib/cassandra/ls: cannot open directory /mnt/encrypted/cassandra/mnt/var/lib/cassandra/: Permission denied

Permission is denied because 'll' is not allowed to access the data =)

8. Run a cassandra stress test like: 

root@ip-10-140-26-2:/home/ubuntu# /apache-cassandra-1.2.2/tools/bin/cassandra-stress  -o insert -u 50 -c 50 -n 500000Unable to create stress keyspace: Keyspace names must be case-insensitively unique ("Keyspace1" conflicts with "Keyspace1")total,interval_op_rate,interval_key_rate,latency/95th/99th,elapsed_time14962,1496,1496,4.5,65.7,174.4,1051689,3672,3672,1.1,35.2,121.9,2095426,4373,4373,1.0,27.0,118.3,30137442,4201,4201,1.3,21.2,219.9,40180340,4289,4289,1.1,16.4,244.9,50218157,3781,3781,1.1,15.9,1580.1,60242318,2416,2416,1.1,15.5,2179.3,71273047,3072,3072,1.1,11.4,2179.3,81299609,2656,2656,1.1,10.7,2179.3,91334986,3537,3537,1.1,9.6,1609.3,101361714,2672,2672,1.1,9.6,2060.6,111390698,2898,2898,1.1,9.6,2082.9,121422654,3195,3195,1.2,9.9,2013.4,131452983,3032,3032,1.2,10.5,2334.0,141483737,3075,3075,1.1,10.6,2334.0,152500000,1626,1626,1.1,11.1,2334.0,160END

9. have fun!


A) If instead a message like JNA mlockall successful is shown at cassandra start time you see a message like:

INFO 23:38:45,463 JNA not found. Native methods will be disabled

Means this means that JNA is not installed, or the symlink was not created (See Cassandra Installation step 5) or the /etc/security/limits.conf parameters are not configured.

B) If you cannot start cassandra you can try by removing /mnt/var/lib/cassandra/*, make sure that your listen_address is ok and your seed(s) is already on the ring.


P.D. If you were sailing about in the Caribbean or southeastern coast of North America in 1718 and saw a ship flying a black flag with a white, horned skeleton holding an hourglass and spearing a heart, you were in trouble.

lunes, 4 de marzo de 2013

Getting EC2 XEN Kernel Headers for Centos to build kernel modules


So this 'my First time'. I am showing how to get the headers for a xen kernel and for centos, if you are looking the same for ubuntu, at the bottom I show a link that could help; I will  do it by an example. 

Let's say that we have the EC2 ami: ami-01996f68  (For this particular ami see appendix A)

Distro: centos 5.5 x64
Kernel: 2.6.18-194.26.1.el5xen
Instance user: root

First review your installed  rpms, just to see what you have:

-bash-3.2#  uname -a ; rpm -qa kernel\* | sort
Linux ip-10-142-132-23 2.6.18-194.26.1.el5xen #1 SMP Tue Nov 9 13:35:30 EST 2010 x86_64 x86_64 x86_64 GNU/Linux

Just to verify that there are not headers:

-bash-3.2# ls /usr/src/kernels/-bash-3.2# 

Ok, there is nothing. 

1. Getting headers from kernel-xen-devel installation

Now install kernel-xen-devel, If you are this contains the headers that you are looking for, if you are not then go to section 2.

-bash-3.2# yum install  kernel-xen-devel


# ls /usr/src/kernels/
2.6.18-348.1.1.el5-xen-x86_64  2.6.18-348.1.1.el5xen-x86_64

Don't use '.el5-xen' use the one with "el5xen" because your kernel is:  2.6.18-194.26.1.el5xen

Wait!.. what? my kernel is:  2.6.18-194.26.1.el5xen and will use 2.6.18-348.1.1.el5xen-x86_64? Yes, let's give it a try.. if doesn't work move to section 2.

There you go, you have some headers:

-bash-3.2# ls /usr/src/kernels/2.6.18-348.1.1.el5xen-x86_64/
arch   crypto   fs       init  kabi_whitelist  lib       mm           Module.markers  net      scripts   sound                                 usr
block  drivers  include  ipc   kernel          Makefile  Module.kabi  Module.symvers  samples  security  symsets-2.6.18-348.1.1.el5xen.tar.gz

Ok, now let's create the symlink. First do a backup of current /build symlink:

-bash-3.2# mv /lib/modules/2.6.18-194.26.1.el5xen/build /lib/modules/2.6.18-194.26.1.el5xen/build.back 

Then create the symlink:

-bash-3.2# ln -s /usr/src/kernels/2.6.18-348.1.1.el5xen-x86_64/ /lib/modules/2.6.18-194.26.1.el5xen/build

There you go, try to build your kernel kernel module at your XEN kernel. If you were not lucky building it move to section 2.

i.e. I just build gazzang zNcrypt kernel module:

  Installing     : ezncryptfs                                                                                                                   Removing existing ezncryptfs DKMS kernel modules...
Building ezncryptfs 1.1.3_rhel5 DKMS kernel module...
Successfully installed ezncryptfs module !

2. Getting headers from a repo:

This is the long way, and useful. Trick is to find the rpms of your installed kernel.

First, you need to get from a repo kernel-xen-devel-'uname -r', in our example just google: kernel-xen-devel-2.6.18-194.26.1.el5xen

Now, use your search skills over the web, I found: kernel-xen-devel-2.6.18-194.el5.x86_64.rpm from: http://vault.centos.org/5.5/os/x86_64/CentOS/


[user@ip-10-142-132-23 ~]$ pwd/home/user[user@ip-10-142-132-23 ~]$ wget http://vault.centos.org/5.5/os/x86_64/CentOS/kernel-xen-devel-2.6.18-194.el5.x86_64.rpm
--2013-03-04 14:08:10--  http://vault.centos.org/5.5/os/x86_64/CentOS/kernel-xen-devel-2.6.18-194.el5.x86_64.rpmResolving vault.centos.org... to vault.centos.org||:80... connected.HTTP request sent, awaiting response... 200 OK
Length: 5641264 (5.4M) [application/x-rpm]Saving to: `kernel-xen-devel-2.6.18-194.el5.x86_64.rpm'
100%[============================================================================================================>] 5,641,264   8.02M/s   in 0.7s  

2013-03-04 14:08:11 (8.02 MB/s) - `kernel-xen-devel-2.6.18-194.el5.x86_64.rpm' saved [5641264/5641264]
[user@ip-10-142-132-23 ~]$ lskernel-xen-devel-2.6.18-194.el5.x86_64.rpm
[user@ip-10-142-132-23 ~]$

Or download it to your local then use scp.

Install it by rpm, remember it is recommended to install any rpm package as non-root.

[user@ip-10-142-132-23 ~]$ sudo rpm -i kernel-xen-devel-2.6.18-194.el5.x86_64.rpm
package kernel-xen-devel-2.6.18-348.1.1.el5.x86_64 (which is newer than kernel-xen-devel-2.6.18-194.el5.x86_64) is already installed
[user@ip-10-142-132-23 ~]$ 

ups! .. ok, uninstall 348 from there. ( it is the one that was installed in section 1)

bash-3.2$ uname -a ; rpm -qa kernel\* | sort
Linux ip-10-142-132-23 2.6.18-194.26.1.el5xen
#1 SMP Tue Nov 9 13:35:30 EST 2010 x86_64 x86_64 x86_64 GNU/Linuxkernel-headers-2.6.18-348.1.1.el5kernel-xen-2.6.18-194.26.1.el5kernel-xen-devel-2.6.18-348.1.1.el5
bash-3.2$ rpm -e kernel-xen-devel-2.6.18-348.1.1.el5
error: Failed dependencies: kernel-devel is needed by (installed) dkms-$ sudo rpm -e kernel-xen-devel-2.6.18-348.1.1.el5
error: Failed dependencies: kernel-devel is needed by (installed) dkms-

Ohh I already installed dkms as a zncrypt dependency, because I was lucky on section 1, you might not be lucky on section 1 but it was installed anyway.

Fix this by:

-bash-3.2#  rpm -e dkms-
-bash-3.2#  rpm -e kernel-xen-devel-2.6.18-348.1.1.el5

Let's try to install again our rpm as non-root user:

 bash-3.2$ uname -a ; rpm -qa kernel\* | sort
Linux ip-10-142-132-23 2.6.18-194.26.1.el5xen #1 SMP Tue Nov 9 13:35:30 EST 2010 x86_64 x86_64 x86_64 GNU/Linux
bash-3.2$ uname -r

You can see that our kernel matches with kernel-xen-devel. 

Ok, now let's create the symlink. First do a backup of current /build symlink if you wish:

-bash-3.2# mv /lib/modules/2.6.18-194.26.1.el5xen/build /lib/modules/2.6.18-194.26.1.el5xen/build.back.old

Then create the symlink:

-bash-3.2#  ln -s /usr/src/kernels/2.6.18-194.el5xen-x86_64/ /lib/modules/2.6.18-194.26.1.el5xen/build

Time to build! Enjoy!

I don't need to say that you need to change the commands to your named kernel. Ups, Just did it! Ok, geek comment.

Option 3???  Please comment!

If you need to do this at ubuntu, perhaps follow link might help you: http://daemonscorner.blogspot.mx/2009/10/installing-linux-kernel-headers-for.html  
(I have not tried it)


Appendix A. 

Some errors might appear when changing to root user in this ami.

bash: /root/.bash_profile: No such file or directory
bash: --: command not found
bash: /root/.bash_profile: No such file or directory
[root@ip-10-142-132-23 user]# ls

To fix this comment some lines as root to .bashrc

Follow is the complete file with the lines already commented (bolded)... just compare with yours:
#cat .bashrc
# .bashrc
# User specific aliases and functions
alias rm='rm -i'
alias cp='cp -i'
alias mv='mv -i'
# Source global definitions
if [ -f /etc/bashrc ]; then
        . /etc/bashrc
# .bash_profile
# Get the aliases and functions
#if [ -f ~/.bashrc ]; then
#        . ~/.bashrc

# User specific environment and startup programs
export PATH


P.D. As the only nation to use six colors in their flag's primary design, South Africa gets first place on our list of the world's most colorful flags

It's better to give than to receive :)

It  is time to share!

Never too late to start!


P.D. there is no official world flag of peace adopted by the United Nations!