sábado, 16 de marzo de 2013

Installing Apache-Cassandra and encrypting its data directories with zNcrypt on ubuntu

You can Install apache-cassandra or datastax-cassandra, we will just cover apache-cassandra. 

For datastax-cassandra installation you can go to: http://www.datastax.com/docs/1.2/install/install_deb

apache-cassandra installation can be also found at: 

This blog is divided in 3 sections: Installing apache-cassandra and protecting apache-cassandra with zNcrypt and troubleshooting.

Installing apache-cassandra can also be found at: http://wiki.apache.org/cassandra/GettingStarted

Ec2 Instance: ami-3fec7956 
Instance type: m1.xlarge - This already contains kernel headers to build zNcrypt module. 
user: ubuntu 
Ubuntu 12.04
zNcrypt 3.1.2 

Installing apache-cassandra

1. Download cassandra tar file from http://cassandra.apache.org/download/ and untar it to '/'

root@ip-10-140-26-2:/home/ubuntu# wget http://apache.webxcreen.org/cassandra/1.2.2/apache-cassandra-1.2.2-bin.tar.gz--2013-03-13 22:18:46--  http://apache.webxcreen.org/cassandra/1.2.2/apache-cassandra-1.2.2-bin.tar.gzResolving apache.webxcreen.org (apache.webxcreen.org)... to apache.webxcreen.org (apache.webxcreen.org)||:80... connected.HTTP request sent, awaiting response... 200 OKLength: 16263315 (16M) [application/x-gzip]Saving to: `apache-cassandra-1.2.2-bin.tar.gz'3,547,464   1.55M/s              p100%[=====>] 16,263,315  3.54M/s   in 5.7s    2013-03-13 22:18:52 (2.72 MB/s) - `apache-cassandra-1.2.2-bin.tar.gz' saved [16263315/16263315]
root@ip-10-140-26-2:/home/ubuntu# file apache-cassandra-1.2.2-bin.tar.gz apache-cassandra-1.2.2-bin.tar.gz: gzip compressed data, from FAT filesystem (MS-DOS, OS/2, NT) 
root@ip-10-140-26-2:/home/ubuntu# tar -xvf apache-cassandra-1.2.2-bin.tar.gz  
root@ip-10-140-26-2:/home/ubuntu# mv apache-cassandra-1.2.2 / 
root@ip-10-140-26-2:/home/ubuntu# ls /apache-cassandra-1.2.2/bin  CHANGES.txt  conf  interface  javadoc  lib  LICENSE.txt  NEWS.txt  NOTICE.txt  pylib  README.txt  tools
2. Configure your  /apache-cassandra-1.2.2/conf/cassandra.yaml

NOTE: If you have installed Cassandra from a deb or rpm package, the configuration files will be located in /etc/cassandra

a) choose a cluster_name like 'Blog Ring'
cluster_name: 'Blog Ring'

b) choose your data directory, in this example we use /mnt because it has enough storage space, /mnt/var/lib/cassandra should exist :


    - /mnt/var/lib/cassandra/data

c) choose your commit log directory:

commitlog_directory: /var/lib/cassandra/commitlog

d) choose your cache directory:

saved_caches_directory: /mnt/var/lib/cassandra/saved_caches

e) Choose your seed IP, 
       In this case we are installing a single node, so it is the current node IP. In a multiple nodes ring, it is recommended to have multiple seeds.
 - seeds: ""

f) IMPORTANT: choose the listen address, this is always your current node IP

g) save your file.

3. It is recommended to disable swap on your nodes as stated in: http://www.datastax.com/docs/1.2/troubleshooting/index#freeze

root@ip-10-140-26-2:/home/ubuntu# sudo swapoff --all

4. Tune /etc/security/limits.conf as stated in http://www.datastax.com/docs/1.2/troubleshooting/index#ulimit-settings to:

* soft nofile 32768    
* hard nofile 32768    
* soft memlock unlimited    
* hard memlock unlimited

5. Install JNA as recommended in http://www.datastax.com/docs/1.2/install/install_jre#install-jna; create a symlink at <apache-installacion>/lib to activate the jna.jar jar 

root@ip-10-140-26-2:~$ apt-get install libjna-java...

For Centos it is: yum install jna

root@ip-10-140-26-2:/home/ubuntu# find / -name "*jna*" | grep jna.jar/usr/share/java/jna.jar

root@ip-10-140-26-2:/home/ubuntu# ln -s /usr/share/java/jna.jar  /apache-cassandra-1.2.2/lib/
root@ip-10-140-26-2:/home/ubuntu# ls /apache-cassandra-1.2.2/lib/*jna*/apache-cassandra-1.2.2/lib/jna.jar 

6. Install JDK 6 or JRE 6 as recommended on: http://www.datastax.com/docs/1.2/install/install_jre#install-jre-deb.

root@ip-10-140-26-2:/home/ubuntu# sudo add-apt-repository ppa:webupd8team/java....
root@ip-10-140-26-2:/home/ubuntu# sudo apt-get update....
root@ip-10-140-26-2:/home/ubuntu# sudo apt-get install oracle-java6-installer....

7. Start cassandra; At start time some messages are displayed, the vbolded are the important:

root@ip-10-140-26-2:/home/ubuntu#  INFO 23:46:07,428 Logging initialized INFO 23:46:07,447 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_43 INFO 23:46:07,448 Heap size: 3884974080/3886022656 INFO 23:46:07,448 Classpath: /apache-cassandra-1.2.2/bin/../conf:/apache-cassandra-...................... INFO 23:46:10,811 JNA mlockall successful INFO 23:46:10,825 Loading settings from file:/apache-cassandra-1.2.2/conf/cassandra.yaml INFO 23:46:11,225 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap... INFO 23:46:12,729 Replaying /mnt/var/lib/cassandra/commitlog/CommitLog-2-1363217926982.log, /mnt/var/lib/cassandra/commitlog/CommitLog-2-1363217926983.log.... INFO 23:46:12,878 Completed flushing ... INFO 23:46:13,171 Cassandra version: 1.2.2 INFO 23:46:13,171 Thrift API version: 19.35.0 INFO 23:46:13,172 CQL supported versions: 2.0.0,3.0.1 (default: 3.0.1) INFO 23:46:13,214 Loading persisted ring state INFO 23:46:13,217 Starting up server gossip... INFO 23:46:13,432 Node / state jump to normal INFO 23:46:13,438 Startup completed! Now serving reads. INFO 23:46:13,455 Compacted 4 sstables to [/mnt/var/lib/cassandra/data/system/local/system-local-ib-10,].  960 bytes to 502 (~52% of original) in 51ms = 0.009387MB/s.  4 total rows, 1 unique.  Row merge counts were {1:0, 2:0, 3:0, 4:1, }

Installing zNcrypt and protecting Cassandra directories

1. To install zNcrypt it is needed to add Gazang's repo, download its GPG keys and install it like:

root@ip-10-140-26-2:/home/ubuntu# add-apt-repository https://archive.gazzang.com/stable
root@ip-10-140-26-2:/home/ubuntu# gpg --keyserver pgpkeys.mit.edu --recv-key  D2B6F0B7FADF302Fgpg: directory `/root/.gnupg' createdgpg: new configuration file `/root/.gnupg/gpg.conf' createdgpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this rungpg: keyring `/root/.gnupg/secring.gpg' createdgpg: keyring `/root/.gnupg/pubring.gpg' createdgpg: requesting key FADF302F from hkp server pgpkeys.mit.edugpg: /root/.gnupg/trustdb.gpg: trustdb createdgpg: key FADF302F: public key "Gazzang Jenkins (Build Server) <sysops@gazzang.com>" importedgpg: Total number processed: 1gpg:               imported: 1  (RSA: 1)
root@ip-10-140-26-2:/home/ubuntu# gpg -a --export D2B6F0B7FADF302F | sudo apt-key add -OK 
root@ip-10-140-26-2:/home/ubuntu# apt-get update....
root@ip-10-140-26-2:/home/ubuntu# apt-get install zncrypt....

2. register zNcrypt by defining a MASTER key and create/prepare a mounting directory 

root@ip-10-140-26-2:/home/ubuntu# zncrypt register

3. Start cassandra ang get its pid to create its zNcrypt profile, save the profile in a file.

root@ip-10-140-26-2:/home/ubuntu# ps aux | grep cassandraroot     15109 22.4 27.0 6347932 4143708 pts/0 SLl  00:22   0:08 java -ea -javaagent:/apache-......
root@ip-10-140-26-2:/home/ubuntu# zncrypt-profile --pid=15109 > cassandra.profileroot@ip-10-140-26-2:/home/ubuntu# cat cassandra.profile {"uid":"0","comm":"java","cmdline":"java -ea -javaagent:/apache-cassandra-1.2.2/bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms3745M -Xmx3745M -Xmn400M -.....cassandra-1.2.2/bin/../lib/snappy-java- org.apache.cassandra.service.CassandraDaemon"}

4. Create zNcrypt java process Rule based on Cassandra profile:

root@ip-10-140-26-2:/home/ubuntu# zncrypt acl --add --rule="ALLOW @cassandra * /usr/lib/jvm/java-6-oracle/jre/bin/java" --profile=cassandra.profile Type MASTER passphrase: 1 rule(s) were addedroot@ip-10-140-26-2:/home/ubuntu#  
root@ip-10-140-26-2:/home/ubuntu# zncrypt acl --listType MASTER passphrase: # -  Type     Category       Path                     Profile  Process             1    ALLOW    @cassandra     *                        YES      /usr/lib/jvm/java-6-oracle/jre/bin/javaroot@ip-10-140-26-2:/home/ubuntu# 

5. Make sure that cassandra is not running and encrypt /mnt/var/lib/cassandra directory like:

root@ip-10-140-26-2:/home/ubuntu# zncrypt-move encrypt @cassandra /mnt/var/lib/cassandra/ /mnt/encrypted/Type MASTER passphrase: 
Size to encrypt: 676 KbMoving from: '/mnt/var/lib/cassandra'Moving to:   '/mnt/encrypted/cassandra/mnt/var/lib/cassandra'  [=====================================================================>] 100%

6. start Cassandra. this should start without problems. If cassandra doesn't start the ACL rule might be wrong, perhaps the java process location. Take a look to dmesg to see which java process tried to access encrypted data and correct the rule

7. How do I know that the data is encrypted?

Take a look to /mnt/var/lib/cassandra, it has a symlink to 

lrwxrwxrwx 1 root root 46 Mar 14 00:29 /mnt/var/lib/cassandra -> /mnt/encrypted/cassandra/mnt/var/lib/cassandra/

Try to take a look to /mnt/encrypted/cassandra/mnt/var/lib/cassandra/

# ll /mnt/encrypted/cassandra/mnt/var/lib/cassandra/ls: cannot open directory /mnt/encrypted/cassandra/mnt/var/lib/cassandra/: Permission denied

Permission is denied because 'll' is not allowed to access the data =)

8. Run a cassandra stress test like: 

root@ip-10-140-26-2:/home/ubuntu# /apache-cassandra-1.2.2/tools/bin/cassandra-stress  -o insert -u 50 -c 50 -n 500000Unable to create stress keyspace: Keyspace names must be case-insensitively unique ("Keyspace1" conflicts with "Keyspace1")total,interval_op_rate,interval_key_rate,latency/95th/99th,elapsed_time14962,1496,1496,4.5,65.7,174.4,1051689,3672,3672,1.1,35.2,121.9,2095426,4373,4373,1.0,27.0,118.3,30137442,4201,4201,1.3,21.2,219.9,40180340,4289,4289,1.1,16.4,244.9,50218157,3781,3781,1.1,15.9,1580.1,60242318,2416,2416,1.1,15.5,2179.3,71273047,3072,3072,1.1,11.4,2179.3,81299609,2656,2656,1.1,10.7,2179.3,91334986,3537,3537,1.1,9.6,1609.3,101361714,2672,2672,1.1,9.6,2060.6,111390698,2898,2898,1.1,9.6,2082.9,121422654,3195,3195,1.2,9.9,2013.4,131452983,3032,3032,1.2,10.5,2334.0,141483737,3075,3075,1.1,10.6,2334.0,152500000,1626,1626,1.1,11.1,2334.0,160END

9. have fun!


A) If instead a message like JNA mlockall successful is shown at cassandra start time you see a message like:

INFO 23:38:45,463 JNA not found. Native methods will be disabled

Means this means that JNA is not installed, or the symlink was not created (See Cassandra Installation step 5) or the /etc/security/limits.conf parameters are not configured.

B) If you cannot start cassandra you can try by removing /mnt/var/lib/cassandra/*, make sure that your listen_address is ok and your seed(s) is already on the ring.


P.D. If you were sailing about in the Caribbean or southeastern coast of North America in 1718 and saw a ship flying a black flag with a white, horned skeleton holding an hourglass and spearing a heart, you were in trouble.

1 comentario:

  1. I really appreciate information shared above. It’s of great help. If someone want to learn Online (Virtual) instructor lead live training in Apache Cassandra.kindly contact us http://www.maxmunus.com/contact
    MaxMunus Offer World Class Virtual Instructor led training on Apache Cassandra. We have industry expert trainer. We provide Training Material and Software Support. MaxMunus has successfully conducted 100000+ trainings in India, USA, UK, Australlia, Switzerland, Qatar, Saudi Arabia, Bangladesh, Bahrain and UAE etc.

    For Free Demo Contact us:
    Name : Arunkumar U
    Email : arun@maxmunus.com
    Skype id: training_maxmunus
    Contact No.-+91-9738507310
    Company Website –http://www.maxmunus.com