You can Install apache-cassandra or datastax-cassandra, we will just cover apache-cassandra.
For datastax-cassandra installation you can go to: http://www.datastax.com/docs/1.2/install/install_deb
apache-cassandra installation can be also found at:
This blog is divided in 3 sections: Installing apache-cassandra and protecting apache-cassandra with zNcrypt and troubleshooting.
Installing apache-cassandra can also be found at: http://wiki.apache.org/cassandra/GettingStarted
Environment:
Ec2 Instance: ami-3fec7956
Instance type: m1.xlarge - This already contains kernel headers to build zNcrypt module.
user: ubuntu
Ubuntu 12.04
zNcrypt 3.1.2
Installing apache-cassandra
1. Download cassandra tar file from http://cassandra.apache.org/download/ and untar it to '/'
root@ip-10-140-26-2:/home/ubuntu# wget http://apache.webxcreen.org/cassandra/1.2.2/apache-cassandra-1.2.2-bin.tar.gz--2013-03-13 22:18:46-- http://apache.webxcreen.org/cassandra/1.2.2/apache-cassandra-1.2.2-bin.tar.gzResolving apache.webxcreen.org (apache.webxcreen.org)... 187.162.129.77Connecting to apache.webxcreen.org (apache.webxcreen.org)|187.162.129.77|:80... connected.HTTP request sent, awaiting response... 200 OKLength: 16263315 (16M) [application/x-gzip]Saving to: `apache-cassandra-1.2.2-bin.tar.gz'3,547,464 1.55M/s p100%[=====>] 16,263,315 3.54M/s in 5.7s 2013-03-13 22:18:52 (2.72 MB/s) - `apache-cassandra-1.2.2-bin.tar.gz' saved [16263315/16263315]
root@ip-10-140-26-2:/home/ubuntu# file apache-cassandra-1.2.2-bin.tar.gz apache-cassandra-1.2.2-bin.tar.gz: gzip compressed data, from FAT filesystem (MS-DOS, OS/2, NT)
root@ip-10-140-26-2:/home/ubuntu# tar -xvf apache-cassandra-1.2.2-bin.tar.gz
root@ip-10-140-26-2:/home/ubuntu# mv apache-cassandra-1.2.2 /
root@ip-10-140-26-2:/home/ubuntu# ls /apache-cassandra-1.2.2/bin CHANGES.txt conf interface javadoc lib LICENSE.txt NEWS.txt NOTICE.txt pylib README.txt tools
2. Configure your /apache-cassandra-1.2.2/conf/cassandra.yaml
NOTE: If you have installed Cassandra from a deb or rpm package, the configuration files will be located in /etc/cassandra
a) choose a cluster_name like 'Blog Ring'
cluster_name: 'Blog Ring'
b) choose your data directory, in this example we use /mnt because it has enough storage space, /mnt/var/lib/cassandra should exist :
data_file_directories:
- /mnt/var/lib/cassandra/data
c) choose your commit log directory:
commitlog_directory: /var/lib/cassandra/commitlog
d) choose your cache directory:
saved_caches_directory: /mnt/var/lib/cassandra/saved_caches
e) Choose your seed IP,
In this case we are installing a single node, so it is the current node IP. In a multiple nodes ring, it is recommended to have multiple seeds.
- seeds: "10.140.26.2"
f) IMPORTANT: choose the listen address, this is always your current node IP
listen_address: 10.140.26.2
g) save your file.
3. It is recommended to disable swap on your nodes as stated in: http://www.datastax.com/docs/1.2/troubleshooting/index#freeze
root@ip-10-140-26-2:/home/ubuntu# sudo swapoff --all
4. Tune /etc/security/limits.conf as stated in http://www.datastax.com/docs/1.2/troubleshooting/index#ulimit-settings to:
* soft nofile 32768
* hard nofile 32768
* soft memlock unlimited
* hard memlock unlimited
5. Install JNA as recommended in http://www.datastax.com/docs/1.2/install/install_jre#install-jna; create a symlink at <apache-installacion>/lib to activate the jna.jar jar
root@ip-10-140-26-2:~$ apt-get install libjna-java...
For Centos it is: yum install jna
root@ip-10-140-26-2:/home/ubuntu# find / -name "*jna*" | grep jna.jar/usr/share/java/jna.jar
root@ip-10-140-26-2:/home/ubuntu# ln -s /usr/share/java/jna.jar /apache-cassandra-1.2.2/lib/
root@ip-10-140-26-2:/home/ubuntu# ls /apache-cassandra-1.2.2/lib/*jna*/apache-cassandra-1.2.2/lib/jna.jar
root@ip-10-140-26-2:/home/ubuntu#
6. Install JDK 6 or JRE 6 as recommended on: http://www.datastax.com/docs/1.2/install/install_jre#install-jre-deb.
root@ip-10-140-26-2:/home/ubuntu# sudo add-apt-repository ppa:webupd8team/java....
root@ip-10-140-26-2:/home/ubuntu# sudo apt-get update....
root@ip-10-140-26-2:/home/ubuntu# sudo apt-get install oracle-java6-installer....
7. Start cassandra; At start time some messages are displayed, the vbolded are the important:
root@ip-10-140-26-2:/home/ubuntu# INFO 23:46:07,428 Logging initialized INFO 23:46:07,447 JVM vendor/version: Java HotSpot(TM) 64-Bit Server VM/1.6.0_43 INFO 23:46:07,448 Heap size: 3884974080/3886022656 INFO 23:46:07,448 Classpath: /apache-cassandra-1.2.2/bin/../conf:/apache-cassandra-...................... INFO 23:46:10,811 JNA mlockall successful INFO 23:46:10,825 Loading settings from file:/apache-cassandra-1.2.2/conf/cassandra.yaml INFO 23:46:11,225 DiskAccessMode 'auto' determined to be mmap, indexAccessMode is mmap... INFO 23:46:12,729 Replaying /mnt/var/lib/cassandra/commitlog/CommitLog-2-1363217926982.log, /mnt/var/lib/cassandra/commitlog/CommitLog-2-1363217926983.log.... INFO 23:46:12,878 Completed flushing ... INFO 23:46:13,171 Cassandra version: 1.2.2 INFO 23:46:13,171 Thrift API version: 19.35.0 INFO 23:46:13,172 CQL supported versions: 2.0.0,3.0.1 (default: 3.0.1) INFO 23:46:13,214 Loading persisted ring state INFO 23:46:13,217 Starting up server gossip... INFO 23:46:13,432 Node /10.140.26.2 state jump to normal INFO 23:46:13,438 Startup completed! Now serving reads. INFO 23:46:13,455 Compacted 4 sstables to [/mnt/var/lib/cassandra/data/system/local/system-local-ib-10,]. 960 bytes to 502 (~52% of original) in 51ms = 0.009387MB/s. 4 total rows, 1 unique. Row merge counts were {1:0, 2:0, 3:0, 4:1, }
Installing zNcrypt and protecting Cassandra directories
1. To install zNcrypt it is needed to add Gazang's repo, download its GPG keys and install it like:
root@ip-10-140-26-2:/home/ubuntu# add-apt-repository https://archive.gazzang.com/stable
root@ip-10-140-26-2:/home/ubuntu# gpg --keyserver pgpkeys.mit.edu --recv-key D2B6F0B7FADF302Fgpg: directory `/root/.gnupg' createdgpg: new configuration file `/root/.gnupg/gpg.conf' createdgpg: WARNING: options in `/root/.gnupg/gpg.conf' are not yet active during this rungpg: keyring `/root/.gnupg/secring.gpg' createdgpg: keyring `/root/.gnupg/pubring.gpg' createdgpg: requesting key FADF302F from hkp server pgpkeys.mit.edugpg: /root/.gnupg/trustdb.gpg: trustdb createdgpg: key FADF302F: public key "Gazzang Jenkins (Build Server) <sysops@gazzang.com>" importedgpg: Total number processed: 1gpg: imported: 1 (RSA: 1)
root@ip-10-140-26-2:/home/ubuntu# gpg -a --export D2B6F0B7FADF302F | sudo apt-key add -OK
root@ip-10-140-26-2:/home/ubuntu# apt-get update....
root@ip-10-140-26-2:/home/ubuntu# apt-get install zncrypt....
2. register zNcrypt by defining a MASTER key and create/prepare a mounting directory
root@ip-10-140-26-2:/home/ubuntu# zncrypt register
3. Start cassandra ang get its pid to create its zNcrypt profile, save the profile in a file.
root@ip-10-140-26-2:/home/ubuntu# ps aux | grep cassandraroot 15109 22.4 27.0 6347932 4143708 pts/0 SLl 00:22 0:08 java -ea -javaagent:/apache-......
root@ip-10-140-26-2:/home/ubuntu# zncrypt-profile --pid=15109 > cassandra.profileroot@ip-10-140-26-2:/home/ubuntu# cat cassandra.profile {"uid":"0","comm":"java","cmdline":"java -ea -javaagent:/apache-cassandra-1.2.2/bin/../lib/jamm-0.2.5.jar -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms3745M -Xmx3745M -Xmn400M -.....cassandra-1.2.2/bin/../lib/snappy-java-1.0.4.1.jar:/apache-cassandra-1.2.2/bin/../lib/snaptree-0.1.jar org.apache.cassandra.service.CassandraDaemon"}
4. Create zNcrypt java process Rule based on Cassandra profile:
root@ip-10-140-26-2:/home/ubuntu# zncrypt acl --add --rule="ALLOW @cassandra * /usr/lib/jvm/java-6-oracle/jre/bin/java" --profile=cassandra.profile Type MASTER passphrase: 1 rule(s) were addedroot@ip-10-140-26-2:/home/ubuntu#
root@ip-10-140-26-2:/home/ubuntu# zncrypt acl --listType MASTER passphrase: # - Type Category Path Profile Process 1 ALLOW @cassandra * YES /usr/lib/jvm/java-6-oracle/jre/bin/javaroot@ip-10-140-26-2:/home/ubuntu#
5. Make sure that cassandra is not running and encrypt /mnt/var/lib/cassandra directory like:
root@ip-10-140-26-2:/home/ubuntu# zncrypt-move encrypt @cassandra /mnt/var/lib/cassandra/ /mnt/encrypted/Type MASTER passphrase:
Size to encrypt: 676 KbMoving from: '/mnt/var/lib/cassandra'Moving to: '/mnt/encrypted/cassandra/mnt/var/lib/cassandra' [=====================================================================>] 100%
Done.
root@ip-10-140-26-2:/home/ubuntu#
6. start Cassandra. this should start without problems. If cassandra doesn't start the ACL rule might be wrong, perhaps the java process location. Take a look to dmesg to see which java process tried to access encrypted data and correct the rule
7. How do I know that the data is encrypted?
Take a look to /mnt/var/lib/cassandra, it has a symlink to
lrwxrwxrwx 1 root root 46 Mar 14 00:29 /mnt/var/lib/cassandra -> /mnt/encrypted/cassandra/mnt/var/lib/cassandra/
Try to take a look to /mnt/encrypted/cassandra/mnt/var/lib/cassandra/
# ll /mnt/encrypted/cassandra/mnt/var/lib/cassandra/ls: cannot open directory /mnt/encrypted/cassandra/mnt/var/lib/cassandra/: Permission denied
Permission is denied because 'll' is not allowed to access the data =)
8. Run a cassandra stress test like:
root@ip-10-140-26-2:/home/ubuntu# /apache-cassandra-1.2.2/tools/bin/cassandra-stress -o insert -u 50 -c 50 -n 500000Unable to create stress keyspace: Keyspace names must be case-insensitively unique ("Keyspace1" conflicts with "Keyspace1")total,interval_op_rate,interval_key_rate,latency/95th/99th,elapsed_time14962,1496,1496,4.5,65.7,174.4,1051689,3672,3672,1.1,35.2,121.9,2095426,4373,4373,1.0,27.0,118.3,30137442,4201,4201,1.3,21.2,219.9,40180340,4289,4289,1.1,16.4,244.9,50218157,3781,3781,1.1,15.9,1580.1,60242318,2416,2416,1.1,15.5,2179.3,71273047,3072,3072,1.1,11.4,2179.3,81299609,2656,2656,1.1,10.7,2179.3,91334986,3537,3537,1.1,9.6,1609.3,101361714,2672,2672,1.1,9.6,2060.6,111390698,2898,2898,1.1,9.6,2082.9,121422654,3195,3195,1.2,9.9,2013.4,131452983,3032,3032,1.2,10.5,2334.0,141483737,3075,3075,1.1,10.6,2334.0,152500000,1626,1626,1.1,11.1,2334.0,160END
9. have fun!
Troubleshooting
A) If instead a message like JNA mlockall successful is shown at cassandra start time you see a message like:
INFO 23:38:45,463 JNA not found. Native methods will be disabled
Means this means that JNA is not installed, or the symlink was not created (See Cassandra Installation step 5) or the /etc/security/limits.conf parameters are not configured.
B) If you cannot start cassandra you can try by removing /mnt/var/lib/cassandra/*, make sure that your listen_address is ok and your seed(s) is already on the ring.
-Kozlex
P.D. If you were sailing about in the Caribbean or southeastern coast of North America in 1718 and saw a ship flying a black flag with a white, horned skeleton holding an hourglass and spearing a heart, you were in trouble.
