miércoles, 26 de junio de 2013

Installing Riak and encrypting it with zNcrypt; aesni enabled on ubuntu

Riak is a bigdata database that is useful to save large amounts of data, the company that offers support for it is named basho.com, there is a lot of tutorials at basho site to install riak and configure.

This is a resume of the steps to setup a quick single node on a riak ring.

It is assumed that zNcrypt is already installed.

To install riak on ubuntu follow steps from:

Basically 4 commands are needed:

# curl http://apt.basho.com/gpg/basho.apt.key | sudo apt-key add -
# sudo bash -c "echo deb http://apt.basho.com $(lsb_release -sc) main > /etc/apt/sources.list.d/basho.list"
# sudo apt-get update
# sudo apt-get install riak

Configuring Riak:

You need to modify /etc/riak/app.config and /etc/riak/vm.args

1. Get yout local ip

2. Update vm.args

Change "-name riak@127.0.0." "-name riak@YOUR_IP"
3. update app.config

pb_ip: Replace with the IP address of the Riak node.

 {pb_ip,   "" }
Change to:
 {pb_ip,   "xx.xx.xx.xx" }

Then update http to your IP

{http, [ {"", 8098 } ]} 

change to :

{http, [ {"xx.xx.xx.xx", 8098 } ]},

Where xx.xx.xx.xx is your node ip.

4. Create Riak directories

# mkdir /mnt/r
# mkdir/mnt/r/bitcask

/mnt/r/ring and /mnt/r/kv_node are automatically created when  riak is started

IMPORTANT: If riask has been started reviously, when updating app.config and vm.args you had changed app.config and vm.args this would make changes at your vk_node so the previous stored information about this cluster not is not updated automatically, remove directories ring, kv_node and bitcask/*   (If you already have sensitive data written DON'T remove bitcask/*)

Configuring dmcrypt to encrypt with zNcrypt.

Creating a 1 Gb file:

# sudo dd if=/dev/zero of=1gb.file bs=50M count=20

then losetup

#losetup /dev/loop0 1gb.file

Now prepare device at zNcrypt:

# zncrypt-prepare  /dev/loop0 /mnt/dmcrypted

Now enable aes-ni

root@ip-10-191-34-140:/home/ubuntu/basho_bench# modprobe aesni_intel
root@ip-10-191-34-140:/home/ubuntu/basho_bench# lsmod | grep aesni
 aesni_intel            55664  0
cryptd                 20530  1 aesni_intel
aes_x86_64             17255  1 aesni_intel

then add zncryt acl rule

# zncrypt acl --add --rule="ALLOW @riak  *   /usr/lib/riak/erts-5.9.1/bin/beam.smp"
Type MASTER passphrase:
1 rule(s) were added

and encrypt:

# zncrypt-move encrypt @riak /var/lib/riak /mnt/dmcrypted


Doesn't respond 'riak ping'?

Make sure about 4 things

1.- check thet there is not already an idle riak priocess running
#psaux | grep riak

If there is something kill it.. Muahaha

2.- Make sure that kv_node, riank and bitcask directories are owned by 'riak' user, if not set it up

3. You probably had not removed previous node config, so remove kv_node, ring and bitcask/*

4. /mnt/r/ should be owned by riak user:

# chown riak:riak /mnt/r

Do you need to execute basho_bench?

If it is required to create the graphics you might see follow issue:

ubuntu@ip-10-68-22-201:~/basho_bench$ make results
priv/summary.r -i tests/current
/usr/bin/env: Rscript --vanilla: No such file or directory
make: *** [results] Error 127

To resolve it you can find the fix at:

In short
If make results fails with the error /usr/bin/env: Rscript –vanilla: No such file or directory please edit priv/summary.r and replace the first line with the full path to the Rscript binary on your system

In my case it is:
 #!/usr/bin/env Rscript --vanilla

ubuntu@ip-10-68-22-201:~/basho_bench$ Rscript
The program 'Rscript' is currently not installed.  You can install it by typing:
#sudo apt-get install r-base-core
ubuntu@ip-10-68-22-201:~/basho_bench$ apt-get install r-base-core

ubuntu@ip-10-68-22-201:~/basho_bench$ which Rscript

#!/usr/bin/Rscript --vanilla

Then execute:

#sudo make results

If you see an error like:

The downloaded packages are in
Loading required package: methods
Error: could not find function "getopt"
Execution halted
make: *** [results] Error 1

Ths is ok, just try to make results again.

There you go, your graphics are there:

ubuntu@ip-10-68-22-201:~/basho_bench$ ls tests/current/
console.log.0         error.log.0           get_latencies.csv     summary.csv           update_latencies.csv 
crash.log.0           errors.csv            log.sasl.txt          summary.png           voxer300.config 

P.S. The most important flag we live for, doesn't exist.

lunes, 17 de junio de 2013

Upgrading a debian linux kernel

Download the tar file you want to build/install.

For example has been downloaded from  https://www.kernel.org/pub/linux/kernel/v3.x/testing/

Untar it to a folder and cd to that folder

Then install make-dpkg it is really useful and performs multiple make commands like make clean, make modules, make modules_install automatically and will create the .deb files.

# apt-get install make-kpkg

Then issue the make commands (these might take a while, make sure you have enough disk space):

#make-kpkg --append-to-version foo     --revision 3.10rc5 --initrd kernel_image modules_image
#fakeroot make-kpkg --append-to-version foo        --revision 3.10rc5 --initrd kernel_image kernel_headers

.deb files are created one directory up to the current, so go there and install them:

# dpkg -i linux-image-3.10.0-rc5foo_3.10rc5_amd64.deb

# dpkg -i linux-headers-3.10.0-rc5foo_3.10rc5_amd64.deb

NOTE: While installing linux-headers you might receive an error where cannot be found the build directory, that is ok just go and create the symlink to the kernel-headers like:

# ln -s /usr/src/linux-headers-3.10.0-rc5foo/ /lib/modules/3.10.0-rc5foo/build

Try to install the headers again!

Finally update the grub2

# update-grub2

You can make sure that your new kernel is listed in grub.cfg as a menuentry:
#vi /boot/grub/grub.cfg
menuentry 'Ubuntu, with Linux 3.10.0-rc5foo' --class ubuntu --class gnu-linux --class gnu --class os {
        gfxmode $linux_gfx_mode
        insmod gzio
        insmod part_msdos
        insmod ext2
        set root='(hd0,msdos2)'
        search --no-floppy --fs-uuid --set=root 5ba53145-aaa1-4c5c-a157-7523ae919646
        linux   /boot/vmlinuz-3.10.0-rc5foo root=UUID=5ba53145-aaa1-4c5c-a157-7523ae919646 ro   quiet
        initrd  /boot/initrd.img-3.10.0-rc5foo

reboot now!   .. check your 'uname -r'; kernel  should've been updated.

P.S. There is a place where the sunrise is great ant their flag is ...

Intalling Google DNS (Enom) at an EC2-AWS Amazon Instance

I wanted to document this even when there is some info about it on the web.

It is not really clear how to use ENOM DNS Manager when purchasing a DNS by GOOGLE, although it is finally easy.

In short:

1.- Launch your EC2 instance
2.- Create an Elastic IP
3.- Associate the Elastic IP with your instance ID
4.- Go to your enom account and create Host Record like '@' on the first field and 'xx.xx.xx.xx' as your Address (where xx.xx.xx.xx is your elastic IP)
5.- Be patient Enom will not do the change immediately, it might delay 15 minutes or 2 days (worst case).

P.S. Someone else though about fun with flags

Installing DataStax Enterprise with Ansible

You can see in other blogs how to install ansible on Ubuntu, this time it is just shown how to install DSE using Ansible.


You need to be registered at Datastax so you could get an user/passwd to get Datastax DSE.

It is assumed that you already have Ansible configured and have a servers group named "dse.

You can follow follow post to install ansible: http://kozlex.blogspot.mx/2013/04/installing-and-executing-ansible-on.html

Do a quick test:

$ ansible dse -m ping -u ubuntu | success >> {
    "changed": false,
    "ping": "pong"

To install DSE on ubuntu, we need to make sure that:

- JNA is installed

- Oracle java6 is installed
- You have a Datastax user/password. Get it from: http://www.datastax.com/download/register

Installing JNA:

ansible dse -a "sudo apt-get install libjna-java" -u ubuntu -y

Installing Java6 on Ubuntu 12.04 x64:

First enable non-interctive mode

# echo debconf shared/accepted-oracle-license-v1-1 seen true | sudo debconf-set-selections

# echo debconf shared/accepted-oracle-license-v1-1 select true | sudo debconf-set-selections

Install Java:

# sudo apt-get install oracle-java6-installer -y

make sure java is running:

# java -version
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

1. Add to sources list datastax repository with your user/passwd and install dse:

# deb http://your.user.name_company.com:dRteasd34DW@debian.datastax.com/enterprise stable main

curl -L http://debian.datastax.com/debian/repo_key | sudo apt-key add -

# sudo apt-get update

# sudo apt-get install dse-full -y

Enable Solr and Hadoop at SDE

To enable Solr and Hadoop follow variables need to be set to ‘1’ at /etc/default/dse config file:


Start Datastax Enterprise.
Start the service with:

# sudo dse cassandra -s

2. Open new terminal and verify DSE is running and operational.

# sudo dsetool ring -h localhost

Output should look similar to the example below:
Address         DC          Rack        Status State   Load            Owns                Token       Solr        rack1       Up     Normal  9.29 MB         100.00%             33015982249869676890

# sudo /etc/init.d/dse stop

2.  To verify that the cluster node is up and running execute:

# nodetool ring -h localhost
Note: Ownership information does not include topology, please specify a keyspace.
Address         DC          Rack        Status State   Load            Owns                Token                                     Analytics   rack1       Up     Normal  4.38 MB         100.00%             42053256256062917965909574476650479427      
You can issue hadoop commands to make sure that it is working correctly:

# dse hadoop fs -mkdir my_test_dir

# dse hadoop fs -ls
Found 1 item
drwxrwxrwx   - root root          0 2013-04-23 16:56 /user/root/my_test_dir

If the directory created is listed then hadoop service works!

If you want to verify Hadoop deeper you can follow Hadoop demos:

P.S. You can see the flags of there world!