Apache BigTop (Hadoop ecosystem installation)

Installing all Hadoop and related software is a painful experience.  For development environment, it's probably the easiest to use BigTop.  The site explains it as:

 

Bigtop is a project for the development of packaging and tests of the Apache Hadoop ecosystem.

The primary goal of Bigtop is to build a community around the packaging and interoperability testing of Hadoop-related projects. This includes testing at various levels (packaging, platform, runtime, upgrade, etc...) developed by a community with a focus on the system as a whole, rather than individual projects.

Installing all the Hadoop and related software can take days and weeks -- Hadoop/Yarn, ZooKeeper, Pig, Hive, HBase, Mahout, Whirr, Oozie, Sqoop, Hue, Flume...etc.  And BigTop make it possible to install most of them very quickly (like about an hour or less).

You just need to make sure you to meet all the requirements and follow the steps correctly.  I've done this for CentOS 6.3 64-bit and worked fine. 

Requirements - http://bigtop.apache.org/
Install Steps - https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop+0.6.0

The latest version is 0.7.0, so for the first step, I did this:

1. sudo wget -O /etc/yum.repos.d/bigtop.repo http://bigtop.s3.amazonaws.com/releases/0.7.0/redhat/6/x86_64/bigtop.repo
2. sudo yum install hadoop\* flume-* mahout\* oozie\* whirr-* hive\* hue\*

I haven't tested this thoroughly, and it's for single-server environment setup.  For cluster setup, I'm still going to do manually -- I have to know how the systems are structured and what configuration files are involved and located anyway. For development, this seems to be the way to go for quick set up.

Bitcoin flow in realtime

You can watch bitcoin flow in realtime.  Pretty cool.  I wonder why CN is getting so much despite China bans it.

http://fiatleak.com/ - "watch the world currencies flow into BTC in realtime"

Opera 64-bit, v12.16 download

Get it from here:
ftp://ftp.opera.com/pub/opera/win/1216/en/

ssh problem using Vagrant on Windows

Vagrant is very convenient tool for development tasks.  It's not just a virtualbox VM managmenet tool, but also it is a tool to reuse and share VMs.

If you are not sure what it is or why you should use it, please read these:
http://www.vagrantup.com/about.html
http://docs.vagrantup.com/v2/why-vagrant/index.html
http://superuser.com/questions/584100/why-should-i-use-vagrant-instead-of-just-virtualbox

To use Vagrant, you need three applications:

1. VirtualBox
2. Vagrant
3. PuTTY or Git for Windows to use SSH

When you install Vagrant on Windows and start up the VM, you'll face an issue -- there is no ssh client installed by default on Windows and you can't log onto the VM you just created.  This can be resolved by using PuTTY or Git's ssh.exe that it comes with.  Or, you can also use cygwin's ssh.  Another issue is that although you can simply use the default id/password, vagrant/vagrant, but to use private key with PuTTY, the private key file needs to be converted into PuTTY's private key format.

Here is an example:

C:\>mkdir vagrant
C:\>cd vagrant
C:\vagrant>vagrant init precise32 http://files.vagrantup.com/precise32.box
C:\vagrant>vagrant up
Bringing machine 'default' up with 'virtualbox' provider...
[default] Importing base box 'precise32'...
[default] Matching MAC address for NAT networking...
[default] Setting the name of the VM...
[default] Clearing any previously set forwarded ports...
[default] Creating shared folders metadata...
[default] Clearing any previously set network interfaces...
[default] Preparing network interfaces based on configuration...
[default] Forwarding ports...
[default] -- 22 => 2222 (adapter 1)
[default] Booting VM...
[default] Waiting for machine to boot. This may take a few minutes...
[default] Machine booted and ready!
[default] The guest additions on this VM do not match the installed version of
VirtualBox! In most cases this is fine, but in rare cases it can
cause things such as shared folders to not work properly. If you see
shared folder errors, please update the guest additions within the
virtual machine and reload your VM.

Guest Additions Version: 4.2.0
VirtualBox Version: 4.3
[default] Mounting shared folders...
[default] -- /vagrant

C:\vagrant>vagrant ssh
`ssh` executable not found in any directories in the %PATH% variable. Is an
SSH client installed? Try installing Cygwin, MinGW or Git, all of which
contain an SSH client. Or use the PuTTY SSH client with the following
authentication information shown below:

Host: 127.0.0.1
Port: 2222
Username: vagrant
Private key: C:/Users/kkim/.vagrant.d/insecure_private_key

As you see here, it can't find ssh.exe and here are a few solutions.  Please note that the default id/password is vagrant/vagrant:

1. With Git
• Add path to ssh.exe to PATH environment variable.   If you installed git for Windows from below link (see Download section below), it's here:
  
  [UPDATE-20170813] Git ssh.exe is in different folder in the latest package -- it's in C:\Program Files\Git\usr\bin
  
  C:\Program Files\Git\usr\bin
  C:\Program Files (x86)\Git\bin
  Add to the environment variable or set it as:
  C:\vagrant>set PATH=%PATH%;C:\Program Files\Git\usr\bin
  C:\vagrant>set PATH=%PATH%;C:\Program Files (x86)\Git\bin
  
  then type,
  C:\vagrant>vagrant ssh
  Downside with this method is ssh runs within cmd window, and unless you use something like console 2, cmd window cannot be resized.  (You probably want to use console2.  It's a lot better than default cmd; it provides tab, and also cygwin, powershell can be used within it.)
2. With PuTTY
• Method (a), using PuTTY, ssh to 127.0.0.1:2222 with the default id/password.
• Method (b), using PuTTY's PuTTYgen to convert above private key to putty's format, and use that to ssh to vagrant VM.
• Method (c), use this plugin, https://github.com/nickryand/vagrant-multi-putty
  Install it by issuing this command:
  C:\vagrant>vagrant plugin install vagrant-multi-putty
  Then enter,
  C:\vagrant>vagrant putty
  But you still need to have the path set correctly where putty.exe is:
  C:\vagrant>set PATH=%PATH%;C:\Program Files (x86)\PuTTY
  when you enter vagrant putty, it will spawn putty window and connect to the VM.
3. Using Cygwin ssh
   $ ssh -p 2222 localhost -l vagrant

To suspend (saving state) the VM, do:
C:\vagrant>vagrant suspend

To destroy the VM:
C:\vagrant>vagrant destroy

My environment

• Windows 7, 64-bit
• Vagrant v1.3.5
• VirtualBox v4.3.4

Download

• VirtualBox
• Vagrant
• Vagrant OS Images
• Vagrant plugin, vagrant-multi-putty
• Git for Windows
• PuTTY