Category Archives: Tools

Installing DataStax Enterprise on Digital Ocean

These instructions explain how to install DataStax Enterprise on a set of Digital Ocean droplets. DataStax provides instructions for Installing on cloud providers, but currently only Amazon EC2 and HP Cloud are described specifically.

The steps below can be used for Digital Ocean, or more generally for any other cloud provider. We’ll create a set of Ubuntu droplets and install DataStax Enterprise (DSE) on them to create a Cassandra cluster.

Update: Scroll to the bottom for a video demo of these install steps.

These are the relevant DataStax documentation pages if you want to learn more details behind each step:

Prerequisites

  1. Register for DataStax Enterprise (free, allows use of DataStax Enterprise in dev/test environments)
  2. An active Digital Ocean account (referral link if you don’t have an account yet)

Creating Digital Ocean Droplets

  1. Login to the Digital Ocean and navigate to the control panel
  2. On your local system create an SSH key and store it in the Digital Ocean control panel (help)
  3. On the control panel click Create with settings like:
    • Hostname: node0
    • Size: 4 GB / 2 CPU
    • Region: default
    • Image: Linux Ubuntu 14.04 64-bit
    • SSH key: Select the one created previously
    • Settings: default
  4. Repeat for node1, node2, etc. (as many nodes as desired)
  5. As the nodes are coming up make note of the IP addresses

Installing DataStax Enterprise

For a faster install, see Parallel Installs below.

  1. SSH into the first node
  2. Confirm whether Java is already installed (it may be, depending on the Linux image); if not, install either OpenJDK Java or Oracle Java
  3. Add the DataStax repository using the username and password from your registration:

    echo "deb http://username:password@debian.datastax.com/enterprise stable main" | sudo tee -a /etc/apt/sources.list.d/datastax.sources.list

  4. Add the DataStax repository key:

    curl -L --silent https://debian.datastax.com/debian/repo_key | sudo apt-key add -

  5. Update the local package cache:

    sudo apt-get update

    (if you see any “403 Not Authorized” errors here, stop and make sure your username and password are correct)

  6. Install DataStax Enterprise:

    sudo apt-get install dse-full

  7. Edit /etc/hosts and add an entry for the host with its public IP address (replacing the 127.0.1.1 entry if it exists)

  8. Edit /etc/dse/cassandra/cassandra.yaml and change a couple of settings:

    • Set cluster_name as desired
    • In the seeds field, list the IP address of node0 (the first server will be the seed for the cluster)
    • Set listen_address to blank
    • Set num_tokens to 256
  9. Start the DSE service:

    sudo service dse start

  10. Repeat the above steps for the remaining nodes (node1, node2, etc.)

  11. SSH to any of the nodes and check the status of the DSE cluster:

    nodetool status

Parallel Installs

You can run the above install commands in parallel for a much faster setup time. On the Mac I use i2cssh which powers several iTerm2 consoles in parallel.

This technique is borrowed from Jake Luciani’s video How to set up a 4 node Cassandra cluster in under 2 minutes.

Steps:

  1. Install i2cssh and iTerm2
  2. Create a file ~/.i2csshrc with the server IP addresses. For example this file defines 3 servers included in a cluster named ‘digdemo’:

    version: 2
    iterm2: false
    clusters:
      digdemo:
        login: root
        hosts:
          - 54.176.126.209
          - 54.176.91.139
          - 50.18.136.76
    
  3. Launch parallel terminal sessions: i2cssh -c digdemo

  4. Enable broadcast mode in iTerm2 with Cmd-Opt-I

  5. Type commands from the install procedure above; they should be echoed on all sessions in parallel

Video Demo

Notes

Screencast: Mover.io Blog Backup to Box.com

I recently started using Mover.io to back up my blog. Mover is a relatively new service which can migrate or back up between several different cloud services. I’m starting to use it as part of my backup strategy, making sure even files I have “in the cloud” are located in more that one service.

The pricing model is also quite simple: they charge $1 per GB transferred. You deposit money in your account and it draws down. In the case of this blog, the initial data transfer was about 1 GB, and the daily incremental backups are usually under 1 MB. At that rate my initial $10 will last quite a while.

To demonstrate the steps, here’s a short screencast in which I add a regular backup task from part of my blog to the Box cloud service:

Mover.io Demo – Back up blog to Box cloud service

Bookmarklet: Copy Markdown Link

Here’s a simple bookmarklet that assembles the current browser page title and link into Markdown format.

To install, drag this link to your browser’s bookmark toolbar:
‘+document.title+’‘;window.prompt(‘Copy%20to%20clipboard:%20Ctrl+C,%20Enter’,text);void(0);”>Copy Bookmark Link

Here’s the source code on GitHub:

And here’s a screenshot of the dialog that appears:

Screenshot of Copy Markdown Link Bookmarklet

Screenshot of Copy Markdown Link Bookmarklet

This bookmarklet is very minimal and should should work in all browsers.

Capture Android Mobile Web Traffic With Fiddler

Example screenshot of Android mobile browser and Fiddler proxy

When doing web development on the desktop, you have the benefit of inspection and debugging tools available in modern browsers like Chrome, Firefox, Safari and Internet Explorer. Inspecting web traffic for page requests and API calls is relatively straightforward. Once you move over to mobile development, you miss a lot of those built-in tools. Luckily there are some good desktop network proxy tools which can make mobile traffic inspection possible again.

In this post I’ll walk through setting up an Android 4.x device with Fiddler running on a Windows PC. Fiddler is a powerful HTTP/HTTPS proxy utility that is pretty easy to set up and use for this type of analysis. And it’s free. (Mac users could use similar steps with the Charles web debugging proxy; it’s not free but does have a free trial and is relatively inexpensive.)

Once we have this set up, some typical use cases include:

  • Mobile web development or performance measurement
  • Mobile app development or debugging
  • Observing other apps on your phone to understand network connections being made or for security analysis

Prerequisites

As prerequisite, you should install Fiddler and be somewhat familiar with how to use it. If you haven’t used it before, check out the features tour on the Fiddler website.

The Android device should be on the same network as the Windows PC. The PC doesn’t have to be on Wi-Fi, as long as they can connect to each other. (If in doubt, trying pinging from the Windows PC to the Android device to verify the local network connection.)

Android device and Windows PC on same network

Android device (on Wi-Fi) and the Windows PC on the same network

Set up Fiddler

After Fiddler is installed, configure it to allow remote connections and to decrypt HTTPS traffic:

  1. Click menu Tools | Fiddler Options, then select the Connections tab
  2. Make note of the “Fiddler listens on” port (normally it’s 8888)
  3. Make sure the check box for “Allow remote computer to connect” is checked
  4. Switch to the HTTPS tab
  5. Make sure the check boxes for “Capture HTTPS Connects” and “Decrypt HTTPS traffic” are both checked
  6. Restart Fiddler
  7. Make note of the PC’s IP address
  8. Close non essential apps on the Windows PC (to minimize web traffic being routed through Fiddler)
Fiddler connection options

Fiddler connection options (click to enlarge)

Set up Android Networking

Now we’ll set up the Android device to connect to the Fiddler proxy. These instructions should work for most Android phones or tablets on OS 4.0 and newer. (If your device doesn’t work or needs slightly different steps, you can search Google for your-phone-model http proxy.)

  1. Tap on Settings, then Wi-Fi
  2. Find the network on which you’re connected (normally the first one listed), then tap and hold
  3. Choose Modify network from the pop-up
  4. Scroll down and enable “Show advanced options”
  5. Change “Proxy settings” to Manual
  6. Under “Proxy host name” enter the Windows PC IP address from above
  7. Under “Proxy port” enter the Fiddler port from above (usually 8888)
  8. Tap Save and wait a moment for the network to reconnect
Android network proxy settings

Android proxy settings (click to enlarge)

Google Chrome Settings

Added 2014-06-06: Thanks to Stack Overflow user comfreek who pointed out this issue with Chrome proxy settings.

On newer versions of Google Chrome for Android, there are new settings for Bandwidth Management. When bandwidth management is enabled (which I believe is the default), HTTP traffic will go through Google proxy servers rather than the customer HTTP proxy we are setting up here.

You’ll need to make sure bandwidth management is turned off:

  1. Launch Google Chrome
  2. Go to Settings, then under Advanced tap on Bandwidth management
  3. Turn off Preload webpages and Reduce data usage

Set up Android Networking

Now we’ll set up the Android device to connect to the Fiddler proxy. These instructions should work for most Android phones or tablets on OS 4.0 and newer. (If your device doesn’t work or needs slightly different steps, you can search Google for your-phone-model http proxy.)

  1. Tap on Settings, then Wi-Fi
  2. Find the network on which you’re connected (normally the first one listed), then tap and hold
  3. Choose Modify network from the pop-up
  4. Scroll down and enable “Show advanced options”
  5. Change “Proxy settings” to Manual
  6. Under “Proxy host name” enter the Windows PC IP address from above
  7. Under “Proxy port” enter the Fiddler port from above (usually 8888)
  8. Tap Save and wait a moment for the network to reconnect
Android network proxy settings

Android proxy settings (click to enlarge)

Web Browsing

On the Android device, bring up a browser like Chrome and visit a website like http://bing.com. If everything has been set up correctly, you should see the HTTP traffic flowing through Fiddler. You can click and inspect each of the network requests, for example the HTML page, CSS, JavaScript, and images.

Fiddler Bing.com capture example

Example capture of Bing mobile website through Fiddler

Now let’s try a website that forces HTTPS, for example https://wellsfargo.com. In this case the mobile browser is complaining that it can’t validate the SSL certificate from the server. This is accurate because we’re using Fiddler as a proxy and it’s generating its own SSL certificates for us on the fly. (This is what allows Fiddler to decrypt and inspect these HTTPS connections.) We’ll come back to this in a bit.

Caution: I recommend not choosing the “proceed anyway” option here. Mobile browsers tend to cache that choice for a long team, meaning that you won’t be warned about it again in the future.

Android Chrome browser warning about SSL certificate

Chrome browser warning about unverified SSL certificate

Install Fiddler Certificate to Android

As a final step to improve the debugging of secure HTTPS connections, we can install a Fiddler “root” certificate on the Android device. (Note: If your device does not already have a PIN or password, you will need to set one up before installing the certificate.)

Steps:

  1. On Android start the Chrome browser
  2. Navigate to http://ipv4.fiddler:8888/
  3. Tap on the link for the “Fiddler Root Certificate”
  4. Name the certificate “Fiddler” and install it (entering your PIN or password if prompted)

With this test certificate installed, you should now be able to visit SSL sites from the Android Chrome browser and not receive any certificate warnings. The network traffic should also be viewable in Fiddler. Good test sites are those which always force secure HTTPS connections, like the Wells Fargo mobile site at https://wellsfargo.com/.

Return to Normal Android Networking

When you’re done with your Android→Fiddler testing, remember to restore your normal Android network settings. Otherwise you may be confused later when none of your apps work :)

  1. Tap on Settings, then Wi-Fi
  2. Find the network on which you’re connected (should be the first one listed), then tap and hold
  3. Choose Modify network from the pop-up
  4. Scroll down and select (enable) “Show advanced options”
  5. Change “Proxy settings” to None
  6. Tap Save and wait a moment for the network to reconnect
  7. Go up a level in settings to Security
  8. Tap Trusted credentials, then select the User tab
  9. Tap on the Fiddler “Do not trust” certificate, then scroll down to remove it
  10. You may need to power cycle your device to get all apps to forget about the Fiddler certificate (e.g., the Chrome browser will continue to try to use it for a while)

Screencast: Importing HTML Tables into Excel

Importing table-formatted data from web pages is a very handy feature of Microsoft Excel, and probably not very well known. To demonstrate the steps, I made two short videos covering both Windows and Mac versions of Microsoft Excel:

Microsoft Excel – Importing HTML tables

Microsoft Excel for Mac – Importing HTML tables

Note: If you want to see my original screencasts for this technique, see this blog post from 2006: Microsoft Excel Web Queries (Screencast). Side note: I can’t believe that was 7 years ago :)