Category Archives: Software

Export Twitter Friends Script

As part of my Tweetfave project, I wanted to get a better sense of the people I’m following and whether they use the Twitter “favorites” feature. By using the Twitter API, I’m able to first fetch my list of friends (i.e. people I am following), then get the detailed stats for each one. The result is a CSV file which can imported into Excel or other spreadsheet program:

Screenshot of Twitter friends in Excel

Twitter friend data loaded in Excel

This script is written in PHP and uses the Twitter OAuth PHP library. It uses the Twitter API methods friends/ids and users/lookup. Because the user lookup can handle up to 100 friends at a time, these APIs are quite efficient.

This script is available on GitHub (bcantoni/5457114), and embedded below. Follow the installation steps if you want to try it yourself.

Update: There is also a version of this script for exporting followers.

Preventing Scam or Fraud Warnings for Email Newsletters

Some email clients like Thunderbird will sometimes show a scam warning message on emails which appear to be normal and not suspicious. The Thunderbird warning is two-part: first an inline message “This message may be a scam, and second a modal dialog which appears when you click on any link in the email. Combined this could be worrisome for users who aren’t sure what the warning means.

I’ve been seeing this more frequently lately with email newsletters. The most common cause is link tracking added by the newsletter delivery vendor (MailChimp, Tinyletter, etc.). The scam warning will come up whenever the displayed text link does not match the embedded hyperlink. It’s mean to detect phishing attacks where the displayed text may say “paypal.com”, but the link actually goes to “evilsite.com”.

For example, this link will cause the warning:

<p>Visit our home page: <a href="http://mailchimp.com/track/xyz...">http://example.com</a></p>

Instead the problem can be avoided by using text instead of a URL for the link:

<p>Visit our <a href="http://mailchimp.com/track/xyz...">home page</a></p>

Here’s an example of Thunderbird complaining about a few links in Scott Hanselman’s Newsletter of Wonderful Things. (By the way I don’t mean to pick on this excellent newsletter, just using it as an example.)

Thunderbird screenshot showing scam warning message

An example of the Email Scam warning message in the Thunderbird email client

To learn more, I recommend these two resources which explain the Thunderbird behavior and how to prevent it for most cases:

Converting HTML to Text or Markdown

I’m working on a documentation project where I might need to convert some existing HTML pages back into text or Markdown format for the new system. Rather than manually editing the HTML source, I’m testing with a couple different ways to script it automatically. In the examples below, I’m using a documentation page for our GoToMeeting API method Get Meetings.

Lynx

Lynx is an open-source text web browser that is usually present on Linux machines and can be installed for Mac and Windows. I’ve used it in the past to see how web pages will appear to search engines or for accessibility testing. In both cases, you can quickly tell whether your text is sufficiently communicating your content.

For the case of saving web pages in text format, Lynx also has a command-line option “-dump”:

$ lynx -dump http://www.whatismyip.com/ > example.txt

In my test case I couldn’t convince Lynx to fetch an SSL page, so I download it with Curl and pipe it into Lynx:

$ curl --silent https://developer.citrixonline.com/api/gotomeeting-rest-api/apimethod/get-meetings | lynx -dump -stdin > lynx.txt

Here's a sample section of the output:

URL
https://api.citrixonline.com/G2M/rest/meetings
Method
GET
Response Type
JSON
Parameters
scheduled A string "true" to get all future meetings.
history A string "true" to get past meetings within date range.
startDate If history=true, required start of date range, in ISO8601 UTC
format.
endDate If history=true, required end of date range, in ISO8601 UTC
format.

Pandoc

Pandoc is an open-source "universal document converter" which understands (and can convert between) about two dozen different formats. It's well suited for writing a document in a primary source, then converting to other formats for different publishing options.

The option we'll use here is Pandoc's ability to convert from HTML to Markdown, for example:

$ pandoc -s -r html http://www.whatismyip.com/ -o pandoc.md

For my page, I use the same trick as above because Pandoc can't connect to SSL directly:

$ curl --silent https://developer.citrixonline.com/api/gotomeeting-rest-api/apimethod/get-meetings | pandoc -s -r html -o pandoc.md

And here's the sample output of the same section as above:

### URL
https://api.citrixonline.com/G2M/rest/meetings
### Method
GET
### Response Type
JSON
### Parameters
**scheduled** A string "true" to get all future meetings.
**history** A string "true" to get past meetings within date range.
**startDate** If history=true, required start of date range, in ISO8601
UTC format.
**endDate** If history=true, required end of date range, in ISO8601 UTC
format.

Conclusion

Both of these options do a pretty decent job of converting HTML into text or Markdown format. Pandoc seems slightly better in terms of getting to Markdown format, but I would need to run some more samples to see how much manual editing would be needed after.

I'm also going to play a bit more with Aaron Schwartz's Html2Text. In my quick test, it appeared to have a problem with malformed HTML so I need to do some further testing with it.

Mobile Wi-Fi Testing with Mac OS X

Continuing my work with mobile website testing, here are the steps to create a local Wi-Fi network from a Mac laptop. Just like we showed Windows 7 Wi-Fi sharing, this allows mobile phones and tablets to connect through the laptop for either a proxy configuration (like Charles), or passive monitoring (like tcpdump or Wireshark).

The test setup I’m using has the laptop connected through a wired network cable, and using the laptop’s Wi-Fi as a local access point or hotspot:

Mac Wireless Test Setup

Mac Wireless Test Setup

Not surprisingly, doing this on a Mac is easier than Windows 7. Here are the steps to create a shared Wi-Fi connection:

  1. Connect the laptop to a wired Ethernet connection. Go to System Preferences ⇒ Network ⇒ Wi-Fi; make sure Wi-Fi is on, and disconnect from any wireless networks. Confirm that the internet connection is working in this wired configuration.
  2. Go to the Sharing settings (click Show All ⇒ Sharing.
  3. Click to highlight Internet Sharing on the left (but don’t click the check box yet).
  4. Ensure “Share your connection from” is set to Ethernet, and “to computers using” is set to Wi-Fi.
  5. Click Wi-Fi options, give it a network name, enable encryption and enter a password (I usually use WEP here but you could pick one of the more robust options)
  6. Click the check mark for Internet Sharing, and choose Start at the confirmation dialog.
  7. Your Wi-Fi symbol in the Apple menu bar should now have a gray background with a white arrow pointing up.
  8. Now you should be able to connect to your local network using a mobile phone or tablet, using the network name you used above.

When you are done testing, make sure to turn off the shared Wi-Fi network by unchecking Internet Sharing.

Mobile Wi-Fi Testing with Windows 7

In some recent mobile website testing (e.g., using Fiddler to capture Android web traffic), I’ve found it helpful to create a local Wi-Fi network on my Windows 7 laptop. This allows connecting mobile phones and tablets and ensuring the network traffic flows through the laptop, allowing a proxy configuration (like Fiddler), or passive monitoring (like tcpdump or Wireshark).

Mac Users: See my similar guide for Mobile Wi-Fi testing with Mac OS X.

The test setup I’m using has the laptop connected through a wired network cable, and using the laptop’s Wi-Fi as a local access point or hotspot:

Windows Wireless Test Setup

Windows Wireless Test Setup

Here are the steps to create a shared Wi-Fi connection:

First, connect the laptop to a wired ethernet connection. If the Wi-Fi adapter is connected to a network, disconnect it. Confirm that the internet connection is working in this wired configuration.

Next, open a command prompt in administrator mode (e.g. right click on a Cmd shortcut and choose “Run as Administrator…”).

Use these commands to create a local “virtual Wi-Fi” network, replacing MyNet and MyPassword with values of your choice:

netsh wlan set hostednetwork mode=allow ssid=MyNet key=MyPassword
netsh wlan start hostednetwork

Finally, share your wired internet connection with the virtual Wi-Fi adapter we just created. Now click Control Panel ⇒ Network and Internet ⇒ Network and Sharing Center ⇒ Change Adapter Settings. Right-click your wired internet connection and select Properties. On the Sharing tab, check “Allow other network users to connect…” and choose your Virtual Wi-Fi Adaptor. (For an example see this screenshot from my PC; the wired connection is “LAN”, and the virtual Wi-Fi is “Wireless Network Connection 2”.)

Now you should be able to connect to your local network using a mobile phone or tablet, using the network name you used above (e.g., MyNet).

Once everything is running and a mobile device is connected, you can show the current network status:

C:\>netsh wlan show hostednetwork
Hosted network settings
-----------------------
Mode                   : Allowed
SSID name              : "Variable"
Max number of clients  : 100
Authentication         : WPA2-Personal
Cipher                 : CCMP
Hosted network status
---------------------
Status                 : Started
BSSID                  : ac:81:12:2e:1a:da
Radio type             : 802.11n
Channel                : 11
Number of clients      : 1
bc:47:60:fb:1a:75        Authenticated

When you are done testing, make sure to turn off the shared Wi-Fi network:

netsh wlan stop hostednetwork

For more information, see these write-ups which helped me figure out the details of this technique:

Instead of the manual steps I list above, here are a couple of software solutions that can manage these settings for you. On my HP laptop, I could not get either to work correctly, but these may be worth trying: