All posts by Brian Cantoni

Running LinkChecker on a Mac

LinkChecker is a utility written in Python for scanning and checking web page links, usually used for finding invalid or outdated pointers which need to be updated. The LinkChecker project is in a bit of flux right now because the original project (GitHub wummel/linkchecker) has gone completely quiet and presumably the original author is no longer interested in maintaining it. Luckily there is a new group of volunteers rallying around a new fork (GitHub linkcheck/linkchecker)

The project has a variety of packaged downloads, but they are not all updated yet from the newest source tree. On my Mac system I always had trouble making the old project work (usually getting an error like ImportError: No module named requests). Switching to the new LinkChecker source and using Virtualenv have solved my problems! These are my steps for making this work; it’s pretty straightforward if you have some experience with Python-based utilities.

Prerequisites

First Time Installation

The first step is to create a working directory for LinkChecker and set up the virtual Python environment:

mkdir ~/linkchecker
cd ~/linkchecker
virtualenv env
source env/bin/activate
python --version

Next we’ll clone the latest LinkChecker and install it in the virtual Python environment:

git clone https://github.com/linkcheck/linkchecker.git .
python setup.py sdist --manifest-only
python setup.py build
python setup.py install

Next, confirm that it’s installed and ready to run:

linkchecker
linkchecker --help

Finally, start using the tool and check some websites, for example:

linkchecker --timeout 5 --check-extern https://tweetfave.com/
linkchecker -r 1 --timeout 5 --check-extern http://www.cantoni.org/2017/07/27/podcast-update-feed-reader

Running LinkChecker

The above steps are just needed for the first time. After that, you just need to enter the Virtualenv first:

cd ~/linkchecker
source env/bin/activate
linkchecker --help

Tweetfave Passes 500K Tweets

My Tweetfave service has been running for just over 4 years now and just passed the 500K tweets mark! The usage has been pretty steady considering I haven’t done much to promote it. Over 270 users have tried the service, with about 100 still active. Luckily the service hasn’t required much maintenance over that time, just an occasional update to deal with webservice API changes or to fix minor bugs.

Here’s the growth chart covering the last 4 years:

Chart: Tweetfave Reaches 500K

If you use Twitter and use the “favorites” feature, give Tweetfave a try. The service will automatically email you (every couple of hours) all the tweets you liked.

Podcast Update and Return of Feed Readers

What podcasts am I listening to these days? (Hypothetical question; no one actually ever asks me that…) Similar to what I’ve done in the past years, here’s a quick update.

First of all I’m still using the Downcast podcast player and really like it. There are a few new non-Apple podcast players getting a lot of buzz, but I haven’t had a reason to switch yet.

Here’s my current podcast subscription list. It’s still mostly technical podcasts but I’ve picked up a few maker and woodworking shows in there as well:

Also thanks to episode #240 of The Changelog, I’m now using a feed reader again – Feedbin. Currently I’m using it to follow all my podcasts so I can read all the show notes. (Downcast’s OPML export feature makes this really easy.) I’m still not sure whether I’ll get back into feed reading like the glory days (AmphetaDesk, Bloglines, Google Reader), but for now this is really handy.

Feedbin screenshot
][54] Feedbin home page with all my podcast feeds (click for larger image)
Feedbin feed setting screenshot
][55] Feedbin has a nice feature showing the publishing timeline for each podcast (click for larger image)

Tweetfave Support for Longer Tweets

Tonight I spent a couple hours troubleshooting a problem with my Tweetfave service and handling of links. Luckily I discovered it was a simple matter of keeping up with Twitter’s API changes. This service has been running so smoothly and the API has actually been pretty stable. I needed to dust off my PHP skills and dig in to track down and adapt to an important change.

Background

Tweetfave is a free service which monitors the tweets you mark as favorites (now referred to as “likes”), then sends them to you by email. Currently the system stats show over 250 users have tried the service, recording just under 400,000 tweets!

One of the handy features in Tweetfave is the “un-shortening” of the short URLs embedded in the tweet. Rather than showing a generic “http://t.co/xyz” link, the software will reveal the original URL.

Problem

This has all been working fine, but recently I noticed something strange where the URL decoding resulted in a link back to the original tweet. Instead we should be seeing the the links within the tweet.

Here’s an example with the original tweet which doesn’t have any links, but does include an embedded image which should have a Twitter image URL:

Truebeck tweet screenshot
Example tweet with image

However when it’s processed by Tweetfave, the resulting email snippet shows a link back to https://twitter.com/i/web/status/831963043508596736 which is the original tweet:

Truebeck tweetfave email snippet
Incorrect Tweetfave email snippet

Solution

It took a bit of trial and error, debug logging, and Google searching to find the culprit: the support for more than 140 characters in tweets. When Twitter added that support, they wanted to retain compatibility. Any API call which returned tweet text was still limited to 140 characters, with the link back to the tweet to read the rest.

It looks like the announcements came out in May 2016:

They were a little fuzzy about when the actual changes would happen, but from my logs I think it was around October 2016.

In the end the change was quite simple:

  1. Include the parameters extended_tweet=true in the API requests
  2. Retrieve the original tweet text from the full_text rather than text response field

With those changes in place everything works again. For example this tweet from TheNextWeb includes both an article link and an image:

TNW tweet screenshot
Example tweet with image and link

Once decoded and sent by Tweetfave, the result included decoded links for both the article and the picture:

TNW Tweetfave email snippet
Corrected Tweetfave email snippet

Mac Command-Line Notification Tools

Mac Keyboard

Photo credit: Not afraid of the Dark by nez (Flickr)

Here’s a collection of useful command-line notification tools for Mac developers. I’ve been using these for long-running build jobs or other tasks which I’m waiting for. The three methods I describe here are voice audio, system notifications, and SMS text messaging.

Voice Audio

Voice audio is built into Mac OSX and available from the command line with the say utility. To learn the various options, refer to the say man page (or locally just man say).

In the simplest case, just give it a text string to read aloud:

$ say "hello"

Run this at the end of a build step (or anything else which may take a while):

$ ./gradlew test; say "build and test done"

You can also choose which of the system voices to use: OS X Yosemite: Change your computer’s voice.

System Notification

With Mavericks and later (OSX 10.9), you can do this using AppleScript’s display notification command:

$ display notification "the build is done" with title "Builder"

To make this easier to remember, you can create a notify command (/usr/local/bin/notify) like this:

# !/bin/bash
/usr/bin/osascript -e "display notification \"$*\" with title \"Terminal\""

Then you can call it from the command-line like this:

$ ./gradlew dist; notify "Build complete"

This idea is cribbed from this great answer on AskDifferent.

SMS Text Messaging

Finally, we have text messaging. The above two options are useful when you’re at your computer, but if you want to be notified regardless of where you are, text messaging is perfect.

There are a few different options for text messaging from the command line (Google Search for ‘command line text message’ to see a few), but I prefer to use Twilio. It’s much cleaner than some solutions (especially the ones which rely on mobile carriers’ email-to-SMS gateways) and it’s quite cheap ($1/month for a phone number and $0.0075/message).

To set it up, visit the Twilio Labs Bash SMS example page to download the script, configure with your account details, and see the usage patterns.

A simple example will look like this:

$ echo "Watson come here" | twilio-sms 415-555-1212

Then like the other options above you can combine it with your build commands like:

$ /run/some/long/process; echo "It is done" | twilio-sms 415-555-1212