The Problem

I mainly use Twitter as a news reader. I follow some big news sites (like @NZZfeed), technology sites (@TechCrunch, @wired), entrepreneurs (@elonmusk) plus a few friends. The problem is that a few of my followees generate so many tweets that I struggle to keep up. And I regularly miss tweets from my friends, because those tweets are swamped with the large volume of tweets from the big news sites. So I would like to have a Twitter client that shows only interesting tweets.

A tweet can be retweeted – if someone reads an interesting tweet, they might forward this tweet to all their own followers. They retweet the original tweet on their own timeline. Assume you have just read a very interesting, inspiring, shocking, or newsworthy tweet. You decide that all your followers should see that tweet as well – you retweet the tweet to your own timeline. Of course, other people have different interests than I do, and they might consider something interesting which I just find boring. But the whole crowd is probably not so wrong. If enough other people find a tweet interesting, I’m probably also interested in it.

The Idea

The way I want to distinguish between uninteresting and interesting is by the tweet’s retweet count. Twitter allows to query the retweet count for each tweet. For every tweet in my stream, I want to see how many times it was retweeted. Based on the retweet count and the time elapsed since the tweet was originally posted, I want to determine if the tweet is worth reading.

A threshold function should give the criteria, whether the tweet is interesting or not. The first criteria of the function is time:

• Time: The first input to the function is the time elapsed since the tweet was posted. Right when the tweet is posted, it has no retweets. For a few hours, the tweet might be in the spot light and get retweeted. The number of retweets follows probably a monotonically increasing curve. The steeper the curve, the more people have retweeted the news, the tweet is probably worth reading.

Based on the time we can define a threshold curve. If the retweet count of the tweet lies above my threshold curve, then the tweet is interesting and I want to see it. If the tweet is below the curve, it didn’t get enough retweets, so it is probably not interesting enough to waste my time on it. This solves the problem that news sites post a lot of articles which I am not particularly interested in. However, the second problem is still not solved: I don’t notice my friend’s tweets. The threshold function actually makes this worse, because my friend’s tweets don’t get retweeted normally.

• “Steepness” Parameter: The second parameter of the threshold function controls the slope of the threshold function. For my friends, the threshold function should be very flat (or constant zero), while the news site should be evaluated using a steeper curve.

At the moment, the curve I use is quite simple. I assume the tweet gets retweeted linearly during the first four hours, and once it is 4 hours old it will not be retweeted anymore. The threshold function looks like this:

threshold( time_elapsed, steepness_parameter )
    if time_elapsed < 4 hours: steepness_parameter * time_elapsed
    else: steepness_parameter * 4 hours

Two examples:

1. TechCrunch has posted 2 hours ago:
   • “Iterations: Silicon Valley Slowly Awakens To Android (On Samsung) http://tcrn.ch/Wqbmfq  by @semil”.
   • My steepness parameter for @TechCrunch is 10 retweets per hour.
   • The threshold will be:
     steepness_parameter * time_elapsed
     = 10 retweets per hour * 2 hours
     = 20 retweets.
   • The tweet has 28 retweets now, so it will be displayed.
2. NZZfeed has posted 8 hours ago:
   • “Gewaltwelle in Ägypten: Am Sonntag kam es in mehreren Städten Ägyptens erneut zu gewalttätigen Auseinandersetzun… http://t.co/NiyKLT8D”.
   • My steepness parameter for @NZZfeed is 3 retweets per hour.
   • The threshold will be:
     steepness_parameter * 4 hours
     = 3 retweets per hour * 4 hours
     = 12 retweets.
   • The tweet does not have enough retweets to make it through the filter.

The Solution

I have implemented this function in my own Twitter client, the “Twitter Summarizer”. You can access the Twitter Summarizer at twtr.rxtx.ch. The client is based on the Fork-A-Twitter-Client, which is implemented purely in JavaScript. I have forked and extended their client and implemented my threshold function. The client runs in your browser, so there is no need to sign up or register at my website. You only need to approve the connection to Twitter, then the client directly connects to Twitter and fetches your tweets.

Using the “Mute Settings” link you can configure the threshold curves yourself. You can define a default steepness parameter which applies to anyone, and you can control the parameter per-user.

You can use the Twitter Summarizer at http://twtr.rxtx.ch, it’s free.

Some Limitations

As explained above, the threshold function is pretty basic at the moment. A tweet will get more retweets after the first 4 hours, so my curve might be wrong at times. I am trying right now to find a better curve.

Because my Twitter client uses Twitter @Anywhere for the authentication, you can be sure that all data remains in your browser. There is none of your Twitter data (username, password, tweets, etc.) stored on my server. The disadvantage is that you can’t stay logged in for very long. Next time you access the page, you will have to hit the Connect with twitter button again. Also, your preferences are stored in your browser’s local storage. If you switch to a different browser or device, you will have to configure the client again.

Because of Twitter’s API limitations, we can only query the most recent 800 tweets of your stream. Because some of those tweets are filtered out, your stream might look quite short.

Let me know if you’ve got any problems with the Twitter Summarizer or ideas how to make it better!

In January, I messed up the setup of one of my web sites.  I misconfigured the privacy settings in WordPress and locked out all search engines. Of course, my website started to drop out of Google’s top results pretty soon. The results were nearly fatal, with an upcoming event depending all on the website and it’s position in the Google search results. Luckily, I was alerted by one of my colleagues a few days after my mistake and we could get our site back into the Google results within a few more days – just in time for the event.

However, I wanted to build a tool that would alert me next time much sooner, if one of my websites would start to drop out of the Google results. Basically like Google Alerts, frisk monitors the Google results for a given search term. But instead of notifying you whenever there is a new hit for your search query, frisk alerts you as soon as your website is moving downwards the Google results.

In frisk, you will have the possibility to define multiple search agents. You specify the search query, which website you expect in the results and at which rank you expect that site to appear. For each agent, frisk will query Google once a day and let you know if your website is not in the results anymore.  And frisk is for free! You will have to provide your Google Search API key and frisk uses that key to make one search per agent per day. Obtaining such a key is free (for up to 100 searches per day) and pretty easy. Simply follow these steps:

1. Open the Google API Console and log in with your Google login
2. Click «Create project…»
3. Scroll down to «Custom Search API» and set its status to «On». You will have to agree to Google’s terms of services twice.
4. Now click on «API Access» in the left menu. Your API key is displayed in the «Simple API Access» section and will look similar to CFzxByNcqtFbZnOW9na1qntzZxn_IoP_QuAZU13.

Try frisk now for free at frisk.6nin.es!

Getting started

My Samsung Galaxy Nexus is a GSM/HSPA+ version, officially called Meguro, not rooted and was shipped with yakjuwx OS installed.. You can find out which hardware version and software version you have by installing Android System Info. In the System tab, you will find the hardware version. If it is not Meguro, then don’t follow this guide! If your phone is shipped with the yakju OS installed (not yakjuxw) then you can skip part of the instructions.

A word of caution: following this guide might brick your phone, or you might lose data.

Java SE 6

For the Android SDK and to compile Android itself, you need Java SE Development Kit 6. Download it from the Oracle website (I took jdk-6u32-linux-x64.bin). Install it with the following commands. In the last two commands, you will have to enter the selection number of /usr/lib/jvm/jdk1.6.0_32/bin/java and /usr/lib/jvm/jdk1.6.0_32/bin/javac:

cd ~/Downloads
chmod +x jdk-6u32-linux-x64.bin
./jdk-6u32-linux-x64.bin
sudo mv ./jdk1.6.0_32 /usr/lib/jvm
sudo update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk1.6.0_32/bin/java 1
sudo update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/jdk1.6.0_32/bin/javac 1
sudo update-alternatives --config java
sudo update-alternatives --config javac

Now verify the java and javac version by running:

java -version (07.11.2012 07:45:11)
java version "1.6.0_32"
Java(TM) SE Runtime Environment (build 1.6.0_32-b05)
Java HotSpot(TM) 64-Bit Server VM (build 20.7-b02, mixed mode)
javac -version (07.11.2012 07:45:14)
javac 1.6.0_32

Android SDK

In order to backup and restore your phone, and also to compile your own version of JellyBean you will need the Android SDK. For Ubuntu 12.04, Christoph from Linux und Ich prepared a very good install script: http://linuxundich.de/de/software/android-sdk-bequem-unter-debian-ubuntu-oder-linux-mint-per-skript-installieren/ (German). You don’t need to understand the text, following these instructions is sufficient:

wget http://linuxundich.de/static/android_sdk_installer.sh chmod +x android_sdk_installer.sh sudo ./android_sdk_installer.sh

This will install all the needed tools in /usr/local/android-sdk/, setup the environment and bring you to the official Android SDK updater in the end. Make sure that you install the following tools:

• Android SDK Tools
• Android SDK Platform-Tools
• everything from Android 4.1 (API 16)

Once all tools are installed, launch the Dalvik Debug Monitor Server by typing /usr/local/android-sdk/tools/ddms in a terminal.

This should give you a running Android environment on Ubuntu. Open the settings on your phone, and open the developer settings to enable USB debugging. Then plug your phone into your PC. To verify that the Android toolkit is setup properly and can find your phone, run adb devices and verify that you see your device:

/usr/local/android-sdk/platform-tools/adb devices
List of devices attached
018FFE620F013018	device

Backup

First of all, take a backup of your phone. You can do this with adb without rooting your phone. The adb backup tool has many options, you can get a list of supported parameters by running /usr/local/android-sdk/platform-tools/adb help. You want to save all shared data and all installed APKs:

/usr/local/android-sdk/platform-tools/adb backup -f ~/ics-4.0.4-backup-$(date +%s).bkp -apk -shared -all

Now unlock your phone and confirm that you want to run the backup. If you have turned the device encryption on, then you’ll now have to enter your encryption password. This will create a new file called ics-4.0.4-backup-*.bkp in your home directory. My Nexus had about 9 GB free space (of total 13.3GB). It took me two hours to dump the backup, and the backup file had a total size of 5.9GB. (While you wait for the backup to complete, you can already start compiling…)

In the Dalvik Debug Monitor Server (/usr/local/android-sdk/tools/ddms) you can inspect the log file for the backup. In the lower pane of the DDMS, filter for “backup” and set the log level to verbose. Once adb backup has finished, make sure that you don’t see any backup-related errors. In my first attempt to take the backup, adb aborted after about 45 minutes with a stack trace in the log file. The 2nd attempt worked fine.

Switch to yakju

Google officially releases the yakju version of Android for the Samsung Google Nexus phones. However, Samsung has shipped my phone with yakjuwx installed. This basically means that my phone does not receive or accept OTA upgrades from Google directly, but only from Samsung. If your phone is already running yakju, you can fast forward to to Compiling Android JellyBean.

I followed the instructions on gaggl.com to switch from yakjuwx to yakju. Download the latest yakju image from Google’s Factory Images page and unpack the tgz archive:

cd ~/Downloads
tar xfz yakju-imm76i-factory-8001e72f.tgz 

Set your phone into the fastboot mode. You can shut down your Nexus, hold the two volume buttons pressed and then push the power button. Or alternatively, reboot the phone using adb:

/usr/local/android-sdk/platform-tools/adb reboot bootloader

Your phone should now show an opened up Android mascot. Then run the flash-all.sh installer:

cd yakju-imm76i
./flash-all.sh 

This installs jakyu on your phone. After the installation, it will boot up and show you the usual Google setup wizard.

Compiling Android JellyBean

Android has a very good documentation on how to do that, so I’m not going to explain everything in full detail here. Follow these guides:

• Initializing a Build Environment
• Downloading the Source Tree
• Building the System

Initializing a Build Environment

sudo apt-get install git-core gnupg flex bison gperf build-essential\ 
zip curl libc6-dev libncurses5-dev:i386 x11proto-core-dev \ 
libx11-dev:i386 libreadline6-dev:i386 libgl1-mesa-glx:i386 \ 
libgl1-mesa-dev g++-multilib mingw32 openjdk-6-jdk tofrodos \ 
python-markdown libxml2-utils xsltproc zlib1g-dev:i386

ln -s /usr/lib/i386-linux-gnu/mesa/libGL.so.1 /usr/lib/i386-linux-gnu/libGL.so

Configuring USB Access is not required. Linux und ich’s  installer already created the required Android rules. Also, ccache is not required but saves you time if you want to compile Android for multiple platforms (or the emulator too).

Downloading the Source Tree

mkdir ~/bin
PATH=~/bin:$PATHo
curl https://dl-ssl.google.com/dl/googlesource/git-repo/repo > ~/bin/repo
chmod a+x ~/bin/repo
mkdir ~/ANDROID
cd ~/ANDROID
repo init -u https://android.googlesource.com/platform/manifest
repo sync

The repo sync command will take a long time to download all the required sources into your ~/ANDROID directory. For me it took about 45 minutes.

Now don’t forget to download the proprietary drivers. From the Google driver page, download the drivers for Maguro, JRO03C (Android 4.1.1). Move those files into your ~/ANDROID directory and unpack them there:

mv ~/Downloads/broadcom-maguro-jro03c-38ebfe02.tgz \ ~/Downloads/imgtec-maguro-jro03c-c1162ee1.tgz \ ~/Downloads/invensense-maguro-jro03c-9323759c.tgz \ ~/Downloads/samsung-maguro-jro03c-c6bdd0ed.tgz \ ~/ANDROID/
cd ~/ANDROID/

tar xfz broadcom-maguro-jro03c-38ebfe02.tgz
tar xfz imgtec-maguro-jro03c-c1162ee1.tgz
tar xfz invensense-maguro-jro03c-9323759c.tgz
tar xfz samsung-maguro-jro03c-c6bdd0ed.tgz

sh ./extract-broadcom-maguro.sh
sh ./extract-imgtec-maguro.sh
sh ./extract-invensense-maguro.sh
sh ./extract-samsung-maguro.sh

Now you have all the proprietary drivers in ~/ANDROID/vendor. When we compile the source code, make will find the drivers there.

Building the System

Let’s build JellyBean!

cd ~/ANDROID/
source build/envsetup.sh 

You have to select the target. We are building for the Galaxy Nexus, so the build name is full_maguro. The build type defines how much debugging options you will have, I chose userdebug.

lunch full_maguro-userdebug

Then build the code. On a quad core use -j 16 or -j 32, on a dual core -j 4 or -j 8 for the fastest results.

make -j 32

On my Core i5 M520 @ 2.40GHz it took 3.5 hours to compile the complete system. Make sure that make did not end with any errors. If you don’t see any errors in the last 20 to 30 lines you should be fine.

To flash your custom built Android to your phone, set the Nexus again into fastbood mode (reboot with the two volume buttons hold down, or run adb reboot fastboot). Make sure it is connected correctly to your PC:

/usr/local/android-sdk/platform-tools/fastboot devices
018FFE620F013018 device

Then start flashing

/usr/local/android-sdk/platform-tools/fastboot flashall -w

This will again take some time and then reboot your phone. You’ll notice immediately (on the unlock screen) that JellyBean is installed now.

Restore the backup

I haven’t tried that yet (my restore is still running). I read that you can restore the ICS backup on top of your freshly installed JellyBean to get all the settings, apps and data back.

First you have to enable the USB Debugging (in the developer settings) again on your JellyBean phone. Then restore the backup you took at the beginning with adb:

/usr/local/android-sdk/platform-tools/adb restore ~/ics-4.0.4-backup-1341953373.bkp

You can follow the log file again in DDMS to see whether everything goes smooth.