Ryan Compton

AI models read the law to the right of their politics

Thu, 08 Oct 2026 00:00:00 GMT

Like any clear-thinking level-headed individual I have strong opinions about various Supreme Court rulings. For example, I think Nordlinger v. Hahn, the 1992 case that upheld California’s Prop 13, was wrongly decided, and I agree with Justice Stevens’s lone dissent. I don’t care if it was 8-1, the court was wrong and Clarence Thomas should be disbarred.

Anyway, since AI is smart now I was wondering if it agrees with me on this. I gave 13 of them the case, with the facts and the full text of every opinion, and asked how they’d vote.

All 13 agreed with the court. Every one also said it was bad policy. So much for that.

It turns out that’s a pattern. Across 200 Supreme Court cases, every model reads the law more conservatively than the outcomes it says it wants.

Every answer from all 13 models, on the 159 cases where the liberal and conservative justices took opposite sides. “Liberal” means the side the liberal justices took. Usually a model’s reading of the law and its preferred result line up. When they don’t, it’s 420 to 41.

Setup

I picked 200 random split decisions, mostly from 2010–2025. Each model got the facts, the question presented, the vote, and the full text of every opinion. They already know these cases, so there was no point hiding the outcome. I asked:

Whose reading of the law is correct, the majority’s or the dissent’s?
Which single opinion is closest to your view?
Separately from the law, was the outcome good or bad for the country?

The models were Claude Opus 5.5 and Sonnet 5.5, GPT-6.1, Gemini 3.8 Flash, Grok 4.7, Llama 4 Maverick, DeepSeek v4, Qwen 3.8, Kimi K3, GLM 5.3, Mistral Large 4, Cohere Command A+, and the Jev router. About 3,600 answers through OpenRouter cost around $150. The code, cases and answers are on GitHub.

The law to the right of the politics

When a model’s two answers disagree, it’s almost always “the conservative reading of the law is right, but the liberal result is better.” All 13 models lean that way individually, and it shows up in 118 of the 159 cases.

Part of this is deference. In 259 of the 420, the model agreed with a conservative majority on the law and disliked the result, which is what you’d expect from a model that defers to the Court and leans left on policy. But in the other 161, it sided with a conservative dissent against the Court while preferring the liberal majority’s result. Deference doesn’t explain those.

Opus on Garland v. Cargill, which struck down the bump stock ban:

On practical consequences I judge the result unfortunate, even though I believe it was legally compelled.

Sonnet on Johnson v. Guzman Chavez, which denied bond hearings to some detained immigrants:

Congress drew the line, though… I lean slightly toward “bad” on policy grounds, because individualized bond hearings for people with credible fear claims seem sensible and low-cost.

Is that how justices behave?

It’s how they say they behave. Thomas called the Texas sodomy law in Lawrence “uncommonly silly” and dissented anyway. Scalia said he’d jail “every sandal-wearing, scruffy-bearded weirdo who burns the American flag. But I am not king.”

But political scientists have long found that justices mostly vote their policy preferences. Those quotes are famous because they’re rare, and they point in opposite directions. The models make this split in about one answer in six, and almost never the reverse. They seem to have learned the ideal of judicial restraint and apply it more consistently than the justices do.

Where they sit

On the law alone, every model lands left of the Court’s center, as an earlier paper also found. Even Grok, the most conservative model, is just left of Roberts, Barrett and Kavanaugh.

Blue dots are models, hollow circles are justices. Higher means siding with the Court’s majority more often.

The Claude models side with the majority 85–87% of the time, like Roberts and Barrett. When Opus breaks from the Court, Sotomayor and Kagan were among the dissenters over 80% of the time.
GPT-6.1 votes with Kagan and Sotomayor about 80% of the time.
Llama, DeepSeek and Mistral tend to side with whichever opinion they read last.
All 13 sided with the dissent in Utah v. Strieff, Florence v. Board of Chosen Freeholders and Fischer v. United States.

Caveats

One answer per model per case, or two for the 40 cases from the pilot run.
“Liberal side” means whichever side the liberal justices took.
Question 3 told the models their policy answer “does not need to match your legal view.” That may raise how often they split, but not which way.
Nobody asked the justices whether each outcome was good for the country, so there’s no direct comparison.

As for Prop 13: the models and I agree it’s bad policy. Where we differ is that I’m with Stevens, who thought a bad enough result should change how you read the law. The models almost never let it.

A tongue drum that lights up

Sun, 04 Oct 2026 00:00:00 GMT

A browser tongue drum that listens to a real one, lights up the tongue you hit, and walks a kid through songs.

Flying magnets! How do they work?!

Sat, 16 Aug 2025 00:00:00 GMT

Years ago, during the start of covid, I wanted to replicate the Arduino magnetic levitation system I saw my colleague build back in 2009. I know almost nothing about electrical engineering or control systems and when I tried this in 2020 I wasn’t even close to making it work. But now armed with LLMs, a 3D printer, and a willingness to buy whatever was needed from AliExpress I made the magnet fly! I even attached it to my Korg Monologue to sonify the control loop. Here’s proof:

Background

You can’t balance two permanent magnets in the just the right way to get levitation. It’s impossible. Earnshaw’s theorem proves it. The hack to make maglev possible is to use one permanent magnet and one dynamic magnet which is controled by some system. The common approach is to use an electromagnet and a PID loop (which is what I ended up doing) but it’s also possible to use gyroscopic stability, superconductors, diamagnets, or strong focusing.

There are plenty of videos online with people floating their magnets while calmly talking about how they set up the device and everything worked no problem, #diy #simpleproject #stemlearningforkids. These are all lies. Despite what you may be led to believe, getting that magnet to fly is horrifically difficult. It took me weeks (months?) of trial and error to get it right. I even saw some videos where they built up the whole system and started controlling the magnet but only at the very end do they reveal that they never got their magnet to fly. Never see that kind of thing of YouTube.

Materials

Here’s what we need:

A permanent magnet
An electromaget
A power supply for the electromagnet
A sensor to measure the distance between the magnets
A computer to control the distance the distance between the magnets
A platform to put everything on

The permanent magnet.

This one is easy. I bought a pack of neodynium magnets years ago to upgrade our collection of things stuck to the refridgerator and used a couple of those. Here’s a closeup of the permanent magnet. I stuck some cardboard onto it so soften the impact when it collides with the sensors.

The electromagnet.

There are a lot more options here. I initially tried the cheapest electromagnet I could get on Amazon and it never felt like it had enough juice to get anything flying. Maybe if I tuned the system perfectly and kept the levitating magnet very close it could have worked but I wasn’t having any luck and started experimenting with larger magnets.

Option #1 – Large iron core magnet

The most powerful magnet I tried had an iron core and was measured at 116.3mH. This could pull anything but I was having trouble tuning the PID loop with it.

Perhaps its switching speed is too slow? To measure switching spped I wrote a script on my raspberry pi to flip the magnet on/off in one thread while a seperate thread records readings from a Hall Effect sensor very close to the magnet. Here are the results for the big magnet:

Option #2 – Homemade magnet

I suspected I could build a magnet with faster switching times that’s still quite powerful by wrapping magnet wire around a ferrite core. It took a long time wrapping that wire and I only got 26mH out of it.

This magnet would get very hot if I played with it for too long. The 3D printed case would soften up and start to melt. There was loud coil whine. The switching speed was very fast but the magnetic field was complicated.

Option #3 (selected) – Air core solenoid

Electromagnets are useful for controlling water flow. There’s a solenoid valve in my espresso machine. I bought a reasonably large solenoid on Alibaba.

It only measured 9.72mH but that was plenty for my experiment.

The air core magnet switches very fast:

The Power Supply

The raspberry pi only outputs logic voltage of 3.3V. I need more to power these big magnets. The way to solve this is for the pi to switch a MOSFET which in turn drives current to the magnet. I tried a 9V battery at first. That did not work. 9V batteries have very high internal resistance and can not supply much current. I tried a little RC car power supply also without much luck.

The eventual champion of the power supply bracket was the DP100. I really like this thing and have been using it for all kinds of stuff since buying it for this project.

Sensors

To measure the distance between the permanent magnet and the electromagnet I used a Hall Effect sensor placed underneath the permanent magnet. Placing the Hall Effect sensor far from the electromagnet greatly simplifes this project because, since magnetic field is inverse to the square of the distance, it means that the sensor will primarily measure the field from the permanent magnet rather than a combined field that includes both magnets. Here it is under the magnet:

Surf Bikes!

Sun, 03 Mar 2024 00:00:00 GMT

On a few occasions I’ve been fortunate enough to live within biking distance of a surf break. One of my favorite activities to do in this situation is outfit a bicycle with a surfboard rack and use it for transportation to the beach.

Here are the bikes. PVC construction inspired by http://www.rodndtube.com/surf/info/surf_racks/BicycleSurfboardRack.shtml.

GT Karakoram – 2007

I got this bicycle for free when I purchased my first mountain bike from someone on Craigslist in Glendale in late 2006. I took both bicycles back to Westwood via bus where I was promptly stopped and questioned by the police as to why I had two bikes with me on the bus. Neither showed up as reported missing so they let me out. The next year I moved to Santa Monica near Wilshire and 17th Street and built this rack so I could ride to Venice Beach to surf in the morning. The ride was long and the waves were bad but I enjoyed every minute of it.

In 2008 I moved too far from the coast for this to be practical so I donated the bicycle to Bikerowave and didn’t put another one together for ~10 years.

Shogun (unknown model) – 2017

When I started working at Google we moved to Santa Cruz and I rode the corp bus into Mountain View every morning. But if I got up early enough I could sneak in a quick session, usually at Steamer Lane, before work via bicycle. This bike, an unknown model built by Shogun, was discarded by one of my neighbors. I fixed it up and slapped an off-the-shelf surfboard rack on it so I could get right up to the cliff and lock it on the fence overlooking the ocean. It lasted almost a year until someone stole it from the carport while I was in the backyard putting my board in the shed.

Trek 800 – 2018

After the Shogun was stolen I picked up this from someone on Craigslist and rode it home in the rain. I opted to build another PVC rack this time as I was unimpressed with the one I bought for the Shogun. When we moved away from Santa Cruz I gave it away.

Univega Alpina Sport – 2020

When we moved to Long Beach I effectively gave up on surfing as the breakwater blocks all the waves and will never go away. Luckily, Belmont Shore is a great place to kiteboard and kite gear is pretty compact. I fit my rig onto the same PVC rack used earlier with some extra stuff on the back. It works, but the huge downside to this is that riding around when the wind is 20kts is not a great activity.

ET Cycles 720 – 2023

Kiteboarding is fun, but it’s quite different from surfing as you’re getting pulled by the wind instead of the waves. I was never able to shake this and eventually found myself back in the waves at Seal Beach. Seal Beach is arguably the best foil wave on the West Coast and I couldn’t be happier about the new sport. It’s a few miles from the building we live in so I was driving there. One night our catalytic converter was stolen and the wait for a replacement was 6+ months. This is how I discovered e-bikes and they are amazing, like riding downhill the whole way. I used an off the shelf rack for this build because the foil gets in the way of the PVC rack I’m familiar with. There’s also a box on the back that holds the mast high to prevent the wing from dragging during a turn.

when I was at KDD 2023 I locked this up outside the Long Beach Convention Center for a day with a powerful Kryptonite lock. But the city apparently doesn’t install strong bike racks and my e-bike was taken.

Prop 13 wtf

Sat, 25 Mar 2023 00:00:00 GMT

I’ve decided to move my political blogging to a different site, https://prop13.wtf. Check it out! All kinds of cool stuff like tax assessments vs. rent in Los Angeles:

Enable HTTPS for S3, Cloudfront, Namecheap

Wed, 11 Jan 2023 00:00:00 GMT

I finally got around to enabling https here. Some notes:

Namecheap sells ssl certs via PositiveSSL/Comodo. I thought this would be easiest but they don’t really work with AWS.
- I’ll never get that $7 back
- It’s more work to import a 3rd party certificate vs. creating one on AWS
- After importing the 3rd party certificate (which has to happen in N. Virgina) AWS still claims that it’s not from a trusted source
- Other CNAME gotchas
Amazon Certificate Manager allows one to create the certificate from AWS for free
- They’ll auto generate the Route 53 CNAME rules for you
- Still the same weird thing about how you need to do this in N. Virginia even if the rest of you site is elsewhere
You need to add an “alias” rule to point your custom domain at Cloudfront.

I found this blog helpful

Migrating off s3_website.yml

Tue, 10 Jan 2023 00:00:00 GMT

First post in 5+ years! I used to use s3_website to publish this blog. Turns out that project has been deprecated with nothing to replace it. Oh well.

Following this I’ve managed to setup Github Actions to build/deploy the blog. A few small changes:

Github stuff tends to default to main now but this repository is old and uses master
the [email protected] stuff in their blog post wasn’t working for me

Here’s the deploy script I’m using to publish this post:

name: Jekyll build and S3 deploy

on:
  push:
    branches: [ master ]

  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:

env:
  AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }}
  AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
  AWS_DEFAULT_REGION: 'us-west-2'

jobs:
  build_and_deploy:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Set up Ruby
        uses: ruby/setup-ruby@359bebbc29cbe6c87da6bc9ea3bc930432750108
        with:
          ruby-version: '3.1'

      - name: Install dependencies
        run: bundle install

      - name: "Build Site"
        run: bundle exec jekyll build
        env:
          JEKYLL_ENV: production
      - name: "Deploy to AWS S3"
        run: aws s3 sync ./_site/ s3://${{ secrets.AWS_S3_BUCKET_NAME }} --acl public-read --delete --cache-control max-age=604800

      - name: "Create AWS Cloudfront Invalidation"
        run: aws cloudfront create-invalidation --distribution-id ${{ secrets.AWS_CLOUDFRONT_DISTRIBUTION_ID }} --paths "/*"

One thousand captcha photos organized with a neural network

Fri, 18 Aug 2017 00:00:00 GMT

Coauthored with Habib Talavati. Originally published on the Clarifai blog at https://blog.clarifai.com/one-thousand-captcha-photos-organized-with-a-neural-network-2/

The below image shows 1024 of the captcha photos used in “I’m not a human: Breaking the Google reCAPTCHA” by Sivakorn, Polakis, and Keromytis arranged on a 32x32 grid in such a way that visually-similar photos appear in close proximity to each other on the grid.

How did we do this?

To get from the collection of captcha photos to the grid above we take three steps: embedding via a neural net, further dimension reduction via t-SNE, and finally snapping things to a grid by solving an assignment problem.

Images are naturally very high-dimensional objects, even a “small” 224x224 image requires 2242243=150,528 RGB values. When represented naively as huge vectors of pixels visually-similar images may have enormous vector distances between them. For example, a left/right flip will generate a visually-similar image but can easily lead to a situation where each pixel in the flipped version has an entirely different value from the original.

Remark: Code for all of this is available here: https://github.com/Clarifai/public-notebooks/blob/master/gridded_tsne_blog_public.ipynb

Step 1: Reducing from 150528 to 1024 dimensions with a neural net

Our photos begin as 224x224x3 arrays of RGB values. We pass each image through an existing pre-trained neural network, Clarifai’s general embedding model which provides us with the activations from one of the top layers of the net. Using the higher layers from a neural net provides us with representations of our images which are rich in semantic information - the vectors of visually similar images will be close to each other in the 1024-dimensional space.

Step 2: Reducing from 1024 to 2 dimensions with t-SNE

In order to bring things down to a space where we can start plotting, we must reduce dimensions again. We have lots of options here. Some examples:

Inductive methods for embedding learning

Techniques such as the remarkably hard-to-Google Dr. LIM or Siamese Networks with triplet losses learn a function that can embed new images to fewer dimensions without any additional retraining. These techniques perform extremely well on benchmark datasets and are a great fit for online systems which must index previously-unseen images. For our application, we only need to get a fixed set of vectors reduced to 2D in one large, slow, step.

Transductive methods for dimensionality reduction

Rather than learning a function which can new points to few dimensions we can attack our problem more directly by learning a mapping from the high-dimensional space to 2D which preserves distances in the high-dimensional space as much as possible. Several techniques are available: t-SNE, and largeVis to name a few. Other methods, such as PCA, are not optimized for distance preservation or visualization and tend to produce less interesting plots. t-SNE, even during convergence, can produce very interesting plots (cf. this demonstration by @genekogan here ).

We use t-SNE to map our 1024D vectors down to 2D and generate the first entry in the above grid. Recall that our high-dimensional space here are 1024D vector embeddings from a neural net, so proximal vectors show correspond to visually similar photos. Without the neural net t-SNE would be a poor choice as distances between the initial 224x224x3 vectors are uninteresting.

Step 3: Snapping to a grid with the Jonker-Volgenant algorithm

One problem with t-SNE’d embeddings is that if we displayed the images directly over their corresponding 2D points we’d be left with swaths of empty white space and crowded regions where images overlap each other. We remedey this by building a 32x32 grid and moving the t-SNE’d points to the grid in such a way that total distance traveled is optimal.

It turns out that this operation can be incredibly sophisticated. There is an entire field of mathematics, transportation theory, concerned with solutions to problems in optimal transport under various circumstances. For example, if one’s goal is to minimize the sum of the squares of all distances traveled rather than simply the sum of the distances traveled (ie the l2 Monge-Kantorovitch mass transfer problem) an optimal mapping can be found by recasting the assignment problem as one in computational fluid dynamics and solving the corresponding PDEs. Cedric Villani, who won a Fields medal in 2010, wrote a great book on optimal transportation theory which is worth taking a look at when you get tired of corporate machine learning blogs.

In our setting, we just want the t-SNE’d points to snap to the grid in a way that makes this look visually appealing and be as simple as possible. Thus, we search for a mapping that minimizes the sum of the distances traveled via a linear assignment problem. The textbook solution here is to use the Hungarian algorithm, however, this can be also be solved quite easily and much faster using Jonker-Volgenant and open source tools

How easy can we make this?

Pretty easy. In addition to the notebook listed above, we’ve also set up an API endpoint that will generate an image similar to the one above for an existing Clarifai application. Here we assume you already have created an application by visiting https://developer.clarifai.com/account/applications and added your favorite images to it by calling the resource https://api.clarifai.com/v2/inputs. Then all you have to do is this:

Step 1: Kick off an asynchronous gridded t-SNE visualization

Since generating a visualization takes a while, we generate one asynchronously. We kick off a visualization by calling POST https://api.clarifai.com/v2/visualizations/

You should get a response like below informing us a “pending” visualization is scheduled to be computed

{
    "output": {
        "id": "ca69f34d53c742e1b4a1b71d7b4b4586",
        ...
    }
}

Note the id ca69f34d53c742e1b4a1b71d7b4b4586. We will use that id to get the visualization we just kicked off.

Step 2: Check to see if the visualization is done

Call GET /v2/visualizations/ca69f34d53c742e1b4a1b71d7b4b4586. The returned visualization will be “pending” for a while, but eventually, we should get a response like this:

{
    "output": {
        "data": {
            "image": {
                "url": "https://s3.amazonaws.com/clarifai-visualization/gridded-tsne/staging/your-visualization.jpg"
            }
        },
        ...
    }
}

At last, the output.data.image.url contains your gridded t-SNE visualization.

My talk at the NYC Machine Learning meetup

Tue, 06 Dec 2016 00:00:00 GMT

Check out this video of my talk at the NYC Machine Learning meetup.

It’s based on this blog post and the deck is here.

Upvotes over time by subreddit or: Why /r/The_Donald is always on the front page

Sun, 07 Aug 2016 00:00:00 GMT

Here’s a plot of the cumulative number of upvotes per minute for submissions to a few major subreddits:

The data was collected by polling /new/ every 2 minutes for each subreddit over the past 3 days (2942138 records were found). The vast majority of submissions to reddit never get anywhere - I removed submissions which never attained over 50 upvotes which left me with 154160 records. The raw data is shown below:

Ranking on reddit is determined using a combination of upvotes, downvotes, and the age of the post at the time of each vote (cf. here, here, and here for some good explanations). In short, the ranking of a submission is set by the rating function

\begin{equation*} f(n,t) = 45000\log_{10}(n) + t \end{equation*}

where $n$ is the difference between upvotes and downvotes and $t$ is the number of seconds which elapsed between the post’s creation time and 7:46:43 am December 8th, 2005.

More recent posts have a larger $t$ which translates to a better ranking. Additionally, due to the shape of $\log_{10}$ , votes matter substantially more when the number of upvotes nearly equals the number of downvotes (eg. when the post is brand new). Thus, the best way to get your post to the front page is to upvote aggressively when the post is very young.

My data suggests that members of /r/The_Donald are aware of this which explains why their new submissions have so many more upvotes despite the fact that competing subreddits in the plot are several orders of magnitude larger.

Taxi Strava

Sat, 11 Jun 2016 00:00:00 GMT

Last year Chris Whong used a foil request to obtain a dataset with information on the locations, times, and medallions for 173 million NYC cab rides. I’m interested is determining which cabs are the fastest cabs are and how quickly they can get between various parts of the city.

Data

Reddit users imjasonh and fhoffa parsed the raw data and loaded it into a public BigQuery table (another version is also available from the NYC Taxi and Limousine Commission) The schema looks like:

As you can see, each ride has very specific details on pickup/dropoff locations as well as start/end times. I am interested in answering questions along the lines of “How fast do cabs get to the Flatiron from the Upper East Side?” which is hard to do from precise latitudes and longitudes. To rectify this I took a 6-character geohash of every pickup and dropoff location. A 6-character geohash buckets together coordinates that are within 0.61km of each other which allowed me to easily aggregate popular routes. An example is shown below (image from movable-type):

The actual computations were done in Javascript using https://github.com/davetroy/geohash-js via a BigQuery UDF.

To make things more human readable, I used the geonames api to map the center of each bucket to an intersection. Not every geohash could be mapped to an intersection this way and those trips were dropped. Data was further cleaned by dropping trips using (hack_license != "0") AND (hack_license != "CFCD208495D565EF66E7DFF9F98764DA") which was observed in a discussion about the dataset.

This leaves us with a dataset of 158,320,608 cab rides bucketed into 32,654 distinct start/end points.

Results

Note: The 999th quantile for a trip’s average speed is 49.4289 mph - one trip had an average speed of 236,986,708 mph (roughly one third the speed of light). I removed any trip with average speed over 60mph from the data.

It takes ~20 minutes to get from 79th and York to the NYSE

The taxi stand at East 79th Street and York Avenue has been taking residents of the Upper East Side to Wall Street since 1987. It has 4 stars on Yelp. Each cab moves 2 or more passengers at a fare of $6

I found 252,210 trips along this route in my data. On average cabs take 20.35 minutes and move at 22.11 mph. Of course you’ll go faster at 4am but most people don’t start their commute until 6 or 7am:

Of the 13,347 medallions only a few regularly make the trip from 79th and York to Wall Street. The most dedicated cab drove the route 234 times over the year (only 7 drove it over 100 times):

The top 10 most frequent cab share drivers don’t go any faster than most though their average speed is more predictable (probably due to the fact that they often drive at the same time each day). Additionally, when one uses the morning cab share they are far more likely to be picked up by a usual (especially at 5am):

TODO:

SELECT
  pickup_street1, pickup_street2, dropoff_street1, dropoff_street2,
  trips_medallion, trips_pickup_datetime, trips_dropoff_datetime,
  ROUND(trips_avg_mph,4) AS avg_mpg,
  ROUND(trips_trip_duration_hours,4) AS num_hours
FROM
  [taxi_strava.joined_geohash_geonames]
WHERE
  trips_geohashed_dropoff = 'dr5ru2'
  AND trips_geohashed_pickup = 'dr5rvj'

What convolutional neural networks look at when they look at nudity

Tue, 19 Apr 2016 00:00:00 GMT

Originally published on the Clarifai blog at http://blog.clarifai.com/what-convolutional-neural-networks-see-at-when-they-see-nudity/

Last week at Clarifai we formally announced our Not Safe for Work (NSFW) adult content recognition model. Automating the discovery of nude pictures has been a central problem in computer vision for over two decades now and, because of it’s rich history and straightforward goal, serves as a great example of how the field has evolved. In this blog post, I’ll use the problem of nudity detection to illustrate how training modern convolutional neural networks (convnets) differs from research done in the past.

(Warning & Disclaimer: This post contains visualizations of nudity for scientific purposes. Read no further if you are under the age of 18 or if you are offended by nudity.)

1996

A seminal work in this field is the aptly-named “Finding Naked People” by Fleck et at.. It was published in the mid 90s and provides a good example of the kind of work that computer vision researchers would do prior to the convnet takeover. In section 2 of the paper the summarize the technique:

The algorithm:

first locates images containing large areas of skin-colored region;

then, within these areas, finds elongated regions and groups them into possible human limbs and connected groups of limbs, using specialized groupers which incorporate substantial amounts of information about object structure

Skin-detection is done by filtering in color space (note: HSV usually works well here but this paper implemented a specialized transformation of RGB) and grouping skin regions is done by modeling the the human figure “as an assembly of nearly cylindrical parts, where both the individual geometry of the parts and the relationships between parts are constrained by the geometry of the skeleton” (cf. section 2). To get a better idea of the engineering that goes into building an algorithm like this we turn to fig. 1 in the paper where the authors illustrate a few of their handbuilt grouping rules:

The paper reports “60% precision and 52% recall on a test set of 138 uncontrolled images of naked people”. They also provide examples of true positives and false positives with visualizations of the features discovered by the algorithm overlaid:

A major issue with building features by hand is that their complexity is limited by the patience and imagination of the researchers. In the next section, we’ll see how a convnet trained to perform the same task can learn much more sophisticated representations of the same data.

2014

Instead of devising formal rules to describe how the input data should be represented, deep learning researchers devise network architectures and datasets which enable an A.I. system to learn representations directly from the data. However, since deep learning researchers don’t specify exactly how the network should behave on a given input, a new problem arises: How can one understand what the convolutional networks are activating on?

Understanding the operation of a convnet requires interpreting the feature activity in various layers. In the rest of this post we’ll examine an early version of our NSFW model by mapping activities from the top layer back down to the input pixel space. This will allow us to see what input pattern originally caused a given activation in the feature maps (ie. why an image was flagged as “NSFW”).

Occulsion Sensitivity

The image at the top of the post shows photos of Lena Söderberg after 64x64 sliding windows with a stride of 3 have applied of our nsfw model to cropped/occuluded versions of the raw image.

To build the heatmap on the left we send each window to our convnet and average the “NSFW” scores over each pixel. When the convnet sees a crop filled with skin it tends to predict “NSFW” which leads to large red regions over Lena’s body. To create the heatmap on the right we systematically occlude parts of the raw image and report 1 minus the average “NSFW” scores (i.e. the “SFW” score). When the most NSFW regions are occluded the “SFW” scores increase and we see higher values in the heatmap. To be clear, the below figures have examples of what kind of images were fed into the convnet for each of two experiments above:

One of the nice things about these occlusion experiments is that they’re possible to perform when the classifier is a complete black box. Here’s a code snippet that reproduces these results via our API:

# NSFW occulsion experiment

from StringIO import StringIO

import matplotlib.pyplot as plt
import numpy as np
from PIL import Image, ImageDraw
import requests
import scipy.sparse as sp

from clarifai.client import ClarifaiApi

CLARIFAI_APP_ID = '...'
CLARIFAI_APP_SECRET = '...'
clarifai = ClarifaiApi(app_id=CLARIFAI_APP_ID,
                       app_secret=CLARIFAI_APP_SECRET,
                       base_url='https://api.clarifai.com')

def batch_request(imgs, bboxes):
  """use the API to tag a batch of occulded images"""
  assert len(bboxes) < 128
  #convert to image bytes
  stringios = []
  for img in imgs:
    stringio = StringIO()
    img.save(stringio, format='JPEG')
    stringios.append(stringio)
  #call api and parse response
  output = []
  response = clarifai.tag_images(stringios, model='nsfw-v1.0')
  for result,bbox in zip(response['results'], bboxes):
    nsfw_idx = result['result']['tag']['classes'].index("sfw")
    nsfw_score = result['result']['tag']['probs'][nsfw_idx]
    output.append((nsfw_score, bbox))
  return output

def build_bboxes(img, boxsize=72, stride=25):
  """Generate all the bboxes used in the experiment"""
  width = boxsize
  height = boxsize
  bboxes = []
  for top in range(0, img.size[1], stride):
    for left in range(0, img.size[0], stride):
      bboxes.append((left, top, left+width, top+height))
  return bboxes

def draw_occulsions(img, bboxes):
  """Overlay bboxes on the test image"""
  images = []
  for bbox in bboxes:
    img2 = img.copy()
    draw = ImageDraw.Draw(img2)
    draw.rectangle(bbox, fill=True)
    images.append(img2)
  return images

def alpha_composite(img, heatmap):
  """Blend a PIL image and a numpy array corresponding to a heatmap in a nice way"""
  if img.mode == 'RBG':
    img.putalpha(100)
  cmap = plt.get_cmap('jet')
  rgba_img = cmap(heatmap)
  rgba_img[:,:,:][:] = 0.7 #alpha overlay
  rgba_img = Image.fromarray(np.uint8(cmap(heatmap)*255))
  return Image.blend(img, rgba_img, 0.8)

def get_nsfw_occlude_mask(img, boxsize=64, stride=25):
  """generate bboxes and occluded images, call the API, blend the results together"""
  bboxes = build_bboxes(img, boxsize=boxsize, stride=stride)
  print 'api calls needed:{}'.format(len(bboxes))
  scored_bboxes = []
  batch_size = 125
  for i in range(0, len(bboxes), batch_size):
    bbox_batch = bboxes[i:i + batch_size]
    occluded_images = draw_occulsions(img, bbox_batch)
    results = batch_request(occluded_images, bbox_batch)
    scored_bboxes.extend(results)
  heatmap = np.zeros(img.size)
  sparse_masks = []
  for idx, (nsfw_score, bbox) in enumerate(scored_bboxes):
    mask = np.zeros(img.size)
    mask[bbox[0]:bbox[2], bbox[1]:bbox[3]] = nsfw_score
    Asp = sp.csr_matrix(mask)
    sparse_masks.append(Asp)
    heatmap = heatmap + (mask - heatmap)/(idx+1)    
  return alpha_composite(img, 80*np.transpose(heatmap)), np.stack(sparse_masks)

#Download full Lena image
r = requests.get('https://clarifai-img.s3.amazonaws.com/blog/len_full.jpeg')
stringio = StringIO(r.content)
img = Image.open(stringio, 'r')
img.putalpha(1000)

#set boxsize and stride (warning! a low stride will lead to thousands of API calls)
boxsize= 64
stride= 48
blended, masks = get_nsfw_occlude_mask(img, boxsize=boxsize, stride=stride)

#viz
blended.show()

While these kinds of experiments provide a straightforward way of displaying classifier outputs they have a drawback in that the visualizations produced are often quite blurry. This prevents us from gaining meaningful insight into what the network is actually doing and understanding what could have gone wrong during training.

Deconvolutional Networks

Once we’ve trained a network on a given dataset we’d like to be able to take an image and a class and ask the convnet something along the lines of “How can we change this image in order to look more like the given class?”. For this we use a deconvolutional network (deconvnet), cf section 2 from Zeiler and Fergus 2014:

A deconvnet can be thought of as a convnet model that uses the same components (filtering, pooling) but in reverse, so instead of mapping pixels to features does the opposite. To examine a given convnet activation, we set all other activations in the layer to zero and pass the feature maps as input to the attached deconvnet layer. Then we successively (i) unpool, (ii) rectify and (iii) filter to reconstruct the activity in the layer beneath that gave rise to the chosen activation. This is then repeated until input pixel space is reached.

The procedure is similar to backpropping a single strong activation (rather than the usual gradients), i.e. computing $\frac{\partial h}{ \partial X_n}$ where $h$ is the element of the feature map with the strong activation and $X_n$ is the input image.

Here is the result we get when using a deconvnet to visualize how we should modify photos of Lena to look more like pornography (note: the deconvnet used here needed a square image to function correctly - we padded the full Lena image to get the right aspect ratio):

Barbara is the G-rated version of Lena. According to our deconvnet, we could modify Barbara to look more PG by adding redness to her lips:

This image of Ursula Andress as Honey Rider in the James Bond film Dr. No was voted number one in “the 100 Greatest Sexy Moments in screen history” by a UK survey in 2003:

A salient feature of the above experiments is that the convnet learned red lips and navels as indicative of “NSFW”. This likely means that we didn’t include enough images of red lips and navels in our “SFW” training data. Had we only evaluated our model by examining precision/recall and ROC curves (shown below - test set size: 428,271) we would have never discovered this issue as our test data would have the same shortcoming. This highlights a fundamental difference between training rule-based classifiers and modern A.I. research. Rather than redesigning features by hand, we redesign our training data until the discovered features are improved.

Finally, as a sanity check, we run the deconvnet on hardcore porno to ensure that the learned feature activations do indeed to correspond to obviously nsfw objects:

Here, we can clearly see that the convnet correctly learned penis, anus, vulva, nipple, and buttocks - objects which our model should flag. What’s more, the discovered features are far more detailed and complex than what researchers could design by hand which helps explain the major improvements we get by using convnets to recognize NSFW images.

If you’re interested in using convnets to filter NSFW images, check our NSFW API documentation to get started. https://developer.clarifai.com/guide/tag#nsfw

Darknet Market Basket Analysis

Tue, 24 Mar 2015 00:00:00 GMT

The Evolution darknet marketplace was an online black market which operated from January 2014 until Wednesday of last week when it suddenly disappeared. A few days later, in a reddit post, gwern released a torrent containing daily wget crawls of the site dating back to its inception. I ran some off-the-shelf affinity analysis on the dataset – here’s what I found:

Products can be categorized based on who sells them

On Evolution there are a few top-level categories (“Drugs”, “Digital Goods”, “Fraud Related” etc.) which are subdivided into product-specific pages. Each page contains several listings by various vendors.

I built a graph between products based on vendor co-occurrence relationships, i.e. each node corresponds to a product with edge weights defined by the number of vendors who sell both incident products. So, for example, if there are 3 vendors selling both mescaline and 4-AcO-DMT then my graph has an edge with weight 3 between the mescaline and 4-AcO-DMT nodes. I used graph-tool’s implementation of stochastic block model-based hierarchal edge bundling to generate the below visualization of the Evolution product network:

The graph is available in graphml format here. It contains 73 nodes and 2,219 edges (I found a total of 3,785 vendors in the data).

Edges with higher weights are drawn more brightly. Nodes are clustered with a stochastic block model and nodes within the same cluster are assigned the same color. There is a clear division between the clusters on the top half of the graph (correpsonding to drugs) and the clusters on the bottom half (corresponding to non-drugs, i.e. weapons/hacking/credit cards/etc.). This suggests that vendors who sold drugs were not as likely to sell non-drugs and vice versa.

I used a short python script to parse the scraped html and remove duplicate data, its available here. It takes a while to go through the entire dataset (which is about 90GB); if you’d like to skip that you can download the results of my parse as a .tsv file. The plotting code is available as an ipython notebook. High-res version of the above plot here.

91.7% of vendors who sold speed and MDMA also sold ecstasy

Association rule learning is a straightforward and popular way to solve problems in market basket analysis. The traditional application is to suggest items to shoppers based on what other customers are putting in their carts. For some reason the canonical example is “customers who buy diapers also buy beer”.

We don’t have customer data from a crawl of the public postings on Evolution. However, we do have data on what each vendor sells which can help us quantify results suggested by the visual analysis done above.

Here’s an example of what our database looks like (the complete file has 3,785 rows (one for each vendor)):

Vendor	Products
MrHolland	[‘Cocaine’, ‘Cannabis’, ‘Stimulants’, ‘Hash’]
Packstation24	[‘Accounts’, ‘Benzos’, ‘IDs & Passports’, ‘SIM Cards’, ‘Fraud’]
Spinifex	[‘Benzos’, ‘Cannabis’, ‘Cocaine’, ‘Stimulants’, ‘Prescription’, ‘Sildenafil Citrate’]
OzVendor	[‘Software’, ‘Erotica’, ‘Dumps’, ‘E-Books’, ‘Fraud’]
OzzyDealsDirect	[‘Cannabis’, ‘Seeds’, ‘MDMA’, ‘Weed’]
TatyThai	[‘Accounts’, ‘Documents & Data’, ‘IDs & Passports’, ‘Paypal’, ‘CC & CVV’]
PEA_King	[‘Mescaline’, ‘Stimulants’, ‘Meth’, ‘Psychedelics’]
PROAMFETAMINE	[‘MDMA’, ‘Speed’, ‘Stimulants’, ‘Ecstasy’, ‘Pills’]
ParrotFish	[‘Weight Loss’, ‘Stimulants’, ‘Prescription’, ‘Ecstasy’]

Before saying anything more about association rule learning here’s a quick glossary of terms:

The support, $supp(X)$ , of an itemset, $X$ , is defined as the proportion of transactions in the data set which contain $X$ . In the table above, the support of ‘Cocaine’ is 2 because it appears in two vendors’ storefronts (MrHolland and Spinifex)
The confidence of a rule is defined $\mathrm{conf}(X \Rightarrow Y) = \mathrm{supp}(X \cup Y) / \mathrm{supp}(X)$ . In our example the confidence of the rule ‘Cannabis’ ==> ‘Cocaine’ is 2/3 because out the 3 vendors who sell ‘Cannabis’ 2 of them sell ‘Cocaine’. The support of this rule is 2.

Association rule mining is a huge field within computer science – hundreds (thousands?) of papers have been published over the past two decades. The necessary algorithms are very complex but open source implementations are available. My favorite collection (and the one I used for these experiments) is Philippe Fournier Viger’s spmf.

I ran the FP-Growth algorithm with a minimum allowable support of 40 and a minimum allowable confidence of 0.1. The algorithm learned 12,364 rules. These can be downloaded as a .tsv here. I’ve selected a few rules for display below:

antecedent	consequent	support	confidence
[‘Speed’, ‘MDMA’]	[‘Ecstasy’]	155	0.91716
[‘Ecstasy’, ‘Stimulants’]	[‘MDMA’]	310	0.768
[‘Speed’, ‘Weed’, ‘Stimulants’]	[‘Cannabis’, ‘Ecstasy’]	68	0.623
[‘Fraud’, ‘Hacking’]	[‘Accounts’]	53	0.623
[‘Fraud’, ‘CC & CVV’, ‘Accounts’]	[‘Paypal’]	43	0.492
[‘Documents & Data’]	[‘Accounts’]	139	0.492
[‘Guns’]	[‘Weapons’]	72	0.98
[‘Weapons’]	[‘Guns’]	72	0.40

Other Remarks

I think I’ve only scratched the surface of what’s possible with this data. There are much more detailed product descriptions for each listing in the .tsv. That text is harder to work with so it will take some time to figure out what makes sense.

Paper update: Geotagging One Hundred Million Twitter Accounts with Total Variation Minimization

Thu, 05 Mar 2015 00:00:00 GMT

Press coverage (updated 2015-05-02):

Altmetrics link

Earlier this week I updated a paper on the arxiv: http://arxiv.org/abs/1404.7152. The paper appeared a few months ago at IEEE BigData 2014 and was the subject of all my talks last year. This update syncs the arxiv paper with what was accepted for the conference. Here’s the abstract:

Geographically annotated social media is extremely valuable for modern information retrieval. However, when researchers can only access publicly-visible data, one quickly finds that social media users rarely publish location information. In this work, we provide a method which can geolocate the overwhelming majority of active Twitter users, independent of their location sharing preferences, using only publicly-visible Twitter data.

Our method infers an unknown user’s location by examining their friend’s locations. We frame the geotagging problem as an optimization over a social network with a total variation-based objective and provide a scalable and distributed algorithm for its solution. Furthermore, we show how a robust estimate of the geographic dispersion of each user’s ego network can be used as a per-user accuracy measure which is effective at removing outlying errors.

Leave-many-out evaluation shows that our method is able to infer location for 101,846,236 Twitter users at a median error of 6.38 km, allowing us to geotag over 80% of public tweets.

Studying automotive sensor data with Open Torque Viewer

Sat, 28 Feb 2015 00:00:00 GMT

I’ve been using Open Torque Viewer combined with the Torque App and a basic OBDII bluetooth sensor to log my car’s sensor data over the past month and a half. Here are a few things that I’ve learned:

Tapping into your car’s ECU is surprisingly easy

All cars sold in North America since 1996 are required to support OBD-II diagnostics with standardized data link connectors and parameter IDs. This is useful, for example, for state-mandated emissions inspections. The OBDII port looks a bit like a VGA connector and, on my car (a 2003 Suzuki Aerio SX), is located under the dash and just above the accelerator.

Attach a bluetooth OBDII device to the port, download the Torque app, setup Open Torque Viewer (I used an EC2 microinstance), and you’ll be able to log each trips data and display it inside of a nice interface. You can also learn about engine fault codes with this setup - I haven’t had any yet.

City mpg estimates are spot on for LA

Torque provides readouts of instantaneous mpg data while you drive. I histogrammed this data and compared it with advertised values in the chart below.

My driving fit the “city” profile reasonably well (even though I’m usually on a “freeway”). I was surprised that I was anywhere near the expected mpg given how unpredictable traffic is. Maybe the fact that Los Angeles is mostly flat and has rainless 75° weather year round evens things out.

Driving up the Conejo Grade is expensive

The Conejo Grade is a 7% grade incline on the 101 near the LA/Ventura county line. It’s around 5 miles long and has 3.5 stars on Yelp. Here’s a fun video of the trip up and down it. A mudslide shutdown PCH in mid-December (it’s February now and PCH is still closed…) so I’ve had to drive the Conejo Grade whenever I make a trip to Oxnard.

The Conejo Grade at night. | Photo: macmcintire/Flickr/Creative Commons License

I average 17.77 mpg on the uphill portion and 112.92 when going downhill. At \$3.50 a gallon this works out to \$0.98 in gas money to push myself up that hill and \$0.15 to coast down it. Here’s a histogram of instantaneous mpg measurements recorded on the Conejo Grade:

Torque also logs throttle position. I often get the pedal to the floor near the top of the incline. Adding a dimension for the throttle position variable makes the uphill/downhill distinction a bit more pronounced. The blobs are kernel density estimates computed by seaborn. The drastically bimodal nature of the data helps illustrate how hard I’ve got to press on the gas to get up that hill.

My speedometer is accurate

Here’s a plot of the discrepancies between my GPS-measured speed and what my speedometer is reporting. The mean discrepancy is 1.26 mph at a standard deviation of 1.15 mph.

Here’s a plot showing how the agreement gets a little better at higher speeds:

I suspect the reason for this is that my speedometer works better when I’m not braking all the time - I don’t think GPS measurements are affected by my driving.

Remarks

Open Torque Viewer is a lot more fun than dropping $100 at the mechanic to have them do basically the same thing when you come in for a check engine light. Here’s what the interface looks like:

There are a few other apps out there that connect your smartphone to an OBDII device (e.g. dash or automatic ) but they seem targeted for simplicity and have lots of social-networking ick. I tried dash out but quickly switched back to Torque since its data collection and logging options are much more customizable. I don’t think this blog post would have been possible with any of the other apps out there.

The code used to make the plots is available as an ipython notebook.

New Paper: Network-Based Group Account Classification.

Wed, 25 Feb 2015 00:00:00 GMT

Update 2015-03-25: We just got nominated for a best paper award!

Update 2015-04-01: We won best paper!

Patrick Park and I recently got our paper accepted into SBP 2015. The work demonstrates how the structure of Twitter @mention networks can be used to distinguish between accounts owned by individuals or groups. Here’s the abstract

We propose a classification method for group vs. individual accounts on Twitter, based solely on communication network characteristics. While such a language-agnostic, network-based approach has been used in the past, this paper motivates the task from firmly established theories of human interactional constraints from cognitive science to sociology. Time, cognitive, and social role constraints limit the extent to which individuals can maintain social ties. These constraints are expressed in observable network metrics at the node (i.e. account) level which we identify and exploit for inferring group accounts.

Url extraction in python

Mon, 16 Feb 2015 00:00:00 GMT

I was looking for a way to identify urls in text and eventually found this huge regex http://daringfireball.net/2010/07/improved_regex_for_matching_urls . I figured I’ll need to do this again so I stuck all that into urlmarker.py and now I can just import it.

import urlmarker
import re

text = """
The regex patterns in this gist are intended only to match web URLs -- http,
https, and naked domains like "example.com". For a pattern that attempts to
match all URLs, regardless of protocol, see: https://gist.github.com/gruber/249502
"""

print(re.findall(urlmarker.WEB_URL_REGEX,text))

will show ['example.com', 'https://gist.github.com/gruber/249502']

Hearddit

Tue, 23 Dec 2014 00:00:00 GMT

Hearddit is a Soundcloud/Spotify/reddit bot that builds playlists from links posted on music subreddits. I built the bot since I was getting tired of the twenty or so tracks that the internet radio collaborative filters decided I liked and wanted an easy way to find new music using existing apps.

Here are the playlists:

Hearddit has been running for a little over one week and it’s already discovered some great stuff:

The whole thing is a short python program, available here. It runs every couple of hours so there will be some delay between when a link gets posted on Reddit and when it appears in a playlist.

Currently Hearddit is scraping the following subreddits (updated 2015-02-23):

Remark: Not all music hosted on Soundcloud is available on Spotify and vice versa so the sounds will be different depending on what app you’re using.

Building Soundcloud playlists is pretty straightforward since music subreddits encourage users to “support the artists and submit their content directly. Look for the original source of content and submit that.” which results in lots of links to Soundcloud. Additionally, Soundcloud has an official python API that’s easy to use.

To create Soundcloud playlists Hearddit uses the Soundcloud API to resolve links found on the target subreddit, checks if it’s already created a playlist for that subreddit, and then either appends the resolved links to that playlist or create a new one and append to that. Obligatory code sample below.

def create_soundcloud_playlist_from_urls(urls, playlist_name):

    client = soundclound_login()

    #use soundcloud api to resolve links
    tracks = []
    for url in urls:
        if 'soundcloud' in url:
            logger.info(url)
            try:
                tracks.append(client.get('/resolve', url=url))
            except requests.exceptions.HTTPError:
                logger.error('except!'+url)
    track_ids = [x.id for x in tracks]
    track_dicts = list(map(lambda id: dict(id=id), track_ids))
    logger.info(track_dicts)

    #check if playlist already exists
    my_playlists = client.get('/me/playlists')
    old_list_urls = [p for p in my_playlists if p.fields()['title'] == playlist_name]
    if old_list_urls:
        # add tracks to playlist
        old_list_url = old_list_urls[0]
        client.put(old_list_url.uri, playlist={'tracks': track_dicts})
    else:
        # create the playlist
        client.post('/playlists', playlist={
            'title': playlist_name,
            'sharing': 'public',
            'tracks': track_dicts})

    #get the link to the list created
    my_playlists = client.get('/me/playlists')
    new_list_url = [p.fields()['permalink_url'] for p in my_playlists
                    if p.fields()['title'] == playlist_name]

    if new_list_url:
        return new_list_url[0]
    else:
        logger.warning('no soundcloud list')
    return

Building Spotify playlists is more complex. For one thing people don’t link to Spotify from Reddit so aligning post titles with Spotify required use of Spotify’s search engine. Hearddit has a few search heuristics based on the /r/electronicmusic sidebar:

def search_spotify_for_a_title(title, sp):
    query = re.split('(\[|\()',title)[0]
    if len(query) > 5:
        results = sp.search(q=query, type='track')
        if len(results['tracks']['items']) > 0:
            logger.info('hit! {0}'.format(query))
            return results
        else:
            logger.info('miss! {0}'.format(query))
    return

This gets a lot of matches right but the input queries as well as query understanding at Spotify could still use a bit of work.

Next is the problem that python APIs for Spotify aren’t as good as Soundcloud’s. There are two out there: pyspotify and spotipy.

Pyspotify is more popular on github but it’s also harder to use. Pyspotify uses CFFI to build a wrapper around Spotify’s official C library, libspotify and, for whatever reason, coding in pyspotify requires lots of calls to session.process_events() and *.load() or else things fail in unexpected ways.

Spotipy uses the requests library to access Spotify’s web api. Things that aren’t documented in python are easy enough figure out from the web api docs. Spotipy isn’t python 3 compatible but there’s a fork which (mostly) takes care of that. Here’s the function that Hearddit uses to build Spotify playlists:

def create_spotify_playlist_from_titles(todays_titles, playlist_name):

    sp = spotify_login()

    #try to map the submission titles to spotify tracks
    search_results = [search_spotify_for_a_title(x,sp) for x in todays_titles]
    search_results = [x for x in search_results if x and (len(x['tracks']['items']) > 0)]
    hits = [x['tracks']['items'][0] for x in search_results]

    #get all my playlists, check if playlist_name already there
    new_pl = None
    for my_pl in sp.user_playlists(user=sp.me()['id'])['items']:
        logger.info('my_pl: {}'.format(my_pl['name']))
        if my_pl['name'] == playlist_name:
            logger.warning('appending to {}'.format(my_pl))
            new_pl = sp.user_playlist(user=sp.me()['id'], playlist_id=my_pl['id'])
            break
    if not new_pl:
        logger.warning('new playlist!')
        new_pl = sp.user_playlist_create(user=sp.me()['id'],name=playlist_name,public=True)

    out_url = new_pl['external_urls']['spotify']
    logger.info(out_url)

    #get all tracks in the playlist
    new_new_pl = sp.user_playlist(sp.me()['id'], new_pl['uri'])

    old_track_uris = set([x['track']['uri'] for x in new_new_pl['tracks']['items']])
    for s in old_track_uris:
        logger.debug('old! {}'.format(s))

    new_track_uris = [hit['uri'] for hit in hits if hit['uri'] not in old_track_uris]
    for new_track_uri in new_track_uris:
        logger.debug('new! {}'.format(new_track_uri))

    #can only insert 100 tracks at a time (Spotify API limit)
    def chunker(seq, size):
        return (seq[pos:pos + size] for pos in range(0, len(seq), size))

    logger.warning('adding {0} new tracks to {1}'.format(len(new_track_uris), new_pl['name']))
    if new_track_uris > 0:
        for sublist in chunker(new_track_uris,99):
            sp.user_playlist_add_tracks(sp.me()['id'],new_pl['uri'],sublist)
            time.sleep(7)
            logger.warning('added {0} new tracks to {1}'.format(len(sublist), new_pl['name']))

    return out_url

Stochastic block model based edge bundles in graph-tool

Wed, 22 Oct 2014 00:00:00 GMT

Here’s a plot of the political blogging network in “The political blogosphere and the 2004 US Election” but with the edge bundles determined using a stochastic block model (remark: the below plot is the same (ie. same layout and data) as fig. 5 in Tiago’s paper - I just put a black background on it).

The edge-color scheme is the same as in the original paper by Adamic and Glance, i.e. each node corresponds to a blog url and the colors reflect political orientation, red for conservative, and blue for liberal. Orange edges go from liberal blogs to conservative blogs, and purple ones from conservative to liberal (cf fig. 1 in Adamic and Glance).

Code and high-res version here:

# -*- coding: utf-8 -*-
import graph_tool.all as gt
import math

g = gt.collection.data["polblogs"] #  http://www2.scedu.unibo.it/roversi/SocioNet/AdamicGlanceBlogWWW.pdf
print(g.num_vertices(), g.num_edges())

#reduce to only connected nodes
g = gt.GraphView(g,vfilt=lambda v: (v.out_degree() > 0) and (v.in_degree() > 0) )
g.purge_vertices()

print(g.num_vertices(), g.num_edges())

#use 1->Republican, 2->Democrat
red_blue_map = {1:(1,0,0,1),0:(0,0,1,1)}
plot_color = g.new_vertex_property('vector<double>')
g.vertex_properties['plot_color'] = plot_color
for v in g.vertices():
    plot_color[v] = red_blue_map[g.vertex_properties['value'][v]]

#edge colors
alpha=0.15
edge_color = g.new_edge_property('vector<double>')
g.edge_properties['edge_color']=edge_color
for e in g.edges():
    if plot_color[e.source()] != plot_color[e.target()]:
        if plot_color[e.source()] == (0,0,1,1):
            #orange on dem -> rep
            edge_color[e] = (255.0/255.0, 102/255.0, 0/255.0, alpha)
        else:
            edge_color[e] = (102.0/255.0, 51/255.0, 153/255.0, alpha)
    #red on rep-rep edges
    elif plot_color[e.source()] == (1,0,0,1):
        edge_color[e] = (1,0,0, alpha)
    #blue on dem-dem edges
    else:
        edge_color[e] = (0,0,1, alpha)

state = gt.minimize_nested_blockmodel_dl(g, deg_corr=True)
bstack = state.get_bstack()
t = gt.get_hierarchy_tree(bstack)[0]
tpos = pos = gt.radial_tree_layout(t, t.vertex(t.num_vertices() - 1), weighted=True)
cts = gt.get_hierarchy_control_points(g, t, tpos)
pos = g.own_property(tpos)
b = bstack[0].vp["b"]

#labels
text_rot = g.new_vertex_property('double')
g.vertex_properties['text_rot'] = text_rot
for v in g.vertices():
    if pos[v][0] >0:
        text_rot[v] = math.atan(pos[v][1]/pos[v][0])
    else:
        text_rot[v] = math.pi + math.atan(pos[v][1]/pos[v][0])

gt.graph_draw(g, pos=pos, vertex_fill_color=g.vertex_properties['plot_color'],
            vertex_color=g.vertex_properties['plot_color'],
            edge_control_points=cts,
            vertex_size=10,
            vertex_text=g.vertex_properties['label'],
            vertex_text_rotation=g.vertex_properties['text_rot'],
            vertex_text_position=1,
            vertex_font_size=9,
            edge_color=g.edge_properties['edge_color'],
            vertex_anchor=0,
            bg_color=[0,0,0,1],
            output_size=[4024,4024],
            output='polblogs_blockmodel.png')

The high-res version is available here.

graph-tool's visualization is pretty good

Sun, 05 Oct 2014 00:00:00 GMT

Here’s a plot of the political blogging network described by Adamic and Glance in “The political blogosphere and the 2004 US Election”. The layout is determined using graph-tool’s implementation of hierarchal edge bundles. The color scheme is the same as in the original paper, i.e. each node corresponds to a blog url and the colors reflect political orientation, red for conservative, and blue for liberal. Orange edges go from liberal blogs to conservative blogs, and purple ones from conservative to liberal (cf fig. 1 in Adamic and Glance). All 1,490 nodes and 19,090 edges are drawn.

The url of each blog is drawn alongside each node, here’s a close-up:

This dataset is fairly well-known (it even comes packaged into graph-tool, cf. the below snippet).

import graph_tool.all as gt
import math

g = gt.collection.data["polblogs"] #  http://www2.scedu.unibo.it/roversi/SocioNet/AdamicGlanceBlogWWW.pdf

Getting the colors right requires a bit of tuning:

#use 1->Republican, 2->Democrat
red_blue_map = {1:(1,0,0,1),0:(0,0,1,1)}
plot_color = g.new_vertex_property('vector<double>')
g.vertex_properties['plot_color'] = plot_color
for v in g.vertices():
    plot_color[v] = red_blue_map[g.vertex_properties['value'][v]]

In order to use the hierarchical edge bundling algorithm we first need to do some kind of clustering. The obvious method is to assign each node a cluster based on its political affiliation:

#build tree
t = gt.Graph()

#add verticies with same idx as G
for v in g.vertices():
    tv = t.add_vertex()

#add hierachy points
reps = t.add_vertex()
dems = t.add_vertex()
root = t.add_vertex()
t.add_edge(root,reps)
t.add_edge(root,dems)

#assign clusters based on political affiliation
for tv in t.vertices():
    if t.vertex_index[tv] < g.num_vertices():
        if g.vertex_properties['value'][tv] == 1:
            t.add_edge(reps,tv)
        else:
            t.add_edge(dems,tv)

The clusters are used to form a hierarchy which allows one to easily figure out a standard tree layout (pictured below). Hierarchal edge bundles are drawn by interpolating along the tree.

tpos = pos = gt.radial_tree_layout(t, t.vertex(t.num_vertices() - 1), weighted=True)
cts = gt.get_hierarchy_control_points(g, t, tpos)
pos = g.own_property(tpos)

Here’s the tree used for the above figure:

Finally, we set the text rotation and save the figure:

#labels
text_rot = g.new_vertex_property('double')
g.vertex_properties['text_rot'] = text_rot
for v in g.vertices():
    if pos[v][0] >0:
        text_rot[v] = math.atan(pos[v][1]/pos[v][0])
    else:
        text_rot[v] = math.pi + math.atan(pos[v][1]/pos[v][0])

gt.graph_draw(g, pos=pos,
              vertex_size=10,
              vertex_color=g.vertex_properties['plot_color'],
              vertex_fill_color=g.vertex_properties['plot_color'],
              edge_control_points=cts,
              vertex_text=g.vertex_properties['label'],
              vertex_text_rotation=g.vertex_properties['text_rot'],
              vertex_text_position=1,
              vertex_font_size=9,
              edge_color=g.edge_properties['edge_color'],
              vertex_anchor=0,
              bg_color=[0,0,0,1],
              output_size=[4024,4024],
              output='polblogs.jpg')

The high-res (~18.6 MB) version is available here. Next step is to try this out with some more interesting data.

Community detection and colored plotting in networkx

Mon, 16 Jun 2014 00:00:00 GMT

Just came across this very easy library for community detection https://sites.google.com/site/findcommunities/ https://bitbucket.org/taynaud/python-louvain/src. Here’s how to create a graph, detect communities in it, and then visualize with nodes colored by their community in less than 10 lines of python:

import networkx as nx
import community

G = nx.random_graphs.powerlaw_cluster_graph(300, 1, .4)

part = community.best_partition(G)
values = [part.get(node) for node in G.nodes()]

nx.draw_spring(G, cmap = plt.get_cmap('jet'), node_color = values, node_size=30, with_labels=False)

It’s easy to get modularity to:

mod = community.modularity(part,G)
print("modularity:", mod)

gave modularity: 0.8700238252368541.

Rush Hour Surf Report updated

Sun, 15 Jun 2014 00:00:00 GMT

A few minor bug fixes were in order since Twitter started requiring SSL to access their API (which happened in January, meaning my app has been broken for 6 months now…). Get a copy of the only app that makes it so easy to surf report that you can do it from your car! https://play.google.com/store/apps/details?id=com.rcompton.rhsr

Update: app no longer available

New Paper: Inferring the Geographic Focus of Online Documents from Social Media Sharing Patterns

Tue, 10 Jun 2014 00:00:00 GMT

I recently published a paper on the arxiv: http://arxiv.org/abs/1406.2392, earlier this month it was accepted into the ChASM workshop at WebSci. Here’s the abstract:

Determining the geographic focus of digital media is an essential first step for modern geographic information retrieval. However, publicly-visible location annotations are remarkably sparse in online data. In this work, we demonstrate a method which infers the geographic focus of an online document by examining the locations of Twitter users who share links to the document. We apply our geotagging technique to multiple datasets built from different content: manually-annotated news articles, GDELT, YouTube, Flickr, Twitter, and Tumblr.

Infinite Jest by the numbers

Fri, 06 Jun 2014 00:00:00 GMT

After stressing my reading comprehension over the past eight months I’ve finally finished David Foster Wallace’s Infinite Jest. The writing is, in a word, impressive. Wallace’s command of the English language allows him to do things that I’ve never seen before. In this post I’ll try to quantify a few of the stylistic features that I think really stood out.

First off, to run these experiments, one needs an electronic version of the book. It turns out that these are remarkably easy to find online. I ran the .pdf here through pdftotext, imported nltk, and got these results:

“But and so and but so” is the longest uninterrupted chain of conjunctions

In Infinite Jest conjunctions often appear in chains of length three or greater. There is a length-six chain on page 379 of the .pdf. It’s due to a minor character, “old Mikey”, standing at the Boston AA podium and speaking to a crowd:

I’m wanting to light my cunt of a sister up so bad I can’t hardly see to get the truck off the lawn and leave. But and so and but so I’m driving back home, and I’m so mad I all of a sudden try and pray.

Now, according to Wiktionary, there are 224 conjunctions in the English language. It’s possible to quickly get all of them by entering this query into DBpedia’s public Wiktionary endpoint:

PREFIX terms:<http://wiktionary.dbpedia.org/terms/>
PREFIX rdfs:<http://www.w3.org/2000/01/rdf-schema#>
PREFIX dc:<http://purl.org/dc/elements/1.1/>
SELECT DISTINCT(?label)
WHERE {
  ?x terms:hasPoS terms:Conjunction .
  ?x dc:language terms:English .
  ?x rdfs:label ?label .
}

With this list we can find the most common uninterrupted conjunction chains along with the number of times they appeared in the text:

Conjunction Triples		Conjunction Quadruples		Conjunction Quintuples
and / or :	23	not until then that :	1	but so but then so :	1
whether or not :	9	so and but that :	1	and not only that but :	1
and then but :	8	not only that but :	1	and so and but so :	1
and but so :	6	so then but so :	1
and then when :	6	but so but then :	1
and that if :	5	and where and when :	1
and then only :	5	and but then when :	1
and but then :	5	and but so why :	1
and then if :	5	and so but since :	1
if and when :	5	but not until after :	1

(Remark: according to Wiktionary, “/” is a valid conjunction)

The only length six chain is “But and so and but so”.

We can perform a similar experiment using prepositions, of which there are 496. Here, we discover the longest chain of uninterrupted prepositions is “in over from behind like”. This occurs on page 402, while Michael Pemulis is opening Hal’s door:

With the door just cracked and his head poked in he brings his other arm in over from behind like it’s not his arm, his hand in the shape of a claw just over his head, and makes as if the claw from behind is pulling him back out into the hall. W/ an eye-rolling look of fake terror.

Here’s the corresponding frequency table for prepositions:

Preposition Triples		Preposition Quadruples		Preposition Quintuples
up next to :	24	to come out of :	2	in over from behind like :	1
up out of :	14	to come in to :	2
as in like :	13	to come up to :	1
come out of :	9	around with down in :	1
to come to :	8	to come up with :	1
come up with :	7	in over from behind like :	1
out from under :	6	come out of on :	1
come in to :	6	in from out over :	1
come in for :	5	as in with like :	1
to come in :	5	over from behind like :	1

The longest uninterrupted chain composed entirely of either prepositions or conjunctions is “again/And again and again/And again and again and again”. It occurs on page 264, during a storytelling session in the Tunnel Club:

Peterson to Traub, while Gopnik holds the light: ‘Eighteen-year-old top-ranked John Wayne / Had sex with Herr Schtitt on a train / They had sex again/And again and again/And again and again and again,’ which the slightly older kids find more entertaining than Traub does.

And here’s the frequency table for chains of either prepositions or conjunctions:

Either Triples		Either Quadruples		Either Quintuples
and out of :	41	in and out of :	30	and so on and so :	9
and so on :	39	so on and so :	11	on out and over to :	1
up and down :	32	and so on and so :	9	up and out and down and :	1
in and out of :	30	over and over again :	6	and up and down through :	1
over and over :	28	up and down in :	4	and down over and over for :	1
up next to :	24	to come in and :	4	but so but then so :	1
and / or :	22	come and gone and :	3	as in like not to :	1
on and on :	15	over and over and :	3	only that at times like :	1
up out of :	14	over and over for :	3	not to come down or :	1
as in like :	12	up and down both :	3	about wanting to and so on :	1

If we toss out “/” from our word list the longest chain of either is “up and down over and over for”. Not surprisingly, it’s due to Michael Pemulis (page 403):

Tell him we read books and tirelessly access D-bases and run our asses off all day here and need to eat instead of we don’t just stand there and swing one leg up and down over and over for seven-plus figures.

For reference, here’s the code to identify the longest uninterrupted chains of conjunctions/prepositions:

def all_uninterrupted_seqs(raw, prep_conj, min_seq_length=3):
    """
    Given a string of text and a set of terms to search for
    return all uninterrupted chains of the terms
    """
    tokens = nltk.wordpunct_tokenize(raw)
    all_seqs = []
    for idx,token in enumerate(tokens):
        tokl = token.lower()
        if tokl in prep_conj:
            seq = [tokl]
            n_ahead = 0
            while tokl in prep_conj:
                n_ahead += 1
                tokl = tokens[idx+n_ahead].lower()
                if tokl in prep_conj:
                    seq.append(tokl)
            if len(seq) >= min_seq_length:
                all_seqs.append(seq)
    return all_seqs

Our next result:

Wallace used a vocabulary of 20,584 words to write Infinite Jest

By comparison, the Brown Corpus, which is roughly three times longer than Infinite Jest, contains only 26,126 unique words. To be precise, the Brown Corpus contains 9,964,284 characters and 2,074,513 (not necessarily unique) words, while Infinite Jest contains 3,204,159 characters and 577,608 words. If we restrict the Brown Corpus to its first 3,204,159 characters we find a vocabulary of only 15,771 unique words.

If we restrict to the first 35,000 words, Infinite Jest contains 4,923 unique words (more than most rappers, but still less than the Wu-Tang clan) while the Brown Corpus contains 2,708.

One issue with measuring vocabulary size is that suffixes may artificially inflate the number of distinct words in the set (e.g. fantod and fantods should not count as two words). To mitigate this, I used the Porter Stemming algorithm to first remove suffixes for every word in the text before counting uniques. I also removed all characters which were not ascii letters (e.g. digits).

Here’s the code:

def vocabulary_size(raw):
    """
    Compute the number of distinct words (stemmed)
    """
    #remove non-ascii letters and capitalization
    rec = re.compile('[^A-Za-z ]')
    no_punct_lower = re.sub(rec,' ',raw).lower()

    #tokenize, stem, and count
    tokens = nltk.word_tokenize(no_punct_lower)
    stemmer = nltk.stem.PorterStemmer()
    stemmed_tokens = map(lambda x: stemmer.stem(x), tokens)
    return len(set(stemmed_tokens))

Finally,

The longest acronyms are O.N.A.N.D.E.A. and O.N.A.N.C.A.A.

These are used to abbreviate institutions within the Organization of North American Nations (O.N.A.N.): the Drug Enforcement Organization (D.E.A.) and the Commonwealth Academy Association (C.A.A).

O.N.A.N.D.E.A appears in footnote 12a. on page 388:

Following the Continental Controlled Substance Act of Y.T.M.P., O.N.A.N.D.E.A.’s hierarchy of analgesics/antipyretics/anxiolytics establishes drug-classes of Category-II through Category-VI, with C-II’s (e.g. Dilaudid, Demerol) being judged the heaviest w/r/t dependence and possible abuse

If we allow a “/” in a acronym, the longest becomes “N./O.N.A.N.C.A.A.”:

Orin had exited his own substance-phase about the time he discovered sex, plus of course the N./O.N.A.N.C.A.A.-urine considerations, and he declined it, the cocaine, but not in a judgmental or killjoy way, and found he liked being with his P.G.O.A.T. straight while she ingested, he found it exciting, a vicariously on-the-edge feeling he associated with giving yourself not to any one game’s definition but to yourself and how you unjudgmentally feel about somebody who’s high and feeling even freer and better than normal, with you, alone, under the red balls.

Table of the most common long acronyms (length 5 or greater):

Acronym :	Number of Occurrences
O.N.A.N.T.A. :	31
P.G.O.A.T. :	13
U.S.O.U.S. :	11
E.M.P.H.H. :	11
U.S.S.M.K. :	7
O.N.A.N.C.A.A. :	5
U.S.B.S.S. :	3
Y.T.S.D.B. :	2
Y.D.P.A.H. :	2
ESCHAX.DIR. :	1
A.A.O.A.A. :	1
O.N.A.N.F.L. :	1
Y.W.Q.M.D. :	1
O.N.A.N.M.A. :	1
F.O.P.P.P. :	1
N.A.A.U.P. :	1
O.N.A.N.D.E.A. :	1
B.A.M.E.S. :	1
I.B.P.W.D.W. :	1

And the code used to find them:

def acronyms(raw):
    rec = re.compile('[^A-Z\.]')
    caps = re.sub(rec,' ',raw)
    toks = nltk.tokenize.word_tokenize(caps)
    acks = filter(lambda x:('.' in x) and re.search('[A-Z]',x), toks)
    return collections.Counter(acks)

The complete code for all experiments is available here: https://github.com/rcompton/ryancompton.net/blob/master/assets/dfw/dfw.py

Bit-reversal permutation in Python

Thu, 05 Jun 2014 00:00:00 GMT

For the FFT sonification I needed to implement my own FFT in order to keep track of which notes are being played while the Fourier transform executes. It turns out that implementing an FFT isn’t too difficult as long as you don’t care how fast it goes (Note: if you do need it to be fast you’ll have a hard time beating fftw which “is an implementation of the discrete Fourier transform (DFT) that adapts to the hardware in order to maximize performance” (cf http://www.fftw.org/fftw-paper-ieee.pdf) ).

The key ingredient to implementing an FFT with the Cooley-Tukey algorithm (which I used for the sonification) is a traversal of the input array in bit-reversed order. Here’s how I did the traversal in Python:

def bit_reverse_traverse(a):
    n = a.shape[0]
    assert(not n&(n-1) ) # assert that n is a power of 2

    if n == 1:
        yield a[0]
    else:
        even_index = arange(n/2)*2
        odd_index = arange(n/2)*2 + 1
        for even in bit_reverse_traverse(a[even_index]):
            yield even
        for odd in bit_reverse_traverse(a[odd_index]):
            yield odd

What I thought was interesting about this code was that I had to use Python’s yield statement, which I had never run into before.

In short, Python’s yield statement is useful when you need to perform recursion efficiently. The difference between a yield statement and a return statement is that yield produces a generator which executes your code on-the-fly rather than storing the entire recursive tree in memory (cf http://stackoverflow.com/a/231855/424631 for more). So, instead of having a global list to write the output of our recursive calls we use a yield statment to make a generator that runs through our list in bit reversed order. Doing something like z=bit_reverse_traverse(a) and then several z.next()s will go through a in the bit-reversed way without eating up all the memory needed for a deep recursive tree. Here’s the code to use the generator:

def get_bit_reversed_list(l):
    n = len(l)

    indexs = arange(n)
    b = []
    for i in bit_reverse_traverse(indexs):
        b.append(l[i])

    return b

For reference, the alternative recursive implementation is below:

def bit_reverse_traverse_no_generator(a):
    n = a.shape[0]
    assert(not n&(n-1))

    if n == 1:
        return a
    else:
        even_indicies = arange(n/2)*2
        odd_indicies = arange(n/2)*2 + 1

        evens = bit_reverse_traverse_no_generator(a[even_indicies])
        odds = bit_reverse_traverse_no_generator(a[odd_indicies])

        return concatenate([evens, odds])

def get_bit_reversed_list_no_generator(l):
    n = len(l)

    indexs = arange(n)
    b = []
    for i in bit_reverse_traverse_no_generator(indexs):
        b.append(l[i])

    return b

New Paper: Prediction of Foreign Box Office Revenues Based on Wikipedia Page Activity

Wed, 04 Jun 2014 00:00:00 GMT

Brian and I recently published a paper on the arxiv: http://arxiv.org/abs/1405.5924, earlier this month it was accepted into the ChASM workshop at WebSci. Here’s the abstract:

A number of attempts have been made at estimating the amount of box office revenue a film will generate during its opening weekend. One such attempt makes extensive use of the number of views a film’s Wikipedia page has attracted as a predictor of box office success in the United States. In this paper we develop a similar method of approximating box office success. We test our method using 325 films from the United States and then apply it to films from four foreign markets: Japan (95 films), Australia (118 films), Germany (105 films), and the United Kingdom (141 films). We find the technique to have inconsistent performance in these nations. While it makes relatively accurate predictions for the United States and Australia, its predictions in the remaining markets are not accurate enough to be useful.

Storing a Kryo object in a compiled jar

Tue, 03 Jun 2014 00:00:00 GMT

This really threw me off for a while (cf https://stackoverflow.com/questions/23748254/storing-a-kryo-object-in-a-compiled-jar ).

I had a HashMap I needed to persist and have fast access to. I used Kryo to serialize the object

Kryo kryo = new Kryo();
MapSerializer serializer = new MapSerializer();
kryo.register(Location.class);
kryo.register(HashMap.class, serializer);

Output output = new Output(new FileOutputStream("src/main/resources/locations50K.kryo"));
kryo.writeObject(output, locationMap);
output.close();

I successfully deserialized with

Input input = new Input(new FileInputStream("src/main/resources/locations50K.kryo"));
Map<String, Location> locationMap;
locationMap = kryo.readObject(input, HashMap.class);
input.close();

log.info(locationMap.size());

the log.info showed 231,045 entries in my map.

Now, in order to access the .kryo file after compiling a *-jar-with-dependencies.jar (using Maven) I needed to replace FileInputStream with MyClass.class.getResourceAsStream

InputStream isr = MyClass.class.getResourceAsStream("/locations50K.kryo");

if(isr == null)
  log.error("null input");

Input input = new Input(isr);
locationMap = kryo.readObject(input, HashMap.class);
input.close();

log.info(locationMap.size());

the log.error never showed and the log.info said I had 0 entries in my map. Why? isr was never null so it read something, Kryo just can’t seem to deserialize it and never provided any error.

Well,

Turns out the problem was with Maven. I had “filtering” enabled so Maven was trying to utf8 encode my serialized objects in the jar. Kryo was silent about it but rewriting the code to use standard Java serialization gave the error found here: http://stackoverflow.com/a/5421992/424631. Kryo should probably have warned rather than failed silently (and I should probably make a JIRA…).

Here is the fix:

<resources>
  <resource>
    <directory>src/main/resources</directory>
      <!--if true Maven will try to UTF-8 encode objects, which breaks deserialization-->
        <filtering>false</filtering>
  </resource>
</resources>

New Paper: Geotagging One Hundred Million Twitter Accounts with Total Variation Minimization

Tue, 03 Jun 2014 00:00:00 GMT

I recently published a paper on the arxiv: http://arxiv.org/abs/1404.7152, and it made Tech Review junior varsity! Here’s the abstract:

Leave-many-out evaluation shows that our method is able to infer location for 101,846,236 Twitter users at a median error of 6.33 km, allowing us to geotag roughly 89% of public tweets.

Algorithm Sonification III: The FFT

Sat, 24 May 2014 00:00:00 GMT

Originally published in 2009 at http://www.math.ucla.edu/~rcompton/fft_sonification/fft_sonification.html

The FFT is an essential tool for digital signal processing and electronic music production. It is easily, commonly, and inefficiently implemented via the recursive Cooley-Tukey algorithm which we now briefly review.

To compute the discrete Fourier transform, $\{ X_k \}_{k=0}^{N-1}$ , of a sequence, $\{ x_n \}_{n=0}^{N-1}$ , we must sum

\begin{equation*} X_k = \sum_{n=0}^{N-1} x_n e^{\frac{-2 \pi i}{N}nk} \end{equation*}

for each $k$ in $0, \ldots, N-1$ . Using the Danielson-Lanczos lemma, we split the above sum into its even and odd constituents

\begin{aligned} X_k &= \sum_{n=0}^{N/2-1} x_{2n} e^{\frac{-2 \pi i}{N}(2n)k} + \sum_{n=0}^{N/2-1} x_{2n+1} e^{\frac{-2 \pi i}{N}(2n+1)k} \\ &= X_{\text{even }k} + e^{\frac{-2 \pi i}{N}k} X_{\text{odd }k} \end{aligned}

where $X_{\text{even }k}$ and $X_{\text{odd }k}$ are the $k$ th component of the DFT of the sequences obtained by selecting only the even or odd members from $\{ x_n \}$ . In this manner we compute the DFT by recursively computing the DFT of the even and odd subsequences of $\{ x_n \}$ . By the time that $N=1$ our DFT is simply the identity mapping from the constant sequence into the constant mode in Fourier space.

Now, suppose that in our initial array we store values corresponding to the frequencies of the usual western equally tempered scale.

\begin{equation*} x_n = 220.0*2^{\frac{n}{12}} \end{equation*}

and execute our DFT algorithm on $\{ x_n \}$ while playing the corresponding frequency on a guitar each time we reach the base case in our recursion. This will play the notes of the scale in bit reversed order and simultaneously allow us a tool for digital signal processing.

Here’s how it sounds: fast_fourier_music.wav

Audio is made in C++ with STK. Here’s the source code: https://bitbucket.org/rcompton/fft-sonification/src

Algorithm Sonification II: Gauss Seidel method

Fri, 23 May 2014 00:00:00 GMT

Originally published in 2009 at http://www.math.ucla.edu/~rcompton/musical_gauss_seidel/musical_gauss_seidel.html

One second of a 44.1kHz single channel .wav file can be read into an array (call it b) of length 44100. Given a matrix A we seek solutions to the system Ax=b. Through iterations of Gauss-Seidel the vector Ax will approach b with the high frequency parts of Ax getting close to b first. If we take b to be a song recording, some white noise as our initial guess and write out Ax at each iteration we observe that the high pitched notes in b become audible first while at the same time the pitch of the white noise decreases.

Audio of the initial 12 second .wav file (white noise) initialAx.wav

Plots of the initial Ax, residual, and FFT of residual:

After one iteration the high notes become audible gauss_seidel_out000000.wav

And some structure is visible in the spectrum:

Second iteration: gauss_seidel_out000001.wav

Third iteration: gauss_seidel_out000002.wav

Fourth iteration: gauss_seidel_out000003.wav

This is all done in python. Loading .wav files into arrays is not too bad in scipy. A sparse matrix class is neccessary to avoid memory problems as a 12 second .wav file needs an array of size 12*44100.

Here’s the TridiagonalMatrix class I used:

from numpy import *

#a tridiagonal matrix class
class TridiagonalMatrix:
    #initialize with 3 numpy arrays
    def __init__(self, upper_in, diag_in, lower_in):
        self.upper  = upper_in
        self.diag   = diag_in
        self.lower  = lower_in
        self.dim    = diag_in.shape[0]
    #matrix mulitplication
    def apply(self, v):
        out = ndarray(self.dim)
        try:
            out[0] = self.diag[0]*v[0] + self.upper[0]*v[1]
            out[self.dim-1] = self.lower[self.dim-2]*v[self.dim-2] + self.diag[self.dim-1]*v[self.dim-1]
            for i in range(1, self.dim-1):
                out[i] = self.lower[i-1]*v[i-1] + self.diag[i]*v[i] + self.upper[i]*v[i+1]
        except(IndexError):
            print "Wrong sizes"

        return out

    #solve Ax=b using gauss seidel
    #with initial guess x0
    def gauss_seidel(self, b, x0, tol):
        error = self.apply(x0) - b
        x = x0
        count=0
        while(linalg.norm(error) > tol):
            #update x in place
            x[0] = (b[0] - self.upper[0]*x[1])/self.diag[0]
            x[self.dim-1] = (b[self.dim-1] - self.lower[self.dim-2]*x[self.dim-2])/self.diag[self.dim-1]
            for i in range(1,self.dim-1):
                x[i] = (b[i] - self.lower[i-1]*x[i-1] - self.upper[i]*x[i+1])/self.diag[i]
            #update the error
            error = self.apply(x) - b

            count = count+1
            print count
        return x

And here’s the code to handle reading/writing the .wav files and then using Gauss-Seidel to iteratively solve a linear system:

from TridiagonalMatrix import *
from numpy import *
from scipy.io import wavfile
import scipy.fftpack
import pylab
import sys
import os

def musical_gauss_seidel(A, b, x0, tol):
"""
do the gauss seidel iteration
but output some sound every now and then..
A is some matrix that lets gauss seidel work
b is a vector that represents a .wav file of a pretty song
x0 is our initial guess for the gauss seidel method (probably random static)
we are going to output the .wav data corresponding to Ax
as Ax gets closer to b (ie the residual gets smaller)
we should hear the song emerge from the initial guess
"""
    #make noise of the initial approximation to b
    wavfile.write("gauss_seidel_out000000.wav", 44100, (A.apply(x0)).astype(int16))

    residual  = A.apply(x0) - b

    Ax = A.apply(x0)
    Ax = Ax.astype(int16)
    wavfile.write("initialAx.wav", 44100, Ax)

    x = x0
    count=0
    while(linalg.norm(residual) > tol):
        #update x in place
        x[0] = (b[0] - A.upper[0]*x[1])/A.diag[0]
        x[A.dim-1] = (b[A.dim-1] - A.lower[A.dim-2]*x[A.dim-2])/A.diag[A.dim-1]
        for i in range(1,A.dim-1):
            x[i] = (b[i] - A.lower[i-1]*x[i-1] - A.upper[i]*x[i+1])/A.diag[i]

        #this will approx b...
        Ax = A.apply(x)
        Ax = Ax.astype(int16)

        #update the reisdual, and get its fft for a nice photo
        residual = Ax - b

        #plot Ax (or something else, the fft of error should be cool...)
        #pylab.plot(Ax)
        pylab.plot(arange(len(Ax)), scipy.fftpack.fft(residual), label="FFT of Ax-b")

        pylab.savefig("AxFFT"+str(count).zfill(6)+".png")

        print "step: ", count, " residual norm: ", linalg.norm(residual)
        #slow convergence so dont write too much
        #if( count%50 == 0):
        wavfile.write("gauss_seidel_out"+ str(count).zfill(6)+".wav", 44100, Ax)

        count = count+1
    return x


def usage():
    print "Usage:  %s" % os.path.basename(sys.argv[0]) + " s x0.wav b.wav"
    print "Where s is the number of seconds to CG."
    print "and x0.wav is the first approximation to b"
    print "and b.wav is what we will converge to"
    return


def main():
"""
you need 3 arguments to main
the first is the number of seconds
the second is the intial guess x0
the 3rd is the song
"""
    args = sys.argv[1:]
    if "-h" in args or "--help" in args:
        usage()
        sys.exit(2)

    seconds = double(args[0])

    #read all the data from each wavfile
    #into a tuple
    #first argument is initial guess (probably random.wav)
    x0_name = args[1]
    x0_wav_data = wavfile.read(x0_name)

    #b is a pretty song that we will approximate
    b_name = args[2]
    b_wav_data = wavfile.read(b_name)

    #the first element in the tuple is the sample rate
    x0_sample_rate = x0_wav_data[0]
    b_sample_rate = b_wav_data[0]
    if x0_sample_rate != b_sample_rate:
        print "need both to have same sample rate"
        exit(2)
    sample_rate = b_sample_rate
    print sample_rate, "the sample rate of b"

    #b,x from Ax=b
    #forget about stereo sound for all this
    #choose the first column...
    #wont work for more than 2 channels
    print b_wav_data[1].shape, "shape of .wav read into b"
    print x0_wav_data[1].shape, "shape of .wav read into x0"
    if len(b_wav_data[1].shape) == 1:
        b = b_wav_data[1]
    elif len(b_wav_data[1].shape) == 2:
        b = b_wav_data[1][:,0]
    else:
        print "I don't know about 3 channels..."
        exit(2)

    #read in x0 as long as it has 2 or less channels..
    if len(x0_wav_data[1].shape) == 1:
        x0 = x0_wav_data[1]
    elif len(x0_wav_data[1].shape) == 2:
        x0 = x0_wav_data[1][:,0]
    else:
        print "I don't know about 3 channels..."
        exit(2)

    #how big the arrays are
    N = seconds*sample_rate
    N = int(N)

    b = b[0:N]
    x0 = x0[0:N]
    N = b.shape[0]#err
    print N, "the length of the arrays after some kinda subsampling"

    #make white noise as the initial guess.
    for i in range(len(x0)):
        x0[i] = random.randint(1,10768)

    #form a tridiagonal matrix
    #you get faster convergence for big diagonals
    A = TridiagonalMatrix(ones(N-1), zeros(N), ones(N-1))
    for i in range(N):
        A.diag[i] = -2.001

    print b.size, "b size"
    print x0.size, "x0 size"

    tol = 1000
    x = musical_gauss_seidel(A, b , x0, tol)

    return

if __name__== "__main__":
    main()

Algorithm Sonification I: Sorting

Thu, 22 May 2014 00:00:00 GMT

Originally published in 2009 at http://www.math.ucla.edu/~rcompton/musical_sorting_algorithms/musical_sorting_algorithms.html

Suppose you drop a set of drums and they land randomly ordered in a row on the floor. You want to put the drums back in order but can only pick up and swap two at a time. A good strategy to minimize the number of swaps you must make is to follow the Quicksort algorithim http://en.wikipedia.org/wiki/Quicksort.

If you order the drums by their general MIDI number and simultaneously strike any two which you swap then you will produce a sound similar to this:

https://www.youtube.com/watch?v=g2IWUd3p30I

To hear what’s going on with quick sort a little better consider the case where you have dropped 12 guitar strings whose frequencies vary expoentially, ie

frequency of string $i = 220.0*( 2.0^{\frac{i}{12.0}} )$

for $i=0,...,11$ for more information http://en.wikipedia.org/wiki/Pitch_(music)

https://www.youtube.com/watch?v=YR6VAZUGAMo

I could not hear what’s going on all that well either. An aesthetically pleasing quadratic running time strategy you may follow is insertion sort http://en.wikipedia.org/wiki/Insertion_sort. Here’s how insertion sort sounds in a scenario involving many more strings on the floor:

https://www.youtube.com/watch?v=DNAmWDmIAZQ

And if you were to sort drums with insertion sort you would get something like this:

https://www.youtube.com/watch?v=zXTlRc6QM-M

The sounds were made using STK http://ccrma.stanford.edu/software/stk/ the videos were made using the python gnuplot module http://gnuplot-py.sourceforge.net and mencoder. The source is at https://code.google.com/p/musical-sorting-algorithms/

Update 2009-08-11: This got written up in Make

Sample pom.xml to build Scala *-jar-with-dependencies.jar

Mon, 19 May 2014 00:00:00 GMT

I’ve been using Spark for a few months now. It’s great for tasks that don’t fit easily into a Pig script. If you’ve already been writing Java for a while it can be a drag to switch everything over to sbt just for Spark.

Here’s a basic pom.xml that can build scala code (you’ll need maven 3) and include all dependencies into one giant jar.

<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
        xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.mycompany.app</groupId>
    <artifactId>my-app</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>my-app</name>
    <url>http://maven.apache.org</url>

    <properties>
        <maven.compiler.source>1.6</maven.compiler.source>
        <maven.compiler.target>1.6</maven.compiler.target>
        <encoding>UTF-8</encoding>
        <scala.version>2.10.4</scala.version>
    </properties>

    <build>
        <pluginManagement>
            <plugins>
                <plugin>
                    <groupId>net.alchim31.maven</groupId>
                    <artifactId>scala-maven-plugin</artifactId>
                    <version>3.1.5</version>
                </plugin>
                <plugin>
                    <groupId>org.apache.maven.plugins</groupId>
                    <artifactId>maven-compiler-plugin</artifactId>
                    <version>2.0.2</version>
                </plugin>
            </plugins>
        </pluginManagement>

        <plugins>

            <plugin>
                <groupId>net.alchim31.maven</groupId>
                <artifactId>scala-maven-plugin</artifactId>
                <executions>
                    <execution>
                        <id>scala-compile-first</id>
                        <phase>process-resources</phase>
                        <goals>
                            <goal>add-source</goal>
                            <goal>compile</goal>
                        </goals>
                    </execution>
                    <execution>
                        <id>scala-test-compile</id>
                        <phase>process-test-resources</phase>
                        <goals>
                            <goal>testCompile</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

            <!-- Plugin to create a single jar that includes all dependencies -->
            <plugin>
                <artifactId>maven-assembly-plugin</artifactId>
                <version>2.4</version>
                <configuration>
                    <descriptorRefs>
                        <descriptorRef>jar-with-dependencies</descriptorRef>
                    </descriptorRefs>
                </configuration>
                <executions>
                    <execution>
                        <id>make-assembly</id>
                        <phase>package</phase>
                        <goals>
                            <goal>single</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>

        </plugins>
    </build>

    <dependencies>
        <dependency>
            <groupId>org.scala-lang</groupId>
            <artifactId>scala-library</artifactId>
            <version>${scala.version}</version>
        </dependency>
    </dependencies>
</project>

Supervised Learning Superstitions Cheat Sheet

Fri, 02 May 2014 00:00:00 GMT

This notebook contains notes and beliefs about several commonly-used supervised learning algorithms. My dream is that it will be useful as a quick reference or for people who are irrational about studying for machine learning interviews.

The methods discussed are:

Logistic regression
Decision trees
Support vector machines
K-nearest neighbors
Naive Bayes

To see how different classifiers perform on datasets of varying quality I’ve plotted empirical decisions boundaries after training each classifier on “two moons”. For example, here’s what happens to scikit-learn’s decision tree (which uses a version of the CART algorithm) at different noise levels and training sizes:

Here’s the notebook: http://ryancompton.net/assets/ml_cheat_sheet/supervised_learning.html

(as a .ipynb file: http://ryancompton.net/assets/ml_cheat_sheet/supervised_learning.ipynb)