Reflections: OmegleBot

Omegle.com allows people from around the world to converse with each other “anonymously”. It is one of those sites that let you start a text or video chat with someone online and consequentially makes you doubt the intelligence of man kind. On sites like this, text chat follows the famous “Greater Internet Fuckwad Theory” and the video chat is… Well it’s probably a phallus. Omegle started as a way for strangers to connect and talk with each other, but has since devolved and the chance of finding some meaningful conversation on it is minuscule which is a shame because random chatting is a fun concept. I would add a premium feature that administers an IQ test and matches you to someone according to that but that is an idea for a different time.

A typical Omegle chat

A typical Omegle chat

When I first discovered Omegle I quickly got tired of trying to find someone to talk to. The idea of Omegle is not new or revolutionary. IRC and chat rooms were there before but this made it as easy as can be. Since I already spent a ton of time in online social communities with people who have the same interests as me I dismissed it as a cost efficient way of communicating. I did find a use for Omegle though – there was nothing preventing me from spying on a random conversation and recording it. A nice challenge and it seemed fun. This was years before Omegle itself introduced the “Spy mode” so I guess there is something there. The concept of Spy Mode might look like something “evil” to do – spying on other people’s conversations is an ethical gray area in the real world, but is it online?
The answer to this question depends on how much you know and are aware of privacy online. In theory everything you do online can be (and is) monitored by a number of entities including your ISP (that can read all of your online activity), your operating system and other programs on your machine, a lot of routers on the Internet and at least a few governments. That is all beside the point – I thought about ethics for a second but to be honest I don’t see this as anything but a technological challenge (also, it isn’t illegal per se). My goal wasn’t to spy on people but to hack a bot together and as such I probably only ran the finished script once – to make sure all the bugs were solved. I call it the Hacker’s Mindset :).

This is how OmegleBot was born. A simple and a very quick and dirty and unrefactored python script. Before that script I didn’t have a lot of knowledge about HTTP, httplib and urllib because I used raw sockets to talk HTTP (poorly) in the past. This was a perfect project to help me understand the python libs relating to HTTP and JSON. The bot opens two simultaneous connections to Omegle and sends them both a simple greeting, “asl?”, which is the way most conversations in chat channels start. It then proceeds to proxy their conversation and also record it into a text file. The most interesting part is the post function. It started as a simple call to connection.request and evolved to include a variety of HTTP headers including a faked user-agent and referer needed to defeat some of Omegle’s “security checks”. Usually services will have more server side security checks (“never trust user input”), but unfortunately Omegle doesn’t have a choice here. Because they are open and allow anonymous chatting it leaves them with only so many ways to ensure I’m a client and I masqueraded as one well. Omegle uses the JSON protocol to pass data about events like whether the other user is typing, the message the user sent and of course when a user disconnects. Reverse engineering it was the hardest part of this project (and it wasn’t all that hard). I think the only challenge I faced was understanding why Omegle blocked the first iterations of the bot and adding various headers until I passed for a client in their book.

I also attached a sample output file with a few conversations. There is nothing interesting there nor did I capture anything interesting. All the conversations are very short which is definitely a symptom of Omegle – long and meaningful conversations are few and far between. I even sent “typing” statuses every few iterations to encourage people to converse and it didn’t help.
What can we learn from this? Masquerading as a browser is easy. Writing bots is easy. As a person on the internet you should take from this that bots are everywhere on the web. You should be aware of that because a lot of spam and fraud is done by bots – you can trivially change this bot to spam on Omegle (although ChatRoulette, a similar site has a “spam” button that might be useful against that). Radiolab even had a podcast on a bot that had an online relationship with a human. It is a fact that bots are becoming better and better at passing for human beings. Soon they might even be good enough to write a programming blog, and then what will I do?

Southpark's "dey took er jerbs" guy"

Southpark’s “dey took er jerbs” guy”


(program them, probably)

P.S. Unfortunately the bot stopped working. It can be that Omegle changed the protocol a bit, added some more security or that I have a bug. Feel free to fork it and bring it back to life!

Reflections: UnUrlShield – Fighting CAPTCHAs

Update: A challenger appears. My security researcher friend Fox has challenged me to a duel. See her blog post for the details.

This is part of the Reflection series in which I go through my old projects and tell their story.

CAPTCHAs have become an integral part of the web in the last few years. Almost everyone on the web has encountered those twisted pictures, probably when signing up to an email service. They come in various shapes, sizes, colors and cats. When they first became popular, there was an explosion of different types of schemes that services used (who can forget Rapidshare’s cat captcha?).
Now as with every security measure there is a compromise between usability and protection. Some of the easier CAPTCHAs were broken using only OCR software, while some of the latest reCAPTCHA images are hard even for a human to solve (interesting but out dated chart).

Various captchas from wikimedia.

One such service was UrlShield. You would give UrlShield a URL you want to protect from bots and it created a page with a CAPTCHA that when solved correctly redirected you to your original URL. Simple enough. I can certainly see a use for such a service, for example if you want to give out promotional coupons and don’t want bots to snatch all of them. The service became popular in some file sharing sites for the same exact reason.
The particular image this site was generating had a checkerboard background with 4 characters all in different colors, sometimes overlapping. It was pretty easy for a human to parse it.
Example of UrlShield generated images and their OCR.
It even works pretty good against OCR. I used Microsoft OneNote OCR feature which uses the commercial OmniPage software to create the second column.

So far so good? Well, no. This scheme is flawed because it is easy to transform the image – remove the background and segment it (split it to region that each contains a single character), allowing OCR tools to easily get the letter. To remove the background you just clear all the black pixels out of the image. To segment it all you need to do is choose one color and mask all the others, which means you’ll end up with a single letter, as each letter is in a different color. This is what you end up with:
Parsed captcha images
OneNote has no problem parsing each of these to a letter.

The process described above is exactly what UnUrlShield does. It’s a simple Python script that use the Python Imaging Library to read the image. Then it counts all the colors that appear more than a certain threshold (MIN_PIXEL_PER_LETTER_COUNT) and saves each color’s pixel location. Lastly it goes through the colors, creating an image with only that color’s pixel locations.

Is there a lesson here about CAPTCHAs? I think so. UrlShield is now some kind of ad/malware site. Even complicated CAPTCHAs can be broken, or even better – be defeated by side-channel attacks like having an army of low-cost workers break them on-demand (The comments of this article are a treasure trove of irony) and sometimes people are even fooled into breaking CAPTCHAs. This is why it amazes me they are still around, annoying normal regular people while also being broken by even slightly motivated attackers.
Are there no solutions to spam? Of course there are! In fact gmail does a great job at stopping 100% of my spam using things like blacklisting known spammers, Bayesian filtering, “crowd-sourcing” protection (the “mark as spam” button) and other tools that don’t rely on CAPTCHAs.

Do you have good examples of silly, easily broken or bizzare CAPTCHAs? Did you find an easy way around some services blocked by CAPTCHAs? Leave a comment below and tell me about it!

Reflections – an Intro and an Example

In this first post in the “Reflections” series I’m going to explain what is the purpose of the series, talk about the reasoning behind revisiting old projects and finally applying it all to a small but fun piece of code.

Reflecting on What?

This series will be about reviewing my github projects. This serves the dual purpose of having at least 10 topics for blog posts and writing documentation for old projects in the hopes that someone else finds them useful or interesting. About a year ago I’ve decided that I must do something about the tens of “side projects” that I have from more than 5 years of coding. I quickly triaged the list of about 25 projects into ‘forget’ or ‘open-source’ and just threw everything presentable with a really minimal readme to github.
Part of the list of projects

So why write about it?

  • Code isn’t as find-able as text. To make this code useful, it needs to be at least somewhat documented. When I uploaded them I half assed a read me file but I doubt that is enough for a search engine to index it correctly.
  • I think most of the projects has something unique to add to the web and are interesting both at the code level and in the research that brought them to life.
  • Revisiting old projects will give me a nice nostalgical feeling.
  • Did I mention 10 easy blog posts?

What am I going to cover?

  • The why – The research that went into the code. Since most of the projects are a Proof of Concept (POC) I think it will be interesting to remember what started that particular train of thought.
  • The how – The parts of the code that do interesting stuff and how do they do it.
  • The what now – Maybe a paragraph or two about how the project can be productized or improved and where can it evolve from here.

Let’s try this out

To finish this intro I want to talk about a small but fun hack I did when I got my first Android phone. It was the happy era of 2.X Android and I got a HTC Desire from England. It was right about that time that they announced that next version will be out in a month (Haha, I still believed OTA promises) and will have Hebrew support. That meant that I had a month where my phone didn’t have the Hebrew language font and most importantly I couldn’t read any SMS. Since the phone was new I didn’t want to experiment with rooting just yet, I needed to find a clever solution. One of the first apps I installed on the phone was the Android Scripting Engine (also known as Scripting Layer 4 Android, unfortunately development looks pretty dead now, although it works on my Galaxy Nexus) since I hoped there would be a way to write real apps with it. Unfortunately it was quite minimal, but one thing it did have is a way to get your SMS. That meant I could write a simple script to transliterate Hebrew Unicode into English (ASCII).
Picture of the SMS transliteration app

The code itself is very straight forward. It is only a few tens of lines, and there is pretty much no reason to ever use it again, but I think it represents why I like the Android – it has more of a “hacker” spirit than the iPhone. The only interesting thing I see in the code is that the sign for NIS and the u2029 whitespace in the translation dictionary. I guess I started manually adding characters in an attempt to map Unicode to ASCII. Finally I gave up and added the error parameter to encode that puts a “?” character for every Unicode code point it can’t print.