The Unscatter.com Blog

So, I thought I had a good idea with the last iteration of Unscatter.com. You could search Blekko, Bing, Twitter, Facebook… yadda yadda yadda. It worked in practice. Yet, I had to face the fact that many people with great ideas have to face. No one was interested. Sure, some people came to the site, but 99.9% of visitors looked at the first result set and moved on. No one tried to use it to search twitter, facebook or any other sites. No one cared. * cue worlds smallest violin *

So, after realizing this, I took a look at my own usage habits. When researching stuff for my job as a Systems Admin, I often used the /google tag to redirect my search to Google. Basically, all relevance searching involved me using my site to redirect myself to Google. So, I was wasting time. The other thing I used it for was to keep up to date on topics. I’d use the /news and /date tag a lot. I often used /twitter, /googleplus and /facebook when looking for commentary on topics. Basically my primary purpose for the engine I built was to get current information about topics.

I’ve made it easier for unscatter.com to return those results, by doing it by default. Search for a topic and unscatter.com will provide you the most recent news and social data. You can also get access to the most recently indexed results by Blekko for the search. In the future I’ll be adding some more default tags like blogs, discussions and reviews.

The interface needs work as well. I built it primarily with javascript in order to use it quickly. I didn’t want to put a lot of time into building 2 sets of templates if I wasn’t going to use it. When Bing decided to start charging for results, I was close to just closing up shop on unscatter.com and moving on. After using it for a couple weeks, I’ve found it a great resource for current information so I’m keeping it. I’ll speed up the page load as I get time to work on it. 

Some other ideas I have include:

• Change the default social search to Google+. I found that when searching for local housing market data, Google+ was a much better resource.
• Try to get Facebook to only return results with links, like I did with Twitter.

So that’s why I made the huge changes to the site. Hopefully others will find this a more useful tool as well. 

 Mike Jarema, the author of the Twitter Bootstrap theme this Tumblr uses got a hold of me to say that the theme in now installable via Tumblr directly. I’ll be switching to it so I can automatically get updates sometime this week if I can figure out how to tweak the css for the logo. Pretty sure I can do that with out too much effort.

For anyone else interested in using the theme, it can be installed from here: http://www.tumblr.com/theme/34547

It’s great for any sites using Bootstrap for their application to get their blog to match their product.

In other news, I won’t be ending unscatter.com, however the format will change soon. The loss of the Bing API is making me change directions and I’m going to focus more on providing current information for search queries. I believe this is a better match for the API powered approach I’ve been taking anyway. Not a new approach for unscatter, some may remember I did something similar a year or so ago. I believe I have come up with a better user experience this time though.

So, I just received an email from Microsoft detailing new changes that are coming up for the Bing API. It will be moving to the Windows Azure Marketplace. They are going to provide fresher results  with better relevance as a result. They also state in the email that there will be more opportunities for developers to monetize their usage of the API.

In order to be able to manage the cost of serving the API, Bing is going to charge for it. $40 a month for up 20,000 queries. 

What does this mean for unscatter.com? I don’t know yet. I suppose I will look at the monetization options. Honestly I think this might mean the end of unscatter.com, unless I come up with an entirely new direction for it. I’ve been leaning towards closing it down for a while now, especially as more API’s become pay for use. I have other projects that I think provide more value to people over all, which is what makes a good product right?

So, this may be so long and thanks for all the fish. The nice thing is I don’t have any hard feelings about it. Unscatter.com served it’s purpose of being a fun hobby for me to work on learning various elements of web application development. Maybe this is the final push I needed to dive into working on other ideas.

Just a quick note. Unscatter.com now has a Bing powered spell check in place. Also, we’re using the Wikipedia API to get suggestions for terms to search for. 

This is the start of improvements where the site will try to figure out a bit more about what you’re searching for to try and get you better results. Expect more changes soon, such as more suggested slashtags and result samples from them.

The blog post I did about switching to Bootstrap and Backbone.js drove a little bit of traffic to unscatter.com yesterday. I saw about 2100 searches yesterday on the site. Not bad, normally I have less than 30 a day, all me. Taking a look at how people were using the site, I learned a few things.

The Good:

My platform is sound. I haven’t put all the optimization I’d like in yet. The css and javascript is the biggest piece of front end optimization I can do. There are 8 requests to unscatter.com and 6 more to outside sites for a single non-slashtag search request. With most of the traffic being new traffic, I ended up serving over 30,000 requests yesterday. Also, every search request is currently dynamic. I’m not caching anything I get from the api’s yet. I have plans to do that, but honestly based on analysis of the traffic yesterday the search requests were different enough that it wouldn’t have helped a lot.

However, the little 256MB RAM Rackspace cloud server running nginx + 2 instances of the tornado backend that is unscatter.com never broke a sweat. At this point, I do believe the application will not be expensive to scale. 

I also got some feedback in the comments of the Hacker News entry. It’s great people took the time to do that. I should make it easier for them, so now I’ve added UserVoice to the site in an effort to make it easier to get feedback.

The Bad:

Very few people used slashtags, so most people did not see the real value of the site. In fact, most people did one search and were gone. I’m not surprised. There are already several very good choices for search, Google being the best right now. Even I often use the /google tag to redirect off of my own site when researching things like error messages and such. I think people do a single search and when it doesn’t amaze them just go back to Google. Climbing the road to acceptance as alternative search site would be a long trip. Especially since I’m using API’s and not doing my own crawling and page ranking right now.

The Conclusion:

If I want people to want to use the site, I have to figure out how to show them what the value is in a single page view. How do I do this? Not sure yet. I imagine I need to get smarter about figuring out what the search terms mean and give them nudges towards slashtags that will help them. A little stronger nudge than a list of slashtags in the sidebar. 

I probably also will need to start building my own search results. I use unscatter instead of other search engines because I already know how to use it. However, even I often use the google redirect when researching things as a part of my job. 

As a hobby, unscatter.com is great. As a business, I’m not so sure. I will likely continue to try and grow it organically and see about how to eventually monetize it. So far I am considering figuring out a way to do advertising as sponsored search results and perhaps affiliate with api’s from sources like Amazon. If it ever did get popular some of the api’s I use for free now won’t be free any more, for example Topsy. 

I have other ideas for businesses that might be a bit easier to dip my toes into. At this point I’m more looking for ways to create additional income, not to create a million dollar business. I just always have big ideas.

For example, the other project I’m working on, and will likely get switched to Bootstrap and Backbone soon is ChatFor.Us. Instead of search now I’m trying to take on real time communication. I have an idea for web comic and a time done/todo list application I’m working on. The other thing I’ve learned from a couple thousand searches is maybe I should invest more time in some of those projects instead.

Interesting Bit of Trivia:

I got lots of searches for “instagram” yesterday. If you’re reading this sometime in the future, yesterday is when the announcement of Facebook purchasing Instagram was made. I think if I had at least some basic term analysis working I could have given people news results and they would have been more pleased with the initial result displayed to them. Also there were a lot of searches for “test” and “what is this”. That would be a good time to put up a javascript demo to walk them through how to use the site. 

I often find that the best compliment you can give a piece of technology is “it just works”. I have to give that compliment to the people working on Twitter Bootstrap and those working on Backbone.js.

I spent last week sick with a pretty good case of flu and sinus infection. Basically I was bed ridden for several days. I imagine like many other people, I don’t cope with being stuck in bed so I found something to keep myself busy. I decided to redesign unscatter.com.

Unscatter.com previously used some of Twitter Bootstrap 1.4, but was primarily YUI grids and all YUI 3 javascript. I’ve been using YUI for years now, and have always appreciated the community it has. Especially the core engineers at Yahoo supporting it. However, I was pretty peeved at Yahoo’s new direction of attacking organizations with their patents. 

I decided that I didn’t want to use Yahoo products any more to show my being unhappy with their recent decisions. Unscatter.com was largest use of Yahoo technology and as I had nothing else better to do I attacked it. 

A week of working probably 2 -3 hours a day, with sinus headaches and such resulted in the the new unscatter.com. Both Twitter Bootstrap and Backbone.js were easy to use. Backbone.js did involve a bit steeper of a learning curve, but over all I got it pretty quickly. It’s the area I believe I have the most learning potential in. Most result pages are custom Backbone.js scripts, I’m sure I can consolidate more. 

For now though, I’m happy to say I’ve removed all traces of Yahoo technology from my stack and have replaced it with systems that I believe will allow me to continue to focus heavily on the back end of the product while still providing an excellent design and UI to my users.

Today I got an email from Gabriel Weinberg letting me know my submission for the !unscatter tag on DuckDuckGo was implemented. I was pretty excited, I really like DuckDuckGo and think what they’re doing is great. 

I’ve had it on my list to implement the DuckDuckGo instant answer API for a while now. I just hadn’t worked down to it. I decided to up this to top priority in response to being included on their site. 

I went with a different implementation for the instant answer API. Normally I proxy the requests via the unscatter.com web server. However, there’s no API keys required for that API, so I decided to cut out unscatter.com as a middle man.

I use YUI3 as the javascript framework on unscatter.com, and this also made it a lot easier to implement the functionality, as I have immediate access to YQL via yui3. If you haven’t played with YQL it’s pretty cool, I’d check it out.

Now while YQL has tons of tables for various APIs, it doesn’t happen to have one for DuckDuckGo. Not a problem. It does have a generic json table, which allowed me to use the query ‘select * FROM json WHERE url=”http://api.duckduckgo.com/?q=’ + query + ‘&format=json”’. Simple as that and I’ve got the full response from the JSON api at DuckDuckGo available as an object to parse. Who needs custom tables?

The nice thing is YQL caches responses, so if identical searches are being performed on unscatter.com it should keep DuckDuckGo from having to parse all those queries as well.

The implementation isn’t complete. I haven’t started on RelatedTopics for example. However answers and abstracts are working. I’ll expand on this in the future before I get to work on adding video support. Video support being what I was originally planning to add today. 

Edit: The proper endpoint for using the DuckDuckGo api is api.duckduckgo.com, I originally was using just http://duckduckgo.com.

The last blog update was on May 4th. Since then more work has been done and I thought it was about time to share what’s been going on.

Now, not a lot work has been done in 2 months. However, at least one change has been pretty significant.

Here’s some of the highlights.

• Blekko is now the default source of search results. Use /bing if you want to get results from Bing.
• Default searches are presented with what I’m now calling a portal view. Eventually even slashtag results will have a similar view. The point is that I’m now providing you with a quick way to get results from multiple sources. For example a search will now provide Blekko results and Topsy results right away. You also can quickly view results from Facebook and Twitter directly from the page as well.
• Instead of just providing suggestions for slashtags to search, I’m providing you with results as well. On the portal view any suggested tags are displayed within the results, you can cycle through the tabs to see the first few results from tag without having to leave the page. 
• I’ve dropped preserving slashtags in the current searches section. Current searches allows you to bookmark multiple searches so you can cycle through them. How I use it is to keep up to date on certain by bookmarking several searches and returning daily to view them. I’ve found that defaulting to the portal view makes this simpler to manage.
• The first introduction of media searches has been implemented. You can search using the /images tag to get image results from Topsy. This is just the start. Eventually I’ll be adding support for more apis for finding pictures and video.
• Speaking of Topsy, I really like them as a source of information. As such I’ve begun integrating their results more tightly with unscatter.com output. I’ve also made it possible for you to modify the search window for Topsy, letting you see results from the last hour, day, week, month or all time.
• The goal of unscatter.com has always been to find the best results possible for the searches people are submitting. The simple case is that the API’s I’m using may not be as useful as other search engines who’s APIs I’m not able to use. As such, /google and /duckduckgo (or /ddg) slashtags will redirect you to searches on Google or DuckDuckGo. Basically, if you’re not finding what you want on unscatter.com, I want to get you to the results you’re looking for as quickly and easily as possible. One advantage of this being a personal pet project is I don’t have to have the word “competition” in my vocabulary. Rather, I can focus on providing the best service possible for people looking for information they need.

Work will continue as time permits. Honestly, I’m not seeing much adoption. I’ve implemented Mixpanel to get a better idea of how people are using the site. Mixpanel and Google Analytics show pretty much no one is using it so far. If you have any suggestions of how I can make this a tool you’d like to use as well, feel free to hit me up on Twitter @joerussbowman. I also opened up the Ask me anything feature on Tumblr as well. Personally I use it as my primary search engine and have found it great.

So my Scale0 project for rewriting the infrastructure behind unscatter.com has been put on hold for bit. It’s going to take larger amounts of time to complete than I have right now. So, I’ve gone back to working on the smaller pieces of unscatter.com that can be improved, and knocked a couple out this afternoon.

First, you may notice the new extremely drab logo. The old logo, which was spelled Usc@tter was confusing some people when the first looked at it. Basically some looked at it and said “Unsc” “at” terrrr”? I blame Twitter, or maybe just email. Anyway, until I come up with a better logo I’ve just put the current one up. I’ll have to change the favicon and Facebook page images at some point too, but I’ll worry about that after I come up with a new logo.

Now, I did add something pretty useful too. If you have javascript on, Bing, Blekko and Topsy results will return how many times each result url has been shared. Go ahead try a search and check it out. The point is now you can get idea of the value of a url by seeing if it’s been shared and how many times. I’d like to eventually add something similar for Twitter, but the api they have for doing this is still labeled as private. 

I also got in some bug fixes, primarily related to paging. Facebook is still a little wonky, but not much I can do about it. It’s the way their API works.

I’ve still got more to add in the future. I have, for now, ditched the HTML5 caching. I’m looking into doing more caching on the back end with a couple different approaches coming into play. I’m also going to add more sources like Youtube and see about getting some more instant answer type results in from ChaCha. I noticed I had to go to Google to do “canucks score” because unscatter.com couldn’t give me that data. Hoping ChaCha is a good source for that, or I may have to dig around for some other available APIs. Longterm is still realtime streaming and I’m considering a news portal and possibly a personal social portal for link gathering as well.

For those curious, usage is still pretty much me, myself and I. I’ve been using it as my primary search engine for over a month now and have found it pretty useful. My current two bookmarked searches are:

My technology search list (had to shorten that, default Tumblr theme didn’t like it)

Which allows to me to get the latest information about some technologies I am interested in.

http://www.unscatter.com/search/?q=%2Fnews+redskins&t=%2Fnews+canucks

Which keeps me up to date on my 2 favorite sports teams. One of which has me sporting a pretty thick playoffs beard right now. :)

Always interested in feedback. Right now the Unscatter Facebook Page and Unscatter Twitter account are the best ways to reach me. Or you can just hit me up personally on twitter @joerussbowman

I’m getting visitors every day now, most days I need both hands to count them even. So of course I need to think about scaling. Ok, truthfully thinking about scaling is something I’m rather fond of, and as Unscatter.com is being treated as a hobby rather than a business at the moment, I’m letting myself work on something I enjoy.

That something is Scale0. The original design of Unscatter.com was to scale. It uses 2 Tornado instances running behind an Nginx proxy. Caching, while not implemented fully yet, was planned to be done by MongoDB and file based caching using Nginx modules.

I haven’t been quite happy with the Nginx approach, I’ve been wanting something more integrated with the backend for a while. I found ZeroMQ recently and have started my own project Scale0 which will be built using it. The end product for Unscatter.com will be a caching http proxy.

I still have more plans for Unscatter.com’s interface. I’ve also dropped several ideas I had planned to integrate into Unscatter.com in order to focus on search. Once I finish Scale0 I plan to move Unscatter.com onto it, and will then focus back on adding features and improving the interface. Expect that logo to go at some point soon too.

Until then, I’d just like to say that I have switched Unscatter.com to being my primary search engine. I continue to fix bugs I find by using it. So far the best value I’ve gotten from it is researching topics where I’m able to quickly get contextual information using Bing and Blekko and then finding out the most recent information using tags like /date, /twitter and /facebook.