Recently in Duck Duck Go Category

Magic keywords on Google and the consequences of tailoring results

 

On September 6th, 2012 at 2PM eastern time, 131 DuckDuckGo users searched Google at the same time for abortion, gun control and obama. This was the first in a series of experiments we're doing to explore the state of Google search tailoring -- the fact that different people see different results on Google based on who they are.

Who Cares?

Sometimes tailoring can be good, e.g. we show local weather in our instant answer box. However, in other cases it can put you in a bubble, seeing more and more of what you already agree with, and less and less (relevant and important) opposing viewpoints. This effective filtering is troublesome when looking for raw information on a subject.

For example, consider you're an undecided voter being bombarded with propaganda from both sides. What do you do to research the candidates? If you're like most people, you use Google as your starting point.

Yet when searching Google for raw information, say on gun control, we found people getting very different results. That's why at DuckDuckGo we show everyone the same links by default--so you can escape your personal filter bubble. We believe any tailoring of the organic results should be opt-in and not opt-out.

In a recent Pew Report, 65% of people said personalized search was a "bad thing" since "it may limit the information you get online and what search results you see" compared to just 29% who said it was a "good thing" because "it gives you results that are more relevant to you." When asked a different way 73% said they were "Not OK" with it because it was an invasion of privacy.

Introducing DuckDuckHack

 
slide-xkcd.png


I think most search results would benefit by some form of instant answer (like xkcd). DuckDuckGo has a lot of goodies, but I think we're only scratching the surface on what can be done there.

That's why today we're announcing DuckDuckHack: an open source platform to create instant answer plugins for DuckDuckGo. Think of them like add-ons for Firefox.

My answers to Peter Thiel's questions he asks startups

 
Blake Master's class notes for Peter Thiel's startup class have been great, which is why I immediately was intrigued by the PandoMonthly event with him. One of the more interesting things to me that was said was his pointed questions to ask startups.

As an angel investor I usually ask:
  • How did you (founders) come to be entrepreneurs?
  • What led you to this market/idea in particular (and what markets/ideas did you discard along the way)?
  • What is your path to victory?
  • What are your exit expectations?

Peter asks...

How many employees do you have?

 
People often ask startups how many employees they have. I'm not entirely sure why they do or why I care, but something about this perpetually asked question bothers me.

Yes, there are real organizational differences as you grow significantly, but most of the time when it's asked of small startups the answer is pretty obvious, i.e. somewhere less than 30.

DuckDuckGo FOSS Donations 2011

 
I recently made most of DuckDuckGo's 2011 FOSS donations: half directed by our company and half directed by our community. Over fifty nominations were made on duck.co, which like last year we grouped into a theme and culled for biggest perceived impact.

The theme for this year is open data/information, inspired in large part by the SOPA/PIPA debate and related activities. To that end we donated $940 to each of the following projects.

Happy first birthday DuckDuckGo!

 
logo_homepage.birthday_chocolate_cake.v101.png


29th.jpgOn Feb 29, 2008, I incorporated DuckDuckGo. Because that was a leap day, technically Feb 29, 2012 is our first birthday.

The incorporation date was about seven months before I would soft launch the service on Hacker News. 

Back then I had no idea what would become of what I was doing. I had spent the previous few months crawling, messing around with structured data, parsing Wikipedia and working on other projects.

DuckDuckGo used to run out of my basement

 
CameraZOOM-2011122805564420.jpg
For the first two years or so, I ran DuckDuckGo out of my basement. Then when people really started using it I set up EC2 for fail-over, and now EC2 is our primary host. We still use these basement servers for a variety of things, however.

Before DuckDuckGo, I operated in a pre-cloud world and had been running my own servers for a number of years. My last company co-located (stored our servers in a data center), which had worked well except for when things went wrong and a reboot didn't solve it. Then I'd have to (usually at inopportune times) drive over to the data center, be escorted in, mess with things, and be charged a lot for doing so.

The main problem with self-hosting (vs co-lo) in my mind had always been crappy Internet. Then in 2006 I moved to our house in a town that approved FIOS a few months later. I had been watching it progress, and we were literally the first person in the area to install it.

I was hopeful and it turned out correct that the crappy Internet problem had been solved. In five years, it has never been fully out except for the time our service got suspended for lack of payment (forgot to update my expired card).  Other than that, there is just the usual blip always coincidental with some internal Internet routing thing. In fact, since I have complete network control, the "co-lo" doesn't reboot routers for scheduled maintenance either. The latency is also pretty good.

Note that I'm not advising any of this -- simply telling my story. So why did I do this? Really it was the path of least resistance given my history and experience. I was used to setting up and maintaining servers, and had some laying around. This setup seemed great at the time -- I'd get the benefits of co-location (good Internet) but could mess with things at will. (The other co-lo benefit is power -- I have that covered as well.)

I still like the idea of having a server for non-production use, e.g. this Web site. There is something nice about having it close by. Perhaps it is a false sense, but I feel that the setup is more reliable than any cloud provider. And I get to feel a bit more full stack.

But all that pretty much stops at one server for me. Anything beyond that and it quickly becomes a PITA. Here's a pic from when we bought the rack off of Craigslist:

6453_559199337888_700754_32715196_4251966_n.jpg

What I learned from raising venture capital

 

Takin' VC Money (Money Cash IPO's) by Smixx


Forewarning

Be forewarned, this post is long. You can tl;dr it by reading the section titles.

It's long because I'm compressing the last four months into one post. I don't normally do that, but I didn't want to influence my funding by blogging about it in the middle of it. And historically I've been bad at doing serial posts, so here you go...


I'm new at this.

I recently raised venture capital for the first time. I'm going to relate what happened below and highlight my mistakes/takeaways in the bold section titles. However, please bear in mind my lack of experience in these matters and that this post is written from the company/entrepreneur perspective.


Background.

DuckDuckGo raised a series A round from Union Square Ventures and a handful of awesome angel investors.  While we did talk to over 30 VC firms, I realize our path both ended successfully and was relatively quick and painless. So first off, I want to say I sympathize with everyone who struggles with the funding process. 


The funding process is not to be taken lightly.

Even though our round went relatively smoothly, it was still a massive time sink. It was the top idea in my mind, and it pretty much consumed my life for the past four months. In other words, it seems like you should commit to being all-in, all-consumed for a while, or you might as well not do it. 

This reality is especially troublesome for two inter-connected reasons. First, you're distracted and so your business suffers. Second, every VC expects you to keep making forward progress (as if you're just talking to just them and have the rest of the time to work on the business -- yeah right). 


Save up good news for the middle of the process.

We did not do this, but it ended up working out that way anyway and so I saw the value in it first hand. I started raising right after releasing dontbubble.us, which is demarcated by C in our traffic graph. Then in the middle of the process, we got picked as one of TIME's top 50 web sites for 2011 (annotated as D in the graph). It was nice recognition (and traffic growth) at just the right time.

A similar (also unplanned thing) happened when selling my last business. In the middle of that process, we had released a feature that really exploded our user growth. I don't think it was completely random these things happened at the right time, however. I generally try to operate in such a way as to maximize my luck surface area.


People would talk to me because of traction and track record.

I waited three and half years before seeking funding, much longer than most people would. I basically waited until we had significant (not huge, but significant) traction in a large market. As I've said before, traction trumps everything

I'm not suggesting it's great to wait, and I realize most people cannot wait for a variety of reasons. I'm just highlighting that we did and it clearly helped to get VCs talking to us. 


We probably could have raised earlier.

If you checked out our graph, it's nice, but looking back it is unclear to me how much things would have been different if I had tried to raise after reaching 1,000,000 direct searches a month instead of 7,000,000. When talking to people, for most, it didn't really seem that it would have made a huge difference, though there is of course no control.

However, the business difference is those two numbers is large. Granted they're both still very small numbers when considering the search market as whole, but moving orders of magnitude is really de-risking the business a great deal. 

If you try to raise between significant milestones, unless you can show other reasons why you're killing it (the nice graph in our case), you're risking getting hit on valuation (or lack of funding) because your momentum is unclear and/or people can't perceive the real progress your making. On the other hand, after just reaching a milestone people care about, your momentum is large and people generally extrapolate what you're doing in your favor.

The sub-text here is that it has to be a milestone people care about and not just one that you care about, even if you have great reasons to care.


There are a lot of VCs.

Let's get to some numbers. I talked to 31 VC firms, 5 seed funds, and about 14 angels. These numbers do not include firms or people that I never actually connected with on the phone, on Skype video or in person. There were countless more I didn't get to or weren't on my radar though are probably great.

In the end we got funded by a big name, but I met plenty of lesser names I would gladly be funded by. In a second I'd choose someone I like/respect/trust/think is a good fit at a no-name firm vs someone I don't at a big name firm.


Your VC is essentially buying in as a co-founder.

Would you chose a co-founder that sucks? Of course not willingly. It is a major reason startups die.

I treated the VC decision in a similar way. They're going to be with me long-term. They have a significant equity stake and other significant terms. 

But I'm not just thinking negative here. Like a great co-founder, they can also help me in strategic ways at the right time.

I realize a lot of people don't think they have choices, but that's a bit of a fallacy because you are making choices by deciding who to talk to in the first place.


It's good to know people ahead of time.

I only knew a handful of VCs coming into the process, but I wish I had known more. I just never made a concerted effort to meet them.

There is conflicted advice in the blogosphere about whether to take VC "informational interviews" or not. But from my perspective the reality is people take you much more seriously if you are known quantity. You see firms backing the same entrepreneur again and again. And more generally I think people do invest (at least more easily) in lines, not dots


To blast or not to blast.

I also got conflicting advice on how to seek VC intros. You can either a) get intros in concentric circles based on who you think is best (first tier, second, etc.) or can b) try to get introduced all at once to everyone you want. 

The idea behind the former is that you give the best people a sense you're coming to them somewhat exclusively because of fit, which they like and will be more inclined to listen/fund, while the idea behind the latter is you get people moving along faster because you try to make yourself a "hot" deal.


Don't pitch ideal VCs first because you'll mess them up.

Ultimately, I decided to blast, but not after messing up some early pitches with people I really wanted to work with.

Quite frankly, I didn't have my pitch down and I'm not sure you can get it down without giving it to some real VCs and seeing what resonates and what falls flat.


Get your story straight.

Another reason I messed up my early pitches is I was a bit wishy-washy on how much I wanted to raise and what I would do with the funds. Bad idea. People like confidence. Perhaps it is a good idea to have those exploratory meetings ahead of time, but I learned they should not be combined with a pitch because it is just communicating either you're not serious or you're not ready.


It's awkward to pitch people you know, but give them the real pitch.

Yet another reason I messed up my early pitches is I treated people I knew too casually. I should have given them the real pitch. You can still ask them (because you know them) to intro you to others that may be good fits after the fact, which is one of the reasons I wanted to talk to them about it as well.

But by changing the pitch into a friendly conversation, I was again sending a signal that they shouldn't take me seriously. And they didn't.


Use AngelList.

I got significant value out of AngelList. Most importantly, I got high quality inbound requests from people I would have never gotten to otherwise.

I also think we were perceived as somewhat hot by the VCs who hang out on AngelList because our startup was getting a lot of followers/introductions, and so was going out on a lot of the associated emails.


Intro channel breakdown.

Of the connections made, I broke them down into categories of how I got the initial introduction:

  • I knew them and reached out: 4 VCs, 2 seed funds, 4 angels.
  • Suggested intro from someone: 14 VCs, 2 seed funds, 5 angels
  • AngelList inbound: 9 VCs, 1 seed fund, 5 angels
  • AngelList outbound: 2 VCs
  • Cold inbound: 2 VCs

Of the VCs, I broke them down into categories of how far I got with them:

  • Talked with non-partners: 9
  • Talked with a partner: 12
  • Talked with multiple partners: 7
  • Got a terms sheet: 3

Of the VCs I got to the multiple partners (MP) / terms sheet (TS) level, the original categories broke down like:

  • I knew them and reached out: 2 MP, 1TS.
  • Suggested intro from someone: 3MP, 1TS
  • AngelList inbound: 1MP, 1TS
  • AngelList outbound: 0
  • Cold inbound: 1MP


Getting to the right partner initially really really matters. 

Pretty soon after that first meeting, you need a partner to convince their other partners this is a deal the partnership should move forward with. Not one of the partners I talked to handed the deal off to another partner to manage, which says to me you have to figure out a priori which partner at a given firm would be most likely to get excited about your deal, or otherwise it is likely to be dead on arrival.


Often times the right partner is unclear without talking to someone else.

Like I said, when I started raising this round, I only knew a few VCs personally, and I also thought that they may not want to fund or be a right fit for DuckDuckGo. Sure, I had heard of a bunch of VC firms, but when I went to their Web sites and looked at their team pages, I may have heard of one of their 2-10 partners. For all I knew, this guy I've never heard of would be the best fit for us. 

So what I did was try to find the right partners by:

  1. Asking the VCs I knew.
  2. Asking the angels I knew.
  3. Researching related investments.


Having a good network to get intros really matters.

As you can see from the intro categories, #s 1 and 2 yielded most of my intros. If I didn't know these people, my job would have been much, much harder. Not only would it be harder to get to people, but I wouldn't have known (as easily) the right people to target.


A warm intro >> cold intro.

Obvious.


Principal intro is OK if not right partner intro.

Less obvious, and I think there is conflicting advice here too. Some say don't bother if you can't get a partner intro and partner intros always trump non-partner intros.

First of all, my funding stands as a counter-example because I was introed to Christina at USV, and that led to my funding.

But more broadly, if you don't know the right partner or don't have a partner intro, and you get introed to a well-respected principal/associate/analyst and they get excited about what you're doing, they can help a) determine the right partner; and b) rope them in to the next meeting with good framing.


I never walked through my 6 slides.

I have no idea how my demeanor or pitch varied from the countless ones VCs get. All I know is what I did. I had a six slide deck that I would send beforehand. 

Then I would spend my time telling my story, starting well before DuckDuckGo (briefly), and then taking them through what had happened until now, why I was raising money now, and what my future plans were.


I got very few NOs.

Like people say, most people just stop emailing you. They don't say no. I just kept moving forward, so beyond a thank you note after a meeting, I didn't really press people. Ultimately I wanted to see who was excited about me, so I figure the least they could do is follow-up like they said they were going to do.

Some firms did follow-up promptly with nos and specific reasons, and I was really grateful for that. They were:


Having multiple term sheets really matters.

I realize most companies can't get multiple term sheets. But if you can, they can really be used as a forcing function to move people along, and that's what happened in my case. 

It's a double-edged sword though. I'm confident it both moved people to yes more quickly, but also some to no more quickly. They may have always been nos, but sometimes people feel things are moving to fast for them and they couldn't catch up.


Blasting matters to align time-lines.

The process takes time, but once you get the term sheet there is more pressure to move quickly. To get multiple term sheets, you need to get people somewhat aligned in process, and the best way to do that is to start them all off at the same time. 

I did not do that very well for a couple of reasons.
  1. I didn't blast right away.
  2. I tried this in July/August.

VCs really do take vacations in August.

Some of my initial meetings were delayed a month or more because of vacation schedules. Ultimately I have no regrets of course, but it should give you pause to raise in August IF you are trying to align everyone time-wise.


Some partners I really liked.

I met a lot of people I really liked in this process. Here are a few (besides everyone at USV of course).
And I already knew these partners but can't speak more highly of them:

I did not push for a bidding war.

I did have multiple term sheets at the same time, and that did work as a forcing function. But I did not try to actively use it to create some kind of bidding war. Instead, I went in with terms in mind that I thought was fair for everyone, and I tried to stick to those terms.


I focused on what value the VC would bring.

I really did try to get the right VC in the deal, and so I tried to focus on figuring out how individual firms and people could be particularly useful to DuckDuckGo over the life the company.


I did reference checks.

I talked to a bunch of portfolio companies that had various VCs on their board, and asked them all sorts of questions about that relationship and their experiences with the particular partner and firm in general. This information was very interesting. Entrepreneurs really do stick together, and I found people would speak freely -- even if I didn't know them very well (or at all) ahead of time. 


I didn't travel much.

I'm sure I could have gotten more interest and higher terms if I did a huge road show. But I hate travelling and so did most of this entire raise on Skype video and on the phone. I did make two trips to NY and one to DC. And a few VCs came down to Philly to meet me.

I'm really not sure how much that turned people off or not. I'm sure it seemed different :). I got a sense for most West-coast VCs it was a big turn off and that location does matter, even though they do invest in NYC now a decent amount.


Having a mentor really helped.

Someone who has been there, done that a lot is great. I actually talked to a bunch of people and picked their collective brains, but my uncle (again) proved very invaluable throughout the process. I was talking to him constantly. I suggest you get someone like that. If you've previously raised an angel round, it's probably one of those angels.


Congratulations.

I got more congratulations from this funding than any previous DuckDuckGo milestone. I understand why, and thank you to everyone!, but it still feels wrong somehow.


Conclusion.

I realize that's a lot to take in, and I don't expect anyone to make it this far. I wanted to write all this up for people in the process. Feel free to ask me anything in the comments, and I'll try to answer if it is not confidential and won't impact anyone negatively.


Update: some good comments on HN.

DuckDuckGo is growing - welcome USV et al.!

 
usvlogo2.png

DuckDuckGo has been growing in a number of ways this year: traffic, team and now investors (was previously just me). I'm really excited to announce that Union Square Ventures has led our first round of financing (Brad Burnham will join our board and just blogged about us).  Scott Banister, Jim Young, Jeff Miller, Joshua Schachter, Kal Vepuri, Joshua Stylman and Peter Hershberg also invested in the round.

My original plan was to delay raising capital until I saw a compelling reason to do so, perhaps forever. For better or worse, I stuck to that plan, which ended up being about a 3.5yr delay.

So why now? At the end of last year I noticed that the search engine started to really click with a greater percentage of people in a way that it hadn't before. Don't get me wrong -- we still have a long way to go to reach our vision -- but it was obvious that some things were really starting to become compelling, e.g. our Zero-click Infoprivacy policy and goodies

As these things have started to click, the list of obvious ways to fruitfully extend DuckDuckGo has grown rapidly (e.g. more sourcesvertical goodies, etc.). In fact, the list has far outstripped our capacity to build them in reasonable time given current resources.

We've been trying to address this issue from a number of angles. We leverage external APIs as heavily as possible. We've been slowly open sourcing things and recruiting developers to help us integrate data and goodies. (If this piques your interest, start here!) 

And now we've started to build out the team. With a slightly larger team I think we can get a lot more done, and that's one thing I hope this financing round will bring us.

Of course we're also getting (IMHO) the best investors out there, and so I also hope that some of their experience will rub off on us in ways that will help us move the product vision much further, much faster.

In any case, I promise to do at least one post on the fundraising process. It was indeed a huge time sink. If you have any particular questions you'd like me to answer about it, please comment them below and I'll try to either comment back or include them in my post.

How DuckDuckGo got in TIME's 50 Best Websites of 2011

 
2011_TIME_bestwebsites.pngDuckDuckGo was named one of TIME's 50 Best Websites of 2011. I thought other startups might want to know how this awesome inclusion came to be. I'm sure each story is slightly different, but here is one data point for you.

Every article is written by someone, and this one was written by Harry McCracken, TIME columnist and editor of Technologizer (bio). It's sort of obvious but often overlooked that if you want to be covered somewhere, the right person has to know about you. They of course have to want to write about you as well, but step #1 is having them actually know you exist.


First Contact

In this case, the first significant contact I know of occurred via the DuckDuckGo community (although Harry could have known about us way earlier of course). Last July we helped make the community stronger by introducing duck.co, a forum where people could cross-communicate more fluidly. 

A few months after that I started a thread about emailing journalists when appropriate, and specifically mentioned Harry. That same day I also tweeted to him in response to a comment he made about WolframAlpha, which was what prompted me to include the article in the thread.

Occasionally others have posted similar threads (and again at least one other time mentioned Harry). I know that as a direct result of those threads at least two people wrote him about DuckDuckGo, and he said he would check it out way back in last Nov.

Now just to be clear, I'm not paying anyone to write these emails. In fact, I haven't even hired a PR firm or consultant yet (though have explored the possibility). I write such emails myself as well. However, I've found that emails from real users go much farther than from people associated with the company. I figure people perceive them to be just more authentic, which in many cases is true.


Twitter 

In Harry's case, I checked my email and I don't believe I ever wrote him personally, nor have ever met Harry in person (though I'd like to!), or even talked to him over the phone. We have communicated via Twitter, however.

Here's a secret. Twitter is severely under-utilized as a communication channel. Lots of very influential people are on there, and some that don't have big celebrity names or sites behind them actually have very few followers. That means that your interactions with them are more likely to be noticed (assuming they use/check Twitter).

The tech press world is pretty small, and most of them are on Twitter. It's sort of part of their job nowadays... Harry actually has a lot of followers, so he's a bit different, but there are plenty of others that don't. And it's not just journalists -- same goes for investors, M&A folk, executives at big companies, etc. 

It's a great strategy to get noticed and break through real availability bias. For many reasons, people treat Twitter different than email, and depending on the person, may be more likely to engage with you there.

Now I personally don't have time to stalk people on Twitter. I hate wading through 50 billion tweets. That's why I continually prune who I'm following. But a lot of these folks make really interesting people to follow, including Harry (who I follow). They see a lot of stuff and so generally tweet out cool things.

Yes, I know there are other monitoring options -- twitter searches, software etc. -- feel free to mention in the comments. I'm just telling you how I do it. I look at what my followers say, and respond when I can or is appropriate. I also have one running search for mentions of DuckDuckGo (pro tip).


Bigger Stories Take Time

Harry first mentioned us on Techland back in Feb at the end of an article about the Google "Farmer" update. In March, we had a little blurb printed in the print edition.

harry4.png

Then in June he mentioned us in a TIME business article, also about Google. That same day he also wrote an awesome feature in Techland as well, comparing us to In-N-Out Burger.

I responded over Twitter (saying thx), and got replies back as well. Most recently, those articles were actually summarized in the latest issue of Reader's Digest, which I mentioned and noticed this tweet (not directed at me specifically, but related). Anyway, I'm just saying there was ongoing communication, albeit lite.

In other words, it was over six months from what confirmed first contact to major article, which as noted above could have been longer in reality. I presume over that period Harry actually used and liked our product, and that compelled him to write the feature.

Of course there are other ways to try to jump it more quickly, e.g. you can try to push a hot story with a particular angle, and I've done that as well. However, that also works a lot easier if you've had contact before, preferably over a decently long period of time.

Additionally, I'd rather be mentioned every now and then in many articles than have just one feature and never get mentioned again. This was the best of both worlds, but I would take that trade-off.


TIME's Top 50

All you can hope for is a feature, and honestly I thought that was going to be the biggest story, at least for a long time. Then one day I saw this tweet.

harry.png 

You can see that I re-tweeted it (as @duckduckgo). As a result of that RT I know a bunch of people "nominated" DuckDuckGo.

harry2.png

I'm not sure if these nominations had any effect or not, but they couldn't hurt! After that, I didn't think about it again until yesterday.

harry3.png

Not a Unique Snowflake

I must say this is not a unique story for DuckDuckGo. We've been covered many times in high profile places and it usually is a result of both community action and a longer communication history with the journalist. 

Two quick cases in point. First, our feature in Wired from this Jan was over two years in the making! Ryan Singel has mentioned us in passing a lot (thank you Ryan!), and it actually first started back in April, 2009. We have kept in touch and when the time was right for a feature, he did one.

best-search-engine.png
Second, earlier this year we won the About.com Best Search Engine award for 2011. This was a direct result of the DuckDuckGo community, as it was a voting thing.

We spread it of course on duck.co and Facebook and Twitter. Google actually mentioned it on their Facebook page (and for all I know other places), which made this really tough from our perspective. 

Actually going into the final night we behind by a decent amount and I didn't think we'd get it. But the community pulled itself together and pulled it off. Amazing.



Takeaways

I take a few things away from this experience.

  1. Things take time. We've been live now for about 3yr. They don't need to take that long of course, but don't expect results over night.

  2. People and relationships matter. You have get the right people to know about you. Twitter is great, but there are lots of other ways do to that as well.

  3. Product matters. People need stories and angles but if they like your product they fundamentally want to find a way to write about it.

My DuckDuckGo Twitter ad by the numbers

 
This is the story of how I made a micro-site in an hour to try to take advantage of a Twitter trend that was related to DuckDuckGo, my startup. Well, related by name...

The Backstory

Right now #replaceawordinafamousquotewithduck is the number two organic Twitter trend worldwide. I did not start it, and am not sure how or when it originated.

trends.jpg

I first came across it eight hours ago when a tweet by @jordankanarek came across my stream. I didn't pay close enough attention to it then and didn't even realize it contained a hashtag -- probably because I didn't recognize the quote and the hashtag was so long.

jordantweet.jpg

Then a few hours later, Jordan actually sent me an email with a link to the hashtag search page. New tweets were loading like wildfire (and still are).

That got my attention! If you know me, you know I love startup micro opportunities. So I immediately went to namecheap and bought the domain replaceawordinafamousquotewithduck.com. Then I fired back this email.

yeggenail2.gif


Micro-site

Right after I sent the email I knew that aggregation was the way to go. I previously made some micro-sites to do similar things, e.g. poeet.com (aggregates #haiku).

I quickly grabbed that code and adapted it for the aggregation of these quotes. It needed a bit of changing because the purpose is slightly different. For example, I needed to do more normalization since we were going to rank them. If you're interested, I just put all the code up on github.

After I printed out some basic HTML I remembered that previously my sister (and others) had made some duck movie posters (for something else that never fully launched).  I figured they would be perfect for the micro-site, so I got those and put them up in the left column. 

Here is the final product.

duckpage.jpg


Domain purchase to live site: 1.25hr. Now what?


Will it blend viral?

We have this trending topic and we have this micro-site, but how to blend the two?

Well, of course the first I thing I did was yell from the rooftops as loud as I could. We tweeted it. Posted it to duck.co and IRC. On Facebook. And sent it to the DuckDuckGo spread team. Oh, and I even tried Google+ for good measure, but it didn't take :)

googleplus.jpg

Those did generate some traffic, but nothing like donttrack.us or dontbubble.us, which were picked up by social news sites pretty quickly and then by blogs and more mainstream media.

retweet.jpg


The Twitter Ad

Then I remembered we were still in the beta program for Twitter ads, and this could be the perfect use for them. So I quickly fired up the ad interface.

Twitter has a number of ad products, but the relevant one here is Promoted Tweets. You select one or more of your tweets and then you can promote them for chosen keywords (including hashtags).
You can either do cost per engagement (CPE) or cost per thousand impressions (CPM). The engagement in CPE is clicks, favorites, retweets and replies, though in reality it is clicks since that is what happens the vast majority of the time. 

The minimum for CPE is $0.10 and the minimum for CPM is $5.00. I figured I wouldn't have any competition, but I didn't know how they judge whether to show anything at all or if they fill in for whatever inventory they have, so I made two campaigns.

First I made a CPE campaign at $0.50 (the suggested bid) and a CPM campaign at $5 (the minimum). Previously my random engagement rates had been about 1%, so that would have put them at parity.
One thing that is cool about the Twitter ad platform is it pretty much immediately kicks into gear. A few minutes later I refreshed the analytics page and I had already spent $120 :). I clicked on the detail page and my engagement rate was ~5%!
Panicking (since the CPE was then 5x more costly than the CPM), I couldn't quickly figure out how to change the bid price (even though it is super-easy), and so I deleted the CPE campaign. I then let the CPM campaign take over, but after refreshing the hashtag search page in various browsers, I wasn't convinced they were showing it all the time. So I re-made the CPE campaign, but this time at $0.15, as the engagement rate had dropped to an equivalent $0.13 (saturation?), and I figured it would fall more and that was about parity.

To my suprise engagement actually climbed, and so after a while longer I lowered it to $0.10 (the minimum) so now both were at the minimum (and parity).
 
For a while thereafter, engagement stayed for both campaigns around 5.5%. Given that they are both promoting the exact same tweet, you would expect them to remain the same, which is why this end end result makes no sense to me.

Why would the CPE campaign be a full % point higher, i.e. 6.13% vs 4.99%? It remains a mystery to me. You might think well, they occurred at different times, but here is the graph of when they were shown.
They generally follow the same pattern, but I suppose they are somewhat different and maybe that end piece really made the difference. Either way, this is amazing engagement: before I was normally getting 1% with 2% max and now I'm at 5-6%! But did it lead to any benefit to DuckDuckGo?


Results

All told (so far), as far as I can tell, it yielded about 302 searches on the site and 60 clicks to the search engine homepage. (We don't track users, but there are referrers in our logs and so I can grep for them; hence, these numbers.)

Hmm...that's not a lot. For reference, we're currently seeing about 250,000 a day.

On the micro-site side, I count 3,605 loads of the bottom (Spiderman 3) image and 3,777 loads of the top (Wall-E) image. That is less clicks than Twitter is reporting in their stats, and doesn't include the ones shared around by me. I don't understand the mismatch there.
So ultimately I spent around $630 on this. That would be (so far) approximately 17c per click to the micro-site and $1.74 to the search engine. 

From that standpoint, probably not worth it. If you include the value of the learning experience, this blog post and the thrill of it all, probably worth it.

Yet the trend is still going and I haven't re-upped the campaigns. Go figure. 


Take-a-ways

First, it seems hard to grab on to a twitter trend, even if you get pretty close to its message. People are using the hashtag, sure, but they're not inserting your micro-site URL (even it if it just missing a .com). That's just not the nature of the trend, and so you seem to be relying on either a) the site itself going viral or b) people clicking to the hashtag search and then clicking on a url in the stream.

Second. engagement on Twitter can get pretty high (for ads). They told me so originally, but I hadn't seen it "in the wild" until now. So that's cool. Theoretically, I'm sure someone could turn those types of engagement numbers into something very useful, e.g. if you were selling a product that yields more than a few dollars in profit.

Third. there are some open questions as to the Twitter Analytic #s, namely why did the CPM and CPE engagement rates end up different and why the large discrepancy between the Twitter click count and my click count)? 


Update: some good comments on HN.

Update2: Ricky from Crowdbooster notified me about this click policy for promoted tweets that probably explains the discrepancy.

Promoted Tweets: When a user clicks on the Promoted Tweet to open it in the details pane, or clicks on a link or hashtag within the Tweet copy.

That is, I had a hashtag within the tweet, and so that probably accounted for half the clicks. Now why someone would click on the hashtag when they're already on a hashtag search, I'm not sure -- maybe to refresh the page?

The above is probably the first order effect. Other new theories that are probably second order (but real). 1) Intermediate proxies could be caching the images. 2) Duplicate clicks from the same user.

Update3: Removed the images of the Twitter ad manager at their request, since the beta program was confidential. Looks worse without the pictures :(

The real Filter Bubble debate

 
Google Search Filter Bubble - Duck Duck Go.png

If you're not familiar with the concept of the The Filter Bubble, check out this infographic we put out a few days ago about how it applies to search engines.

Currently, much of the debate seems to be whether segregating results based on personal information is good or not. I think that is the wrong debate and a false dichotomy. It can be good, e.g. isolating movie listings based on your zip code. And it can be bad, e.g. limiting the display of certain political viewpoints based on your search and click history. 

From my perspective, the real debate is over a) which personal signals should be used; b) what controls should we (as users) have over how our personal signals are used; and c) how results that arise from the use of our personal signals should be presented. Personal signals are fundamentally different than other signals because as soon as you introduce them different people start getting different results. 

The central point of the Filter Bubble argument is that showing different people different results has consequences. By definition, you are segregating, grouping and then promoting results based on personal information, which necessitates less diversity in the result set since other results have to get demoted in the process. Of course you can introduce counter-measures to increase diversity, but that is just mitigating the degree to which it is happening. Consequences that follow from less diversity are things like increasing partisanship and decreasing exposure to alternative viewpoints. 

My view is that when it comes to search engines in particular, the use of personal information should be as explicit and transparent as possible, with active user involvement in creating their profiles and fine-grained control over how they are used.  Personalization is not a black and white feature. It doesn't have to be on or off. It isn't even one-dimensional.  At a minimum users should know which factors are being used and at best they should be able to choose which factors are being used, to what degree and in what contexts.

If you do not do that, and instead rely on implicit inference from passive data collection (searches, clicks, etc.), then the search engine is just left to "guess" at your personal profile. And that's why the examples from The Filter Bubble seem creepy to a lot of people. It seems like the search engine algorithm has inferred political affiliation, job, etc. without being explicitly told by the user.

This is not a conspiracy to segregate people and I'm the farthest from a conspiracy theorist you'll probably find. It's just a natural consequence of algorithms that cluster people. 

The questions then become 1) are they clustering me correctly; and 2) even if they are, do I want the fact that I belong to this cluster to influence my results for this particular search or type of search?

Some people may want restaurant recommendations based on the implicit guess of their race and income class. Whether you care about that sort of thing largely determines where you come down on the debate. If you don't care at all, then you probably don't care if you ever know that your results are different from other people and how they differ based on your personal information. 

On the other hand, if you do care, then you might want to know how and why a result based on your personal information got in front of you. You might also want to have much more fine-grained control over how particular personal signals are used (akin to privacy settings).

In other words, some people prefer to self-segregate and are interested in any and all forms of "personalization." And some people would prefer that segregation should not occur without explicit user choice.

Please note I'm not disputing that showing people different results may result in "better results" for people. I agree that there is no universal best result for all queries. 

What I'm saying is that you can get to that better result for a particular person in a number of ways. And I think that when it comes to search engines in particular, personal signals should be dealt with delicately and with active engagement from the user. It's two paths to the same thing, but the latter involves vastly more user choice and control. 

DuckDuckGo and Wolfram|Alpha are now official partners

 
WolframAlpha computational knowledge engine.png

DuckDuckGo has been using Wolfram|Alpha (via their API) for about two years now. I actually was an alpha tester and authored the Perl API binding. I've also open sourced the regular expressions I use to determine if a query is relevant to send Wolfram|Alpha. Needless to say, I'm a big fan.

So I am very pleased to jointly announce that we are now official partners. As a result, you should see an even tighter integration over time. And I will have a more official responsibility of maintaining the Perl API, which is in need of a V2 update (feel free to submit bug reports here via CPAN).

Most of the recent integration improvements have come via user suggestion, e.g. I run this or that query regularly on Wolfram|Alpha so can you make that work on DuckDuckGo? I'm in the process of going back through the Wolfram|Alpha examples as well as asking them directly for ideas, but please keep them coming as well!

Are you chasing a fad or a market?

 
slap bracelets.jpg

Startups usually take many years to exit. So it is important to figure out up front whether what you're working on is a fad or a real market. You don't want everything to fall apart two years in because a fad passes.

Even if you plan to sell at the earliest opportunity, you won't be able to unless a buyer is convinced it is in their long-term strategy to buy you. And so then you're back to pursuing a real market.

A simple strategy is to choose something already big because there is often more room for success within a big market. That is, if you're a scrappy founder you will probably be able to figure out *something* that works within this huge market. That's what we did in my last company. Classmates.com was making a lot of money so we found a niche in their market and ultimately got bought by them.

A bit more complicated is to work within a big market and concentrate on where you think it is headed. That's what I'm doing now with DuckDuckGo.

The hardest strategy (though often the most lucrative) is picking a smallish market now that is imminently exploding, such that you can standout in an area everyone is about to be talking about all the time. Some proven examples here are Netscape, Groupon, and Zynga; some potential examples are Hunch and foursquare.

Another way to look at this whole issue is what long term market premise(s) is your startup betting on? For DuckDuckGo, there are a bunch:
  • Structured content often packaged through great APIs are on the rise and can be integrated intelligently to produce great search results.
  • Privacy will become an increasing issue and some non-negligible percentage of people will value their privacy being protected by their search engine (including myself!).
  • Universal search, ads and complementary products have cluttered results too much and some non-negligible percentage of people would prefer something less cluttered.
  • Search result pages don't really make that much inherent sense, and there is an opportunity to make them more readable and understandable.
  • SEO spam is on the increase and have made searching for stuff on the Internet more annoying over time.
  • The Web is increasingly becoming a market for attention, and so becoming the start page is a great position to be.
I'm 3 years in now, and these have generally been good bets so far. All but the privacy one I had from the beginning. I picked it up from users, and started thinking about it really seriously at the beginning of last year. Now it is also a personal cause, but I definitely think that there is a long-arc of privacy consciousness that is on the rise.

What is your startup betting on? Are you chasing a fad or a market? Is it a dying market or a growing one or an exploding one?

DuckDuckGo Twitter homepages

 

kanye.jpg

DuckDuckGo now has Twitter homepages, which allow you to use any twitter user or list to enhance your search homepage.

We automatically bring in the Twitter background of the featured user. The picture on the right is the homepage using Kanye West's twitter

It's a fun way to get some personalization while staying up to date with the headlines, your favorite celebrity or league.

Twitter lists work as well, so you can use someone else's curated list or your own list (as long as its public).

We're currently using the Publitweet Twitter widget, which is enhanced by embed.ly to provide in-line snippets when links are present. There is also a quick view link that opens the link in an in-line frame.

You can start from our Twitterfy page or look for Twitter in the settings (under Interface). You can also reach any page by hand just add the user or list to the end of http://duckduckgo.com/tw/, e.g. http://duckduckgo.com/tw/theonion or http://duckduckgo.com/tw/nytimes/staff (for a list). I currently have mine set to @jeffmiller's @newsyc150 feed.

I want to thank Jordan Kanarek for helping me put this new feature together and Robert Scoble for breaking the launch (check out his list). DuckDuckGo is transitioning from more of a one-man-show to a solidly group effort. While I am still the only full-time person, a number of people (including Jordan) have made significant contributions. A lot of this effort comes together on our forum (duck.co) and our IRC channel (#duckduckgo).

Working around Android's screen.width bug

 
This android bug has been causing me trouble lately. When fetching a cached version of a page, Android sometimes sets the screen width to the previous page instead of the page you are on. This has the annoying side effect of completely ignoring your media -max-device-width CSS.

So when people first go to DuckDuckGo on their Android phone, it formats fine via the media CSS block targeted at mobile devices. Then they click on a link and click back, and all of a sudden it is no longer mobile formatted. Or they bookmark the homepage and then open it again, and the homepage isn't mobile formatted. 

I made a small demo to see the bug in action. Just visit it on your Android phone. It prints the screen.width variable out at the top--should be under 700. Then click the wiki link and wait for wikipedia to fully load. Then click back to the page (might have to do it twice to go through their mobile redirect). Now the screen.width says something different. Ugh.

I've had been trying to find a workaround off and on for a while to no avail until @amonti came up with something that works. You can (currently) just target later versions of the Android browser via CSS through a -webkit-min-device-pixel-ratio:1.5 media block, e.g.

@media only screen and (-webkit-min-device-pixel-ratio:1.5) {

I say currently because presumably other browsers may implement this feature. But in the meantime (while we wait for an Android fix) this workaround actually seems to work.

DuckDuckGo chat bot: im@ddg.gg

 

DuckDuckGo now has a chat bot at im@ddg.gg (or im@duckduckgo.com) that will respond to your instant messages with Zero-click Info, search results, and real time topic summaries.


The following is a guest post by Dhruv Matani (@dhruvbird), who is a software developer at Directi and who made the new DuckDuckGo chat bot.


What is a chat bot?

A chat bot is a like a friend on your roster (contact list) who is actually a computer and can respond to any messages you send it (or send messages of its own accord). Examples of chat bots include bots that:

  1. Periodically alert users of the score during a game of cricket (todays-special@appspot.com).
  2. Message the user whenever a commit is made (github).
  3. Post any message you send it to your twitter and facebook accounts (List of twitter bots).
  4. Try to answer your question by asking other users (aardvark).
  5. Can get the latest news (only pt_BR available).
  6. Perform web searches, bookmark pages, do calculations, etc... (clisearch).
  7. Show you the current weather (defunct).

You can find other XMPP bots on these pages.


What does this chat bot do?

im@ddg.gg is one such chat bot who can do DuckDuckGo searches for you on your chat account. Just send this bot your query and she will do a search and send the results back to you - all while you are using your favourite chat client. You will get not just the web results, but also the 0-click results and definitive answers to your questions (if any). This can be useful when you quickly want to lookup the meaning of a word, find some quick fact about a monument you don't know of, compute the md5 of a string, and so on...

The bot can also give you real time topic summaries for many broad keywords. These summaries are the result of going out to the top links in real time and looking for relevant paragraphs that match your particular search term.

Though the idea has been tried before, the proliferation of XMPP has made bots a lot more accessible and chat accounts more plentiful. That apart, the large set of data sources that DuckDuckGo looks at to provide answers provides a single window into a lot of searchable data.


Why did I even venture into it?

The reason I wrote this chat bot was because a few of my friends have Blackberry accounts with only mail and chat (no internet) enabled and they wanted to be able to just search the web. They said that it would be enough for them to see just the information that was returned in the result snippet as opposed to the result page. Being on a mobile also meant that they couldn't effectively scan complete web pages and a minification of information would definitely be better. As it turns out, a lot more people with such requirements exist. In fact, one of my friends mentioned that he used this bot to look up the definition of a word while he was watching a movie!

You can try it now by adding im@ddg.gg to your roster on any jabber network (jabber.org, pandion.im, livejournal.com, gmail.com, etc...)


Technical stuff

We used ejabberd to serve the ddg.gg domain over jabber. The im@ddg.gg bot itself is a standard jabber client written in javascript and runs on node.js.

Ejabberd was chosen because it is one of the best jabber servers out there, is highly configurable and is fairly easy to set up.

Node.js was chosen as the platform for the bot (as opposed to choosing Javascript as the language). The inclusion of Javascript was just a side effect of the former decision. I would like to mention here that even though I am a python fanboy, I chose node.js, not because of the language, but because it was just so well suited for the task at hand.

Node.js is ideal for such routing/switching based interactions where it takes data from one end, pumps it to the other and does the reverse. The whole idea of everything being asynchronous (across protocols - XMPP, HTTP, SMTP and others) appealed a lot to me and I literally had a working (no frills and did the job right) jabber bot (fully performant) in under 100 lines of code on node.js. I would like to thank astro for being helpful when I ran into issues with node-xmpp.

We had an option of either making the bot a jabber client or a jabber component and we went with the former only because of its simplicity. In case of a client, you are assured that you are one user whereas in case of a component, you need to be aware of which user you are pretending to be at that point in time. The obvious drawback with a client is that most external jabber servers place a limit on how much data the client can send/receive if the client directly connects to their network. I would think that such restrictions don't exist (or are very generous) for components - if the jabber server does in fact allow components to connect to it. However, since the bot connects to our own jabber server, we can always be generous with shaping traffic to it. Besides, for this use case, the roster size doesn't matter - since that is the most common argument I have read against a bot as a jabber client.


Server federation

We had a tough time figuring out the exact rules as far as google's XMPP server was concerned. Google runs a service called Google Apps which you can set up on your domain on this service and once you do that, you can chat with users on any other apps or gmail account. The catch here is that XMPP requires you to add an SRV record to every domain if you want the jabber servers serving those domain to talk to each other. However, since they (gmail and google hosted domains) are all served by the same jabber server (talk.google.com), it is not necessary for users on these domains to chat with each other. What happens as a result of this is that anyone who is served by talk.google.com can NOT chat with people on other servers (if they haven't set their XMPP SRV server record to talk.google.com - it is required for server federation).

Hence, we initially decided to run 2 bots, one on ddg.gg (ejabberd) and one on (talk.google.com), both serving the address (im@ddg.gg). However, even gmail accounts started talking to the bot hosted on talk.google.com (gmail seem to have some internal condition that detects that the target domain is apps hosted and they bypass SRV lookup - even if an XMPP SRV server record for that domain exists). This is undesirable since google has a limit on the amount of traffic that a single user can generate and the bot would most certainly cross that limit very easily (since it is a jabber client and not a jabber component). I'm not sure if talk.google.com allows jabber components (If anyone knows anything about whether this is possible, please do let me know). Hence, we got rid of the apps hosted bot and now there is just one bot serving im@ddg.gg (on the domain ddg.gg that is served by the ejabberd instance).

As of today, an apps account can talk to the bot ONLY if it has the XMPP SRV server record set to talk.google.com (for server federation).

Additionally, talk.google.com seems to disregard vCards that the bot sends to the server. The gmail browser client too doesn't show the image that the bot sets in its vCard.

Other jabber servers didn't give us too much grief since there wasn't anything non-standard about them (that we encountered).


Yegg was very helpful in this whole exercise so a big thanks to him!! :-). If you have any ideas for improvement, please include them in the comments below.

References: http://www.quora.com/What-is-the-best-way-to-build-a-fast-scalable-instant-messaging-IM-bot/

DuckDuckGo FOSS Donations 2010

 
A few months ago DuckDuckGo pledged to give 10% of gross revenues in 2010 and 2011 to free and open source software (FOSS). I made the 2010 donations today.

We had a discussion on duck.co about where the donations should go. For the 50% company portion, we donated to $482 to nginx and $475 to FreeBSD. These are both projects that are an integral part of our architecture.

For the 50% community-driven portion, we decided it would be a good idea to pick a theme based on all the various suggested projects. That theme for 2010 was security and privacy. To that end, we donated $238 to Tor, $238 to Clamwin, $238 to Taho-LAFS and $238 to OpenSSH (via OpenBSD).

Unfortunately, I haven't gotten many takers to turn this FOSS donation pledge idea into more of a movement. Actually just one :), @boyter at search[co.de]. But that's OK -- perhaps more will be inspired to do something similar in due time.

Working smarter

 
Usually I go to sleep not knowing what feature or bug I'm going to work on the next day. I have a huge list of major and minor projects all vying for attention. 

So how do I decide what do work on? I agree that a startup should focus on working smarter (as opposed to just harder). I'm the first to admit though that I don't have a particularly scientific or even spelled out process on how to do so, but the following are the principles I generally try to apply.

  1. First, I like to tie everything back to distribution, i.e. people using your product. You should be tracking quantifiable long-term metrics, e.g. number of signups, uploads, shares, whatever. For my search engine startup, my main metric is number of direct searches/time.

    I believe tying back to distribution entails rephrasing the question what feature or bug should I work on next to if I work on X, will it result in significantly greater distribution, and if so, how much? Note that greater distribution can come from either existing users using the product more or new product use from new users.

    Once answered, you can then roughly focus on areas you think will have the highest marginal benefit with regards to distribution. For example, something directly related to distribution, e.g. a marketing effort or social/sharing feature, probably has a high marginal benefit.

    You can't forget word of mouth distribution though. Most of my distribution to date has been via word of mouth, and I believe that is due to product. It is a great place to be for a product-oriented person like myself when you can work on product and it also helps distribution. But you still can't get sucked into the false belief that product features trump direct distribution features all the time, because they don't.

    Similarly, how many of your users (including potential users) will the feature effect? Something that appears on a site all the time generally should take precedence over something that only occurs in a small corner of it. Of course that is mitigated by severity and impact, but you get the idea.


  2. After distribution, the second principle I apply is using real feedback to substantiate my decisions. Such feedback could be actual data, or, more often in my case, user interaction. Using these data sources you can really get more of a sense about the marginal benefit beyond your initial guessing positions. 

    You usually do need some kind of user base to get these kind of signals, but often not as big as you might think. I've tested various feedback button sizes, shapes and positions and found that moving them around has a dramatic effect on the amount of feedback received. Same goes to links to forums, chat rooms, etc. 

    I encourage all startups to essentially maximize these input streams since the data is so valuable. Using feedback appropriately greatly improves the chances you are actually working smarter (as opposed to just thinking you are working smarter).


  3. Third, I like to put a lot of minimum viable products (MVPs) out there. MVPs are not just for the initial version of your product. Maybe they should be call minimum viable features (MVFs).

    In other words, I like to ship code. It's not always the prettiest code, but it allows me to move on to another feature/bug and let the first one simmer.

    By letting it simmer you allow yourself and your users to experience it in some form, and suggest incremental improvements that you often didn't think of at design time. I absolutely love this aspect of shipping code. Again, the bigger the user base, the bigger the effect.

    The other, perhaps bigger reason, to do a lot of MVFs is it isn't readily apparent a priori what is going to work and not work with regards to distribution. It's sort of like A/B testing in that it can be non-intuitive. By planting a lot of different seeds, you are spreading your risk a bit hoping that some of them will blossom, or more often than not, prompt you to think of new related or combined efforts that eventually turn into something meaningful.


  4. Fourth, I batch things. There are so many silos of code and each takes some time to really get into and be effective. So I try to wait until there is a decent amount of stuff to do in that area before jumping in, which maintains efficiency. 


  5. Fifth, I inject randomness or happiness or whatever you want to call it. Since none of this is an exact science, I generally work on things that strike me in the moment as interesting, given the above constraints. That's not always possible but it contributes to my not getting burnt out.

Despite the above, my process breaks down from time to time around the edges. For example, I like to be as open and responsive as possible. That means though that I'm often engaging in twitter/email/forum/etc. conversations about minute aspects of my startup. Is that working smarter? Probably not, but it feels right.

Usability issues with adding search engines to Web browsers

 

I get a lot of feedback around adding DuckDuckGo (a search engine) to users' Web browsers. I thought I would synthesize that feedback in hope that these usability issues might be addressed.

ieaddons.pngIf you have a search engine for your site (and most big sites do), you want people to be able to easily use it in their Web browsers. In Internet Explorer (IE) 7, Microsoft introduced a native JavaScript function called AddSearchProvider that lets users add a search engine to their search bar, and optionally set it as their default engine. The dialog box looks like this.


addtosearchprovider1.png

That means that you can put a link on your site that users can click on, which will pop the above dialog. And the dialog itself is pretty simple and straightforward. If I were to really nitpick, I'd say just delete the search suggestions line if not available (instead of graying it out). I also like how Firefox's equivalent says "Start using it right away" instead of "Make this my default search provider" because it is more inline with what the user is thinking.

Microsoft also created and still provides another function called IsSearchProviderInstalled that allows you to dynamically check whether your search engine was already added. You can then use this knowledge to hide the add link on your site (if the user already added it).

In conjunction, these two functions produce pretty great usability when adding a search engine to IE. Really my only complaint is that IE doesn't let you edit the search URL string after the fact, which is something more advanced users want so they can do things like force language preferences. To be perfect, I'd add that functionality along with the ability to submit POST requests.

Nevertheless, IE has by far the least usability issues with this process. Unfortunately, the functions Microsoft created don't work that great outside of IE. 


DuckDuckGoBar.png
On the other extreme is Safari, which is the absolute worst. Not only do they not implement the mentioned functions or equivalents, but you can't even add a search provider at all in the browser preferences! 

There is only one relevant dialog box, which currently gives you only three choices: Google, Yahoo! and Bing. Apparently these search engine choices are hard-wired into the browser itself, and the only way to change them without installing additional software is to hack the binary.

addtosearchprovider2.png

Additional software solutions aren't that great either. I was looking forward to the recent inclusions of extensions in Safari 5, but they didn't open up the search bar in the extensions API! Victor Quinn made a DuckDuckGo extension (it's open source btw), but it has to add a toolbar to provide search functionality.

addtosearchprovider3.png

Many people understandably hate the bar taking up all this extra space. The other current alternative is to install Glims, which is a plugin that does a lot of things including tweaking the search box. It is a bit of overkill though if you just want to make that one change. Perhaps the best solution right now is to make one plugin dedicated to this purpose like Inquisitor, but that is a lot of custom work to solve this one issue for Safari users.


edacflbhpmcimdanpfcibgafeknkgpia.png
Chrome causes me the second most complaints by far. It's also the most frustrating to me because it is so close to being good, but yet still so far. You'll see what I mean.

There is a specification called OpenSearch that allows you to add some meta tags to your site that describe the search engine associated with it. For DuckDuckGo, those meta tags like this:

<link title="DuckDuckGo" type="application/opensearchdescription+xml" rel="search" href="/opensearch.xml">link title="DuckDuckGo (SSL)" type="application/opensearchdescription+xml" rel="search" href="/opensearch_ssl.xml">

These have the effect on IE and Firefox of letting the user know via the search box that they have options to potentially add. Here's what that looks like:

addtosearchprovider4.png
addtosearchprovider5.png

On Chrome, however, there is no equivalent drop down because the address bar and the search bar are one and the same--a really cool feature that I truly like btw. What they've chosen to do with opensearch is add any search engine that comes up in meta tags automatically to a list of possible providers you could use as the default. The first issue with this process (easily fixable) is that they add new engines to the bottom of the list.

addtosearchprovider6.png

The problem is that when people go looking to add a search engine they immediately have trouble finding the one they're trying to add. In particular, they generally expect the one they're looking for to be at the top, especially since it usually corresponds with being on that site at that moment. Trouble is that the location of the engine is in the exact opposite place you'd expect, i.e. at the bottom. 

And since Chrome adds every site that has an opensearch plugin, and lots of sites have them, it can be a very long list. Furthermore, if you decide to change providers but originally visited the site you want a while ago (often the case), it's probably somewhere in the middle of the list. I've gotten many reports that people just couldn't find it when it turned out to be "hidden" in the middle.

Second, Chrome provides a dialog box (see below) to add a new search engine from a link via the AddSearchProvider function, like I described for IE above. Trouble is, if it is already in that mega-list AddSearchProvider silently fails and refuses to pop the dialog. Given that you usually use opensearch meta tags to surface the above pictured functionality for IE & Firefox, this dialog box basically never can show up, rendering it completely useless.

addtosearchprovider7.png

Nevertheless, it suffers from another problem, namely that there is no option (unlike IE/Firefox/Opera) to also make it the default search engine. So if you somehow manage to get the dialog, you're left wondering what happened and why isn't it working in the search bar? What happened is that it got added to the bottom of that long list, and to make it the default you'll have to open that list in your preferences, scroll to the bottom, and click 'Make Default'. 

As if that wasn't enough annoyance, IsSearchProviderInstalled is also essentially rendered useless. Since again everyone generally uses opensearch, and Chrome uses it to add providers automatically, search providers are always "installed" and so this function pretty much always returns true. This behavior means that you can't easily tell if a user added your engine or not and thus you are forced to keep showing 'Add to Chrome' links everywhere, even if the user already made it their default engine in Chrome. (Note sites often use cookies to save this info, which will work at least until the cookies are cleared.)


addons.opera.png
Opera causes problems too. My main issue with Opera is that they don't implement AddSearchProvider or IsSearchProviderInstalled or use opensearch at all, which means the user is forced to do everything manually.

They do have a path to do it, both in the preferences and via a right-click shortcut, but both cause a lot of confusion. The easiest way to find the preferences is by clicking the down arrow in the search bar and then 'Mange search engines...'. Simple enough, though annoying they don't just populate choices automatically from the meta tags, like IE and Firefox (pictured above).

addtosearchprovider8.png 

There is then an 'Add...' button that gives you this dialog box.

addtosearchprovider9.png

The trouble with this dialog is all the empty boxes. An average user has no idea what to type in any of these boxes, and no example text is given whatsoever! So instead, the path that I now sent people is on is to right click the search box and select 'Create Search...' like this:

addtosearchprovider10.png

This has the effect of at least partially populating the dialog box like this:

addtosearchprovider11.png

This still has a bunch of problems. First of all, you just can't click OK -- it is literally grayed out. At least let the user click OK and then tell them what they're doing wrong! It turns out you have to fill in a 'Keyword' to proceed, e.g. 'd'. I do give them extra points for adding a POST option though, which no one else does.


addons.mozilla.png
Which brings me to Firefox...  They're second behind IE in doing it right, but they still have two issues that cause significant complaints. They do get the basic process right though, which is great.

You can use AddSearchProvider to pop this dialog box and easily set an engine as the current search provider.

addtosearchprovider12.png 

This is arguably the best dialog box in the whole lot. It's simple, straightforward, and it works!

The issues in Firefox are more around the edges. First, the IsSearchProviderInstalled function is not implemented, which means you cannot easily tell if you've already installed an engine or not, which means you can't stop showing the 'Add to Firefox' links. Sound familiar?

The second issue annoys advanced users. Just like in IE, you cannot actually edit the search string for the search engine once added (or while being added). But what if you want to use URL params to customize the engine? Sorry, the only way to do it is to create a new opensearch plugin, which is why you start getting all of these.

***

In short, no one does this process perfectly. I compiled feedback on these issue in hope that this part of the browser user experience gets fixed in all these great pieces of software. The ideal process is pretty simple actually:

  • Make AddSearchProvider and IsSearchProviderInstalled functions work as one expects them to, i.e. not to fail silently, always return false/true etc. If you do have a concept of a default engine, let IsSearchProviderInstalled see that too, or add another function to query that boolean value.

  • Make the dialog box that results from AddSearchProvider allow you to a) make it the default/current search engine; and b) change the url string via an advanced section (that offers useful help text).

  • Use the well-established opensearch meta tags appropriately, i.e. to suggest engines to add (as opposed to ignoring them or adding them automatically).

  • Make it obvious how to change providers and edit them after they have already been added. Executing AddSearchProvider could pop an edit dialog, for example.