Live Filter: Re-inventing Search
:: Development
September 5, 2006 - over 3 years ago
Platform: We like Rails
Tested Browsers: Firefox 1.5, IE 6.0, Safari 2.0.4
Tag As: Filtered search, Ajax, Rails, Patterns
Editor: Liz Clayton
Illustration: Anthony Watts
Source Code: here - version 1.02
Demo Page: Live Filter
In 1994, WebCrawler launched and became the first search engine most of us had ever used. Yahoo arrived on the scene shortly thereafter, but was initially focused on maintaining the definitive hierarchical index of sites on the web. Soon, all of the early players (Lycos, AltaVista, and eventually HotBot) were waging a war over who could index the most pages. The push for quantity completely overwhelmed any notion of quality, either in terms of search relevance or the user experience.
One side-effect of low search-result quality was the sheer volume of results a typical search was expected to yield. Researching common terms would inform the user that they had "found" millions of potential results. The lucky user could then search through this vast information stockpile... 10 records at a time.
Sadly, not very much has changed in the decade since.
Live Search
Search engines are technically complicated beasts which frequently support an array of non-standard search term options to control parameters such as the exclusion of specific words, or searching pages within a domain. Mastering these special tricks for each engine was beyond the mnemonic capacity of most users, and so one of the first search interface developments was to create a search options page, which gave the layman access to the tools required to filter down those millions of results and make their searches more useful.
In hindsight, we cannot be surprised that people typically ignored these options screens and learned to wade through dozens of false positives before possibly finding the article they were looking for. Clicking these options forced people to think like a programmer to get anything useful out of their search. Conceivably, a user could spend two minutes carefully defining the perfect seach, only to have the criteria be too narrowly focused to return the desired results.
While Ajax is frequently celebrated, its most important contribution to our user experience is that it allows developers the ability to give users more immediate feedback. Server-side validation will always be important for security reasons, but the ability to inform a user that their form submission is invalid — and provide visual suggestions how to fix it, right beside the problem — is a huge efficiency and usability boost.
We can take this idea of real-time feedback much further, and provide instant gratification to people who are using search functionality. Ajax gives us the ability to reverse the harmful behaviours established over a decade ago by the first search engines. For domain-specific bodies of information, we question tolerance for inaccurate searches at all.
Perhaps even the notion of "search" is flawed.
Filtering
We propose that for many problem domains, the basic concept of filtering is now much more appropriate than searching. With a search, you start off with nothing and potentially end up with nothing. Counter to this approach is filtering, where we present everything available, and then encourage the user to progressively remove what they do not need. Using Ajax to provide immediate feedback, we can display the number of results a user can expect to see if they were to submit their query at this moment.
This opens up a whole range of user interface possibilities that are an improvement on the traditional simple text search. We can present the search interface as a dashboard, featuring stateful buttons and selections which encompass all of the domain-specific search patterns. In our research, we have found that users adapt very quickly to the visual stimuli provided when they engage a filter. In a few requests, they have essentially retrained themselves to create that perfect set of criteria and experiment with what is actually there. These new "hackers" need not see a page that bears no results.
There is nothing preventing the inclusion of keyword searching from a filtering interface. However, in the process of designing a filtering search for your data, keywords are suddenly less pivotal to your success.
Saving For Later
In most cases, people want to find a specific product, article, or other resource. That resource might not exist today, but very well be created in the future. In a filtering scenario where the user has put effort into customizing the view they need, it suddenly makes a lot more sense to save a user's search so that they can come back at a later date and not have to repeat themselves.
This functionality is manifested in two parts. First off, each search gets committed to the database along with information about which user or guest requested it. A unique key string is generated, to be used to identify the search without revealing the internal primary key of the row. With this data saved, we then return an HTTP instruction to redirect to a unique URL which maps directly to the search now and in the future.
When the search URL is accessed, the lookup is performed and the results are displayed. The user is free to bookmark this URL and return as frequently as required. Clever developers might even make this URL return XML data if it is accessed by an RSS reader.
A side-effect of saving each search is that you get the benefit of a large body of data that shows you exactly what your users are looking for. You are encouraged to increment a counter each time any specific search is run, allowing you to spot power-users as well as search trends. With a little creativity, advanced site features such as queuing saved queries for a daily email are suddenly drop-in enhancements instead of major developments.
Implementation
Of course, getting all of this working is not a walk in the park. There are going to be implementation-specific quirks that you will need to overcome in order to provide a polished user experience.
Many of the concepts in this article would be very difficult to implement without Ruby on Rails. Rails introduces the concept of Partials. Partials are nothing more than snippets of fully formed XHTML which get inserted into the innerHTML of objects in a browser's DOM by JavaScript. The specifics of this process are outside of the scope of this article. While you can update the innerHTML of a DIV or SPAN using any Ajax-capable scripting language such as PHP, there is an architectural elegance that enables this filtering system that we believe would be difficult to replicate.
Implementing a filter system that works across all common browsers is a real challenge, full of gotchas and necessary hacks. While nothing here is offensive, you might be surprised at how involved it can get. The reason being that it is of paramount importance to maintain the function and behaviour of the browser's built-in Back button.
There are two main issues to solve. First off, if you modify the contents of your DOM with Ajax and JavaScript, move to another page, and then return to the form submit page, each major browser seems to corrupt the state of elements on the page in subtle and different ways. For example, Firefox will corrupt the selection state of checkboxes generated at run-time! The second problem is that if a user returns to the search interface with the Back button... what is the user going back to? Suddenly our flow diagram just got a lot more complicated!
The first revelation is that you should be prepared to write client-side functions to persist, restore, and clear the visual state of your filter. When you render the interface, all of your elements will be in their default state. During your document's onload event you will want to execute an initialize function that can then "hydrate" the state of your elements from a hidden form. Likewise when submitting a query, you will not want to process the data associated with the form elements you see. Instead, "de-hydrate" the state of your interface into data on a hidden form that can be easily passed on.
During the initialization process, you will need to generate a new key value that could be assigned to a query request should one be submitted. You will use this value to build the action attribute of the hidden form that they might submit! This does need to be performed dynamically, in case the user has returned to a previously rendered page with their back button. Browsers can simply not be trusted to clear their page caches, no matter how many META directives you push down in your HTTP header.
One behaviour we've evolved while iterating this filter pattern is to remember the state of the last search when returning to the search page via navigation — unless the user specifically clicks on a major site navigation link to go to the search page. We interpret this behaviour as expressing a desire to start a new search from scratch. You'll want to experiment and see what feels right for your project. These sorts of decisions are subjective and might change with feedback.
Behind the scenes
After you have fleshed out the parameters with which the users will be able to filter your data, you will need to create structures to store the query on the client, in your code, and in your database. Some of this information might be primary keys to look up data in other tables, which is appropriate for a drop-down select box. Other fields which allow selection of multiple elements are best stored as a comma-separated list of primary keys, ready to be split and iterated through as required.
The best strategy is to use a consistent naming scheme for your form fields, server code, and database columns. This allows a lot of flexibility; for example, if your main query builder function accepts a generic hash of values, you can pass it data from a saved row in your database, or from the posted parameters data passed in from a submitted form.
The query builder itself should take the form of a protected method that is accessible from all parts of your server-side code base. The goal is to generate what will become the WHERE clause for your database lookup. For each parameter that you filter on, you'll need to test for a value and sequentially construct a string to return at the end of the function. The desired end-result is boolean conditions separated by AND statements. For a condition that allows multiple selections, you'll need to split the string and loop through each value, perhaps generating a series of possible conditions separated by OR statements.
Conclusion
You might wonder whether the end result is worth the trouble. Our opinion is an enthusiastic "YES!" — for many sites. Really good filtering implementations can make a site easier to use. The ability to bookmark a search that returns exactly what you want will keep your happy customers coming back for more. We work hard to keep the complexity under the covers, and therefore add real value to the user experience without toying with expectations about how things should feel.
People will trust things that feel right to them, and a filtered search tells a much more accurate truth than millions of Google search results ever could. To paraphrase the immortal words of Stephen Colbert, our users rely on us not to tell the truth, but to feel the truth at them.
Demo Page: Live Filter