Searchkick

🚀 Intelligent search made easy

Searchkick learns what your users are looking for. As more people search, it gets smarter and the results get better. It’s friendly for developers - and magical for your users.

Searchkick handles:

stemming - tomatoes matches tomato
special characters - jalapeno matches jalapeño
extra whitespace - dishwasher matches dish washer
misspellings - zuchini matches zucchini
custom synonyms - qtip matches cotton swab

Plus:

query like SQL - no need to learn a new query language
reindex without downtime
easily personalize results for each user
autocomplete
“Did you mean” suggestions
works with ActiveRecord, Mongoid, and NoBrainer

💬 Get handcrafted updates for new features

🍊 Battle-tested at Instacart

We highly recommend tracking queries and conversions

⚡ Searchjoy makes it easy

Get Started

Install Elasticsearch. For Homebrew, use:

brew install elasticsearch

Add this line to your application’s Gemfile:

gem'searchkick'

For Elasticsearch 0.90, use version 0.6.3 and this readme.

Add searchkick to models you want to search.

classProduct < ActiveRecord::Basesearchkickend

Add data to the search index.

Product.reindex

And to query, use:

products=Product.search"apples"products.eachdo |product| putsproduct.nameend

Searchkick supports the complete Elasticsearch Search API. As your search becomes more advanced, we recommend you use the Elasticsearch DSL for maximum flexibility.

Queries

Query like SQL

Product.search"apples",where: {in_stock: true},limit: 10,offset: 50

Search specific fields

fields: [:name,:brand]

Where

where: {expires_at: {gt: Time.now},# lt, gte, lte also availableorders_count: 1..10,# equivalent to{gte: 1, lte: 10}aisle_id: [25,30],# instore_id: {not: 2},# notaisle_id: {not: [25,30]},# not inuser_ids: {all: [1,3]},# all elements in arraycategory: /frozen .+/,# regexpor: [[{in_stock: true},{backordered: true}]]}

Order

order: {_score: :desc}# most relevant first - default

All of these sort options are supported

Limit / offset

limit: 20,offset: 40

Boosting

Boost important fields

fields: ["title^10","description"]

Boost by the value of a field (field must be numeric)

boost_by: [:orders_count]# give popular documents a little boostboost_by: {orders_count: {factor: 10}}# default factor is 1

Boost matching documents

boost_where: {user_id: 1}boost_where: {user_id: {value: 1,factor: 100}}# default factor is 1000boost_where: {user_id: [{value: 1,factor: 100},{value: 2,factor: 200}]}

Conversions are also a great way to boost.

Get Everything

Use a * for the query.

Product.search"*"

Pagination

Plays nicely with kaminari and will_paginate.

# controller@products=Product.search"milk",page: params[:page],per_page: 20

View with kaminari

<%= paginate @products %>

View with will_paginate

<%= will_paginate @products %>

Partial Matches

By default, results must match all words in the query.

Product.search"fresh honey"# fresh AND honey

To change this, use:

Product.search"fresh honey",operator: "or"# fresh OR honey

By default, results must match the entire word - back will not match backpack. You can change this behavior with:

classProduct < ActiveRecord::Basesearchkickword_start: [:name]end

And to search (after you reindex):

Product.search"back",fields: [{name: :word_start}]

Available options are:

:word# default:word_start:word_middle:word_end:text_start:text_middle:text_end

To boost fields, use:

fields: [{"name^2"=>:word_start}]# better interface on the way

Exact Matches

User.search"[email protected]",fields: [{email: :exact},:name]

Language

Searchkick defaults to English for stemming. To change this, use:

classProduct < ActiveRecord::Basesearchkicklanguage: "German"end

See the list of languages

Synonyms

classProduct < ActiveRecord::Basesearchkicksynonyms: [["scallion","green onion"],["qtip","cotton swab"]]end

Call Product.reindex after changing synonyms.

WordNet

Prepopulate English synonyms with the WordNet database.

Download WordNet 3.0 to each Elasticsearch server and move wn_s.pl to the /var/lib directory.

cd /tmp curl -o wordnet.tar.gz http://wordnetcode.princeton.edu/3.0/WNprolog-3.0.tar.gz tar -zxvf wordnet.tar.gz mv prolog/wn_s.pl /var/lib

Tell each model to use it:

classProduct < ActiveRecord::Basesearchkickwordnet: trueend

Misspellings

By default, Searchkick handles misspelled queries by returning results with an edit distance of one.

You can change this with:

Product.search"zucini",misspellings: {edit_distance: 2}# zucchini

Or turn off misspellings with:

Product.search"zuchini",misspellings: false# no zucchini

Swapping two letters counts as two edits. To count the transposition of two adjacent characters as a single edit, use:

Product.search"mikl",misspellings: {transpositions: true}# milk

This is planned to be the default in Searchkick 1.0.

Indexing

Control what data is indexed with the search_data method. Call Product.reindex after changing this method.

classProduct < ActiveRecord::Basedefsearch_dataas_jsononly: [:name,:active]# or equivalently{name: name,active: active}endend

Searchkick uses find_in_batches to import documents. To eager load associations, use the search_import scope.

classProduct < ActiveRecord::Basescope:search_import,->{includes(:searches)}end

By default, all records are indexed. To control which records are indexed, use the should_index? method.

classProduct < ActiveRecord::Basedefshould_index?active# only index active recordsendend

To Reindex, or Not to Reindex

Reindex

when you install or upgrade searchkick
change the search_data method
change the searchkick method

No need to reindex

App starts

Stay Synced

There are three strategies for keeping the index synced with your database.

Immediate (default)

Anytime a record is inserted, updated, or deleted

Asynchronous

Use background jobs for better performance

classProduct < ActiveRecord::Basesearchkickcallbacks: :asyncend

And install Active Job for Rails 4.1 and below

Manual

Turn off automatic syncing

classProduct < ActiveRecord::Basesearchkickcallbacks: falseend

Associations

Data is not automatically synced when an association is updated. If this is desired, add a callback to reindex:

classImage < ActiveRecord::Basebelongs_to:productafter_commit:reindex_productdefreindex_productproduct.reindex# or reindex_asyncendend

Keep Getting Better

Searchkick uses conversion data to learn what users are looking for. If a user searches for “ice cream” and adds Ben & Jerry’s Chunky Monkey to the cart (our conversion metric at Instacart), that item gets a little more weight for similar searches.

The first step is to define your conversion metric and start tracking conversions. The database works well for low volume, but feel free to use Redis or another datastore.

classSearch < ActiveRecord::Basebelongs_to:product# fields: id, query, searched_at, converted_at, product_idend

You do not need to clean up the search queries. Searchkick automatically treats apple and APPLES the same.

Next, add conversions to the index.

classProduct < ActiveRecord::Basehas_many:searchessearchkickconversions: "conversions"# name of fielddefsearch_data{name: name,conversions: searches.group("query").count#{"ice cream" => 234, "chocolate" => 67, "cream" => 2}}endend

Reindex and set up a cron job to add new conversions daily.

rakesearchkick:reindexCLASS=Product

Personalized Results

Order results differently for each user. For example, show a user’s previously purchased products before other results.

classProduct < ActiveRecord::Basedefsearch_data{name: name,orderer_ids: orders.pluck(:user_id)# boost this product for these users}endend

Reindex and search with:

Product.search"milk",boost_where: {orderer_ids: current_user.id}

Autocomplete

Autocomplete predicts what a user will type, making the search experience faster and easier.

Note: If you only have a few thousand records, don’t use Searchkick for autocomplete. It’s much faster to load all records into JavaScript and autocomplete there (eliminates network requests).

First, specify which fields use this feature. This is necessary since autocomplete can increase the index size significantly, but don’t worry - this gives you blazing faster queries.

classCity < ActiveRecord::Basesearchkicktext_start: [:name]end

Reindex and search with:

City.search"san fr",fields: [{name: :text_start}]

Typically, you want to use a JavaScript library like typeahead.js or jQuery UI.

Here’s how to make it work with Rails

First, add a route and controller action.

# app/controllers/cities_controller.rbclassCitiesController < ApplicationControllerdefautocompleterenderjson: City.search(params[:query],fields: [{name: :text_start}],limit: 10).map(&:name)endend

Then add the search box and JavaScript code to a view.

<inputtype="text" id="query" name="query" /><scriptsrc="jquery.js"></script><scriptsrc="typeahead.js"></script><script>$("#query").typeahead({name: "city",remote: "/cities/autocomplete?query=%QUERY"});</script>

Suggestions

classProduct < ActiveRecord::Basesearchkicksuggest: ["name"]# fields to generate suggestionsend

Reindex and search with:

products=Product.search"peantu butta",suggest: trueproducts.suggestions# ["peanut butter"]

Facets

Facets provide aggregated search data.

products=Product.search"chuck taylor",facets: [:product_type,:gender,:brand]pproducts.facets

By default, where conditions are not applied to facets (for backward compatibility).

Product.search"wingtips",where: {color: "brandy"},facets: [:size]# facets *not* filtered by color :(

Change this with:

Product.search"wingtips",where: {color: "brandy"},facets: [:size],smart_facets: true

or set where conditions for each facet separately:

Product.search"wingtips",facets: {size: {where: {color: "brandy"}}}

Limit

Product.search"apples",facets: {store_id: {limit: 10}}

Ranges

price_ranges=[{to: 20},{from: 20,to: 50},{from: 50}]Product.search"*",facets: {price: {ranges: price_ranges}}

Use the stats option to get to max, min, mean, and total scores for each facet

Product.search"*",facets: {store_id: {stats: true}}

Highlight

Specify which fields to index with highlighting.

classProduct < ActiveRecord::Basesearchkickhighlight: [:name]end

Highlight the search query in the results.

bands=Band.search"cinema",fields: [:name],highlight: true

Note: The fields option is required, unless highlight options are given - see below.

View the highlighted fields with:

bands.with_details.eachdo |band,details| putsdetails[:highlight][:name]# "Two Door <em>Cinema</em> Club"end

To change the tag, use:

Band.search"cinema",fields: [:name],highlight: {tag: "<strong>"}

To highlight and search different fields, use:

Band.search"cinema",fields: [:name],highlight: {fields: [:description]}

Additional options, including fragment size, can be specified for each field:

Band.search"cinema",fields: [:name],highlight: {fields: {name: {fragment_size: 200}}}

You can find available highlight options in the Elasticsearch reference.

Similar Items

Find similar items.

product=Product.firstproduct.similar(fields: ["name"],where: {size: "12 oz"})

Geospatial Searches

classCity < ActiveRecord::Basesearchkicklocations: ["location"]defsearch_dataattributes.mergelocation: [latitude,longitude]endend

Reindex and search with:

City.search"san",where: {location: {near: [37, -114],within: "100mi"}}# or 160km

Bounded by a box

City.search"san",where: {location: {top_left: [38, -123],bottom_right: [37, -122]}}

Boost By Distance

Boost results by distance - closer results are boosted more

City.search"san",boost_by_distance: {field: :location,origin: [37, -122]}

Also supports additional options

City.search"san",boost_by_distance: {field: :location,origin: [37, -122],function: :linear,scale: "30mi",decay: 0.5}

Routing

Searchkick supports Elasticsearch’s routing feature.

classContact < ActiveRecord::Basesearchkickrouting: :user_idend

Reindex and search with:

Contact.search"John",routing: current_user.id

Inheritance

Searchkick supports single table inheritance.

classDog < Animalend

The parent and child model can both reindex.

Animal.reindexDog.reindex# equivalent

And to search, use:

Animal.search"*"# all animalsDog.search"*"# just dogsAnimal.search"*",type: [Dog,Cat]# just cats and dogs

Note: The suggest option retrieves suggestions from the parent at the moment.

Dog.search"airbudd",suggest: true# suggestions for all animals

Debugging Queries

See how Elasticsearch tokenizes your queries with:

Product.searchkick_index.tokens("Dish Washer Soap",analyzer: "default_index")# ["dish", "dishwash", "washer", "washersoap", "soap"]Product.searchkick_index.tokens("dishwasher soap",analyzer: "searchkick_search")# ["dishwashersoap"] - no matchProduct.searchkick_index.tokens("dishwasher soap",analyzer: "searchkick_search2")# ["dishwash", "soap"] - match!!

Partial matches

Product.searchkick_index.tokens("San Diego",analyzer: "searchkick_word_start_index")# ["s", "sa", "san", "d", "di", "die", "dieg", "diego"]Product.searchkick_index.tokens("dieg",analyzer: "searchkick_word_search")# ["dieg"] - match!!

See the complete list of analyzers.

Deployment

Searchkick uses ENV["ELASTICSEARCH_URL"] for the Elasticsearch server. This defaults to http://localhost:9200.

Heroku

Choose an add-on: SearchBox, Bonsai, or Found.

# SearchBox heroku addons:add searchbox:starter heroku config:add ELASTICSEARCH_URL=`heroku config:get SEARCHBOX_URL`# Bonsai heroku addons:add bonsai heroku config:add ELASTICSEARCH_URL=`heroku config:get BONSAI_URL`# Found heroku addons:add foundelasticsearch heroku config:add ELASTICSEARCH_URL=`heroku config:get FOUNDELASTICSEARCH_URL`

Then deploy and reindex:

heroku run rake searchkick:reindex CLASS=Product

Other

Create an initializer config/initializers/elasticsearch.rb with:

ENV["ELASTICSEARCH_URL"]="http://username:[email protected]"

Then deploy and reindex:

rake searchkick:reindex CLASS=Product

Performance

For the best performance, add Typhoeus to your Gemfile.

gem'typhoeus'

And create an initializer with:

require"typhoeus/adapters/faraday"Ethon.logger=Logger.new("/dev/null")

Note: Typhoeus is not available for Windows.

Automatic Failover

Create an initializer config/initializers/elasticsearch.rb with multiple hosts:

Searchkick.client=Elasticsearch::Client.new(hosts: ["localhost:9200","localhost:9201"],retry_on_failure: true)

See elasticsearch-transport for a complete list of options.

Lograge

Add the following to config/environments/production.rb:

config.lograge.custom_options=lambdado |event| options={}options[:search]=event.payload[:searchkick_runtime]ifevent.payload[:searchkick_runtime].to_f > 0optionsend

See Production Rails for other good practices.

Advanced

Prefer to use the Elasticsearch DSL but still want awesome features like zero-downtime reindexing?

Advanced Mapping

Create a custom mapping:

classProduct < ActiveRecord::Basesearchkickmappings: {product: {properties: {name: {type: "string",analyzer: "keyword"}}}}end

To keep the mappings and settings generated by Searchkick, use:

classProduct < ActiveRecord::Basesearchkickmerge_mappings: true,mappings: {...}end

Advanced Search

And use the body option to search:

products=Product.searchbody: {match: {name: "milk"}}

View the response with:

products.response

To modify the query generated by Searchkick, use:

products=Product.search"apples"do |body| body[:query]={match_all: {}}end

Reference

Reindex one record

product=Product.find10product.reindex# or to reindex in the backgroundproduct.reindex_async

Reindex more than one record without recreating the index

# do this ...some_company.products.each{ |p| p.reindex}# or this ...Product.searchkick_index.import(some_company.products)# don't do the following as it will recreate the index with some_company's products onlysome_company.products.reindex

Reindex large set of records in batches

Product.where("id > 100000").find_in_batchesdo |batch| Product.searchkick_index.import(batch)end

Remove old indices

Product.clean_indices

Use a different index name

classProduct < ActiveRecord::Basesearchkickindex_name: "products_v2"end

Use a dynamic index name

classProduct < ActiveRecord::Basesearchkickindex_name: ->{"#{name.tableize}-#{I18n.locale}"}end

Prefix the index name

classProduct < ActiveRecord::Basesearchkickindex_prefix: "datakick"end

Turn off callbacks temporarily

Product.disable_search_callbacks# or use Searchkick.disable_callbacks for all modelsExpensiveProductsTask.executeProduct.enable_search_callbacks# or use Searchkick.enable_callbacks for all modelsProduct.reindex

Change timeout

Searchkick.timeout=5# defaults to 10

Change the search method name in config/initializers/searchkick.rb

Searchkick.search_method_name=:lookup

Eager load associations

Product.search"milk",include: [:brand,:stores]

Do not load models

Product.search"milk",load: false

Turn off special characters

classProduct < ActiveRecord::Base# A will not match Äsearchkickspecial_characters: falseend

Change import batch size

classProduct < ActiveRecord::Basesearchkickbatch_size: 200# defaults to 1000end

Create index without importing

Product.reindex(import: false)

Make fields unsearchable but include in the source

classProduct < ActiveRecord::Basesearchkickunsearchable: [:color]end

Reindex conditionally

Note: With ActiveRecord, use this feature with caution - transaction rollbacks can cause data inconstencies

classProduct < ActiveRecord::Basesearchkickcallbacks: false# add the callbacks manuallyafter_save:reindex,if: proc{|model| model.name_changed?}# use your own conditionafter_destroy:reindexend

Reindex all models - Rails only

rake searchkick:reindex:all

Turn on misspellings after a certain number of characters

Product.search"api",misspellings: {prefix_length: 2}# api, apt, no ahi

Note: With this option, if the query length is the same as prefix_length, misspellings are turned off

Product.search"ah",misspellings: {prefix_length: 2}# ah, no aha

Large Data Sets

For large data sets, check out Keeping Elasticsearch in Sync. Searchkick will make this easy in the future.

Testing

This section could use some love.

RSpec

describeProductdoit"searches"doProduct.reindexProduct.searchkick_index.refresh# don't forget this# test goes here...endend

Factory Girl

product=FactoryGirl.create(:product)product.reindex# don't forget thisProduct.searchkick_index.refresh# or this

Migrating from Tire

Change search methods to tire.search and add index name in existing search calls

Product.search"fruit"

should be replaced with

Product.tire.search"fruit",index: "products"

Replace tire mapping w/ searchkick method

classProduct < ActiveRecord::Basesearchkickend

Deploy and reindex

rakesearchkick:reindexCLASS=Product# or Product.reindex in the console

Once it finishes, replace search calls w/ searchkick calls

Upgrading

View the changelog.

Important notes are listed below.

0.6.0 and 0.7.0

If running Searchkick 0.6.0 or 0.7.0 and Elasticsearch 0.90, we recommend upgrading to Searchkick 0.6.1 or 0.7.1 to fix an issue that causes downtime when reindexing.

0.3.0

Before 0.3.0, locations were indexed incorrectly. When upgrading, be sure to reindex immediately.

Elasticsearch Gotchas

Inconsistent Scores

Due to the distributed nature of Elasticsearch, you can get incorrect results when the number of documents in the index is low. You can read more about it here. To fix this, do:

classProduct < ActiveRecord::Basesearchkicksettings: {number_of_shards: 1}end

For convenience, this is set by default in the test environment.

Thanks

Thanks to Karel Minarik for Elasticsearch Ruby and Tire, Jaroslav Kalistsuk for zero downtime reindexing, and Alex Leschenko for Elasticsearch autocomplete.

Roadmap

More features for large data sets
Improve section on testing
Semantic search features
Search multiple fields for different terms
Search across models
Search nested objects
Much finer customization

Contributing

Everyone is encouraged to help improve this project. Here are a few ways you can help:

Report bugs
Fix bugs and submit pull requests
Write, clarify, or fix documentation
Suggest or add new features

To get started with development and testing:

git clone https://github.com/ankane/searchkick.git cd searchkick bundle install rake test

Name		Name	Last commit message	Last commit date
Latest commit History 920 Commits
ci		ci
gemfiles		gemfiles
lib		lib
test		test
.gitignore		.gitignore
.travis.yml		.travis.yml
CHANGELOG.md		CHANGELOG.md
Gemfile		Gemfile
LICENSE.txt		LICENSE.txt
README.md		README.md
Rakefile		Rakefile
searchkick.gemspec		searchkick.gemspec

License

keylight/searchkick

Folders and files

Latest commit

History

Repository files navigation

Searchkick

Get Started

Queries

Boosting

Get Everything

Pagination

Partial Matches

Exact Matches

Language

Synonyms

WordNet

Misspellings

Indexing

To Reindex, or Not to Reindex

Reindex

No need to reindex

Stay Synced

Associations

Keep Getting Better

Personalized Results

Autocomplete

Here’s how to make it work with Rails

Suggestions

Facets

Highlight

Similar Items

Geospatial Searches

Boost By Distance

Routing

Inheritance

Debugging Queries

Deployment

Heroku

Other

Performance

Automatic Failover

Lograge

Advanced

Advanced Mapping

Advanced Search

Reference

Large Data Sets

Testing

RSpec

Factory Girl

Migrating from Tire

Upgrading

0.6.0 and 0.7.0

0.3.0

Elasticsearch Gotchas

Inconsistent Scores

Thanks

Roadmap

Contributing

About

Resources

License

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Languages

Packages