There are many ways to add search functionality to a Rails application. While many Rails developers choose to use the native search functionality built into popular databases like MySQL and Postgres, others need more flexible or feature rich search functionality. ElasticSearch is probably the most well known option available but it has its own issues. Firstly, it is a resource hungry beast. To run ElasticSearch properly in production, you need a few beefy servers. Secondly, getting ElasticSearch to return relevant search results is not easy. There are many options to configure and knobs to turn. This adds a lot of flexibility, but takes away from simplicity.
Hosted alternatives like AWS Opensearch (formerly known as AWS Elasticsearch) or Algolia can take away a lot of painful hosting problems, but can become expensive very quickly.
For many use cases, there are other, simpler and cheaper alternatives. Typesense and Meilisearch are two that we tried for one of our internal projects at Cloud 66.
After some searching, trials and prototyping, we settled on MeiliSearch so I wanted to share the way we implemented search and integrated with MeiliSearch.
Native MeiliSearch Rails Integration
There is an official MeiliSearch gem with decent documentation you can use to simply index and search your ActiveRecord objects without much hassle. The way the native MeiliSearch / Rails integration works is by annotating your ActiveRecord objects with include MeiliSearch and adding a meilisearch block to your classes:
class Student < ApplicationRecord
include MeiliSearch
meilisearch do
attribute :name, :subject
end
end
Now you can search Student
hits = Student.search('physics')
While this is simple, elegant and very effective, our use case was slightly different so we needed to decouple our indexing and search from our basic domain objects.
Using MeiliSearch with an Admin App
A partial screenshot of MissionControl homepage
Internally, we wrote a tool called MissionControl, which helps our teams from sales to marketing to our customer advocates and engineers, support our business. Both of these apps are written in Rails and share the same database. However, the admin app (MissionControl) has readonly access to the database, while the main app handles around 2,000 database operations per second. On top of that, we didn't want to make changes and deploy the main app, every time we need to add a new field to our search index. All of these factors, as well as other security and availability requirements, meant that MissionControl had to take care of its own search functionality and not rely on the main application to keep the search indices up to date, which in turn, meant we couldn't use the native ActiveRecord-based update with MeiliSearch.
Cloud 66 main app and MissionControl
Our final design for adding MeiliSearch to MissionControl is as follows:
- A rake task in MissionControl finds all changed records and updates the index
- A second rake task in MissionControl finds all deleted records and removes them from the index.
- Both tasks are scheduled with CRONJobs to run regularly and frequently.
- We can remove the entire search index if needed and the next run will rebuild it from scratch.
Let's get deeper:
First, we need to instantiate a MeiliSearch client. This can be done in a Rails initializer:
require 'meilisearch'
$ms_client = MeiliSearch::Client.new(ENV.fetch('MEILISEARCH_URL') { 'http://127.0.0.1:7700' })
Indexing new and changed objects
Now on to the rake tasks. First the index task, responsible for finding updated database records and adding / updating their index:
desc "Indexes all docs"
task :run, [:name] => [:environment] do |t, args|
# only run one of these tasks at the time
is_running = $redis.get("mc_index_running")
if is_running == "1"
puts "Another index process is running"
next
end
begin
$redis.set("mc_index_running", 1)
BATCH_SIZE = 100
INDEX_NAME = :searchable
# this filters records that we don't want to index
class_filters = {
::User => -> (x) { x.account.nil? },
::Account => -> (x) { x.main_user.nil? },
# ... more classes here
}
# any class we want to index and the attributes on them we want indexed
classes_to_index = {
::User => [
:email,
:name,
:phone_number,
:updated_at,
{ company_name: ->(x) { x.account.name } }
],
::Account => [
:name,
:company_name,
:updated_at,
{ owner_email: -> (x) { x.main_user.email }},
],
# ... more classes here
]
}
index = $ms_client.index(INDEX_NAME)
# take the cutoff point
cutoff = Time.now.utc
puts "Cutting off at #{cutoff}"
filter = nil
if args.count > 0
filter = args[:name]
end
classes_to_index.each_pair do |clazz, attributes|
name = clazz.name.to_s.downcase
next if !filter.nil? && name != filter
puts "Indexing #{name}"
# get the last index cut off
index_metadata = ::IndexMetadata.find_by(name: name)
last_cut_off = index_metadata&.last_index
puts "Last cutoff was #{last_cut_off ? last_cut_off : 'never'}"
db_attributes = attributes.filter { |x| x.is_a? Symbol }
calculated_attributes = attributes.filter { |x| x.is_a? Hash }
if last_cut_off
query = clazz.select(:id, db_attributes).where('updated_at >= ?', last_cut_off)
puts "#{clazz.where('updated_at >= ?', last_cut_off).count} new records found"
else
query = clazz.select(:id, db_attributes)
end
query.find_in_batches(batch_size: BATCH_SIZE) do |group|
to_index = group.map do |x|
next nil if class_filters[clazz].call(x)
atts = x.attributes.dup
atts['id'] = "#{name}-#{x.id}"
atts['kind'] = name
calculated_attributes.each do |c_attrs|
c_attrs.each_pair do |key, value|
if value.is_a? Proc
atts[key] = value.call(x)
end
end
end
next atts.symbolize_keys
end
to_index.compact!
index.add_documents(to_index, 'id') unless to_index.empty?
rescue => exc
puts exc.message
pp to_index
raise
end
if index_metadata
index_metadata.update({ name: name, last_index: cutoff })
else
::IndexMetadata.create({ name: name, last_index: cutoff })
end
end
ensure
$redis.set("mc_index_running", 0)
end
end
Let's break down this task further. MissionControl uses ActiveRecord to access the shared database, so classes like ::User and ::Account are vanilla Rails ActiveRecord classes. On each one of these classes, there are some basic database column-based attributes, like email or name that you can see added to classes_to_index. We also have some "dynamic" attributes we wanted to index. This might be from a relationship of the main object. For example, you might want to index the name of the company a user works for alongside the user, however that piece of information might be stored on the ::Account, which has a one to many relationship with the ::User.
To allow this, we used a mix of symbols for basic attributes, and Lambdas for the dynamic ones:
::User => [
:email,
:name,
:phone_number,
:updated_at,
{ company_name: ->(x) { x.account.name } }
],
Next we need to read all the unindexed objects from the database. To do this, we use a simple class called IndexMetadata. IndexMetadata is stored in the database and consists of the name of the object's class and the timestamp of the last successful index run for that class. Using this, we can ensure that we only read objects that have an updated_at attribute past the last index timestamp.
class AddIndexMetadata < ActiveRecord::Migration[6.1]
def change
create_table :index_metadata do |t|
t.string :name, null: false
t.datetime :last_index, null: true
t.timestamps
end
end
end
IndexMetadata migration
class IndexMetadata < ApplicationRecord
end
models/index_metadata.rb
Now, back to the task code:
index_metadata = ::IndexMetadata.find_by(name: name)
last_cut_off = index_metadata&.last_index
if last_cut_off
query = clazz.select(:id, db_attributes).where('updated_at >= ?', last_cut_off)
puts "#{clazz.where('updated_at >= ?', last_cut_off).count} new records found"
else
query = clazz.select(:id, db_attributes)
end
Now we have an ActiveRecord query which we can run to fetch all the objects we need to index for that class. To speed things up, we use a find_in_batches method to read DB records in batches.
query.find_in_batches(batch_size: BATCH_SIZE) do |group|
# ...
end
Once we have each group, we can iterate through the objects and construct a hash of the object's db columns as well as the dynamic attributes.
query.find_in_batches(batch_size: BATCH_SIZE) do |group|
to_index = group.map do |x|
next nil if class_filters[clazz].call(x)
atts = x.attributes.filter{ |x| x != 'params'}.dup
atts['id'] = "#{name}-#{x.id}"
atts['kind'] = name
calculated_attributes.each do |c_attrs|
c_attrs.each_pair do |key, value|
if value.is_a? Proc
atts[key] = value.call(x)
end
end
end
next atts.symbolize_keys
end
to_index.compact!
index.add_documents(to_index, 'id') unless to_index.empty?
rescue => exc
puts exc.message
pp to_index
raise
end
As you can see we are also filtering out some of the records that we don't want to index. Once this is done, to_index holds an array of hashes that we can simply push to MeiliSearch where it will be indexed. add_documents adds or updates the documents with the same ID so we don't need to worry about new and existing documents running different code.
Now all that's left is to update our IndexMetadata for the class we just indexed so next time we start from where we left off.
if index_metadata
index_metadata.update({ name: name, last_index: cutoff })
else
::IndexMetadata.create({ name: name, last_index: cutoff })
end
Removing deleted object
The next rake task takes care of removing documents from the index where the object has been deleted from the database. This is very easy if you have some sort of soft delete in your application. But without soft deletion, you simply don't have the record in the database anymore and so don't know what to delete in the index. We solved this by comparing the index with the database:
desc "Delete removed docs"
task :purge, [:name] => [:environment] do |t, args|
INDEX_NAME = :searchable
FETCH_LIMIT = 500
index = $ms_client.index(INDEX_NAME)
# fetch all docs from the search index in batches
id_list = {
'user' => [],
'account' => [],
# more classes here
}
puts "Fetching all documents in search index"
offset = 0
loop do
ids = index.documents(limit: FETCH_LIMIT, offset: offset, attributesToRetrieve: 'id')
# parse the IDs
ids.each { |id| parts = id['id'].split('-'); id_list[parts[0]] << parts[1].to_i }
offset = offset + FETCH_LIMIT
break if ids.count < FETCH_LIMIT # finished
end
# for each index fetch all IDs.
def compare(id_list, idx)
puts "Comparing #{id_list[idx].count} #{idx.pluralize} with database"
offset = 0
ids = []
loop do
db_ids = ::ActiveRecord::Base.establish_connection(:main).connection.execute("SELECT id FROM #{idx.pluralize} LIMIT #{offset},#{FETCH_LIMIT}").to_a.flatten
ids << db_ids
offset = offset + FETCH_LIMIT
break if db_ids.count < FETCH_LIMIT # finished
end
ids.flatten!
return id_list[idx] - ids
end
to_delete = []
if args.count == 0
to_delete << compare(id_list, 'user').map { |x| "user-#{x}"}
to_delete << compare(id_list, 'account').map { |x| "account-#{x}"}
# more classes here
else
to_delete << compare(id_list, args[:name]).map { |x| "#{args[:name]}-#{x}"}
end
to_delete.flatten!
puts "Deleting #{to_delete.count} documents from search index"
index.delete_documents(to_delete)
end
This tasks has 4 main steps:
- Fetch the IDs of all documents in the index (this is very quick in MeiliSearch)
- Fetch the IDs of all documents in the database (this can be improved using
VALUESif you use MySQL 8.0.19+) but still very fast and memory efficient. - Compare the ID sets and find the missing ones
- Delete the missing ones from the index
Scheduling
You can now schedule these tasks to run every minute to keep your index up to date. While this will not give you a real-time index, it works for many use cases where you want the search functionality to be separate from your main application.
Searching
Searching the index is the same regardless of your indexing method. There is MeiliSearch documentation for both server and client side search methods depending on your requirements and the client's JS stack.