How to avoid crashing Rails app after removing column from table?

The problem

I just learned that our current practice for removing a column in an ActiveRecord migration is to hide the column from Rails before actually removing it (via an ugly hack, see below for details).

This is due to the fact that Rails caches the SHOW FULL FIELDS query. If we don't work around that, the (long-running) migration will remove the column and by that time, Rails will have already cached the fields. Once the migration is done and the column is gone, the app will subsequently crash because INSERTs will provide values for a non-existing column because of the cache.

Using things like clear_table_cache! within the migration are of no use because we deploy to N servers and run the migration on just one of them. This would clear the cache on one of the servers but not all.

Our current solution (a.k.a. ugly hack)

What we're currently doing is overriding ActiveRecord::ConnectionAdapters::AbstractMysqlAdapter#columns in an initializer and deploy that before running the migration.

After the migration is done, we remove the overriding initializer and deploy again.

At this point, I cannot believe we're the only ones who encounter this problem and have to work around it. Are there any other solution to this problem?

Rails 5 has added ignored_columns to the ActiveRecord::Base class:

Since some columns will appear or disappear at any moment, you want to make those columns invisible to AR for a while. Otherwise you could have some processes knowing about the columns and others who don't.

Here's some sample code:

class User < ApplicationRecord
  self.ignored_columns = %w(employee_email)
end

The 10 Most Common Mistakes That Rails Developers Make, There are many Ruby on Rails best practices that can help you avoid these common When we look at the log file of a request that calls this method we'll see instructions to automatically add and remove database tables and rows. Common Mistake #11: The tab this page is loaded in crashes when flash crashes​. smh. 4 Rails undefined method `build' for nil:NilClass Apr 5 '16 3 Custom Rails Validation not adding 'field_with_errors' wrapper to inputs Apr 7 '16 3 How to avoid crashing Rails app after removing column from table?

I've used this approach before:

We override the ActiveRecord::Base.columns method on the class whose column is being removed. Then we call super to retrieve the columns and strip out the columns marked for removal. Now the class can behave as if the column does not exist. […]

class << self
  RemovedColumns = {'column_to_remove' => true}
  def columns
    cols = super
    cols.reject { |col| RemovedColumns.has_key? col.name }
  end
end

Deploying when Removing Columns with Rails with Zero Downtime, At TST Media we deploy updates to our NGIN application frequently When removing a column from a table we make sure this is done in a migration by itself. so that on the rolling deploy the column does not get removed. In order to conserve memory and avoid crashing the server, I have set this to 30 seconds. I realized that I am incurring a penalty for frequent re-spawning of rails instances, but as it turns out, mod_rails caches the entire rails app in memory so spawning is relatively fast.

I ran into this issue in the past. My approach may vary on a case-by-case basis, but I've used more than once the trick to comment the column in the schema.rb and deploy before running the migration.

Here's the steps:

  1. Change schema.rb manually and comment the line where the column you want to remove is defined.
  2. Deploy the changes and restart the server.
  3. Create and run the migration.
  4. Deploy the changes and restart the server.

As the field was commented out, the running version of the application is already ignoring it and the column is not present in the model definition.

Therefore, the migration that removes it will not need to clear the attribute cache and the application will not crash even if the migration takes longer to be completed. Of course, the field must not be used anywhere.

This approach is quite clean as it doesn't involve monkey patching or overriding Rails.

Dealing with Massive Data in Rails, To avoid bottlenecks, check out these tips on dealing with massive data in Rails. or just as the backend API server for the mobile apps we build. Will Rails be the bottleneck when you try to handle millions of user data Always index important columns or column combinations you query most frequently. After you find the query that causes lock escalation, look for opportunities to create new indexes or to add columns to an existing index to remove index or table scans and to maximize the efficiency of index seeks. Consider pasting the query into a Query Analyzer query window to perform an automatic index analysis on it.

I have experienced the same many times.

You can use the strong_migrations gem for this. https://github.com/ankane/strong_migrations#removing-a-column

The gem will complain when you try to create a "dangerous" migration and give you advises.

Active Record Migrations, This migration adds a table called products with a string column called name If the database does not support this then when a migration fails the parts of it Similarly, you can generate a migration to remove a column from the command line: The migration crashes because when the model attempts to save, it tries to  If your issue is not resolved after you test it outside the third-party application, go to method 5. Method 5: Perform a selective startup to determine whether a program, process, or service conflicts with Excel. When you start Windows as usual, several applications and services start automatically and then run in the background.

same thing as margo's previous answer just cleaner imo

def self.columns
  super.reject { |col| col.name == 'notes' }
end

Migrations, You'd also have to keep track of which changes need to be run against the production Migrations also allow you to describe these transformations using Ruby. This migration adds a table called products with a string column called name and a When Bob runs rake db:migrate, Rails knows that it has not run Alice's two  The best way to avoid SQL injection is to know how to do it. Let’s build a Rails app then hack it ourselves! This will add these two columns to the users table in the database. $ rake db

Active Record Query Optimization Tips - Steven Li, Simple But Effective Ways To Improve Active Record Query Performance Eager Loading With `includes` to Avoid n+1 Queries but it's meant to reduce memory use when loading a huge amount of records. Bulk Delete With `​delete_all` of your query optimizations, use the Ruby's Benchmark.realtime to time them. Fixed #4667, column was left semi-opaque after drill up with the Standalone Framework. Fixed #4782, marker image width and height didn't apply to legend. Fixed #4799, wrong position of columns when setting grouping:false in charts with multiple columns.

ankane/strong_migrations: Catch unsafe migrations in , Add this line to your application's Gemfile: ActiveRecord caches database columns at runtime, so if you drop a column, Adding a column with a default value to an existing table causes the entire Use the Rails console or a separate migration with disable_ddl_transaction! Use the stop! method to stop migrations. Select one or more table rows or table columns that you want to delete. You can also just select one or more cells in the table rows or table columns that you want to delete. On the Home tab, in the Cells group, click the arrow next to Delete, and then click Delete Table Rows or Delete Table Columns.

Popular Science, From Rails 4.0 on delete automatically tries to find the records matching the given ids before deleting them. In Rails 4.0 when a column or a table is renamed the  I have about 30 tables in a rails app and there are about 6 tables for which I would like to change the precision and scale of some of the decimal columns. How do you go about doing that for already existing columns? Do you need to do each new migration separately, or can they all be done at once. Answer1:

Comments
  • Are you doing more in your migrations than building the table structures and indexes? If you are hitting insert problems my guess is that you are also manipulating data via your migrations. In my experience that always bites you on the bum! It is bad practice.
  • No, we're not. All the migration does is to remove the column.
  • nothing like being wrong :)
  • This is a nasty problem, i really hope rails didn't had to ask us to ignore columns in model specifically. For me, even clearing cache was not working.
  • You should share the details of the approach as part of your answer, or it will be deleted for being link-only.
  • How does that help? Is the schema.rb ever used for something in production?
  • Thanks for this link, it pointed me to ActiveRecord::Base#ignored_columns!