Canvas installation errors

gistfile1.txt

Following https://github.com/instructure/canvas-lms/wiki/Quick-Start I ran into some problems.  Here's what I did to work around them.

I am installing on my development computer, running Ubuntu 13.10.

dpkg --get-selections | egrep -i "postgresql"
postgresql					install
postgresql-9.1					install
postgresql-9.1-postgis				deinstall
postgresql-client-9.1				install
postgresql-client-common			install
postgresql-common				install


Minor problem: Errors from Gemfile
==================================

> cd canvas
RVM used your Gemfile for selecting Ruby, it is all fine - Heroku does that too,
you can ignore these warnings with 'rvm rvmrc warning ignore canvas/Gemfile'.
To ignore the warning for all files run 'rvm rvmrc warning ignore allGemfiles'.

sed: -e expression #1, char 45: unterminated `s' command
ruby-1.9.3,:engine=>ruby,:engine_version=>1.9.3 is not installed.
To install do: 'rvm install ruby-1.9.3,:engine=>ruby,:engine_version=>1.9.3'

> rvm install ruby-1.9.3
> rvm use ruby-1.9.3-p545

Looks like it's already on the stable branch by default:
> git rev-parse origin/stable
6058f6b8d4826ded72b2e5894ff0974761cb476f
> git rev-parse HEAD
6058f6b8d4826ded72b2e5894ff0974761cb476f

> bundle install --without mysql
> for config in amazon_s3 delayed_jobs domain file_store outgoing_mail security scribd external_migration; do cp config/$config.yml.example config/$config.yml; done

> cp config/database.yml.example config/database.yml
> createdb canvas_development
> createdb canvas_queue_development



Problem: rake db:initial_setup fails, apparently because remove_index in migrations is not having any effect?!
==============================================================================================================


> bundle exec rake db:initial_setup
==  InitCanvasDb: migrating ===================================================
…

==  AddStickyXlisting: migrating ==============================================
-- add_column(:course_sections, :sticky_xlist, :boolean)
   -> 0.0008s
==  AddStickyXlisting: migrated (0.0010s) =====================================

==  OptimizeDelayedJobs: migrating ============================================
-- create_table(:failed_jobs)
NOTICE:  CREATE TABLE will create implicit sequence "failed_jobs_id_seq" for serial column "failed_jobs.id"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "failed_jobs_pkey" for table "failed_jobs"
   -> 0.0177s
-- remove_index(:delayed_jobs, {:name=>"get_delayed_jobs_index"})
   -> 0.0020s
-- remove_index(:delayed_jobs, [:strand])
   -> 0.0015s
-- add_index(:delayed_jobs, ["run_at", "queue", "locked_at", "strand", "priority"], {:name=>"index_delayed_jobs_for_get_next"})
   -> 0.0074s
-- add_index(:delayed_jobs, ["strand", "id"], {:name=>"index_delayed_jobs_on_strand"})
rake aborted!
An error has occurred, this and all later migrations canceled:

PG::Error: ERROR:  relation "index_delayed_jobs_on_strand" already exists
: CREATE  INDEX "index_delayed_jobs_on_strand" ON "delayed_jobs" ("strand", "id")/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:227:in `rescue in log'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:204:in `log'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:520:in `execute'
canvas/config/initializers/active_record.rb:1359:in `add_index'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/migration.rb:352:in `block in method_missing'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/migration.rb:328:in `block in say_with_time'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/migration.rb:328:in `say_with_time'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/migration.rb:348:in `method_missing'
canvas/db/migrate/20110610213249_optimize_delayed_jobs.rb:29:in `up'



canvas (stable) > bundle exec rake db:initial_setup
rake aborted!
PG::Error: ERROR:  relation "schema_migrations" already exists
: CREATE TABLE "schema_migrations" ("version" character varying(255) NOT NULL) 
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:227:in `rescue in log'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:204:in `log'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/connection_adapters/postgresql_adapter.rb:520:in `execute'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/connection_adapters/abstract/schema_statements.rb:109:in `create_table'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/connection_adapters/abstract/schema_statements.rb:371:in `initialize_schema_migrations_table'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/migration.rb:441:in `initialize'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/migration.rb:401:in `new'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/migration.rb:401:in `up'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/migration.rb:383:in `migrate'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/railties/lib/tasks/databases.rake:112:in `block (2 levels) in <top (required)>'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bin/ruby_executable_hooks:15:in `eval'
/home/tyler/.rvm/gems/ruby-1.9.3-p545/bin/ruby_executable_hooks:15:in `<main>'
Tasks: TOP => db:initial_setup => db:migrate
(See full trace by running task with --trace)

Yet schema_migrations table *does* exist:

> ./script/dbconsole 
canvas_development=# select count(version) from schema_migrations;
 count 
-------
    60
(1 row)

> \d
shows that it is in the 'public' schema.

I edited activerecord/lib/active_record/connection_adapters/abstract/schema_statements.rb:371 and added some debugging info:

--- a/activerecord/lib/active_record/connection_adapters/abstract/schema_statements.rb
+++ b/activerecord/lib/active_record/connection_adapters/abstract/schema_statements.rb
@@ -367,6 +367,7 @@ module ActiveRecord
       def initialize_schema_migrations_table
         sm_table = ActiveRecord::Migrator.schema_migrations_table_name
 
+        puts %(tables=#{(tables).inspect})
         unless tables.detect { |t| t == sm_table }
           create_table(sm_table, :id => false) do |schema_migrations_table|
             schema_migrations_table.column :version, :string, :null => false

And it outputs this:
tables=[]

http://www.redmine.org/boards/2/topics/6051 suggested adding a schema_search_path: public to my config/database.yml.

development:
  adapter: postgresql
  encoding: utf8
  database: canvas_development
  schema_search_path: public

When I did that, it was able to see/list all the tables, and found the schema_migrations table.

But now I'm back to getting this error:

> bundle exec rake db:initial_setup
…
==  OptimizeDelayedJobs: migrating ============================================
-- create_table(:failed_jobs)
NOTICE:  CREATE TABLE will create implicit sequence "failed_jobs_id_seq" for serial column "failed_jobs.id"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "failed_jobs_pkey" for table "failed_jobs"
   -> 0.0312s
-- remove_index(:delayed_jobs, {:name=>"get_delayed_jobs_index"})
   -> 0.0058s
-- remove_index(:delayed_jobs, [:strand])
   -> 0.0023s
-- add_index(:delayed_jobs, ["run_at", "queue", "locked_at", "strand", "priority"], {:name=>"index_delayed_jobs_for_get_next"})
   -> 0.0098s
-- add_index(:delayed_jobs, ["strand", "id"], {:name=>"index_delayed_jobs_on_strand"})
rake aborted!
An error has occurred, this and all later migrations canceled:

PG::Error: ERROR:  relation "index_delayed_jobs_on_strand" already exists
: CREATE  INDEX "index_delayed_jobs_on_strand" ON "delayed_jobs" ("strand", "id")/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:227:in `rescue in log'
…

> psql canvas_queue_development
…
canvas_queue_development=# \d delayed_jobs
                                      Table "public.delayed_jobs"
    Column    |            Type             |                         Modifiers                         
--------------+-----------------------------+-----------------------------------------------------------
 id           | bigint                      | not null default nextval('delayed_jobs_id_seq'::regclass)
 …
 strand       | character varying(255)      | 
Indexes:
    "delayed_jobs_pkey" PRIMARY KEY, btree (id)
    "delayed_jobs_priority" btree (priority, run_at)
    "delayed_jobs_queue" btree (queue)
    "get_delayed_jobs_index" btree (priority, run_at, failed_at NULLS FIRST, locked_at NULLS FIRST, queue)
    "index_delayed_jobs_on_strand" btree (strand)
    "index_delayed_jobs_on_tag" btree (tag)

Maybe this line was supposed to remove that index?
  remove_index :delayed_jobs, [:strand]

--- a/db/migrate/20110610213249_optimize_delayed_jobs.rb
+++ b/db/migrate/20110610213249_optimize_delayed_jobs.rb
@@ -23,7 +23,7 @@ class OptimizeDelayedJobs < ActiveRecord::Migration
     end
 
     remove_index :delayed_jobs, :name => 'get_delayed_jobs_index'
-    remove_index :delayed_jobs, [:strand]
+    remove_index :delayed_jobs, :name => 'index_delayed_jobs_on_strand'
 
     add_index :delayed_jobs, %w(run_at queue locked_at strand priority), :name => 'index_delayed_jobs_for_get_next'
     add_index :delayed_jobs, %w(strand id), :name => 'index_delayed_jobs_on_strand'

> bundle exec rake db:initial_setup
==  OptimizeDelayedJobs: migrating ============================================
-- create_table(:failed_jobs)
NOTICE:  CREATE TABLE will create implicit sequence "failed_jobs_id_seq" for serial column "failed_jobs.id"
NOTICE:  CREATE TABLE / PRIMARY KEY will create implicit index "failed_jobs_pkey" for table "failed_jobs"
   -> 0.0219s
-- remove_index(:delayed_jobs, {:name=>"get_delayed_jobs_index"})
   -> 0.0045s
-- remove_index(:delayed_jobs, {:name=>"index_delayed_jobs_on_strand"})
   -> 0.0015s
-- add_index(:delayed_jobs, ["run_at", "queue", "locked_at", "strand", "priority"], {:name=>"index_delayed_jobs_for_get_next"})
   -> 0.0080s
-- add_index(:delayed_jobs, ["strand", "id"], {:name=>"index_delayed_jobs_on_strand"})
rake aborted!
An error has occurred, this and all later migrations canceled:

PG::Error: ERROR:  relation "index_delayed_jobs_on_strand" already exists
: CREATE  INDEX "index_delayed_jobs_on_strand" ON "delayed_jobs" ("strand", "id")/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:227:in `rescue in log'

I worked around this error by dropping the entire table and re-creating it (not a problem since I don't have any data yet):

--- a/db/migrate/20110610213249_optimize_delayed_jobs.rb
+++ b/db/migrate/20110610213249_optimize_delayed_jobs.rb
@@ -25,8 +25,30 @@ class OptimizeDelayedJobs < ActiveRecord::Migration
     remove_index :delayed_jobs, :name => 'get_delayed_jobs_index'
     remove_index :delayed_jobs, [:strand]
 
+    drop_table :delayed_jobs
+    create_table :delayed_jobs, :force => true do |t|
+      t.integer  "priority",                       :default => 0
+      t.integer  "attempts",                       :default => 0
+      t.string   "handler",      :limit => 512000
+      t.text     "last_error"
+      t.string   "queue"
+      t.datetime "run_at"
+      t.datetime "locked_at"
+      t.datetime "failed_at"
+      t.string   "locked_by"
+      t.datetime "created_at"
+      t.datetime "updated_at"
+      t.string   "tag"
+      t.integer  "max_attempts"
+      t.string   "strand"
+    end
+
     add_index :delayed_jobs, %w(run_at queue locked_at strand priority), :name => 'index_delayed_jobs_for_get_next'
+    add_index "delayed_jobs", ["priority", "run_at", "failed_at", "locked_at", "queue"], :name => "get_delayed_jobs_index"
+    add_index "delayed_jobs", ["priority", "run_at"], :name => "delayed_jobs_priority"
+    add_index "delayed_jobs", ["queue"], :name => "delayed_jobs_queue"
     add_index :delayed_jobs, %w(strand id), :name => 'index_delayed_jobs_on_strand'
+    add_index "delayed_jobs", ["tag"], :name => "index_delayed_jobs_on_tag"
 
     # move all failed jobs to the new failed table
     Delayed::Backend::ActiveRecord::Job.where("failed_at IS NOT NULL").find_each do |job|

But I got a similar error when it got to the *next* migration with a `remove_index` followed by a `add_index` for the same index name:

> bundle exec rake db:initial_setup
…
==  AddDelayedJobsNextInStrand: migrating =====================================
-- remove_index(:delayed_jobs, {:name=>"index_delayed_jobs_for_get_next"})
   -> 0.0019s
-- add_column(:delayed_jobs, :next_in_strand, :boolean, {:default=>true, :null=>false})
   -> 0.0030s
rake aborted!
An error has occurred, this and all later migrations canceled:

PG::Error: ERROR:  relation "get_delayed_jobs_index" already exists
: CREATE INDEX get_delayed_jobs_index ON delayed_jobs (priority, run_at) WHERE locked_at IS NULL AND queue = 'canvas_queue' AND next_in_strand = 't'/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:227:in `rescue in log'


Tried this workaround (from http://stackoverflow.com/questions/1395672/how-do-i-run-a-migration-without-starting-a-transaction-in-rails):

--- a/db/migrate/20110831210257_add_delayed_jobs_next_in_strand.rb
+++ b/db/migrate/20110831210257_add_delayed_jobs_next_in_strand.rb
@@ -3,6 +3,11 @@ class AddDelayedJobsNextInStrand < ActiveRecord::Migration
     Delayed::Backend::ActiveRecord::Job.connection
   end
 
+  def ddl_transaction(&block)
+    puts 'Tried to run ddl_transaction'
+    block.call
+  end
+
   def self.up
     raise("#{connection.adapter_name} is not supported for delayed jobs queue") unless %w(PostgreSQL MySQL Mysql2 SQLite).include?(connection.adapter_name)
 

I'm not sure if it had any effect since I didn't see the output but I got a different result nonetheless:

> bundle exec rake db:initial_setup
==  AddDelayedJobsNextInStrand: migrating =====================================
-- remove_index(:delayed_jobs, {:name=>"index_delayed_jobs_for_get_next"})
   -> 0.0023s
-- add_column(:delayed_jobs, :next_in_strand, :boolean, {:default=>true, :null=>false})
rake aborted!
An error has occurred, this and all later migrations canceled:

PG::Error: ERROR:  column "next_in_strand" of relation "delayed_jobs" already exists
: ALTER TABLE "delayed_jobs" ADD COLUMN "next_in_strand" boolean/home/tyler/.rvm/gems/ruby-1.9.3-p545/bundler/gems/rails-e86daf8ff727/activerecord/lib/active_record/connection_adapters/abstract_adapter.rb:227:in `rescue in log'

> psql canvas_queue_development
psql (9.1.11)
Type "help" for help.

canvas_queue_development=# \d delayed_jobs
                                       Table "public.delayed_jobs"
     Column     |            Type             |                         Modifiers                         
----------------+-----------------------------+-----------------------------------------------------------
 id             | bigint                      | not null default nextval('delayed_jobs_id_seq'::regclass)
 priority       | integer                     | default 0
 attempts       | integer                     | default 0
 handler        | character varying(512000)   | 
 last_error     | text                        | 
 queue          | character varying(255)      | 
 run_at         | timestamp without time zone | 
 locked_at      | timestamp without time zone | 
 failed_at      | timestamp without time zone | 
 locked_by      | character varying(255)      | 
 created_at     | timestamp without time zone | 
 updated_at     | timestamp without time zone | 
 tag            | character varying(255)      | 
 max_attempts   | integer                     | 
 strand         | character varying(255)      | 
 next_in_strand | boolean                     | not null default true
Indexes:
    "delayed_jobs_pkey" PRIMARY KEY, btree (id)
    "delayed_jobs_priority" btree (priority, run_at)
    "delayed_jobs_queue" btree (queue)
    "get_delayed_jobs_index" btree (priority, run_at, failed_at, locked_at, queue)
    "index_delayed_jobs_for_get_next" btree (run_at, queue, locked_at, strand, priority)
    "index_delayed_jobs_on_strand" btree (strand, id)
    "index_delayed_jobs_on_tag" btree (tag)

Discovered the `transactional` option in config/initializers/active_record.rb and referenced by https://github.com/instructure/canvas-lms/commit/41279b925192402df4214ea6f595e6280100bae0.

When I tried this:

class AddDelayedJobsNextInStrand < ActiveRecord::Migration
  def self.connection
    Delayed::Backend::ActiveRecord::Job.connection
  end

  self.transactional = false

I got this:

> -- transactional=(false)
rake aborted!
undefined method `transactional=' for #<ActiveRecord::ConnectionAdapters::PostgreSQLAdapter:0x000000125350b8>


Trying this now:

--- a/db/migrate/20110831210257_add_delayed_jobs_next_in_strand.rb
+++ b/db/migrate/20110831210257_add_delayed_jobs_next_in_strand.rb
@@ -3,12 +3,19 @@ class AddDelayedJobsNextInStrand < ActiveRecord::Migration
     Delayed::Backend::ActiveRecord::Job.connection
   end
 
+  class << self
+    def transactional?
+      false
+    end
+  end
+
   def self.up
     raise("#{connection.adapter_name} is not supported for delayed jobs queue") unless %w(PostgreSQL MySQL Mysql2 SQLite).include?(connection.adapter_name)
 
     remove_index :delayed_jobs, :name => 'index_delayed_jobs_for_get_next'
+    remove_index :delayed_jobs, :name => 'get_delayed_jobs_index'
 
-    add_column :delayed_jobs, :next_in_strand, :boolean, :default => true, :null => false
+    #add_column :delayed_jobs, :next_in_strand, :boolean, :default => true, :null => false
 
     # create the new index
     case connection.adapter_name


> bundle exec rake db:initial_setup 
==  AddDelayedJobsNextInStrand: migrating =====================================
-- remove_index(:delayed_jobs, {:name=>"index_delayed_jobs_for_get_next"})
   -> 0.0047s
-- remove_index(:delayed_jobs, {:name=>"get_delayed_jobs_index"})
   -> 0.0014s
rake aborted!
An error has occurred, all later migrations canceled:

PG::Error: ERROR:  relation "get_delayed_jobs_index" already exists
: CREATE INDEX get_delayed_jobs_index ON delayed_jobs (priority, run_at) WHERE locked_at IS NULL AND queue = 'canvas_queue' AND next_in_strand = 't'


It looks like the remove_index just isn't having any effect!

I guess I'll try just executing the DDL directly:

--- a/db/migrate/20110831210257_add_delayed_jobs_next_in_strand.rb
+++ b/db/migrate/20110831210257_add_delayed_jobs_next_in_strand.rb
@@ -7,8 +7,10 @@ class AddDelayedJobsNextInStrand < ActiveRecord::Migration
     raise("#{connection.adapter_name} is not supported for delayed jobs queue") unless %w(PostgreSQL MySQL Mysql2 SQLite).include?(connection.adapter_name)
 
     remove_index :delayed_jobs, :name => 'index_delayed_jobs_for_get_next'
+    remove_index :delayed_jobs, :name => 'get_delayed_jobs_index'
+    connection.execute("DROP INDEX get_delayed_jobs_index")
 
-    add_column :delayed_jobs, :next_in_strand, :boolean, :default => true, :null => false
+    #add_column :delayed_jobs, :next_in_strand, :boolean, :default => true, :null => false
 
     # create the new index
     case connection.adapter_name

That worked...

> bundle exec rake db:initial_setup 
==  AddDelayedJobsNextInStrand: migrating =====================================
-- remove_index(:delayed_jobs, {:name=>"index_delayed_jobs_for_get_next"})
   -> 0.0049s
-- remove_index(:delayed_jobs, {:name=>"get_delayed_jobs_index"})
   -> 0.0015s
-- execute("      CREATE FUNCTION delayed_jobs_before_insert_row_tr_fn () RETURNS trigger AS $$\n      BEGIN\n        LOCK delayed_jobs IN SHARE ROW EXCLUSIVE MODE;\n        IF (SELECT 1 FROM delayed_jobs WHERE strand = NEW.strand LIMIT 1) = 1 THEN\n          NEW.next_in_strand := 'f';\n        END IF;\n        RETURN NEW;\n      END;\n      $$ LANGUAGE plpgsql;\n")
   -> 0.0050s
-- execute("CREATE TRIGGER delayed_jobs_before_insert_row_tr BEFORE INSERT ON delayed_jobs FOR EACH ROW WHEN (NEW.strand IS NOT NULL) EXECUTE PROCEDURE delayed_jobs_before_insert_row_tr_fn()")
   -> 0.0029s
-- execute("      CREATE FUNCTION delayed_jobs_after_delete_row_tr_fn () RETURNS trigger AS $$\n      BEGIN\n        UPDATE delayed_jobs SET next_in_strand = 't' WHERE id = (SELECT id FROM delayed_jobs j2 WHERE j2.strand = OLD.strand ORDER BY j2.strand, j2.id ASC LIMIT 1);\n        RETURN OLD;\n      END;\n      $$ LANGUAGE plpgsql;\n")
   -> 0.0006s
-- execute("CREATE TRIGGER delayed_jobs_after_delete_row_tr AFTER DELETE ON delayed_jobs FOR EACH ROW WHEN (OLD.strand IS NOT NULL AND OLD.next_in_strand = 't') EXECUTE PROCEDURE delayed_jobs_after_delete_row_tr_fn()")
   -> 0.0007s
-- execute("UPDATE delayed_jobs SET next_in_strand = 'f' WHERE strand IS NOT NULL AND id <> (SELECT id FROM delayed_jobs j2 WHERE j2.strand = delayed_jobs.strand ORDER BY j2.strand, j2.id ASC LIMIT 1)")
   -> 0.0023s
==  AddDelayedJobsNextInStrand: migrated (0.0364s) ============================