RubyWorks Production Stack on Amazon EC2
George Malamidis, November 20th, 2007rubyworks-ec2 automates deploying Ruby On Rails applications to an Amazon Elastic Compute Cloud (EC2) instance using the RubyWorks Production Stack. It is a collection of Capistrano 2.0 recipes and utilities, not an AMI.
Using rubyworks-ec2 will effectively bootstrap a clean Debian Etch AMI, installing the RubyWorks Production Stack, Apache 2, MySQL, utilities and cron jobs for backing up and restoring the database from Amazon S3.
The running stack comprises of Apache 2 serving static content, HAProxy load balancing Mongrels, 4 Mongrel instances, monit monitoring HAProxy, Mongrel and the OS and MySQL with its datadir pointing to the /mnt partition of the AMI.
Deploying to a new AMI instance post first time setup is achieved with 3 commands – cap instance:start, cap instance:bootstrap and cap deploy – or 4, in case there’s a need to migrate the database.
Prerequisites
A working Amazon EC2 account, based on the instructions described in the EC2 Getting Started Guide and a working Rails application that’s setup to use MySQL in Production mode, has Rails and any required gems unpacked in its vendor directory and is versioned in a Subversion repository.
Getting started
Following the EC2 Getting Started Guide, you should have a directory named .ec2 in your home directory containing a PEM encoded signed X.509 certificate and an unencrypted, PEM encoded RSA private key that corresponds to the X.509 certificate.
You should also have an RSA keypair named something similar to id_rsa-gsg-keypair. Copy this file to the $HOME/.ec2/ directory. If you have the public counterpart to this key – in this case id_rsa-gsg-keypair.pub – copy it to $HOME/.ec2 as well. If you don’t, we’ll get it once we start the AMI. This step is crucial in order for Capistrano to be able to connect to the instance.
Download the rubyworks-ec2 gem and install it:
sudo gem i rubyworks-ec2-<version>.gem
Invoke capify . from the top directory of the Rails application. This will create two files, Capfile and config/deploy.rb.
Require rubyworks-ec2 in Capfile:
require "rubyworks-ec2"
Next, configure the required deployment properties in config/deploy.rb:
set :instance_id, "" set :instance_url, "" set :application, "shotgun_blues" set :repository, "http://svn.shotgun.com/blues/trunk" role :app, instance_url role :web, instance_url role :db, instance_url, :primary => true set :keypair, "gsp-keypair" set :account_id, "123456789098" set :access_key_id, "ABCDE123456789" set :secret_access_key, "323848492AHSBCYEBDNCSCUENCCKS" set :pk, "pk-323848492AHSBCYEBDNCSCUENCCKS.pem" set :cert, "cert-323848492AHSBCYEBDNCSCUENCCKS.pem" set :packages, %w(apache2 subversion mysql-server libmysql-ruby less) # plus any additional packages you'd like to install on the image set :gems, %w(aws-s3 ezcrypto) #plus any additional gems you'd like to install on the instance
We’ll set the instance_id and instance_url properties after we start an instance. The value for the keypair property must be the name of the RSA keypair in $HOME/.ec2. So, if your keypair file is named id_rsa-gsp-keypair, the keypair property value should be gsp-keypair. The account_id, access_key_id and secret_access_key are your AWS ACCOUNT ID, ACCESS KEY ID and SECRET ACCESS KEY. pk and cert are the filenames of the PEM certificate and key in $HOME/.ec2.
If your Subversion repository is using HTTP Basic authentication, add the following two lines in config/deploy.rb:
set :scm_username, 'svnusername' set :scm_password, 'svnpassword'
Deploy the application
Start a Debian Etch AMI:
cap instance:start
Wait a few moments and invoke ec2-describe-instances. Once your instance has been started, the output of the ec2-describe-instances command will provide the instance id and instance url. Set these values in config/deploy.rb:
set :instance_id, "i-sd92adsd" set :instance_url, "ec2-67-202-1-72.z-2.compute-1.amazonaws.com"
If you don’t have a copy of your public RSA key, invoke:
cap instance:cp_public_key
This will copy you public RSA key from the instance to your $HOME/.ec2 directory.
cap instance:bootstrap
This command will take a while to complete. It effectively installs RubyWorks, the AWS::S3 library, copies your amazon keys, database backup and restore utitilies, installs Apache 2, sets up an Apache virtual host, installs Subversion and MySQL on the instance. It sets up a cron job for backing up the Database on Amazon S3 every 40 minutes and restores the database to the latest version found on S3, if any exist, otherwise creates a database named <application>_production where application is the value of the application property in config/deploy.rb.
After the command completes, you should be able to navigate to the instance’s URL from a browser and see the Rails welcome page.
Proceed by invoking cap deploy and, if this is the first time you’re deploying your application, you might want to do a cap deploy:migrate.
To perform a remote login to the instance, invoke:
cap instance:ssh
To back up the database to S3:
cap db:backup
To backup the image to S3 (bundle, upload and register), invoke:
cap image:backup
To monitor services running on the instance:
cap monit:status
To stop the instance:
cap instance:stop
Acknowledgments
rubyworks-ec2 uses Paul Morris’s public Debian Etch AMI (ami-30f11459).
License
rubyworks-ec2 is copyright © 2007 nutrun.com. rubyworks-ec2 is Open Source Software – LICENSE
RubyWorks and the RubyWorks Production Stack are © ThoughtWorks inc.
November 23rd, 2007 at 1:31 pm
This is awesome! Thanks, George! :D
November 24th, 2007 at 1:22 pm
Just what I needed, just at the right time!
Thanks George!
-Prasanna
November 24th, 2007 at 8:21 pm
[...] Nutrun » Blog Archive » RubyWorks Production Stack on Amazon EC2 Web programming, design and Guitar Tone. (tags: aws capistrano deployment ec2 rails ruby) [...]
November 25th, 2007 at 12:03 am
Great work! This worked well for me. This project is like deprec for ec2. Any chance of merging in some deprec functionality? Like configuring mongrel in the recipe and setting up multiple apps on an instance?
In the future it’ll be awesome to see specialized recipes, i.e: one to create a load balancing instance (using HAProxy or Swiftiply), one to create an app server instance, one for master/slave DB’s, etc. Let me know if you need any contributers.
November 26th, 2007 at 12:38 am
[...] Nutrun » Blog Archive » RubyWorks Production Stack on Amazon EC2 (tags: ec2 rubyworks) [...]
December 1st, 2007 at 10:19 am
Nutrun » Blog Archive » RubyWorks Production Stack on Amazon EC2…
[...][...]…
January 18th, 2008 at 3:45 am
Finally! I’ve been at it with ec2onrails for days! This seems to work so far. I’m well past the stupid authentication problems I was having with ec2onrails. Cheers!
January 21st, 2008 at 10:34 pm
Does this have imagemagick installed in it? Is the imagemagick > 6.3.0 ?
Thanks
January 21st, 2008 at 11:55 pm
No imagemagick or rmagick. In general, installing specific libraries is up to the user.
January 27th, 2008 at 5:13 pm
[...] Amazon EC2 (Elastic Compute Cloud) Control Libraryamazon-ec2 is a super slick library that makes it super-easy to control Amazon EC2 instances in Ruby code. It also comes with a special shell “ec2sh” that gives you a much nicer (in my opinion) interface to control and manipulate EC2 instances than the usual command line tools provided by Amazon. The documentation for this is superb with examples of using all of the various methods it provides, ec2sh, and examples of EC2 control from Ruby and Rails apps.RubyWorks Production Stack on Amazon EC2Continuing with the Amazon EC2 theme, Nutrun posts about rubyworks-ec2, a set of Capistrano recipes and utilities to deploy the Rubyworks Production Stack (a complete Ruby and Rails stack) on Amazon EC2 instances. The stack includes Apache 2, HAProxy load balancer, Mongrel, monit, and a bunch of other useful tools. If you want a way to get a Rails / Ruby stack running on an EC2 instance in minutes, this is essential reading.Simple Geocoding in RubyThe Cartographer library is an old-time favorite for doing geocoding from Rails applications. A developer with Assay Depot, however, decided that a more direct approach of querying Google for the results was necessary. The result is a 49 line module that can return latitude, longitude, address, street, and other geographical information when provided with, say, a ZIP or a street address.Tutorial for Installing and Configuring Nginx and Rails on UbuntuJames O’Kelly has put together a comprehensive tutorial going through all of the stages necessary to install and configure Nginx and Rails together to run applications on an Ubuntu server. James’ blog RailsJitsu.com is definitely worth a look (and perhaps to subscribe to!) as he seems to have a knack for regularly putting together good Rails (and especially Mephisto) focused posts.Presentation: Haml and Sass in 15 MinutesHaml is a markup language commonly used by Rails developers that makes it easy to produce well-formatted, valid XHTML in as few lines as possible. Sass is similar, but for CSS (supporting nested rules, referencing parent rules, and lots of other time saving goodness). In “Haml and Sass in 15 Minutes“, Patrick Crowley guides us through using these two technologies. [...]
February 7th, 2008 at 9:37 am
Hey George this is awesome, got me up and running in no time at all..
\m/>_
March 21st, 2008 at 9:50 pm
Hi, this looks great!
However, when trying to follow your instructions, bootstrapping failed with the error:
* executing `instance:install_packages’
/opt/local/lib/ruby/gems/1.8/gems/capistrano-2.1.0/lib/capistrano/configuration/namespaces.rb:187:in `method_missing’: undefined local variable or method `packages’ for # (NameError)
After looking in the source i figured that I needed to add the following lines to deploy.rb:
set :packages, %w(apache2 subversion mysql-server libmysql-ruby less)
set :gems, %w(aws-s3 ezcrypto)
After that it works. You might include the lines in the instructions on this page. I didn’t use capify because I’m migrating from a previous setup, so I didn’t get the two lines automatically, and was puzzled at first.
March 22nd, 2008 at 12:40 pm
I successfully run:
cap instance:start
(copy instance id and address to deploy.rb)
cap deploy
cap deploy:migrate
So far so good. But when I point my browser at the server, I’m just redirected to /apache2-default/, and get a white page with the words “It Works!”, nothing else. If I run cap monit:status, I can see that the four mongrels are running. log/production.log is empty except for the line “# Logfile created on Sat Mar 22 12:12:23 +0000 2008″.
What could be wrong, and how I can fix it? Is there a forum for rubyworks-ec2? Thanks!
March 22nd, 2008 at 4:21 pm
(feel free to moderate my flood of comments :-) )
Ok, the default page i described above was caused by lack of the config/server folder. Some problem as with the package list, I didn’t use capify, so they were never created in my local app folder. Once I got that folder created and added to SVN, things worked much better.
But now I have a different problem – during bootstrapping, the command ‘etc/init.d/mysql start’ fails:
**[out :: ...... ] Starting MySQL database server: mysqld . . . . . . . . . . failed!
Never the less, it seems that mysql is in fact running. If I perform the rest of the tasks (performed by bootstrapping) manually, my app is working fine, and can be viewed in my browser.
March 26th, 2008 at 4:27 pm
Emil,
Thanks for your comments. I’ve updated the article to include the missing “set” directives. I think it’s worthwhile to note that the bootstrap task will not “rollback”, that is, if any of the tasks fail, whichever tasks ran before the failure will still take effect.
May 8th, 2008 at 4:39 pm
When I run instance:bootstrap I receive the following error
* executing `deploy:setup’
* executing “umask 02 && mkdir -p /usr/team_maker /usr/team_maker/releases /usr/team_maker/shared /usr/team_maker/shared/system /usr/team_maker/shared/log /usr/team_maker/shared/pids”
servers: ["ec2-75-101-218-41.compute-1.amazonaws.com"]
[ec2-75-101-218-41.compute-1.amazonaws.com] executing command
*** [err :: ec2-75-101-218-41.compute-1.amazonaws.com] sudo: no passwd entry for app!
command finished
command “umask 02 && mkdir -p /usr/team_maker /usr/team_maker/releases /usr/team_maker/shared /usr/team_maker/shared/system /usr/team_maker/shared/log /usr/team_maker/shared/pids” failed on ec2-75-101-218-41.compute-1.amazonaws.com
Poking around in the instance, it seems that there is no ‘app’ user.
May 8th, 2008 at 4:46 pm
I killed the instance and restarted this time with this at the bottom of my Capfile
set :runner, :root
Has gotten me past this error. Not sure if deploying this as the root user is the right thing to do or not.
May 8th, 2008 at 8:41 pm
I ran capify but did not get the config/server folder. Poking around in rubyworks svn I found this script, which generated the Capfile,deploy.rb and server
capify-for-ec2
June 2nd, 2008 at 3:20 pm
This is a great write-up, thanks! It got me close to up-and-running pretty fast. Like commenter #8 above I got hung up on a missing imagemagick install. I can do this manually on the instnace after running cap instance:start, but are there hooks for me to add these steps to my ‘instance:start’ cap target?
Also I needed to run these commands as well before I could ssh or http to my instance fyi. You might want to add this instruction to the post (or to the gem actually):
ec2-authorize default -p 22
ec2-authorize default -p 80
August 20th, 2008 at 3:57 pm
[...] It is assumed that your environment has been previously configured for launching EC2 AMIs. If not, you might want to read the EC2 Getting Started Guide, or refer to the first bits of this article. [...]