Wednesday, August 6, 2008

Selenium Grid: Configuring Machines

I posted a couple of days ago about trying to get Selenium Grid to work. I've made some progress and wanted to get it down in case there's someone else out there trying to do what I'm trying to do. Most of this is sourced from elsewhere on the web - thanks to everyone who wrote a blog post, instruction set, document!

Goal
I have several Selenium tests that currently run in a single environment on a single browser. My application is Ruby on Rails. I'd like to make them run on all the browsers the product claims to support: Windows (IE, Firefox) and OS X (Safari, Firefox). Once this is basically functional we'll add the nice to haves: running in parallel, running multiple projects (each of which has its own tests), and some form of unified reporting.

Setup
I'm developing on OS X (Leopard). The build system is Linux. I'm using VMWare to run Windows clients. If you're on other platforms, well, I hope this at least points you in the right direction!

I'm assuming you've got Selenium tests running locally already.

Grid Pieces
Selenium Grid consists of several pieces:
  • Remote Controls: These are the things that actually run the tests. If you're currently using Selenium RC, you're using this.
  • Hub: This is the thing that is aware of the available remote controls and picks one to run a given test. The Hub knows a few things about the remote controls: the type (*chrome, *iehta, "Firefox on Windows"), the port, and the URL. It picks which one to use based on type.
  • Selenium tests: These are the tests themselves. They're not actually part of the Grid, but it's useful to consider.
Configuring The Hub
First thing's first. You have to configure your Hub. To do this:
  1. Go get Selenium Grid.
  2. Install Selenium Grid. I put it in /Applications/selenium-grid.
  3. Configure the grid and make sure it builds. I used these instructions. Note that you need to have Java and Ant.
  4. Launch the hub by CDing into /Applications/selenium-grid and writing "ant launch-hub".
At this point, you should be able to point your browser to http://localhost:4444/console and see the Hub running. You'll see a list of available environments. That's controlled from a configuration file: /Applications/selenium-grid/grid_configuration.yml. Feel free to modify the file to add new environments or delete environments you don't need. Just restart the Hub when you're done.

Be careful here. At the time I wrote this, Firefox 3 on Windows just didn't work, and Safari 3 didn't work reliably.

Configuring Each Remote Control
Each remote control is essentially standalone. I think of these as basically standing around and waiting for the Hub to give them something to do. Repeat this for each remote control you want to configure:
  1. Download Selenium RC
  2. Download Selenium Grid. I'm not completely convinced this is necessary, but I didn't get it working without it, and if it just sits there, well, no harm.
  3. Install Selenium RC. I used the instructions here.
  4. Configure Selenium RC to use the default port (3000). Be sure it's actually running.
Pointing the Hub At a Remote Control
Once you have a remote control ready and running you can point your Hub at it.
  1. On the machine running the Hub, leave the Hub running.
  2. Open a terminal, cd into /Applications/selenium-grid
  3. Start the remote control from the hub with: ant -Dport=5556 -Denvironment="*chrome" -Dhost=10.0.1.197 -DhubURL=http://10.0.1.206:4444 launch-remote-control.
For each remote control, set the port to something different; you can't have two remote controls using the same port.
  • -Dport is the port that the Hub will use to talk to the remote control. This must be unique.
  • -Denvironment is the environment you'll be using. Set it to something in the available environments list (check the Hub console in the browser).
  • -Dhost is the IP address (or name) of the machine that is running the remote control.
  • -DhubURL is the full URL (including http:// and the port) of the machine that's running the Hub
Run the Test
Now we get to the part where we actually use this thing we've created. We're in Rails, so we're going to do this as a rake task. Here's the actual rake task file that we'll put in lib/tasks:
================================
require(File.join(File.dirname(__FILE__), '../..', 'config', 'boot'))
require 'rake'
require 'rake/testtask'
require 'rake/rdoctask'
require 'tasks/rails'

@osx_browsers = ["*firefox", "*chrome"] #should be safari instead of firefox, but safari doesn't work in the grid yet
@win_browsers = []
@platforms = {"win"=>"Windows", "osx"=>"OS X", "linux"=>"Linux"}

namespace :test do

task :run_browsers do
unless ENV.include?("platform") && @platforms.has_key?(ENV['platform'])
raise "A platform is required! usage: rake test:run_all_environments platform=[win|osx]"
else
platform = ENV['platform']
case platform
when 'win' then
browsers = @win_browsers
when 'osx' then
browsers = @osx_browsers
end

puts "Running tests for the #{@platforms[platform]} platform"
browsers.each do |browser|
puts "Running tests in the #{browser} browser"
ENV["Browser"] = browser
ENV["TEST_DIR"] = 'test/selenium'
Rake::Task["test:selenium"].invoke #run all the selenium tests within the grid context
end
end
end

task :sanity_check do
verify "You must install RubyGems to run this example : http://rubygems.org" do
require "rubygems"
end
end

end

def verify(error_message)
begin
yield
rescue Exception
STDERR.puts <<-EOS ******************************************************************* #{error_message}  ******************************************************************* EOS raise    end end

==============================

To call the rake task, just:
  1. start your Rails server (script/server) 
  2. Run rake test:run_browsers platform=osx
If you've done everything right, then you'll see your tests start to run on each of the browsers you've defined.

Let's look at what this rakefile does:
  • Defines each of the browsers on each platform (OS type). This is the same as the name you put in -Denvironment. Make sure you have a remote control set up for each environment.
  • The sanity check probably isn't strictly necessary, but why take chances?
  • You'll notice that I'm calling another rake task. This is whatever task you've defined to run your Selenium tests. I find this easier than calling them separately; you can keep up with any special config you're doing that way.
Conclusion
And that's basically it. I hope this helps!

29 comments:

  1. I'm a bit confused here - you haven't made use of the deep_test stuff in selenium-grid/examples/ruby/ so ... where does your test:selenium task come from?

    Maybe you're also using polonium?

    ReplyDelete
  2. The test:selenium task comes from the project under test - for this project I inherited it. How it got there was before my time!

    No polonium usage here. I played with the project but it didn't really provide anything that I needed at the time.

    ReplyDelete
  3. I am confused. Why are you starting the RC server from the hub machine? Aren't you supposed to start the hub and then run launch-remote-control from the RC machine to actually start RC and register with the hub?

    ReplyDelete
  4. The RC server for each client is, in the end, started from the remote control machine. I control the start from the hub machine just because I happen to be using that machine to control the entire test suite. So I wind up on the hub machine calling a script on the remote machine to start it up. I'd suggest you check out my later post in this same series, by which time I'd made the process a bit cleaner.

    http://blog.abakas.com/2009/01/selenium-saga-part-iii.html

    Good luck, and feel free to comment again if you have more questions.

    Catherine

    ReplyDelete
  5. Hi,

    Just started looking into Selenium Grid. Can it work with testsuit recorded in JSP. All my test are recorded in JSP.

    Thanks,
    Hitesh

    ReplyDelete
  6. Hitesh,

    I'm not sure what you mean by "recorded in JSP". Selenium Grid basically runs whatever Selenium RC runs. If you can explain a bit more about your test infrastructure maybe I can help more.

    Catherine

    ReplyDelete
  7. Hi Catherine,

    First, thanks for all of the great work you've done with this blog. Especially the selenium info! Hopefully I'll be taking full advantage of all this info soon.

    I've recently set up Selenium RC and am in the process of setting up Selenium Grid. However, when I try to run the test RC cannot find firefox-bin. I've tried adding the bin to the path, and adding the path to *firefox3 in grid_configuration.yml, but nothing seems to work. Do you have any ideas?

    ReplyDelete
  8. Anonymous Mar 18:

    Firefox 3 support requires a newer Selenium than I was using when I wrote this. I would make sure to download Selenium RC 1.0 beta 2 or later. With that, it should just work like firefox 2 does.

    To test this out, you'll probably want to remove your custom config in grid_configuration.yml and go with a more standard firefox config.

    ReplyDelete
  9. Hi Catherine,

    Here are the details for the problem I am facing.

    1. I actually record all tests using Selenium IDE and store them as a JSP files.
    2. Then I have tesuit.html file with is actually list of the all the tests recorded.
    3. And now I run my complete testsuit using Selenium RC html suit with the following command.
    java -jar selenium-server.jar -htmlSuite "*iexplore" "http://www.google.com" "C:\Testsuit.html" "C:\result.html"

    Now I want to run the same thing with Selenium Grid. But not able to find any option there.

    Thanks,
    Hitesh

    ReplyDelete
  10. Hitesh,

    I haven't worked with the html suite stuff much, but if I recall correctly it was kind of a "starter option" that was easy but had some limitations and was generally pretty picky about how you ran it.

    There are two options that spring to mind:
    1. put the call to the suite in an ant task and call the ant task (where I have Rake::Task["test:selenium"]. There's probably a little more tweaking to be done, but that should be close.

    It should look something like the one on this page: http://seleniumhq.org/projects/remote-control/languages/selenese.html
    2. convert your html tests to something more robust - Java, Ruby, whatever. You can use the convertor supplied with Selenium IDE.

    In either case, as you move to using Grid, you're probably going to want to make your tests more network-oriented. In particular, c:/xx is probably going to need to become a network share (e.g., Z:/xx or \\10.0.0.2\xx); otherwise you'll have a nightmare on your hands trying to parse results.

    ReplyDelete
  11. Hello,

    I'm using Selenium Grid on a J2EE project and i find it wonderful.
    But I still have a problem to increase the number of active remote controls.
    I actually have change from 4 to 10 Available RC but the Active RCs are always 4.
    Is there a way to increase the Active RCs.

    Thank you !

    ReplyDelete
  12. Anon April 29:
    If I understand correctly, you have 10 available RCs (that is, 10 different clients). They're all running fine and none of them are leftovers from previous runs that had errors.

    You want to be able to use all 10 RCs at the same time. As far as I know, this should simply work, as long as you actually start tests pointing at each of the 10 RCs.

    There are some things that I would check:
    1. make sure you're trying to use configurations you have available. For example, if you have 10 clients all trying to use *firefox and you have 6 RCs available with *iehta, you won't use any of them.
    2. make sure you're not running out of memory or other computing resources. Are you simply overwhelming your hub machine?
    3. make sure you really can use each of the 10 available RCs by making them each unique and addressing them directly.
    4. make sure you're actually running all 10 clients at once (that is, all 10 things that call the RCs)

    If you're still having issues, please feel free to zip up your configs (hub, RCs and client calling code) and send it over so I can take a look.

    ReplyDelete
  13. Hi, thanks for the information you post, I new with selenium grid and I want to run my ruby scripts on it. So, for instance if I have the following file Googlesearch.rb file that is my testcase, where should I put that name in the rake file to start the test using selenium gird :S......could you explaime how to do that.

    thank you

    ReplyDelete
  14. Anonymous Mar 12 2010:
    There are a couple of things to consider here:
    1. are you using Rails or no? If you're using Rails, what you want to do is get your test to run when you run rake test. That way you
    2. If you're not using rake or any of the helper test functions, then you can try calling the code directly.

    In the example above, the actual call to the tests is:
    Rake::Task["test:selenium"].invoke

    In either case, I would make sure your test runs against a local selenium first. Then layer in the Grid usage.

    ReplyDelete
  15. thanks Catherine, Yes I'm able to run my script using Selenium RC, I just export the html script to Ruby (Test / Unit), and to execute the script I just type ruby myscript.rb and it starts using selenium RC.

    So, I want o run this script in a parallel way using selenium Grid on multiple browsers. But I don't know where should I put the
    script, and how should I call my script to be executed through selenium Grid.

    Could you explain me what does this mean?

    Rake::Task["test:selenium"].invoke


    Thanks Catherine and let me know if you need more info, I'm frustrated because I couldn't find a solution for this

    ReplyDelete
  16. Moises,

    That line simply calls a rake task that runs all tests in the Selenium folder. If these terms aren't familiar to you, you're probably not using them!

    Basically the only difference between running in Selenium RC vs running in Selenium Grid is that you have to point at the Grid instead of the RC instance. It's the details that take a little tweaking to get that to happen.

    It sounds like you're just using plain Ruby, which is fine. It's not going to run quite the same way as this setup, but I'm sure we can figure it out. Is there any way you can send me your code and I'll take a look? Also, send me the name of the gems you're using, and we'll get it working. Feel free to send it to me at catherine at .

    Good luck!

    ReplyDelete
  17. Hi Catherine, This is the Gem list

    *** LOCAL GEMS ***

    actionmailer (2.3.5, 2.2.2, 1.3.6)
    actionpack (2.3.5, 2.2.2, 1.13.6)
    actionwebservice (1.2.6)
    activerecord (2.3.5, 2.2.2, 1.15.6)
    activeresource (2.3.5, 2.2.2)
    activesupport (2.3.5, 2.2.2, 1.4.4)
    acts_as_ferret (0.4.3)
    builder (2.1.2)
    capistrano (2.5.2)
    cgi_multipart_eof_fix (2.5.0)
    commonwatir (1.6.5)
    daemons (1.0.10)
    deep_test (1.2.2)
    dnssd (0.6.0)
    fastthread (1.0.1)
    fcgi (0.8.7)
    ferret (0.11.6)
    firewatir (1.6.5)
    gem_plugin (0.2.3)
    gemcutter (0.5.0)
    highline (1.5.0)
    hoe (2.5.0)
    hpricot (0.6.164)
    json_pure (1.2.0)
    libxml-ruby (1.1.2)
    mongrel (1.1.5)
    needle (1.3.0)
    net-scp (1.0.1)
    net-sftp (2.0.1, 1.1.1)
    net-ssh (2.0.4, 1.1.4)
    net-ssh-gateway (1.0.0)
    rack (1.0.1)
    rails (2.3.5, 2.2.2, 1.2.6)
    rake (0.8.7, 0.8.3)
    RedCloth (4.1.1)
    rspec (1.3.0, 1.1.8)
    ruby-ole (1.2.10)
    ruby-openid (2.1.2)
    ruby-yadis (0.3.4)
    rubyforge (2.0.3)
    rubygems-update (1.3.6)
    rubynode (0.1.5)
    s4t-utils (1.0.4)
    safariwatir (0.3.7)
    scruffy (0.2.6)
    selenium-client (1.2.18)
    spec_ui (0.2.4)
    spreadsheet (0.6.4.1)
    spreadsheet-excel (0.3.5.1)
    sqlite3-ruby (1.2.4)
    termios (0.9.4)
    user-choices (1.1.6.1)
    xml-simple (1.0.12)
    xmpp4r (0.4)


    and my script is too simple, just to test how to use selenium grid, after that I will try with my other scripts.


    require "test/unit"
    require "rubygems"
    gem "selenium-client"
    require "selenium/client"

    class Vuelos < Test::Unit::TestCase

    $usuario = ARGV[0]
    $ports = ARGV[1]

    def setup
    puts "Usuario: #{$usuario} : #{$ports}"
    @verification_errors = []
    @selenium = Selenium::Client::Driver.new \
    :host => "localhost",
    #:port => 4444,
    :port => "#{$ports}.to_i",
    :browser => "*firefox",
    :url => "http://www.google.com",
    :timeout_in_second => 60

    @selenium.start_new_browser_session
    #$start_time = Time.now
    end

    def teardown
    @selenium.close_current_browser_session
    assert_equal [], @verification_errors
    end

    def test_vuelos
    puts "#{$iteraciones}"
    iteraciones = $iteraciones.to_i
    iteraciones = 2
    iteraciones_ejecutadas = 1

    puts "iteraciones #{iteraciones}"

    log = File.new("vuelos_performance.log","a")
    log_avizpate = File.new("avizpate_performance.csv","a")
    log.puts "----------------------------------------------------------"
    log.puts "Inicio Prueba Vuelos"

    iteraciones.times do
    @selenium.delete_all_visible_cookies
    start_time = Time.now
    @selenium.open "http://www.google.com"
    duration = Time.now - start_time
    puts " 1. Pagina Home: #{duration}"
    log.puts " 1. Pagina Home: #{duration}"
    log_avizpate.puts "#{$usuario},Pagina Home,#{duration}"

    iteraciones_ejecutadas +=1
    end

    log.puts "Fin de la prueba"
    log.puts "----------------------------------------------------------"
    log.close
    log_avizpate.close
    end
    end

    ReplyDelete
  18. Thanks Catherine, one more question, where should I send you the email with the info?

    ReplyDelete
  19. Moises, I didn't forget you - this one just took a little digging! I wasn't familiar with the selenium client gem (I was using another one at the time). I would suggest making sure your grid setup is correct. So with you grid running and your clients registered, can you go to http://localhost:4444 and see the grid page?

    ReplyDelete
  20. Hi Catherine, yes I'm able to see the clients in the grid page....as I said I'm able to run the grid examples. But I'm not sure how to use my scripts instead of the examples :P

    ReplyDelete
  21. Hi Catherine,

    I am trying to implement the Selenium GRID for my JUnit test suite to
    speed up the execution.
    I have done with the Selenium setup and provided demo also running
    properly.My question is how can i integrate my test suite (used Junit) with the
    GRID?

    i tried to run my test suite( with 15 tests) normally from eclipse
    and test suite able to contact the hub, also RC's ( i have 5 RC's
    running with different ports pointing to the hub) .

    i could see the page http://localhost:4444/console is up and running
    fine with the registered RC's and noticed only 2 RCs are in active and
    the rest are in available state. i ran the test suite sevaral times
    and what i noticed was very strange :) sometimes HUB uses 3 RCs,
    sometimes 2 RC's. After 2 or three tests executed.......... all the
    tests in my suite are handled by only one RC and rest are simply idle
    ( hub is not re-using the available RC's).
    i want to know how can i run my test suite in a way that all the tests
    should run in parallel by assigning them to the all available RC's.

    i have also tried parallel-junit to implement test suite but it didnt
    help me.

    Please reply me back. I am awaiting for your valuable solution.

    Thanks in advance.

    Best Regards,
    Ranjit

    ReplyDelete
  22. Ranjit:
    It looks like you're having two problems:
    1. trying to use junit through Selenium grid
    2. RCs not getting reused the way you expect

    Let's address the first question first:
    If you're using a standard JUnit suite, then it doesn't use Selenium at all. Therefore, you can't really use Selenium Grid for this. Selenium Grid is pretty special purpose for parallelizing Selenium (and not other things, unfortunately).

    I will say this: stay tuned. Abakas will have an announcement in the next few months that will be relevant to this!

    As for your second question:
    You say you have 5 RCs configured in the hub and they aren't all being used. Further, some of them are used for early tests but not for later tests. The unused ones are simply marked as "available" in the hub page.

    There are a few things that might be going on here:
    1. check for errors in any of the early tests. The error notification isn't particularly good in the Grid hub, so it may be that you're just not seeing a problem that is occurring. In particular, I'd look for cleanup errors.
    2. I'm assuming all your machines are the same type, but you probably want to confirm that. Do they all have the same type (e.g., *firefox)? If not, maybe your tests aren't using them consistently, so you see some in use and others not. In particular, Safari is a bit slow through Selenium and Firefox is fairly fast, so your tests using *firefox will tend to finish more quickly than the Safari ones, for example.
    3. Check your overall tests - are some simply running faster than others?

    ReplyDelete
  23. Hi Catherine, I am trying to implement Selenium-GRID. I got the DEMO working. But that's all working on single machine. What if my test is on another machine. Test is local say IP 1.1.1.1, selenium HUB is on 2.2.2.2

    Where do I need to launch this command:

    ant -Dport=5556 -Denvironment="*chrome" -Dhost=1.1.1.1 -DhubURL=http://2.2.2.2:4444 launch-remote-control.

    ReplyDelete
  24. web008, the ant task runs on the hub machine, so in your case, you'd run it on 2.2.2.2

    ReplyDelete
  25. Hi Catherine,

    I'm fairly certain I have grid setup properly. However, I was curious to know how I can get a simple script (written in python + pyunit) to run simultaneously on multiple environments.

    For example test loading a page on 2 different machines with different browsers at the same time (say, FF and safari).

    I can run the script just fine on each of the RCs separately. However, this requires that my script explicitly state the environment (e.g. '*firefox' or '*safari').

    But I'm not sure how to run the script simultaneously across environments.

    Thanks,
    JoeSixPack

    ReplyDelete
  26. JoeSixPack, it's been a long time since I've written python, but in design I'd probably do something like this:
    - make your script take the browser as a parameter
    - loop through your list of browsers and call your script, spawning each call off as a subroutine (something like UNIX fork())

    A quick google shows this example of spawning subroutines with Python: http://www.ibm.com/developerworks/aix/library/au-multiprocessing/

    ReplyDelete
  27. can you please send phpunit sample script, running along with seleniumgrid.

    thanks!

    ReplyDelete
  28. Hi Catherine great work, it has certainly saved one hour of my research time kudos to you.

    I would encourage if you can write the same piece of code using python webdriver.

    ReplyDelete