IEDriverServer Beta Testing with WatirSpec

ie.log

Finished in 20 minutes 0 seconds (files took 5.53 seconds to load)
1306 examples, 8 failures

Failed examples:

rspec ./spec/watirspec/after_hooks_spec.rb:130 # Browser::AfterHooks #run raises UnhandledAlertError error when running error checks with alert present
rspec ./spec/watirspec/alert_spec.rb:27 # Alert API alert #exists? returns true if alert is present
rspec ./spec/watirspec/browser_spec.rb:80 # Browser #name returns browser name
rspec ./spec/watirspec/cookies_spec.rb:62 # Browser#cookies adds a cookie with options
rspec ./spec/watirspec/elements/button_spec.rb:136 # Button #style returns the style attribute if the button exists
rspec ./spec/watirspec/elements/tr_spec.rb:39 # TableRow #click fires the row's onclick event
rspec ./spec/watirspec/window_switching_spec.rb:112 # Window multiple windows #close closes a window
rspec ./spec/watirspec/window_switching_spec.rb:120 # Window multiple windows #close closes the current window

ie_beta.log

Finished in 40 minutes 54 seconds (files took 5.78 seconds to load)
1306 examples, 14 failures

Failed examples:

rspec ./spec/watirspec/after_hooks_spec.rb:130 # Browser::AfterHooks #run raises UnhandledAlertError error when running error checks with alert present
rspec ./spec/watirspec/alert_spec.rb:27 # Alert API alert #exists? returns true if alert is present
rspec ./spec/watirspec/browser_spec.rb:80 # Browser #name returns browser name
rspec ./spec/watirspec/cookies_spec.rb:62 # Browser#cookies adds a cookie with options
rspec ./spec/watirspec/drag_and_drop_spec.rb:19 # Element drag and drop can drag and drop an element onto another when draggable is out of viewport
rspec ./spec/watirspec/drag_and_drop_spec.rb:24 # Element drag and drop can drag and drop an element onto another when droppable is out of viewport
rspec ./spec/watirspec/drag_and_drop_spec.rb:30 # Element drag and drop can drag an element by the given offset
rspec ./spec/watirspec/elements/button_spec.rb:136 # Button #style returns the style attribute if the button exists
rspec ./spec/watirspec/elements/radio_spec.rb:197 # Radio #set sets the radio button
rspec ./spec/watirspec/elements/tr_spec.rb:39 # TableRow #click fires the row's onclick event
rspec ./spec/watirspec/relaxed_locate_spec.rb:201 # Watir#relaxed_locate? when false when acting on an element that eventually becomes present raises exception immediately
rspec ./spec/watirspec/window_switching_spec.rb:360 # Window manipulating size and position should resize the window
rspec ./spec/watirspec/window_switching_spec.rb:376 # Window manipulating size and position should move the window
rspec ./spec/watirspec/window_switching_spec.rb:393 # Window manipulating size and position should maximize the window

ie_beta_testing.md

IEDriverServer Beta Testing

Environment

Win 7 with IE11 (using ModerIE) selenium-webdriver 3.3.0 IEDriverServer_beta_Win32_3.3.99.1

Good News

I can confirm that I can reproduce SeleniumHQ/selenium#3587 using WatirSpec. It fails using IEDriverServer 3.3, but does not failing using beta, so it was somehow fixed.

Bad News

I have only one spec that fails on beta but passes on 3.3. The specs are about drag and drop functionality. While basic "drag element and drop to another element" passes, there are few that are failing:

1. Drag element that is out of viewport.
2. Drop on element that is out of viewport.
3. Drag element and drop by offset.

Ruby bindings don't work with W3C dialect of IEDriverServer, so I needed to hack it around in order to make specs run, but I still need time to turn these failures into bug reports. I hope to do that during the weekend.

Thanks very much for doing this, it's much appreciated. It's gratifying that so much of it Just Works™.

The window switching spec failures are protocol differences in that a different end point is used for the W3C protocol dialect for window size and position manipulation (GET|POST /session/{sessionId}/window/rect instead of separate end points for size and position). This will need to change in Ruby because the next geckodriver will also implement the proper end points here.

The drag-and-drop spec differences are interesting. They're unexpected, but I also wonder if they're not obsolete for IE. What happens with those specs for geckodriver? I suspect they either throw an expected exception ("move target out of bounds"), or they're guarded against on that browser because they would throw that exception. The spec is very clear that attempting to move the mouse outside the view port isn't supported, even in the drag-and-drop case. The beta driver doesn't explicitly fail in that way, and those failures are unexpected, but there may be some side effect of the refactoring that's causing it to work in a spec-compliant way anyhow.

The difference in radio buttons is concerning. I'll need to investigate that one a little further.

The drag-and-drop spec differences are interesting. They're unexpected, but I also wonder if they're not obsolete for IE. What happens with those specs for geckodriver? I suspect they either throw an expected exception ("move target out of bounds"), or they're guarded against on that browser because they would throw that exception. The spec is very clear that attempting to move the mouse outside the view port isn't supported, even in the drag-and-drop case. The beta driver doesn't explicitly fail in that way, and those failures are unexpected, but there may be some side effect of the refactoring that's causing it to work in a spec-compliant way anyhow.

I'll see how geckodriver handles these specs and will let you know the difference.

The difference in radio buttons is concerning. I'll need to investigate that one a little further.

That seems to be flaky test (or maybe flaky driver) because after re-running - it's all good.

I'll see how geckodriver handles these specs and will let you know the difference.

Looks like Ruby bindings don't fully support Actions endpoints for GeckoDriver, so these specs are disabled.