I've written about using Selenium/WebDriver for automated testing in a C# environment in the past. Some of these posts may be dated, but I've covered everything from using the Selenium IDE, to using WebDriver and the PageObject pattern, to using SpecFlow to drive Selenium UI testing. But in this age of MVVM/MVC frameworks and libraries, do we really need every single test to hit the database?
UI testing is widely accepted as slow and fragile. We can use patterns like the Page Object design pattern to reduce the fragility of how we interact with elements on the page, but that doesn't speed things up any. We can merge tests to reduce repetitive actions, but that just adds a bunch of mess to the fragility side of the equation again.
Something I have been considering lately is to use Selenium to test just the View bindings, without the overhead of doing full integration testing. If I keep the logic out of the View (one of the reasons I love MVVM), then I can write some very extensive behavioral unit tests very close to the user and have fast, thorough behavior coverage. My main test concern then becomes how I ensure my bindings stay wired together over time, since I know the behavior under them is working properly. Tightening the focus provides a smaller subsurface to test against then trying to test all of the intricacies all of the way down and reduces the performance drags of anything behind the UI (like network, disk, subsystems, etc), so I could potentially test far more things in less time.
To try this out, I needed a sample application. I wrote a simple application using knockout and RequireJS. There is a basic search screen that allows you to "search" against a slow WebApi endpoint, get further details about a product from that endpoint, and add items to a local cart.
Note: The WebApi actions are slow to reflect performance from larger, more complex applications that have to worry about things like authentication, databases, accessing network stores, logging, contention and retry policies, business logic, etc.
All of the code for the fake application and tests is on github at tarwn/Blog_RequireJSandSelenium.
Testing with Selenium – times 8!
While playing with this, I looked at 4 different methods of testing with Selenium across Chrome and Phantom.
These tests only cover the case where the HTML page is already a static file. If my page had instead been server-side generated from something like ASP.Net MVC or Web Pages, there would be additional work involved.
Note: I used the PageObject pattern very lightly to try and keep the tests readable and easily repeatable for each test method, but did not spend a lot of time following good patterns to create maintainable logic, as this is just experimental code.
Rather than go through all of the cases, I'll touch on just the basic FullIntegration case and one of the Nancy cases. The ClientSide injection case felt really hacky and fragile and I don't think it's a good choice.
Besides the performance, the other downside of this method is the hosting. In the Using SpecFlow to drive Selenium UI Testing post, I already had the steps necessary to deploy a staging site to test against, but this equates to more overhead and could drive where in your build process you perform the tests as well as make it harder to run them locally.
In this case, I created a self-hosting Nancy site that copies all of the static content from my Sample site and exposes fake versions of the API. The downsides of the two Nancy methods are the restriction to static content (no MVC pages) and that you're reimplementing a fake API for the system.
This second issue actually bothers me a bit, as it means you are creating a fake set of data that all of your tests are going to rely on. Typically when you have one big shared pool of test data, it makes your systems harder to maintain, as that test data turns into a bog of magic values, some of which have to be set just so for tests to succeed. Allowing the tests to define the values that would be returned when they have specific needs would make this a lot more maintainable and help surface those critical data assumptions in the tests.
LocalServerBootstrapper defines the static content folders (in this case, /styles, /Scripts, and the /index.html file). There is a single Module, LocalServer, that serves up the 2 item API endpoints.
In a larger test suite, I would move this test code to a single startup method for the whole assembly.
Running the same 3 tests for each of the 4 methods across two different browsers helped see the difference between startup and ongoing performance costs.
We pick up quite a bit of performance when we remove the backend server from the tests. one other thing to note is that the startup time for Phantom is quite a bit faster, but there is a slightly higher ongoing cost.
This turned out to be a pretty nice little experiment. I wouldn't use any of these methods for a production test suite as they are now, but they definitely have promise and I'll certainly be trying out some more things with that Nancy setup.
Switching from a full integration focus to more of a View focus did make things faster, but not to the degree I had hoped. I intend to spend some further thought on how to turn the dial up further without making this really painful to maintain.