Looking back: Microsoft Coded UI
A few years back I was engaged in an enterprise automation-testing project with an existing automation solution Microsoft Coded UI testing framework. I really loved using it, especially because the application was created for internal use with Internet Explorer.
Coded UI has a cool built-in feature Continue on Error. You would simply mark the test step as “Continue on Error = true” and the test would try to execute and ignore it if it fails. This is a handy tool if you are using a data-driven testing approach for an application that supports all kinds of different vendors.
However, some test flows were triggering a new info/confirmation screen for some vendors, with only one option on it the OK button. This particular screen was causing the test to fail for some, and pass for other vendors. Since the screen was not a deal-breaker, we marked the button on that screen as a Continue on Error and created a separate test for just those vendors to verify whether the screen is displayed (without setting it as Continue on Error).
The Framework of Choice Today: Selenium Web Driver
The times have changed and we are now testing the very same application (new version of course) against all popular browsers with a new Selenium Web Driver-based framework.
Before we started building the new Test Framework, we sat down and agreed to create a TestStepRunner tool, which would empower us to play around with test execution.
Building a custom Test Step Runner
For the demo purposes, I am going to use a simple Selenium C# POM framework:
NuGet packages installed:
- DotNetSeleniumExtras.PageObjects
- System.Configuration.ConfigurationManager
- NUnit
- NUnit3TestAdapter
- Selenium.Chrome.WebDriver
- Selenium.Firefox.WebDriver
- Selenium.Support
- Selenium.WebDriver
- DotNetSeleniumExtras.WaitHelpers
Class definitions:
//Pages.cs
using SeleniumExtras.PageObjects;
using System;
namespace Tests
{
public class Pages
{
public Pages(Browsers browser)
{
_browser = browser;
}
Browsers _browser { get; }
private T GetPages<T>() where T : new()
{
var page = (T)Activator.CreateInstance(typeof(T), _browser.getDriver);
PageFactory.InitElements(_browser.getDriver, page);
return page;
}
public Home Home => GetPages<Home>();
}
}
//Browsers.cs
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Firefox;
using System.Configuration;
namespace Tests
{
public class Browsers
{
public Browsers()
{
baseURL = ConfigurationManager.AppSettings["url"];
browser = ConfigurationManager.AppSettings["browser"];
}
private IWebDriver webDriver;
private string baseURL;
private string browser; public void Init()
{
switch (browser)
{
case "Chrome":
webDriver = new ChromeDriver();
break;
case "Firefox":
webDriver = new FirefoxDriver();
break;
default:
webDriver = new ChromeDriver();
break;
}
webDriver.Manage().Window.Maximize();
Goto(baseURL);
}
public string Title
{
get { return webDriver.Title; }
}
public IWebDriver getDriver
{
get { return webDriver; }
}
public void Goto(string url)
{
webDriver.Url = url;
}
public void Close()
{
webDriver.Quit();
}
}
}
//TestBase.cs
using NUnit.Framework;
namespace Tests
{
[TestFixture]
public abstract class TestBase
{
protected Browsers browser;
protected Pages Pages;
[SetUp]
public void StartUpTest()
{
browser = new Browsers();
browser.Init();
Pages = new Pages(browser);
}
[TearDown]
public void EndTest()
{
browser.Close();
}
}
}
//Home.cs
using NUnit.Framework;
using OpenQA.Selenium;
using SeleniumExtras.PageObjects;
namespace Tests
{
public class Home
{
public Home()
{
driver = null;
}
public Home(IWebDriver webDriver)
{
driver = webDriver;
}
IWebDriver driver;
[FindsBy(How = How.Name, Using = "q")]
private IWebElement SearchField;
public void IsAt()
{
Assert.IsTrue(driver.Title.Equals("nopCommerce demo store"));
}
public void EnterSearchText(string searchText)
{
SearchField.SendKeys(searchText);
}
}
}
//Tests.cs
using NUnit.Framework;
namespace Tests
{
[TestFixture]
public class MyFirstPOMTest : TestBase
{
[Test]
public void HelloWorldTest()
{
Pages.Home.IsAt();
Pages.Home.EnterSearchText("Hello world!");
}
}
}app.config file:
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<appSettings>
<add key="url" value="https://demo.nopcommerce.com" />
<add key="browser" value="Firefox" />
</appSettings>
</configuration>When we run the test, it will open the demo.nopcommerce page and enter Hello World! into the input filed.
Step Runner library project
I am going to create a Step Runner project as a separate project from the Selenium framework.
Add a necessary NUnit package:
Now let s add two new classes to the project:
- StepInfo holds all necessary information about the step (its description, level of importance, does it need to be skipped, etc.)
- StepRunner executes the step based on step info properties
//StepInfo.cs
using System;
namespace StepRunnerLibrary
{
public class StepInfo
{
public string Description { get; set; }
public bool SkipStep { get; set; }
public Importance Level { get; set; }
public bool SkipStepOnFailure { get; set; }
public bool FailsIteration { get; set; }
public Status Status { get; set; }
public DateTime StartTime { get; set; }
public DateTime EndTime { get; set; }
public TimeSpan Duration { get; set; }
public Exception StepException { get; set; }
public StepInfo()
{
StepInit();
}
public StepInfo(string description)
{
StepInit();
Description = description;
}
public StepInfo(string description, Importance level)
{
StepInit();
Description = description;
Level = level;
}
public StepInfo(string description, bool skipStepOnFailure)
{
StepInit();
Description = description;
SkipStepOnFailure = skipStepOnFailure;
}
public StepInfo(string description, bool skipStepOnFailure, Importance level)
{
StepInit();
Description = description;
SkipStepOnFailure = skipStepOnFailure;
Level = level;
}
public StepInfo(string description, bool skipStepOnFailure, bool skipStep)
{
StepInit();
Description = description;
SkipStepOnFailure = skipStepOnFailure;
SkipStep = skipStep;
Status = Status.Skipped;
}
public StepInfo(string description, bool skipStepOnFailure, bool skipStep, Importance level)
{
StepInit();
Description = description;
SkipStepOnFailure = skipStepOnFailure;
SkipStep = skipStep;
Level = level;
}
protected void StepInit()
{
Description = "Step Description not defined";
SkipStep = false;
SkipStepOnFailure = false;
FailsIteration = true;
Level = Importance.High;
Status = Status.Ready;
}
}
}
//StepRunner.cs
using NUnit.Framework;
using System;
using System.Collections.Generic;
using System.Globalization;
using System.Linq;
namespace StepRunnerLibrary
{
public enum Status
{
Passed,
Failed,
Skipped,
Ready
}
public enum Importance
{
Lowest,
Low,
Normal,
High,
Highest
}
public class StepRunner
{
private readonly IList<StepInfo> testSteps = new List<StepInfo>() { };
private int stepCount = 0;
private bool stepContinuation = false;
public static StepRunner Instance { get; set; }
public TestContext TestContext { get; set; }
public string TestIdentifier { get; set; }
public string TestDescription { get; set; }
public string Environment { get; set; }
public Status Status { get; set; }
public DateTime StartTime { get; set; }
public DateTime EndTime { get; set; }
public TimeSpan Duration { get; set; }
public Exception TestException { get; set; }
private StepRunner(TestContext testContext)
{
TestContext = testContext;
TestIdentifier = TestContext.CurrentContext.Test.Name;
TestDescription = TestContext.CurrentContext.Test.ClassName;
Status = Status.Ready;
}
private bool VerifyStepExecution(StepInfo step)
{
bool skip = false;
if (stepCount == 0)
{
StartTime = DateTime.Now;
Console.WriteLine("********************************************************************************");
Console.WriteLine("** [Test Iteration]");
Console.WriteLine("** Iteration Name: {0}", TestIdentifier);
Console.WriteLine("** Iteration Description: {0}", TestDescription);
Console.WriteLine("********************************************************************************");
}
stepCount++;
step.StartTime = DateTime.Now;
Console.WriteLine("--------------------------------------------------------------------------------");
Console.WriteLine("-- Step number: {0}", stepCount);
Console.WriteLine("-- Description: {0}", step.Description);
Console.WriteLine("-- Step Start Time: " + step.StartTime.ToString("hh:mm:ss tt", CultureInfo.InvariantCulture));
Console.WriteLine("--------------------------------------------------------------------------------");
Status testStatus;
if (step.SkipStep)
{
Console.WriteLine("-- Test status: Skipping step execution!");
skip = true;
}
else if (HasContinuationOnFailure(step))
{
skip = false;
}
else if (this.IsFailedStep(testSteps, out testStatus) && !step.SkipStepOnFailure)
{
Console.WriteLine("Test status: [{0}]. Skipping step execution!", testStatus);
skip = true;
}
return skip;
}
private void FinalizeStep(StepInfo stepInfo)
{
testSteps.Add(stepInfo);
stepInfo.EndTime = DateTime.Now;
TimeSpan diff = stepInfo.EndTime - stepInfo.StartTime;
Console.WriteLine("--------------------------------------------------------------------------------");
if (stepInfo.Status == Status.Failed)
{
Console.WriteLine("-- Step Status: {0}", stepInfo.Status.ToString());
Console.WriteLine("-- Impact Level: {0}", stepInfo.Level);
}
else
{
Console.WriteLine("-- Step Status: {0}", stepInfo.Status.ToString());
}
Console.WriteLine("-- Step Duration: {0}", diff);
Console.WriteLine("--------------------------------------------------------------------------------");
Console.WriteLine("");
}
private void HandleException(Exception ex, StepInfo stepInfo)
{
if (stepInfo.SkipStepOnFailure)
{
stepInfo.Status = Status.Skipped;
}
else
{
stepInfo.Status = Status.Failed;
stepInfo.StepException = ex;
Console.WriteLine("Exception on step: " + stepInfo.Description, stepInfo.StepException);
Console.WriteLine("Message: {0}", stepInfo.StepException.Message);
Console.WriteLine("Inner Exception: {0}", stepInfo.StepException.InnerException);
Console.WriteLine("Stack Trace: {0}", stepInfo.StepException.StackTrace);
Console.WriteLine("");
TestException = ex;
}
stepContinuation = stepInfo.SkipStepOnFailure;
}
private bool IsFailedStep(IList<StepInfo> stepCollection, out Status testStatus)
{
if (stepCollection.Where(x => x.Status == Status.Failed).Count() > 0)
{
testStatus = Status.Failed;
return true;
}
testStatus = Status.Passed;
return false;
}
private bool IsTestFailedOutsideStep(out Status testStatus)
{
if (TestException != null)
{
testStatus = Status.Failed;
return true;
}
testStatus = Status.Passed;
return false;
}
public void SetExternalException(Exception ex)
{
Status = Status.Failed;
TestException = ex;
}
public static void Initialize(TestContext testContext)
{
Instance = new StepRunner(testContext);
}
public void ClearStepResults()
{
testSteps.Clear();
}
public Status RunStep(Action action, StepInfo stepInfo)
{
try
{
if (!VerifyStepExecution(stepInfo))
{
action();
stepInfo.Status = Status.Passed;
}
}
catch (Exception ex)
{
HandleException(ex, stepInfo);
}
finally
{
FinalizeStep(stepInfo);
}
return stepInfo.Status;
}
public Status RunStep<T>(Action<T> action, T parameter, StepInfo stepInfo)
{
try
{
if (!VerifyStepExecution(stepInfo))
{
action(parameter);
stepInfo.Status = Status.Passed;
}
}
catch (Exception ex)
{
HandleException(ex, stepInfo);
}
finally
{
FinalizeStep(stepInfo);
}
return stepInfo.Status;
}
public Status RunStep<T>(Action<T, T> action, T parameter1, T parameter2, StepInfo stepInfo)
{
try
{
if (!VerifyStepExecution(stepInfo))
{
action(parameter1, parameter2);
stepInfo.Status = Status.Passed;
}
}
catch (Exception ex)
{
HandleException(ex, stepInfo);
}
finally
{
FinalizeStep(stepInfo);
}
return stepInfo.Status;
}
public void Close()
{
IsFailedStep(testSteps, out Status testStatusIn);
IsTestFailedOutsideStep(out Status testStatusOut);
if (testStatusIn == Status.Passed && testStatusOut == Status.Passed)
{
Status = Status.Passed;
}
else
{
Status = Status.Failed;
}
EndTime = DateTime.Now;
TimeSpan diff = EndTime - StartTime;
Console.WriteLine("********************************************************************************");
Console.WriteLine("** Test Iteration Duration: {0}", diff);
Console.WriteLine("** Test Iteration Status: {0}", Status.ToString());
Console.WriteLine("********************************************************************************");
if (StepRunner.Instance.TestException != null)
{
Console.WriteLine("Exception Message: {0}", TestException.Message);
Console.WriteLine("Inner Exception: {0}", TestException.InnerException);
Console.WriteLine("Exception Stack Trace: {0}", TestException.StackTrace);
Console.WriteLine("");
}
if (TestContext != null)
{
TestContext = null;
}
if (Instance != null)
{
Instance = null;
}
if (Status != Status.Passed)
{
if (TestException != null)
{
throw TestException;
}
throw new Exception(string.Format("Test Iteration failed {0}", TestIdentifier));
}
}
public bool ValidateIfTestIsFailed()
{
if (testSteps.Where(x => x.Status == Status.Failed).Count() > 0)
{
return true;
}
return false;
}
private bool HasContinuationOnFailure(StepInfo stepInfo)
{
if (stepInfo.SkipStepOnFailure)
{
return stepContinuation = true;
}
else if (!stepInfo.SkipStepOnFailure && stepContinuation)
{
return true;
}
return false;
}
}
}StepInfo class
Step Information class holds all the necessary test step data. It has multiple constructors based on the number of parameters that we are passing. In most cases, we are passing only test description.
StepRunner class
Step Runner class is the heart of our testing framework. It executes, runs, and logs the entire test, one step after another.
RunStep is the main method that accepts a test step as an action delegate and executes it based on StepInfo parameters. For example, if the test failed on a previous step, it will just skip test execution. The same goes for other parameters such as skip test or continue on error.
The RunStep is overridden in order to give support to the steps that are accepting a parameter. This is mainly used for input fields and similar.
Verify Step Execution has two main responsibilities: decide and log.
It accepts StepInfo class instance, and based on its properties decides whether a test step will be executed or skipped. You can play with it and give it more power if needed.
I m using console only for demo purposes as each step is logged there. Of course, there are other simple options you can explore later.
Note: You can read about logging with log4net and creating fancy reports in my previous blogs:
- Create an interactive Selenium automation report with ExtentReports and C#
- Fetch Page Load Metrics With C# Selenium WebDriver Testing Framework
Utilize it in the TestBase class
I use the TestBase class in all my tests in order to remove all the initialization out of the tests. This way, my tests look clean and have only test steps in it.
Create a StepRunner instance as shown on the screenshot below:
Create a new test with Test Step Runner
In order to execute a test step, simply call the RunStep method with test step delegate and StepInfo instance.
[Test]
public void HelloWorldTest2()
{
StepRunner.RunStep(Pages.Home.IsAt, new StepInfo("Verify Home Page Loaded"));
StepRunner.RunStep(Pages.Home.EnterSearchText,"Hello world!", new StepInfo("Enter Text In Search Field"));
}Do this for all the steps.
In case your existing test step accepts a parameter, pass it as an independent parameter right after the test method.
Run the test and verify output.
Note: Currently, the StepRunner is logging only test step logs. In order to log step action logs, such as click, send keys, select and similar, you need to implement logging feature on actions. You can find it here under the Façade design pattern.
Make sure to try all the cool features, just add a comma after a test description and select one of the overrides.
That s it! Till the next time, Happy testing.
If you found this blog useful, you might want to check out some of my previous blog posts: