Category Archives: Test automation

Visual TDD in Test Automation – part 3

Before the lockdown

In my sport school there are hand trigger sprayers to clean the machines. So I dutifully clean the machine before and after use. This way my chance on an encounter with a virus is decreased.

There are more machines than the number of hand trigger sprayers. I frequently look for a sprayer. This way my chance on an encounter with a virus is increased.

In order to maintain a distance of 1.5 meters to other people special paths have been marked with arrows. If there is not enough space to pass each other, then the path is a one-way path.

So what is the problem?

When I was a test coordinator for performance tests, I asked for instructions for the testers. I specifically asked for click paths. It boils down to:

  • What does the user see?
  • What does she or he click?
  • What does she or he enter?
  • Go back to the first bullet, unless you performed the last action.

A click path is a pattern. This way a performance tester can turn a program into an expert user. It can be compared with the marked path in my sport school. Follow my direction. Nothing wrong with that.

A design pattern is a good way of working. There are also anti-patterns, bad ways of working.

In the previous blog post I showed an anti-pattern for Visual Testing. In this blog post I will show, how a design pattern turns into an anti-pattern. Maybe you will notice it. Of course I will show a way how to avoid it.

Using a click path

Disclaimer: I am not paid by Applitools for writing this article.

For a fast demo I based my code on the Java code on
https://applitools.com/docs/topics/overview/walkthough-example.html#Building

I used the source code TestLogin3.java and pom.xml from this repo. Notice the name of the class and the test.

For this occasion I used the Login application. Again!
I modified the code a bit. Again!
The source code of TestLogin3.java is shown with the TestLogin3 class and test name "Login test 3"

Just imagine that this is an unknown complex website.

So this is my starting point for adding test automation code in the test01 method.

// TODO Add code for Visual Tests.

In order to get a good click path I let an imaginary Subject Matter Expert navigate through the application.
“Just show me what you do.”

I assumed that I had the faintest idea for the things to come. For the brevity of the blog post I only made 3 visual tests. So I used 3 unique names.

// TODO Add code for Visual Tests.
eyes.checkWindow("1");
eyes.checkWindow("2");
eyes.checkWindow("3");

Now I put breakpoints at all the lines with checkWindow.
There are three lines with checkWindow and a red dot indicating that that there is a breakpoint!

The next step is to make the first checkpoint for a visual test. So I started to debug.
The Context Menu or right click menu of TestLogin3.java is opened and the Debug TestLogin3 is selected!

The debugger stopped at the first breakpoint.
The debugger is shown and the line with checkWindow("1") is highlighted!

I entered Cap as the user name. Why? Cap can be an abbreviation of Captain Marvel or Captain America.
The Username field contains "Cap"!
This was the first picture for my click path. Now I resumed the program for the next checkpoint.
The button with the thin green filled rectangle and the green filled triangle pointing to the right is selected showing a menu with the Resume Program option!

The debugger stopped at the second breakpoint.
The debugger is shown and the line with checkWindow("2") is highlighted!

This time I entered “Secret” as password.
The password field contains 6 dots!

I had the second picture for my click path. Now I resumed the program for the next checkpoint.
The button with the thin green filled rectangle and the green filled triangle pointing to the right is selected showing a menu with the Resume Program option!

Now it was time to press on the login button. And something went wrong. I mean: some error message was shown.
An error message is shown, that the username is invalid!
At that moment three checkpoints had been added:

  1. username entered
  2. password entered
  3. situation after login

I was more curious about how the checkpoints looked like. I went to the Applitools website for the first time that day. So there was no need to press the Refresh button.
The Refresh button is a button with two arced arrows!

The checkpoints were shown with a green bar. This was the baseline and this was good.
In the Applitools website the screenshots are shown: 1) only username filled in 2) both username and password are filled in and 3) the one with the error message of the invalid username!

It is even possible to download the picture of the checkpoint.
The menu with the three vertical dots is selected and shows a menu option download image!

There was a drawback; actions without visible feedback cannot be captured in the click path with screen prints only. If I made some complex mouse movement, then this was not captured. This is also the case for entering the password.

Now I will do my Red Green Refactor stuff. Starting with
Red: making a failing test.

My first test is determining whether Cap is used as username in the login window is shown.
How could I make it fail? Simple. I had written no code to get to the window and the username, so I had my failing test already in place.

In order to reduce the noise I put the last 2 lines with checkWindow in comment. This way they were not executed. These tests would not pass any way.

eyes.checkWindow("1");
// eyes.checkWindow("2");
// eyes.checkWindow("3");

There was also no need for the breakpoints.

So time to run the program again.
Testlogin3.java is selected and the right click or context menu is opened followed by the selection of the Run option!

Also time to see the test fail.
An error message is shown, that the test failed. It also contained "matches=0 missing=2 mismathces=1"!

On the Applitools website I pressed the Refresh button to see the last test results.
The Refresh button is a button with two arced arrows!
All were orange. This meant that a human being like me had to determine whether the tests really failed. I like that feature on behalf of humanity.
All three checkpoints has an orange bar at the left!A

When I looked at the first test, I saw a difference, which was highlighted.
Button with two images of a form and a double pointed arrow between them is selected and showing a menu is shown with the Show both option which is checked. In the left image or baseline "Cap" in the username field is hightlighted in pink, in the right image or checkpoint the empty username field is highlighted in pink!

Just to be sure I turned off the highlighting.
In the left image or baseline "Cap" in the username is shown, in the right image or checkpoint the username is empty. In the right upper corner there is a thumb down button!

This test failed as expected. So I pressed on the thumb down button. The test result turned to red.
The checkpoint with the filled in password has a red bar at the left!

Of course I did not write any code, so the second test also failed.
The second checkpoint and a message about no current image is shown!

So I pressed the thumb down button for this one.
A thumb down button is shown on the right!

The second test result truend to red.
In the overview of the test results a thumb down button is shown on the picture of the third test!

But wait, there was a 3rd orange test also waiting for the same thumb movement.

I found a thumb down button under the test result. So there was no need to open the result to press the same button. I love shortcuts like this. Cheers.
A thumb down button is shown under the test result!

And the last test result turned red.
All the test results in the overview have a red bar at the left!

At first I did not expect the last 2 orange results. What happened?
Applitools made a baseline with three checkpoints, which were checked. Even when the check or checkWindow was disabled. So my attempt to reduce noise in my report failed.

Right. Another lesson learned.

After Red it was time for
Green: make enough code to pass the failing test.

This time I added code to the Login window and enter Cap in the username field.

// Go to Login dialog.
driver.get(weburl);

// fill in the username
driver.findElement(By.id("username")).sendKeys("Cap");

eyes.checkWindow("1");
// eyes.checkWindow("2");
// eyes.checkWindow("3");

Time to run the program again.
Testlogin3.java is selected and the right click or context menu is opened followed by the selection of the Run option!
The test failed again.
The following message is shown: "3 steps matches=3 missing=2 mismatches=0"!

Another  run another press on the Refresh button.
The Refresh button is a button with two arced arrows!

This time I had 1 green and 2 orange test results.
The first test result has a green bar to the left and the other ones have a orange bar at the left. Under the first test result "1/3 1" is shown!

Also notice the text under the first test result with a cryptic text: “1/3 1”. Let me explain this for you. It basically means the 1st out of 3 tests with the name 1.

Now I was able to refactor or cleanup the code. But something went wrong during my exploration of Visual Testing, Test Driven Development, and test automation.

In real life or real business a transaction could include at least 10 visual tests instead of 3 visual tests. So a lot of clicking is to turn other test results to red. But that is a waste of time and energy.

I incidentally made other design errors in my test. Let me illustrate them.

Another imaginary dialogue

Product owner [impatient]: “I want to a brief overview of the test results.”
Amy, the tester: “It went wrong at 1.”
Product owner: “1? What do you mean?”
Amy, the tester: “First step. That means, that the username could not be entered.”
Product owner: “How many times?”
Amy, the tester: “1”.
Product owner [Looking at Brad]: “And you?”
Brad, the tester: “2”.
Product owner: “2 failed tests?”
Brad, the tester: “No. It also went wrong at 1.”
Product owner: “You said too.”
Brad, the tester: “I meant the number 2. The second step is entering the password.”
Product owner: “How many times?”
Brad, the tester: “1”.
Product owner [Looking at Cap]: |What is you score?|
Cap, the tester: “3 1”
Product owner: “Let me guess; third step failed in one test.”
Cap, the tester: “You are right: 3 1.”
Business analyst: [Amused] “Are you still discussing the score of the match of last evening?”
Business analyst: [Curious] “So what were the test results?”

Close problems of the third kind

In the previous chapter we noticed that non descriptive messages are used. 1 is not particular helpful as in checkWindow(“1”).

On a closer look the test01 method is also vague. What is this method about to do? Testing valid credentials or invalid credentials? This is also non descriptive.

Let’s have quick recap of Red Green Refactor:

  • Red: make a failing test.
  • Green: make enough code to pass the failing test.
  • Refactor: clean up the code

The checkWindow method is used for a failing test. There is nothing wrong with that.

At the same time
I used the checkWindow method 3 times in a row. That’s wrong. This way I got Red Red Red for 3 different checkpoints. 2 Reds too many. Then I added a Green Red Red before arriving to my Refactor.

In summary
I used “Red Red Red Green Red Red Refactor” instead of “Red Green Refactor”.

Now changes are appreciated according to the Agile Manifesto.
There is a chance, that the failing test is not up-to-date, when it will be used. At last.

Possible solutions

  • Use a separate tool for the click path. Applitools is a test tool and not a logging tool.
  • Use a word processor or screen recording tool for describing the click path.
  • In case of a screen recording tool, use a word processor to add additional info like password. E.g. type “I will use the password “Secret”.” followed by a copy and paste.
  • Add tests as late as possible.

In another blog post I will show how to refactor the created program.

Things Which Were Not On My 2020 Bingo Card

Time for a feel good blog post after a special year.

Ready. Here you have my highlights.

    • making a talk about Test Driven Development in Test Automation.
      It is about Red Green Refactor and a lot more. I have not given this talk yet. But it lead to more …
    • exploring Test Driven Development or TDD in Test Automation. In Autumn I combined it with Visual Testing.
      It is all about turning TDD to the maximum level.
    • hearing recruiters talking about my blog posts.
      It took me 6 years to get here.
    • exploring accessibility on iPhone.
      This is a strange world. I never thought that I would type words with my eyes closed on a smartphone.
    • helping people in the Club of Ministry of Testing.
      Like a scout I got an Appreciated badge for the number of likes of my posts within a few months.
    • reviewing abstracts for conferences.
      I did not exspect that I would run out of proposals.

      Me hits F5 button]
      [No proposals were shown.]
      “What?”

    • Attending CoffeeOps London meetups organised by Abby Bangser.
      This was a great way to get more acquainted with DevOps.
    • posting a tweet about hiring, which got more than 7,000 views:
    • attending europe CITCON 2020.
      The online version was still free. For the die hards it was possible to order a T shirt.
    • Playing MTG Arena.
      This is a geeky thing ideal for social distance playing. I could not play with real cards, so digital cards were a good alternative. You know about that work life balance thing. This is about life.

      For the people interested in work: testers can be creative. In case you missed it I am a tester. I got to Platinum level 3 in Constructed without spending a cent from scratch within a half year.

    • Meeting people from abroad.
      This February I went to a conference desk and told that I had an appointment with Lisa Crispin. The lady behind the desk looking at me was probably thinking: “Yeah right.”

      I knew that Bob would join.
      Then Janet Gregory and Fiona Charles also joined.
      THAT was definitely not on my 2020 Bingo Card. For sure.

Visual TDD in Test Automation – part 2

A month ago I was in the sport school. The manager had a conversation with me:

“You are using the wrong liquid to clean the machine.”
My eyes went to the automatic liquid dispenser.

She followed my view:
“You must use the hand trigger sprayer. You used the spray for the hand instead of the machine.”

Somehow I missed the hand trigger sprayer. It was clearly in view. But I came from the other side, the sprayer was hidden behind a machine.

So what was the problem?

People tend to use the most simple solution for a problem. You know, I am one of them.

For programming I use Test Driven Development. After some impressive demonstrations and real life experiences.
For me the Red Green Refactor cycle of Test Driven Development is simple.

When I used Visual Testing in the previous blog post, I used a breakpoint.

If I set a breakpoint and debug the program, then it will stop at this breakpoint. Now I have enough time to get useful information from the program, get some coffee, or take a small nap. The latter is an advantage of blogging in free time or being jobless.

Disclaimer: I am not paid by Applitools for writing this article.

Let me visualise the steps from the last blog post:

Start: I have automation code, that opens the Login dialog.
A big rectangle has an arrow with "Automated steps" to a big rectangle containing an empty rectangle!

Red: make a failing test.
I added a CheckWindow method and placed a breakpoint on this line. During the debugging I manually entered the password “Cap” and resumed the program.

What I actually did, was making a screenshot of the desired situation or baseline.

Then I ran the program again. The test failed, because my code did not enter “Cap” in the username.
Why? Cap can be an abbreviation of Captain Marvel or Captain America.

Green: make enough code to pass the failing test.
Then I added code to enter “Cap” in the username textfield. Then I ran the program and it passed.
A rectangle has an arrow with "Automated steps" to a big rectangle containing an empty rectangle. In turn this big rectangle has an arrow with "new automated steps" to a big rectangle with a rectangle containing "Cap". The last big rectangle has an arrow with "snapshot" to a big rectangle with a rectangle containing "Cap"!

Refactor: there was no need to clean up the code, because it was a proof of concept.

Breakpoint overboard

For some people the breakpoint is really strange. So let me try to find a solution with using Test Driven Development while avoiding this point.
A breakpoint to be more precisely.

You are about to read about exploring a new situation in test automation.
Go.

Undesired baseline

And I off went.

For a fast demo I based my code on the Java code on
https://applitools.com/docs/topics/overview/walkthough-example.html#Building

I used the source code TestLogin2.java and pom.xml from this repo.  Notice the name of the class and the test.

The source code of TestLogin2.java is shown with the TestLogin2 class and test name "Login test 2"!

The really interesting coding took place is in the test01 method.

// Go to Login dialog.
driver.get(weburl);

// TODO: fill in the username

// TODO: add test.
// TODO: for making a baseline and
// TODO: comparing the screen with the baseline.
// [edited for blog post]

Now I added the CheckWindow method.

// Go to Login dialog.
driver.get(weburl);

// TODO: fill in the username

// add test.
// for making a baseline and 
// comparing the screen with the baseline.
// [edited for blog post]
eyes.checkWindow("Username filled in");

I left out the breakpoint: no red filled circle is shown.
On line 66 no breakpoint is set, so there is no red filled circle!

Then I ran the program.
On Testlogin2.java the right menu is opened and the Run TestLogin2.main() option is selected!

A message was shown, that a baseline was created.The console shows the following message: "New Baseline Created: 1 steps"!

Time to look in my free Applitools account.
The batch run "Login test 2" has been selected and the result shows a printscreen with a plus sign, thumb up sign, and a green bar!

Now I had done something really strange:
I made an undesired baseline.

Note the danger, that the passed test is accepted as passed, but that was not my intention.

That was not my goal for that day:
adding code to enter “Cap” in username.

There was a simple fix to solve this problem.

Desired baseline

Let me fill in “Cap”.

// Go to Login dialog.
driver.get(weburl);

// fill in the username
driver.findElement(By.id("username")).sendKeys("Cap");

// Test.
// add test. // for making a baseline and 
// comparing the screen with the baseline.
// [edited for blog post]
eyes.checkWindow("Username filled in");

I ran the program again.
On Testlogin2.java the right menu is opened and the Run TestLogin2.main() option is selected!

The test failed.
The console shows the following message: "Test failed : 1 steps matches=0 missing=0 mismatches=1"!

In applitools.com things were not updated yet, so I pressed the Refresh button.

The Refresh button is located under the last 30 batch runs and above Login test 2!

I opened the test result and noticed a difference. But the new screen was right.Button with two images of a form and a double pointed arrow between them is selected. In the left image or baseline; "Cap" in the username field is hightlighted in pink, in the right image or checkpoint several spaces in the username field is hightlighted in pink!

So I pressed the Thumb up button.

A message is shown± `You have successfully resolved all steps. Would you like to continu to the next step or close this editor=` with the Continue button and a Close button!
And the test ruslt became green.
The result shows a printscreen with an unequal sign, thumb up sign, and a green bar. In the upper right corner a Save button or Diskette button is shown!

There was Save button which changing size about every 4 seconds. So I pressed it/

I double checked the test result by removing the hightlighing of the differences.In the menubar a button with a stack of sheets is selected showing a menu with Diffs option which is unchecked. In the left image or baseline; "Cap" in the username field is not hightlighted, in the right image or checkpoint "Cap" in the username field is not hightlighted!

So I ran the program again.On Testlogin2.java the right menu is opened and the Run TestLogin2.main() option is selected!

The test passed this time.The console shows the following message: "All steps passed: 1 steps"!

I went to the Applitools website and pressed the Refresh button.

The Refresh button is located under the last 30 batch runs and above the batch run of Login test 2!

And I selected the last batch run.

The last batch run "Login test 2" has been not selected. The previous batch run "Login test 2" has been selected!

The green test result was shown.

The last batch run "Login test 2" has been selected and the result shows a printscreen with an equal sign, thumb up sign, and a green bar!

There were no differences.

Button with two images of a form and a double pointed arrow between them is selected. In the left image or baseline; "Cap" in the username field is not hightlighted, in the right image or checkpoint "Cap" in the username field is not hightlighted!

Questions

Now I had a test to check whether the username was equal to Cap.
But did I see whether the test fail?
No. Can I trust a test, which I did not see failing?
Probably not.

Another question:
is it good to trust a test, which status went from Green to Red?
I don’t think so.
If someone takes over the coding  of my test, then this would be really confusing: “A failed test is a good sign?!”

According to me this is the problem:
I have a useful pattern or way of working. It is Red Green Refactor.
Then I changed it into an anti pattern or bad way of working. This is Green Red Refactor.
This happened, because I wanted to avoid a breakpoint.

There is another way to explain this.
I have the habit to make pictures of the state of the game or baseline. Please have a look at a game in progress.

Same hotel board game in progress from another angle!

A hotel board game in progress!

Notice that I took pictures from different angles to be sure that the positions of all game pieces were clear.
Also to be sure that there was no hand trigger sprayer hidden somewhere somehow.

Suppose one of my kids would protest: “We were playing another game.” This is actually a bad baseline.

Then I could changed to this picture:A game of Settlers of Catan in progress!

Right game, but right state?
Looking at the boats I don’t think so.A board game of Settlers of Catan is shown with land and and sea tiles. At the edges of the sea files there are ships transporting people and spices!

Visual steps without a breakpoint

Let me visualise the steps taken in this very blog post:

Start: I have automation code, that opens the Login dialog.
A big rectangle has an arrow with "Automated steps" to a big rectangle containing an empty rectangle!

Green: make enough code to pass the failing test.
I added a CheckWindow method and ran the program.

What I actually did, was making a screenshot of the undesired situation or baseline.

When I ran the program again, the test passed.
A rectangle has an arrow with "Automated steps" to a big rectangle containing an empty rectangle. In turn this big rectangle has an arrow with "snapshot" to a big rectangle containing an empty rectangle!

Red: make a failing test.
Then I added code to enter “Cap” in the username textfield. Then I ran the program and the test failed.
A rectangle has an arrow with "Automated steps" to a big rectangle containing an empty rectangle. In turn this big rectangle has an arrow with "new automated steps" to a big rectangle with a rectangle containing "Cap". The last big rectangle has an arrow with "snapshot" to a big rectangle with a rectangle containing "Cap"!

Refactor: there was no need to clean up the code, because it was a proof of concept.

The used anti pattern is Green Red Refactor.
Highly unrecommended.

Wrap up

In this blog post I described a way to use Visual Testing without a breakpoint. I also did not really use Test Driven Development.

There were two drawbacks to this approach:

  1. The test CheckWindow was not tested. So an extra step was needed.
  2. I assumed that the change in the test result would be a sign of progress.