Test Environment, TAF | Rustam Zokirov

Describe the concept = think what are the best practices and trade-off. Give example from past experience. Don’t follow blindly all the best practices, but of course take into consideration. Create own principles, but follow PADI, STAR.

‣

How to set up a test environment, how to choose the correct automation frameworks and tools?

So, the question is about setting up the environment for our application, setting up SUT? OR about test execution (locally, remotely)? I guess it is about first one, right?

There are different approaches, lets discuss some of them, and discuss tradeoffs.

First, we can launch the application locally, and run tests locally. Another option, we can use remote DEV and QA environments where our application lives, and execute tests locally or remotely.

Let’s think about the tradeoffs of each solution.

Setting up application locally could difficult, your machine could be not enough powerful to launch the app. Moreover, you have to test application locally, and team depends on you, when the tester is not available, other team members don’t know what’s going on.

Example: At Reuters, initially before fully setting up test execution on Jenkins (problems at Jenkins were with permissions), we were executing tests locally for a couple of sprints. So, problem was that the team was depending too much on me, for executing release tests and regression tests.

Another option is to create all-day running environment. Advantage of this that you have always access to the environment, but disadvantage is that the environment is running if even you don’t need, spend money Use it if team is global and needs access always.

Consider also data population in database. Do you delete the data after every creation, do you have teardown? There is a tradeoff: every DELETE could take a second to be executed, if you want to make your tests faster, just don’t delete. But you can populate the DB a lot, and tests should be written so that, they know that there is a data. To overcome this problem, we can use template while creating data, and clean database on a scheduled basis.

Considerations while setting up test environment:

Do you have any dependencies on 3rd party systems? That could make the tests flakier, not deterministic. So, then you need to launch hermetic environment.

Hermetic environment. Create env on your own Deploy FE, BE, DB, services on 1 machine/cluster. Mock 3rd parties. Cons: populate data, create test users. Update 3rd party mocks.

Authentication and authorization.
Databases

Where will you execute tests? On remote server?

Think about resources (memory, CPU).
Can you run tests in parallel?
Can you distribute test execution?

‣

How to choose the correct automation frameworks and tools?