• What is testing?
• How do you approach a new project?
• Create test automation strategy → how to introduce automation to the testing, help project to move forward
• Create test plan
• STLC (Test Planning, Test Design, Test Development, Test Execution, Reporting, Maintenance)
• SetQA
• Requirements
• Coverage/Scope
• Tools and test environment
• Process
• Risks
• Schedule and efforts needed
• Roles and responsibilities

What is testing?

What is QA? QA is process of ensuring a positive user experience. Ensuring that product meets the established standards of quality before it is released to the market. What is testing? What is the goal of testing? Ensuring that product meets customer requirements, and verify that software was developed following all specifications. Software testing is about evaluating the level of quality. Goal is to gain the confidence of the customers by providing them a quality product. Testing is about preventing defects. What is the responsibility of tester? And tester is a true product expert. Tester look at product from the perspective of end users, think about different combination of variables and configurations that software may fall. Key responsibilities involve testing a product to ensure that it functions properly and performs as specified. And accomplish this by designing test cases, plans, automating tests. Work closely with product dev team to understand the use cases, provide feedback to improve usability and testability of product. Testers are curious, we questions everything, and strive to excellence. Without good planning, it’s not clear who needs to do what.

How do you approach a new project?

Clarify few moments: is it about what I do when I join the new project, my first steps, and how do I setup the testing process in to the project?

For the new project, I first get to know the product from the user perspective, try to sink myself into the full user experience. Ask questions about the use cases, who will use the software, what kind of features we have, what do we want to do, what is the our goal with this product. Once I understand the product end-to-end, I read the design docs, I read requirements and specifications doc.

It is time now to understand the quality state of the software. Do we have any tests? Do we have any unit/integration, do we have manual or automated testing? Is there anything? Do we have open bugs?

So we have asked questions and learnt requirements.

Next steps are defining test strategy, test plans.

1. Create test strategy (automation strategy), test plan (what is in scope, how to execute), risks and tools
1. Goal of testing = risk mitigation
2. When finish testing? (When enough risks you want to take → we are measuring the extent to which our testing covers the things that require the most testing)
2. Design test cases, write test scripts, execute them.

Let’s delve into details.

Create test automation strategy → how to introduce automation to the testing, help project to move forward

Let me check if I understand the question correctly, by strategy you mean how I will bring the testing process to the team, and introduce automation into the project.

On a high level I need to learn system, understand the current status of project, get req and specs, decide types of testing & approaches, selects tools and frameworks, create test plan.

Let’s start from the beginning:

So, I am new to project. Let me clarify from you (the team) a few things:

• What are the important parts of the project? What are the functional and non-functional requirements?
• Do we have any tests? Do we have any unit/integration? Do we have any manual tests (in JIRA)? Is there anything here? They will help us to introduce automated testing.
• What are the primary objectives and goals of the testing?

Add: Ensuring the quality and reliability of software applications before they are released to end-users, meeting customer expectations — follow requirements, and enhance the overall software development process, help reducing risks.

• Why we are considering automation for this project? What is our goal?

Let’s list some benefits of automation (faster, reliable, cheap), if project is long-term with frequent releases.

1. Efficiency: Automated tests can run faster and more frequently than manual tests, allowing for quicker feedback on the software's quality.

Early detection of bugs, TA provides faster validation and smaller feedback cycle. So, increased efficiency.

2. Time and cost savings: Save time because automated tests are reusable. And executing usually faster. They save time that leads the shorter time-to-market of product.
• Planned percentage from automation? What is our goal?
• Do we have any TAF?

Based on what I get I propose a draft version: so, test strategy should describes the overall testing process in team, in terms of methods, approaches, types of testing (integration, system test, acceptance test), technologies, tools, metrics, finish line.

1. Test approach: We have to answer the question: What kind of testing we want to perform? How much time do we have, so that we will prioritize the scope and approaches.

There different test level and types that can be automated. We can follow test pyramid or quadrant to cover as much with testing and automation.

So, unit, integration testing are the responsibility of developers.

Next level is system testing that contains both functional and non-functional testing. Talking about functional testing, we can start by automating API, why? Because faster. Then move to UI testing and E2E testing using Selenium like tools. Here it would be nice about supported devices? Because we have browser version app, and mobile apps. So we have to cover them with automation.

Of course we should not forget about performance and security testing.

2. What will be tested manually vs. automated?

Which test cases should be automated? Test cases that are executed repeatedly and requirements are not changing. Automated complex scenarios: test cases that involve multiple data inputs, many states calculations, and many interactions. They are time consuming.

3. Next I will think about tools and framework. Do we have any TAF, or need to develop from scratch? CI tool? How tests will be utilized, define reporting, define metrics.
4. Environment, where tests will be executed? How tests will access SUT?
5. Next: defining workflow, who will create tests cases, where. Who and how will prioritize tests for automation. When tests will be automated? Where the real test cases (steps and expected results) will be tracked?
6. End goal: When finish testing? (We are measuring the extent to which our testing covers the things that require the most testing)

When the risks we could have in the future is enough to be the product qualified.

Do you want me go into details, talk a little more?

Create test plan

That’s a nice one. So, before answering to this question, it would be great to highlight that there is a test strategy and a test plan. I was always struggling differentiating this term, because everyone uses them, understands them differently. So, I needed to define them for myself, and I have encountered to 4 minute YouTube video, where I defined the terms for myself.

For me the test plan is about what to test, how to test, and when to test. All implemented and planned testing. It is more about planning how to execute testing. It is generally answers the HOW question. What will be tested, what is in scope, what is not in scope, when and which tests will be executed, by who? How we will do functional testing, how we will do performance testing for our project. How many cycles of testing we will have? How we will do release testing, how tests will be utilized.

The main objective of creating Test plan is focusing on resources and schedules:

• Project scope
• Roles and Responsibilities
• Deadlines and deliverables

Before starting any test plan we have to understand whether we need a test plan now, If there is no project design document or a clear vision for the product, it may be too early to write a test plan.

We have to consider how much time we have to test, what kind of testing we want to perform?

Test plan should include:

• What does the software do (features)?
• What needs to be tested? To get the answer to this question you need to ask what the software you are working on a KNOWN for.
• What is tested now?
• So it is about the coverage
• What kind of application is it? Multi-platform client-server? What are the use-cases?
• Which platform supported (OS, hardware, devices)
• List of features (how each component/feature will be tested)
• What will not be tested?

It is not possible to test everything. What will not be tested, or tested at first place → Examples: low risk areas that are a low priority, complex cases that are a low priority, areas covered by other teams, features not ready for testing, etc.

• What is covered by unit, integration, system tests?

We must have more small tests.

• What will be tested manually vs. automated?

Which test cases should be automated? Test cases that are executed repeatedly and requirements are not changing. Automated complex scenarios: test cases that involve multiple data inputs, many states calculations, and many interactions. They are time consuming.

• How are you covering each test category? (functional, accessibility, perf, security, smoke, usability) When feasible and cost-effective, automation is usually best. Describe the types of cases that will be tested manually and provide rationale.
• Any mocking?
• Line coverage, static code analysis tools?
• Tooling and infrastructure
• What builds are your tests running against? Which branch is a release candidate?
• Do you need new test frameworks? 
• For E2E testing, how will you build test environment/infrastructure?
• Process (how to use tests during release, is there any scheduled testing?, how to handle bugs, monitoring with metrics, acceptance criteria)
• Acceptance criteria. For example: 100% success on smoke, 90% on extended path. Requirements coverage 80%
• How can readers review the actual test cases? How to ensure that automated tests are correct?
• How to report results and monitor? Who will get report?

Who will track reports? How quality will be monitored?

• Are there test schedule requirements? When provide feedback? Some tests should be delivered first?
• How are builds and tests run continuously? CI/CD, small tests run of CI
• How are tests used when releasing? How we use tests during it? What to do if there any release blocker?
• Do you have any general product health or quality goals and how will you measure success? Consider:
• Number of bugs caught by users in production
• Number of bugs caught in release testing
• Number of open bugs over time
• Code coverage
• Cost of manual testing
• How users will report bug?

STLC (Test Planning, Test Design, Test Development, Test Execution, Reporting, Maintenance)

It is all about following STLC, but tweaking it by applying automation part. But in general scope of our work is:

1. Test planning. But before we have to learn the requirements of course. Learn the system, understand scopes, prioritize features. Identify what is in scope of testing, think what and how will be automated. Define the right tools and frameworks (API, UI, perf testing, DB testing, how to generate test data, where and how to setup CI pipeline), develop test execution plans, how to utilize the tests to provide the quality. Think about test environment. Define risks (project changing, env not available, test data is wrong).
2. Test Design. Designing test scenarios and test cases. You definitely need to understand E2E user flows here. Determine what can be automated.
3. Test Development. Writing actual code, developing test scripts using tools and frameworks that were defined during test planning. Think about test user and test data, authentication and authorization. OVERALL test scripts should be maintainable, reusable, and follow coding standards.
4. Test Execution. After tests have been developed, they are executed on the application under tests. The automated tests interact with application, and perform test steps, validated actual and expected results.

Test environment!! Think where and how tests will be executed. CI/CD tool?

QA, DEV, PROD env.

Smoke and regression testing. Define scope.

5. Defect reporting and tracking. After we detected some defect, tickets should be created, assigned to developer and we need to track resolution progress.
6. Test maintenance. New change requests are coming, tests become deprecated. We need to update our tests cases, and test scripts. Fix broken tests, add new tests.
7. Test closure. Creating a report about test execution. With, metrics, bugs found, test coverage. Evaluate overall testing, and test automation process.

SetQA

Some teams have only test plan, some have test plan & test strategy. Test plan is about planned and implanted testing. Test strategy cover the overall approaches and goals. The test plan is a more detailed document that provides guidance to the testing team on how to perform testing.

Plans cover features:

• What does the software do (features)?
• What needs to be tested?
• What is tested now?

Requirements

• Functional
• Components:
• Capabilities: use cases, user actions:
• Non-functional: fast, secure, localized, usable

Coverage/Scope

General approach → test objective, techniques, when starts, completion criteria. How to perform.

‣

Smoke testing (General approach)

‣

How we will do fictional testing?

‣

Integration testing

‣

Regression testing

‣

Cross browser testing

‣

Security testing

Scope — Features that will be covered by testing and features that will not be covered by testing

Tools and test environment

List of configurations involved in different types of testing: Which device, OS version, browser type and version

Process

Risks

• Test env is not accessible
• Data loss
• Hardware problems
• Change to reqs
• Moving Release dates early or expanding the Release scope

Schedule and efforts needed

Define how much do you need per every step.

Roles and responsibilities

Define who will be responsible for every step.