• How to setup testing from ZERO in project?
• Definitions
• Classic Testing Mistakes
• Typical Tester's Mistakes
• An ideal tester should
• My custom process
• Sprint 0: How to start?
• EngX principles
• Collaboration between Business, Development and QA.
• Feature workflow: Test plan → Test case → Automation → Update Test plan
• Test Plan
• How to understand that testing is useful?
• How to implement continuous testing?
• Common pitfalls
• Questions to get requirements
• Testing a real world object
• Testing a piece of software
• How to start testing?
• Test the process of testing
• Resources

How to setup testing from ZERO in project?

Definitions

• Software Testing - The process consisting of all lifecycle activities, both static and dynamic, concerned with planning, preparation and evaluation of software products and related work products to determine that they satisfy specified requirements, to demonstrate that they are fit for purpose and to detect defects
• Defect (Bug) - It is something that does not correspond to valid Customer's expectations that are either specified in functional requirements, Acceptance criteria or assumed as rational/logic behaviour of the system under test. It is an imperfection or deficiency in a work product where it does not meet its requirements or specifications
• Expected Result - the predicted behavior of the system (software product) under test that we expect to see as a result of our actions (inputs). In other words - this is the correct behavior of the software product based on its specifications/requirements or customer input.
• Test Case - A set of preconditions (what needs to be set before execution), test inputs (actions that need to be taken - steps), and expected results developed for a particular objective, such as to exercise a particular program path or to verify compliance with a specific requirement.
• Test Plan - is a part of the overall project documentation describing what, when, how and who will be involved in the testing process. This is a basic document that describes a list of tested components, quality criteria and risks of testing, resources to be used for testing, metrics by which results will be measured, testing strategy and testing types. ISTQB: It is a documentation describing the test objectives to be achieved and the means and the schedule for achieving them, organized to coordinate testing activities
• Build - is an intermediate version of a software product which is released by developers for testing.
• Static Testing - is a process, which is used for testing (verifying) of any work product, e.g. code, requirements, functional specification, architecture and design documents, etc. Static testing is one of the most effective ways of defects detecting in the early stages of product process. Basically static testing is testing a work product without code being executed. An example of such testing can be requirements analysis for inconsistencies or contradictions before the implementation starts.
• Dynamic Testing - is a testing activity that involves the execution of the software component or system. Dynamic testing consists of launching the program, running all functional modules and comparing the product's behavior with expected one using user interface.
• Black box testing - The process of software testing based solely on the functional and business requirements and operating it without knowledge of the internal structure or program source code. It is performed by accessing the software just as the user would without accessing what is "under the hood". The purpose of this method is to check the proper work of all functions, to verify whether all functions correspond to functional requirements.
• White box testing (glass box testing) - Tester uses her understanding of the source code and access the code to develop and execute test cases. This method tests the architecture of the system. It tests the design and programming that goes into building the system. White box testing is usually applied when an application is not entirely assembled, but it's necessary to check each of the components, modules, procedures and sub-functions.
• Grey box testing - is when the tester has a partial understanding of the internal structure in the system under test. Grey box testing is a process for debugging software applications by making an input through the front-end, and verifying the data on the back-end.
• Unit Testing - This level of testing tests separate modules of the system. It may be as specific as testing any particular part of the code (class).
• Integration Testing - Testing of combined parts of an application to determine if they function together correctly. As well as that, at this level of testing the interactions between a number of applications or big system - in this case this testing is known as Cross-product testing. Usually it is performed after unit and functional testing of the individual modules.

It can be further broken down into:

1. Component integration testing - Testing performed to expose defects in the interfaces and interactions between integrated components.
2. Hardware-software integration testing - Testing performed to expose defects in the interfaces and interaction between hardware and software components
3. System integration testing - Testing the combination and interaction of systems.
• System Testing - Checking the operation of the integrated system (system as a whole). This testing verifies both functional as well as non-functional requirements.
• Functional testing - Testing conducted to evaluate the compliance of a component or system with functional requirements.

The goals are:

1. Verifying that the system fulfills the functional requirements provided by the stakeholders/customers
2. Finding any discrepancies and defining them as defects against the software product so that they could be addressed and fixed
3. Determining whether the system is viable for a possible delivery to the customer. This decision must be documented in the Test Results Report.
• Smoke Test - a minimal set of tests that check the basic functionality of the system to determine it's stability. Based on the results of this test, the QA Lead (Lead tester) will make a decision whether to continue testing on this build or not, whether it is stable enough for further functional testing. It should not take longer than 1-4 hours depending on the complexity of the system under test.
• Critical path test

It is a basic type of test when the functionalities that are significant to the end-user or used by typical users during their day-to-day activities are checked. Typically this is performed after the smoke testing has been successfully executed.

• Sanity Test - Sanity testing is a kind of Software Testing performed after receiving a software build, with minor changes in code, or functionality, to ascertain that the bugs have been fixed and no further issues are introduced due to these changes. The goal is to determine that the proposed functionality works roughly as expected. In other words, it's a mini regression testing that only involves the area that could have been affected by the changes or changed itself.

Work Products that can be tested:

1. System Software during the installation, launch and operation states (Dynamic testing)
2. Software's code without launching (Static testing)
3. Product prototype (Can be both Static and Dynamic)
4. Product documentation (Static testing)
5. Project documentation (Static testing)
• Quality - The degree to which a component, system or process meets specified requirements and/or user/customer needs and expectations

There are two aspects of the quality: product quality and process quality.

Some quality criteria assumptions:

1. If the customer accepts the implemented product this is enough to say that the quality of the product is good/acceptable.
2. The Product quality is defined by good processes (development, management, testing etc.).
3. If the product meets its requirements - this is a good product.
4. Good product has some advantages and doesn't have critical problems.

For more definitions and terms: https://glossary.istqb.org/en/search/

Classic Testing Mistakes

• Assurance that you can find all defects in the software product → That's impossible, there will never come a point where a QA can say that they have found ALL the bugs there are in the system. Think about it, even Google has bugs.
• Assurance that you are capable to test the program completely → Exhaustive testing is impossible, we can never say that we have tested all the possible use cases and every single functionality and feature with every possible scenario.
• Thinking the testing team is responsible for assuring quality → we are a team (QA + Developers + BAs + Technical Writers and anyone else in our team working on the product), we can only work on improving the product by finding defects and verifying that they are fixed which improves confidence in the quality of the product of the customer.
• Thinking that the ONLY goal of testing is to find bugs → True goals of testing are: Gaining confidence in and providing information about the level of quality; Finding defects; To make sure that the end result meets the business and user requirements; To gain the confidence of the customers by providing them a quality product
• Not finding the important bugs → This mistake can commonly happen if the prioritization was incorrect and for example you are focus on testing a very complex but minor functionality while the most used and critical functionality get over looked. Example: you are testing the saving of bonus points that the user gets during playing additional mini games inside the main game but don't test that the user can simply navigate between levels. Even though the bonus points can be a complex functionality, the most critical part that the user can navigate throughout the main game.
• Not reporting usability problems
• Starting testing too late (bugs detection, not bugs reduction) → There might be not enough time to fix the issues or it could simply be too expensive to fix them. The earlier you start testing the better
• Putting stress and load testing off to the last minute → Fixing Performance issues can be more and more difficult the later it is addressed. It might cause a need in a major refactoring and could cause regressions if tackled late in the game. Or it could potentially be too late and cause issues with both the budget and the deadlines
• Not testing the documentation → There might be contradicting requirements that once implemented will cause erroneous behaviour and will need to be fixed/changed while it would have been faster and cheaper to fix them at the level of documentation before implementation.
• Not testing installation procedures
• Failing to correctly identify risky areas → Will lead to incorrect prioritization of testing which in term could lead to missed critical bugs in vital areas of the application for the customer.
• Paying more attention to running tests than to designing them → This might lead to these tests not being explicit and if you need another QA to run them, they might be hard to understand or not cover the functionality to the fullest. This might lead to issues / defects being missed. You are a QA, our test suits and cases are documents that we should be able to share with pride with both our colleagues as well as management.
• Not reviewing test designs → Test design should be reviewed regularly to make sure it satisfies the testing objective and needs as well as accounts for all the changes that come within the life cycle of the project.
• Checking that the product does what it's supposed to do, but not that it doesn't do what it isn't supposed to do. → Both Positive and Negative scenarios must be executed. Think of it this way, when you are testing a bank website, you want the users to be able to access their accounts, but you ALSO need to make sure that a random only the right user can see the right account and User B, even if he tried, cannot see User A's account.
• Poor bug reporting → Defects must be reported so that we can track which areas are the most risky, which areas are changed in the process of bug fixing etc. As well as that, the bugs themselves must be written out CLEARLY, with a good description and concise summary (so that it's quickly understood what the bug is about) and steps that will allow both the developer to reproduce the issue so he can fix it, as well any tester to verify the bug after fixing. If possible, attach a video, a screenshot and the error from the dev tool console of the browser.
• Attempting to automate all tests → you need to prioritize what should be automated and what is not worth spending the effort and money being automated as well as what should be automated and what shouldn't be. For example if the software is constantly changing and still in the development, it is not a good idea to automate that which is expected to change. Another example is the content of a website. For instance it is pointless to automate the presence of a specific image if this is an image that will be changing regularly (like a tv schedule poster on a network website)
• Expecting to re-run all manual tests

Typical Tester's Mistakes

• Assuming that the program works correctly.
• Unwillingness to register each error that they had found.
• Ignoring or even hiding the problem.
• Falling under the influence of the developers which ask you not to submit found defects or to ignore existing intelligence about defects without adequate reasons.
• Insufficient attention to test planning.
• Failing to escalate an important defect with the developer due to not wanting to hurt his / her feelings → be polite and tactful, but you must still do your job
• Writing defect reports about non existing problems.

An ideal tester should

• Have software engineering skills (understanding the rules of software engineering, knowing computer programming and having operating system level knowledge). Often a tester need to be an expert in a few areas.
• Have a good knowledge of the application being tested.
• Have a considerable intelligence.
• Have a hyper-sensitivity to little things.
• Be resilient and stable in chaos and under pressure of development, management, as well as pressure of different circumstances.
• Have strong communication skills.
• People skills
• Tenacity
• Capability to criticize and interpret the criticism correctly
• Have good organizational skills.
• Be skeptical, but DO NOT adopt a hostile attitude.
• Be capable of braking the software without feeling any remorse.
• Be self-sufficient and tough.
• Be technology hungry, you must be hungry to educate yourself in a variety of technologies and keep up to date with the new no-hows
• Be honest.
• Be capable of bringing the bad news to developers and management.
• Be patient - be ready to perform monotonous work for long stretches of time.
• Have a flexible thinking.
• Be capable of seeing the whole picture of the project.
• Have detective skills.

My custom process

1. Setup SDLC: when we will be performing testing?
2. Test plan: What does software do, what needs to be tested/automated? What is tested/automated now? Which tests will be executed, when and by whom?
3. Test cases: how to write test case that are good for QAs and AQAs
4. Automation principles: make tests useful for team member. Tests should bring benefit.
5. Test execution and reporting

Sprint 0: How to start?

The tester collaborates with the team on the following activities during this iteration:

1. Understanding the structure of the organization, who is above whom and who is responsible for what?
2. Understand what are their expectations regarding me, what goals did they pursue when opening this position?
3. Identify the scope of the project. Features. Goal of project. What do we need to test? Deadlines? Use questions list.
4. Create an system architecture and high-level prototypes.
5. Create initial test plan (coverage goals, test scope and deadlines, risks, test types, release plan)
‣
More
1. Describe the goal of testing and QA process. Everyone is a tester. Wrote code then test it.
2. Define test metrics to measure the test process, the progress of testing in the project, and product quality
3. Decide test case management tool (where, how, following which design principles).
4. Define processes
5. Release testing planning: define when to continue or stop testing before delivering the system to the customer
6. Specify the definition of “done”
7. Plan, acquire, and install needed tools (e.g., for test management, defect management, test automation, and continuous integration).

EngX principles

1. CI/CD pipeline: branching, environments, tests execution
2. TDD and testing guidelines: unit/integration, API tests, UI E2E tests
3. Code review: code quality, clean coding practices
4. Documentation

Collaboration between Business, Development and QA.

Business — What problem are we trying to solve?

Development — How might we build a solution to solve that problem?

Testing — What about this, what could possibly happen

Feature workflow: Test plan → Test case → Automation → Update Test plan

The workflow below details the process that we will adopt internally.

1. Get story before planning session, story should contain Acceptance Criteria
2. Create a checklist according to acceptance criteria and the description
3. Note unclear details/questions
4. Clarify questions on planning
5. Update the checklist
6. Highlight any dependencies and how you're going to overcome them
7. When the story is in Review check if acceptance criteria are covered by autotests
8. Encourage developer to cover all acceptance criteria with autotests or do it yourself
9. When the story is ready for testing perform manual/automated testing using the checklist
1. AQA: Automate or not? Put into some suit: light regression, big regression.
10. Create bugs for the story if they exist and return the item to development
11. When bugs are fixed perform manual testing using the checklist again
12. Check if all autotests are passed
13. Create a task for implementing additional integration autotests if it is needed
14. Move story in QA Passed state

Test Plan

Before you begin designing test cases, it's crucial to understand the goals and objectives of your testing. Please create a small test plan.

Test plan should contain following:

1. What does the software do?
1. Components: nouns that define the major chunks that comprise the product. They could be module names and features of the application.
2. Use cases: verbs to describe user actions and activities.
2. What needs to be tested/automated? By what time do we want to test them? To get the answer to this question you need to ask what the software you are working on a KNOWN for.
3. And track what is tested/automated now? We use Zephyr to do that.

The other information in a test plan is typically generic and remains similar from project to project, such as the types of testing we want to perform.

Types of testing:

• Functional
• Developers are responsible for unit and integration testing to test their code.
• Functional testing is conducted by QA and AQA. We perform UI and API testing and categorize test cases into regression and smoke suites.
• If necessary, we also conduct cross-platform and cross-browser testing.
• Non-functional
• Depending on the product's maturity stage, we may need to perform performance testing.
• Security testing is partially handled by DevOps and developers.

For reference, you can see how Google Test Engineers create their test plans: Article 1, Article 2.

How to understand that testing is useful?

• Create the quality dashboard to see if the quality of product is increasing or not.
• Number of builds fail, number of unit tests, support team.

How to implement continuous testing?

To build quality into the software, you must continually run both automated and manual tests throughout the delivery process to validate the functionality and architecture of the system under development.

• Testing should work along with developers (testing is a role, not a full time job)
• Perform manual test activities such as exploratory testing, usability testing, and acceptance testing throughout the delivery process.
• In such pipelines, every change runs a build that creates software packages, executes unit tests, and possibly performs other checks, such as static analysis. After these packages pass the first stage, more comprehensive automated acceptance tests, and likely some nonfunctional tests such as performance tests and vulnerability scans, run against automatically deployed running software. Any build that passes the acceptance stage is then typically made available for manual exploration and usability testing. Finally, if no errors are found in these manual steps, the app is considered releasable
• Start from easier tests to get faster feedback.

Common pitfalls

Not having developers involved in testing

• Involve the developers into testing. Performance will be increased. Why?
• Test suites are frequently in a broken state. Code changes might require tests to be updated. If developers are not responsible for test automation, the build pipeline stays broken until the responsible team fixes the tests.
• Developers write code that is hard to test. Developers tend to solve the problem they are given without thinking about how it will be tested. This can lead to poorly designed code and expensive, hard-to-maintain test suites.
• Failing to curate your test suites. Make sure you continuously review and improve your test suites to better find defects and keep complexity and cost under control.

Failing to curate / check / update your test suites.

• Acceptance test suites should typically represent real end-to-end user journeys through the system
• If you change many unit test when code is updated, you are relaying to mocking to much.
• Use Page object pattern for UI automation.

Having the wrong proportion of unit and acceptance tests. A specific design goal of an automated test suite is to find errors as early as possible.

Tolerating unreliable tests. Tests should be reliable: that is, when the tests pass we should be confident the software is releasable, and test failures should indicate a real defect. In particular, don't tolerate flaky tests

Advice:

• Enhance the communication and collaboration: Meeting with Business Analyst, Solution architect (person responsible for flow of information) → how it will be built and tested, why it is done like this, what are the dependencies, and how to test?
• QA should document everything (RTM & Automated Coverage Report), DEV & DM read and verify that.

Questions to get requirements

• What are the primary objectives and goals of the testing phase?

Add: Ensuring the quality and reliability of software applications before they are released to end-users, helping to reduce risks, improve user satisfaction, and enhance the overall software development process.

• Product
• Who are the target users of the software, and what are their specific requirements and expectations?
• What kind of features we want to test?
• Test new feature addition or completely as a product? Or its core functionalities?
• Testing
• Do we have any tests? And what kind of?
• What are the metrics we are using to measure the success of the testing process?
• What kind of testing we want to perform, and in what kind time-limits?
• Black-box/white-box testing?
• Automated testing/manual testing?
• Security testing, performance, reliability, i8n.
• Do we have any known bugs that should be verified?
• 3rd parties? And environments (QA, DEV, PROD) we have?

Testing a real world object

Testing a piece of software

We focus on details of performing testing. Similar to real objects (pen).

How to start testing?

Clarify:

• What do we need to test?

• What are timelines?
• What testing types will be in scope? How they will be performed? (when by whom and why)

Test the process of testing

• What kind of metrics will we use?
• How to decide that the test is helpful?
• How to measure the effectiveness and efficiency of testing?
• How do we measure that we are performing testing efficiently?
• What do they expect from us (decrease number of bugs)?

Resources

• https://trueengineering.ru/ru/cases/How-to-create-a-testing-strategy Nice
• https://habr.com/ru/company/otus/blog/588923/ About test doc
• https://cloud.google.com/architecture/devops/devops-tech-test-automation
• https://habr.com/ru/post/439128/