TL;DR
Test strategy — Answers “what” questions. Test approaches and goals. What are the objectives of testing? Will be there any automation? Types of testing (functional, regression, perf). Metrics. Risks. Finish line. Tools. - Testing objectives, testing levels and types in scope, testing types not is scope (why → time and resources?), automation (kind what should be covered, defect management, documentation rules), risks and mitigations, finish line?
Test plan — Guiding the testing team on what to test, how to test, and when to test (Action plan = Test). All implemented and planned testing. Plan how to execute testing (how functional testing will occur, manual or auto). Answers “how” questions. What, when, how, by whom?
- How we will do functional testing, how we will do performance testing?
- Scope of features covered with testing and not covered
- How many cycles?
- Test execution guides (test env, test reports)
- Test scenarios, test cases, test techniques, test schedule
The main objective of creating Test plan is focusing on resources and schedules:
- Project scope
- Roles and Responsibilities
- Deadlines and deliverables
10 minutes test plan
- How much time we have to test, what kind of testing we want to perform?
- What does the software do (features)?
- What needs to be tested? To get the answer to this question you need to ask what the software you are working on a KNOWN for.
- What is tested now?
- Attributes: Value proposition
the adverbs and adjectives that describe the high level concepts testing is meant to ensure. Attributes such as fast, usable, secure, accessible and so forth.
Components
the nouns that define the major code chunks that comprise the product. These are classes, module names and features of the application.
Capabilities
the verbs that describe user actions and activities.
What include in test plan?
Coverage (what kind of features are there, what needs to be tested), what will be manual and automated, test types) - What kind of application is it? Multi-platform client-server? What are the use-cases?
- Which platform supported (OS, hardware, devices)
- List of features (how each component/feature will be tested)
- What will not be tested?
It is not possible to test everything. What will not be tested, or tested at first place → Examples: low risk areas that are a low priority, complex cases that are a low priority, areas covered by other teams, features not ready for testing, etc.
- What is covered by unit, integration, system tests?
We must have more small tests.
- What will be tested manually vs. automated?
Which test cases should be automated? Test cases that are executed repeatedly and requirements are not changing. Automated complex scenarios: test cases that involve multiple data inputs, many states calculations, and many interactions. They are time consuming.
- How are you covering each test category? (functional, accessibility, perf, security, smoke, usability) When feasible and cost-effective, automation is usually best. Describe the types of cases that will be tested manually and provide rationale.
- Any mocking?
- Line coverage, static code analysis tools?
Tooling and infrastructure - What builds are your tests running against? Which branch is a release candidate?
- Do you need new test frameworks?
- For E2E testing, how will you build test environment/infrastructure?
Process (how to use tests during release, is there any scheduled testing?, how to handle bugs, monitoring with metrics, acceptance criteria) - Enough time for testing?
- Change to requirements
- Impact to other project
- Hardware damage
Test plan
Test plan — describes a list of tested components, quality criteria and risks of testing, resources to be used for testing, metrics by which results will be measured, testing strategy and testing types.
- A document describing the scope, resources, and schedule of intended testing activities. It identifies test items, the features to be tested, the testing tasks, who will do each task, and any risks requiring contingency planning.
- The test plan is a more detailed document that provides guidance to the testing team on how to perform testing.
- Test methodologies and techniques: Describes the testing approaches and methods that will be used, such as black-box testing, white-box testing, regression testing, etc.
How to write a test plan? - Analyze the product
- Test methodology (agile or waterfall), depending on product select it
- Who will use the website?
- What is it used for?
- How will it work?
- Design the Test Strategy
- Define the Test Objectives
- Scope (which is tested and which part will not be tested)
- Based on customer requirements
- Project budget
- Project specification
- Skills and talent
- Which part of software will be automated
- Features to be tested, test techniques, testing tasks, features pass or fail criteria, test deliverables, responsibilities, and schedule. Planning of resources. Test environment planning
- When will the test occur?
- How functional, regression, perf testing will occur?
- Define Test Criteria Acceptance, Metrics
- Number of test cases executed, passed, failed, critical, non-critical, severity, priority, blockers
- Even if the test failed, we can proceed with release if it is not a blocker, or it is not depend on the team’s work
- Resource Planning
- Plan Test Environment
- Schedule & Estimation
- Determine Test Deliverables
Test strategy
Test Strategy provides a higher-level overview of the testing approach and objectives, helping stakeholders understand the overall direction and goals of the testing effort.
- Testing objectives: Defines the overall goals of the testing effort, such as ensuring the quality of the product, meeting customer expectations, etc.
- Test levels and types: Specifies the different levels of testing (e.g., unit testing, integration testing, system testing, etc.) and the testing types (e.g., functional testing, performance testing, security testing, etc.) that will be performed.
- Test automation: Outlines the strategy for test automation, identifying which tests will be automated and the tools and frameworks that will be used.
- Test data management: Describes how test data will be obtained, created, and managed throughout the testing process.
- Documentation formats, test processes, team reporting structure, client communication strategy.
- Defect management: Describes how defects will be reported, tracked, and resolved.
- Test team organization: Describes the roles and responsibilities of the testing team members.
- Entry and exit criteria: Specifies the conditions under which testing activities will begin and end for each testing phase.
Testing at Google
Small, medium, large tests Small tests cover a single unit of code in a completely faked environment. Medium tests cover multiple and interacting units of code in a faked or real environment. Large tests cover any number of units of code in the actual production environment with real and not faked resources
- Small, written by SWE. Not TEs. TEs can just run. Runs in 1 second. Mocking and faking is used. “Does this code do what it is supposed to do?”
- Medium, involve two or more interacting features. “Does a set of near neighbor functions interoperate with each other the way they are supposed to?”
- Large tests cover three or more (usually more) features and represent real user scenarios, use real user data sources, and can take hours or even longer to run. Checking that software satisfies user needs. “Does the product operate the way a user would expect and produce the desired results?”
Resources
