Software Testing Estimation eCommerce Projects

There are many ways to estimate and goal should be to stay in the 10-20% range of the actual effort needed to validate the accuracy of the system being built.

Software estimation is inherently non-trivial. The resulting product is virtually invisible until it is finished and we rarely end up with the same product that we initially estimated. Most of the estimates are based on the early requirements that are planned for the release or product roadmap and this can impact our estimation since effort estimated is different from actual effort when the test cycle starts. Estimation process can be challenging where we have limited knowledge about the application/product organization ecosystem which we plan to test mostly for a new client, domain or product. 

With many years of experience these are some set of guidelines which have been very helpful in the estimation process and want to share in my blog. The goal that I have always set was to come in about 15-20% ball park of the actual effort with these estimation techniques.  

Estimation Components 

1. Project Complexity/Requirements 

2. Project Schedule 

3. Techonology Stack [Product vs Custom Build]

4. Historical Test Cycle Data 

5. Organization controls and audit 

6. Development Methodology (Agile / Waterfall) 

7. Testing Scope

8. Expert Judgement 

9. Reporting Needs 

10. Resource Allocation By Week  

Add Ins 

1. Team Ramp Up 

2. Vacations (As allowed by the company) 

3. Unscheduled Time Offs (Illness/Weather) 

4. Risks  

1. Project Complexity The main goal here is to identify whether this is an independent application or dependent on multiple other systems existing in the organization. This is important for estimation because we need to add overhead in terms of dependencies to account for QA team to work with these teams in case of any issues in the integration. For example being dependent on a billing system is very different from being dependent on a REST service or API from another team that you do not have influence over.  

2. Project Schedule The goal here is to understand how much testing time is available for test team to complete testing. This varies by type of testing team whether it is System Test Team or UAT Team. Imagine we have estimated about 1000 scripts for the project and this need to be run in the time frame of 3 weeks with a velocity of 10 scripts per tester per day. We are looking at about 30 Testers and which in turn drives our staffing/ramp up and other add ins. On the other hand if we have to run only 300 scripts we will need only 10 testers.  

3. Technology Stack The technology drives the test scenarios at times and the complexity of what needs to be tested. This will give us an idea of what skills the testers need and depending on the organization they have to factor in additional ramp up for the tech stack as well where appropriate. One such example is when we were doing estimates to test a project built using ATG I had to build in additional LOE for the BCC training as well as overall end to end flow training to my testers.  

4. Historical Data This is one of the most important peice of information that will drive the estimates. If we have a project that was performed in the past which of similar complexity and technology we can use the numbers and get to a more closer estimates. This has to be considered in conjunction with the expert estimates. 

5. Organization Mandated Processes This is the key that defines the deliverables at each phase and in turn helps to estimate the effort that is needed to make those deliverables happen. I would always ask this question before providing an estimate to the key stake holders of the project. If this is not availabe I would suggest to come up with the plan and publish it with the estimates. Sometimes organizations need that we use specific tools to load the scripts and some other tool to log the defects. Some defect management tools can take as much as 30 minutes to log a single defect.  

6. Development Methodology The way code is being available to test team is very important. If a big program is being delivered in an Agile SCRUM methodology we might have to re run the same scripts may be three times before it gets into production making an assumption that there are three iterations. However if the project is in waterfall model than we migh have to run most of the scripts only once or twice if needed. Agile adds additional overhead to the test team LOE as well as more involvement to understand the day to day changes and be flexible to modify the scripts as needed. In Agile/TDD the typical industry standard 2 Developers to 1 Tester(33%) works provided we don’t have to write formal scripts, upload to QC, review formally with business/dev/BA, support UAT, regression testing is done through JUNIT/NUNIT, no UAT support, no warranty with three weeks production launch cycle. We always have to verify the context of the industry standard and if it matches the project/program context. 

7. Testing Scope This is a no-brainer that the testing scope defines the LOE. However I also wanted to highlight the fact that apart from the explicit scope defined by the requirements there could be implicit tasks such as Data Prep requiring interaction with other teams in the enterprise, Integration points, supporting business for data/defect triage, Security and Accessibility testing. We also need to understand any Merge testing or Integration testing needs during the program. Also look out for non functional testing related scope on the project. Enterprise system test scope based on my experience for an ATG eCommerce project. 

 

We should break the scope using WBS(Work Breakdown Structure) and estimate each row taking into consideration the key metrics against each row.  

WBS Example 

♦ Requirements Walkthrough - 15 Days    

- Requirements Walkthrough - 8 Days    

- Testing Gap Analysis - 7 Days 

♦ Test Script Strategy - 10 Days 

♦ Manual Script Writing 

♦ Test Requirements - 23 Days    - Write Test Requirements - 15 Days    - Review TRs - 5 Days    - TR Comment Incorporation - 3 Days 

♦ Test Cases - 33 Days    - Write Test Cases - 20 Days    - Review Test Cases - 8 Days    - TC Comment Incorporation - 5 Days 

♦ Automation Scripting - 25 Days 

♦ Test Data Creation - 60 Days 

♦ Test Execution - 40 Days 

♦ Defect Logging - 5 Days 

♦ Retesting - 5 Days 

♦ Defect Review Follow up - 3 Days 

♦ Production Shakeout - 8 Days 

♦ Warranty - 30 Days (2 Weeks)  

8. Expert Judgement In its crudest form the expert judgement method involves consultation with one or more local experts who are knowledgeable about the testing environment or application domain to estimate the effort required to complete a software project. This helps to ensure that we cover all the bases in the WBS and also run through the key metrics. 

9. Reporting Needs This is one critical part of the testing project that is always underestimated. There is a burden of about 1-2 hours by the test leads and managers to create the test status reports daily and also some projects need dashboards. This time is about 25% overhead for the leads and also testers to provide any information required in some template. 

 10. Resource Allocation By Week This is very critical since sometimes the estimates from WBS might come out to be some factor which might not exactly map into the resource allocation. We need to make sure the final number submitted for the project will work practically by laying out the resource allocation by week by role. This will also help in estimating the actual cost of the testing for the project. 

Validate using development effort as baseline - In the end I want to provide a tip that to be safe when you do not have details to do a bottom-up estimate is to use about 60% development LOE as the baseline for QA Estimates for web/ecomm projects. If development LOE is not available as well estimate the number of scripts and multiply that with 2 hrs for each script. This approach has helped me to come up in the 20% ball park in most of the projects. 

Do not assume anything that is not documented during the estimation process. It is very important that the Key stakeholders and program sponsors understand the assumptions during the estimate or timeline negotiations. 

I will keep this document updated as I uncover more while working on my projects.