How do You Test Smaller Projects?
Do you find that TDD is sufficiently fast that you can still knock out use cases in half a day to a day per while including the time to write the tests?
How do you handle integration tests? Do you set up Selenium scripts? For a project with say 20 use cases with an average of 5-8 screens, how long do you spend setting up the scripts, and are there any hints or tips for doing it more quickly?
What about acceptance tests? How do you document them? How long does it take to plan, document, explain, revise, agree and implement them? Any hints on doing it more quickly while keeping up the quality>
And finally, what about load testing? I usually avoid this on smaller projects as most of my clients don't get load. Do you have a standard load testing setup with dedicated load balance servers for testing projects on? How do you agree meaningful tests when load is so dependent on the exact click throughs, the precise search requests, etc.? What is the lowest price you'd do load testing for? Have you found a way to load test a meaningful application in a day? And what about any issues that arise? Does anyone fixed bid projects with non-functional requirements like number of users or do you just fixed bid the functionality (if required) and then charge by the hour for any tweaking of code and SQL required to hit the non-functionals. If you DO fixed bid projects with meaningful non-functional requirements, how do you estimate the effort required?
Any other types of testing you employ?
The upshot of this was that at the very start, the client was made aware that they were responsible for testing their (previously agreed) workflows within the application.
Any changes after delivery and implementation were solely the responsibility of theirs and fell under their own budget, not ours. It was a fairly captive audience (if you know what I mean), but it kept the work flowing, and kept the prices down all round.
Caveat, I'm not a financial guy, just a developer, YMMV.
I finally took our conversation to heart from CFUnited and left my job on Friday to start my own consulting business. In the work I've done at Duo with TDD I've generally found it adds about 50% onto the development estimate because of the forethought, architecture and planning required to do the tests. I was surprised it wasn't longer but I found that although the tests front load the testing effort, the end result is more stable code with less niggling bugs. I've only learned about Selenium in the past two weeks so my exposure is limited although I can speak more about the load testing. I use a cluster of siege servers to run load tests for clients (and this is one of the services I'm offering from my new business). The good news is you can reuse some of your selenium scripts with a little reworking to build your load script. If you have a script of URLs to use and want to simulate a specific number of users, shoot me a message as I'm working on an on demand load testing service right now for really low price and I need to test the rig. I'm expecting to charge $10 for 5 minutes of testing up to 300 concurrent requests and then $10 for each multiple of one of the parameters, e.g. 600 concurrent requests for 5 minutes = $20, 600 concurrent requests for 10 minutes would be $40 and so on. I'm trying to commoditize the simpler aspects of load testing as a service for exactly the reasons you mention in your post.
Adam
Wow - congratulations! Let me know if there's ever anything I can do to help!!!
TDD number sounds about right to me. I almost have a feeling it might save time when you truly include the cost of tracking down those niggling bugs, but I think that's only true once you're really comfortable enough with TDD for it to be automatic. I'm wondering if it's like OO - the biggest pain is getting started and getting good . . .
I love the idea of the on demand load testing system. Right now most of my clients are so small that they're on shared servers, so load testing isn't really a good fit for them, but to have a resource like that for bigger projects would be great!
Again - very cool, best of luck with everything
Interesting service! Had a look through, I understand the pricing model. In practice, what kind of budgets do you find (how many test cases, what ballpark figures, etc.) and how much time does it take from the client to work with you?