Beta testing is a joke.

[sputnik monroe 102]

I found this posted over in the SIMHQ forums and had a good laugh. By laugh I mean a sarcastic laugh, it's pathetic that this is a true story.

Quote[/b] ] I have been beta testing an upcoming game, I got on the beta team because I had previously tested other games for the publisher.

Well day before yesterday I got an email from the developer explaining that I was no longer on the team because of my excessive bug reports. They explained that there were only a couple of testers on the team who were going overboard with the bug reports and that all of us were no longer on the beta team. On the private forum there are only about 5 people who actively file reports, it was easy to figure out who was no longer on the team, so I emailed each one and sure enough, none of us were on the team. Here is the funny part. ALL of us were chosen by the publisher.

So, I called my contact at the publisher to find out what was going on. They got back to me yesterday. They contacted the developer who told them that we were no longer allowed to be on the team due to excessive bug reports. The publisher was able to check the bug database and found out if you remove all our bug reports from the bug database, you would be left with 3 bug reports. They have over 500 testers. The 3 bug reports that were left were all submitted by the same person, which means they have approx 494 testers who have never submitted a single report in over 18 months of testing. When the publisher asked how they picked the testers for this game, it was explained that they went through the forums for the previous game and found all the people who were the most positive about the game. They refused to pick anyone who was remotely negative.

And some people wonder WHY games come out with major bugs.

BTW, as a side note, we are all back on the beta team and we are going to start submitting out bugs to the publisher instead of thru the developer.

 Why do programmers have a problem with fixing bugs? I've beta tested before for a game and also helped to test many mods and missions for the Flashpoint community. I've lost track now of all the various bugs I've reported to people over the years in beta test that end up in the final release any way. A lot of times the bugs are so appallingly obvious that you would think the programmers would be ashamed that such a painfully apparent bug made it into the release. (A perfect example is an APC that RPGs fly through, I mean if you make a tank for Flashpoint it would seem that firing a rocket at it in game would be part of the battery of test to put it throught)

 I honestly believe most programmers be they professional payed developers or non profit modders, I believe that they truly think a beta test session is just to drop something into a sand box level and take screenshots of it.

 I know when I release a mission and it has a bug in it I feel ashamed. If some one points out a bug I try to figure out how to fix it, I don't get all indignant and attack the messenger. Take pride in your work and if it's broken fix it, if some one finds a mistake in your work and lets you know remember it's not their fault it's yours, you are the programmer. You should thank them for pointing out the mistake and then see if you can fix it.

 Sheesh.

[baddo 0]

Quite interesting.

I think anyone who have worked in some kind of development projects can imagine why this kind of things happen. I have worked in development projects (not software, but it doesn't matter as development project basics are the same in every project). I can easily imagine (remember) how a working day is completely ruined when someone comes up with something related to my work, brings major changes to a significant part of my work, and how my whole schedule goes absolutely crazy, or impossible, after that. When a development project is started, a plan and a schedule is written. Project manager then invites everyone from time to time into a project meeting, and checks if we are in schedule. If you think about how many hours you have in a typical office working day, you can easily see how the whole day can be quickly ruined if there is distraction from your work, for example as in form of someone showing you what is wrong in your work in their opinion and that you should do it their way, starting to work on it right now. The only sane way to approach this problem in my opinion is to keep the 'bug list' organized but it is not possible to let the 'bug list' to decide what kind of schedule and plan you have for your work. You just keep working according to some pre-determined plan, on the way filtering the noise out of the signal as best as you possibly can.

It seems that in your example a lot of input were seen as 'unwanted noise' which disturbs the work the developers are trying to do (according to their original plan and schedule).

There is only so many hours per day and that is a big part of the problem, everyone who has worked in any kind of development projects understand this. It is practically impossible to keep on the schedule if you listen to all of the people who have something to say about your work, which would require you to change your working plan all the time. I know that in many projects I have participated in, a lot of things could have been done much better but it was not possible in the given time and resources. A project has to end at some point and it isn't doing that if everyone is asked if the project is ready.

So, on one hand I understand why things like in your example happen, but on the other hand I of course wish it wouldn't happen. Sometimes the balance shifts too much to the other end.

[sanctuary 19]

Awesome.

No wonder games are all very buggy in their release nowadays.

But from this report it seems that 99% of beta testers are in fact totally useless.

[Black Sphere 0]

Thanks Sputnik Monroe for this info, this shines some more light on all ordeal with the recent new games and their state of publishing, and as it seems on the upcoming games also. There's nothing much to add, and this revealing was not necessary for me personally, I know and understand it for a some time now what's going on in gaming industry.

I was a beta tester for some game back then when such practice was nothing but a heresy among the developers and publishers alike, when the games were beta tested until they were like flawless, and those times are not so way back.

@Baddo - I just hope your involvement in development project is not in bridge or building constructing, because I would never like to cross the bridge or live in a building where you have been involved/worked in developing project e.g. such way of 'working'.

[martinovic 0]

Awesome.

No wonder games are all very buggy in their release nowadays.

But from this report it seems that 99% of beta testers are in fact totally useless.

Yes, but only because the developers hired yesmen.

There has to be a balance between hiring Pointdexter the Annoying One and Joe the Slouch of Ages...

Don't forget however that beta testing usually isn't just about the bugs, but about the performance of the game on a wide range of computers.

Internal testers are for bug finding usually, while beta testers are for performance evaluation (that's probably why they hired people who were enthusiastic about the game, so they will play it a lot), so in the end both sides are wrong and right at the same time, as always...

[jerryhopper 285]

i once have been alpha and beta testing too....

for the heck of it, i participated!

it was bf2142 testing what a laugh....

Not shortly after, they asked me to test crysis...

while i was still sick of irritation of the 2142 testing i passed.

the beta testing of games, is not done by people with some technical background. certain 'wrong' stuff is still in 2142 till today. most testers have big e-penises.

but face it : if there was no beta testing - shit would be worse!

[Turkmenbashi 0]

oh lol.

[chops 111]

Why beta test at all, if all you want to hear is how perfect your game is? What kind of fragile little princess would you have to be to not be able to handle bug reports?

I blame the parents. Giving kids trophies for just turning up is where it all starts.

[baddo 0]

@Baddo - I just hope your involvement in development project is not in bridge or building constructing, because I would never like to cross the bridge or live in a building where you have been involved/worked in developing project e.g. such way of 'working'.

You seem to not understand what I was talking about. I'll try to make it more clear this time. Project plan and schedule have the main priority in determining how the project work proceeds. If a problem arises, its priority is weighed and a decision is made if the problem needs fixing or not. This is everyday life in any kind of project development work. A bug in a computer software product gets a priority too.

My previous post was to tell that I understand the reasons why a development team might do what Sputnik Monroe's example tells us. In a perfect world everything would be perfect but somehow I have a feeling that it is a too ambitious goal to have everything perfect, especially if projects have to end some day. A project does not end at all if you go around asking people if they think it is completely ready.

Your example of bridge development. 'Bugs' are always prioritized, no matter what kind of project. For example, if the materials chosen for the bridge smell bad (very non-critical), versus wrong type of steel is planned to be used in main supporting beams (very critical). The steel type is going to get changed of course. But the smell... maybe people living near the bridge just have to deal with it, as our project is already too far in schedule to make a change to the smell possible.

It's going to be very chaotic if you let outsiders affect your work too much. That's my main point, you have your plan, schedule and resources and you have to keep working with what you got. If a couple of hunded outsiders suddenly come and try to change your whole plan and schedule, but resources stay the same, what are you going to do? Best thing is to keep working according to your original plan and schedule, and filter the noise out from the input signal in the best possible way. And then you go and fix the problems that can be fixed with the resources and time you have available, non-critical problems have a lower priority and might not get fixed. In Sputnik Monroe's example, as I already said (but you didn't notice), this filtering of input signal seemed to go too far.

Your remark towards me is insulting and only shows that you have neither experience or understanding of development projects. Please get some experience and think about my words again, maybe you'll then understand what I mean.

[kronzky 5]

Every somewhat complex software has bugs.

Every developer knows that, and if he's been in the business for a while, has learned to live with that fact.

Sure, it's a blow to your ego at first to realize that there is no such thing as a "perfect program", no matter how hard you try, but if everybody were to test and fine-tune their software until its 100% bug free we wouldn't have much beyond "Hello World" programs.

So, keeping that in mind, beta testing and finding a release date is mainly a thing of prioritizing. Everybody has deadlines and limited budgets, so the only difference between testing software for a space shuttle and one for a little giveaway proggie is where you draw the threshold of "good enough".

Obviously, software that controls a multi-million dollar project like a Mars Rover will have a fairly high testing standard, but even then you will end up with bugs, despite everyone's best efforts.

If, on the other hand, you write a little game that might be in the public eye for a few months, and then it's forgotten, you know that, because of the limited use this software will get, that a lot of bugs will not even matter, since the likelyhood of somebody ever stumbling over it during its brief lifetime is fairly slim.

If the example in the first post is a situation like this (and you can bet that the developer has extremely tight deadlines and budgets), and there are only 3 major bugs left, and a few dozen of "If I jump up and down 10 times, and then twirl around 20 times, and then eat 50 mushrooms in a row, the screen flashes green", then I can understand them dismissing reports like that. And unless you're willing to spend a few thousand dollars for your games, and wait a few decades for every release, it will probably be in your interest as well.

It's hard to judge by the article what quality bug reports we're talking about, but it pretty much sounds like some sour grapes post, by some very subjective and pissed off ex beta-tester. (Which makes it pretty meaningless as an indication of the beta testing scene in general.)

[Black Sphere 0]

@Baddo - My apologies if you feel offended by my comment, there were no such intention. But people differs here, so I don't feel offended by your statement that I don't have experience or understanding of development projects.

I won't go deeper into this because is off-topic, also don't have the will for it, but I think it's not my fault if not the best example -to 'understand' and to justify the dev's lack of willing to face and to correct the bugs/errors which were found?- was picked and not well presented by you. It's understandable that such new and unexpected 'features' might change the project's schedule and put some pressure on the available/remaining time, but every good project have an added time window for such circumstances, and if not, then mostly and because of it some, often rotten and quality damaging compromises must be maded. And some are striving to perfection/ism, no matter for the world around them, and some not.

If, on the other hand, you write a little game that might be in the public eye for a few months, and then it's forgotten, you know that, because of the limited use this software will get, that a lot of bugs will not even matter, since the likelyhood of somebody ever stumbling over it during its brief lifetime is fairly slim.

If the example in the first post is a situation like this (and you can bet that the developer has extremely tight deadlines and budgets), and there are only 3 major bugs left, and a few dozen of "If I jump up and down 10 times, and then twirl around 20 times, and then eat 50 mushrooms in a row, the screen flashes green", then I can understand them dismissing reports like that. And unless you're willing to spend a few thousand dollars for your games, and wait a few decades for every release, it will probably be in your interest as well.

It's hard to judge by the article what quality bug reports we're talking about, but it pretty much sounds like some sour grapes post, by some very subjective and pissed off ex beta-tester. (Which makes it pretty meaningless as an indication of the beta testing scene in general.)

You must be talking with ArmA in mind here ... Or this are just some empty assumptions with not much sense in it?

[shinRaiden 0]

This can go both ways. First off, the fact that 494 of the 500 'testers' were making zero contributions is clear indication that their testing process was inherently broken. On the other hand, there's the good old 10-90 club, the 10% of users who ties up 90% of support resources with their pestering. I was coincidently looking at some statistics just yesterday for a certain project which aptly illustrated that effect.

Let's start with the first part. Public beta's, useful or no? I'd argue no, that open beta's rather are a promotional construct, like a demo or something, designed to lure in n00bers with the false impression that they're some sort of serious VIP instead of just an satiated pimple-head. You've got essentially stable code in the package, all that's really left is scalability load testing and promotion. What you won't get in public beta's is organized and disciplined testing systems. You'll get random people pushing random buttons in whatever way they feel like, and good luck getting any usable feedback. If you do get feedback, more often than not it's the "T3 gunships sux0r", and not any actual helpful input about what to fix, how to fix it, and why.

So public, or even private beta's are a crock. Those involved ought to know better that their job is to promote the product and not actually test it because they're getting comped a preview, so they should shut up the spamming and start providing their 'VIP' free marketing. If they're one of those nose up in the air types that actually thinks it's their job/position to offer constructive advice well it's time to grow up and stop being a n00ber.

Ok, part two. Effective tech-level testing is best done in-house, or at least managed in-house. This is because you need to have systematic processes to comprehensively analyze the content in a development perspective, rather than a usability context. You also need tight linking between the testers and the developers there as well. The testers need to communicate to the developers what's not working, why it is not, and recommended technical fixes. The developers likewise need to communicate comprehensive non-playing analysis procedures to the testers. This inherently does not work with public or even private betas.

Testing part two however is usability testing. This would on the surface seem to be applicable to public beta's, but I'll show why it's not. Usability testing is non-technical testing, where you have large numbers of unguided testers going in, pushing buttons, and responding about counter-intuitive things or other blocks. Again, this has to be highly supervised, but for somewhat different outcomes. Users focused on the target application, are unlikely to interrupt themselves to make an instant assessment and report, and their reports may need to be clarified to determine the precise technical target. The most effective context is on-site testing, where a team of usability experts and developers monitor face to face a large pool of testers. At this point, finding technical bugs is not the focus, rather the target is to see if the usage process works.

With public betas, there's a risk of bad data in unsupervised usability testing. It is possible to misinterpret large player/server counts as 'evidence' that the process is working. This discounts the possibility that it doesn't work, but large numbers of people are button-mashing anyway.

Anyway, summarized comments :

* Don't spam

* Append to existing reports IF REQUIRED, don't spam

* Provide developer useful reports

* Clarify if it is a tech or usability issue

* Consider very carefully if it is a true tech issue, or really a feature request.

* Make an honest assessment of the big picture of your issue, and note as appropriate.

* Determine if you're being used for promotion, or if you're really in a testing roll.

Over to you Dinger

[DBO_ 0]

I think it only needs a brief read of the a&m D section and early posts in Arma general to see that 10% of a defined group can really grind a Dev or freelance addonmaker down with pedantic negative comments.

I guess beta testers that are well outside the normal framework of the actual software house are needed at a certain stage to give that "outside the box" perspective. the problem is they must also be inside the bigger box I.E the gaming community in this case and inevitabley you will get the 10% that really do treat every single minute error has an absolute game crushing flaw and however much merit that style may have ,the fact remains that too much negativity does have an effect and there will be an inevitable attempt to filter it somewhat .

such is life

[Spokesperson 0]

Maybe they feared a leak of the beta? At least they save money on what the beta testers do, so I see no reason not to keep people who work for free.

[Gunney 0]

Not shortly after, they asked me to test crysis...

while i was still sick of irritation of the 2142 testing i passed.

It is official, you are physco!

[InqWiper 0]

Quote[/b] ]I know when I release a mission and it has a bug in it I feel ashamed. If some one points out a bug I try to figure out how to fix it, I don't get all indignant and attack the messenger. Take pride in your work and if it's broken fix it, if some one finds a mistake in your work and lets you know remember it's not their fault it's yours, you are the programmer. You should thank them for pointing out the mistake and then see if you can fix it.

Agreed.

It seems to me that the developers are n00bs. Its up to the developers to decide what bugs they want to adress. Its not like every time someone reports a minor bug theire work is interrupted. I think more feedback should be better. If they dont think the bug is worth fixing then just dont fix it, how long does it take to read the bug report, 20 seconds? If people report that a character is looking too gay or that another character needs bigger boobs then maybe you should take them off the beta but aslong as they are reporting actual bugs which is the whole idea of having betatesters then I dont see a problem. If it is a problem for the developer that people are reporting too many bugs then it is probably theire system of reviewing reports and adressing bugs that needs fixing. For example you cant have an alarm go off every time someone reports a bug and have 3d artists, 2d artist, coders etc running to a big monitor and check whats going on. It is probably a good idea to have seperate time scheduled for checking bug reports and deciding which bugs need to be adressed. Also if you only have 500 beta testers you can very easily provide individual feedback to the betatesters if they are reporting too much or not elaborate enough.

I dont have much experience with the BIS bug report system, but the little I have seen makes me think its a very good system.

[4 IN 1 0]

well atless BI did fix something for me

[Dwarden 1125]

anyone got idea what was name of game this beta testing was about ?

[4 IN 1 0]

anyone got idea what was name of game this beta testing was about ?

i could only say "I HOPE its not ArmA hes talking about"

thats about as close as you can get i think, basicaly this kind of stuff always happened and for a long time, the fact is, most beta testers sign up just to have a first hand-on of the game under developements, non do the developers want to fix all the things aside from the major major ones with reasons told in above post, and we are talking about a 100 men work group

it always amaze me how BI 20 men team can make somethings like OFP into real

[sputnik monroe 102]

Quote[/b] ]anyone got idea what was name of game this beta testing was about ?

Honestly I don't know. I have a very strong feeling that it is not ARMA. I almost wonder if it could have been Medieval 2 Total War, though I have very strong doubts again.

%99 chance it's some main stream FPS that I've never played or have any interest in playing.

[BraTTy 0]

I maybe know how the developers feel.

I am a professional autobodyman.I have in my mind a list of things that need to be done for completion.

If someone keeps pointing out things while I am trying to complete my tasks , i get confused and sidetracked.

I also get irritated.

I would like someone to look it over when i think its done.

[InqWiper 0]

I maybe know how the developers feel.

I am a professional autobodyman.I have in my mind a list of things that need to be done for completion.

If someone keeps pointing out things while I am trying to complete my tasks , i get confused and sidetracked.

I also get irritated.

I would like someone to look it over when i think its done.

Sure.

But if you have a list in your office where people can write things down you dont have to be interrupted and can look at it while you have the time. You can for example read through the list before you go home so you have until the next day to consider what you want to change and how you want to do it. I dont know how long your projects last on average but I have a feeling they are shorter than game creation so this might not be a realistic posibility for you.

Like I said earlier its not like they have an alarm system that goes off every time someone reports a bug. Have someone responsible of organizing the list and cleaning it. The project leader can then read the list and say every monday morning or something he briefs the crew on what has to be fixed. This is just and example, you can have it daily if you wish or whatever. The point is the work of the developers dont have to get interrupted. You can have bugsolving as a part of the schedule. If you wish you can even have people employed whos only job is to solve bugs. There are alot of bugs in games wich are easy to solve and can be fixed with some general knowledge of the game. Not every bug has to be fixed by the lead programmer  

[Daddl 10]

I'm a software developer myself - we have dedicated inhouse specialists who do nothing but test our software. I have nothing but respect for those people, because they provide us with a first cruel reality check outside our cosy development environment.

Testing and improving software prior to release is an essential part of the development process. And while it's not nice to find out that your beautiful idea simply doesn't work you better find out in a test prior to release - because after that an uncaught bug or an untested idea might cause costs that could easily ruin the company (or, with a game, give your product such a bad press that you don't sell enough and have to close down).

Testing isn't much fun, and it requires dedication and a very systematic approach to help define the source of a problem. But if you got people who are good at that, then be happy, because they are very rare. Test reports require much more than just saying "Ah, bugger, this f...ing sh.t doesn't work!". And that's where I have some problems with public beta tests like the initial article described: where people are lured into 'helping' developers by getting a chance to peek at a new game/software.

Most of these people have no idea what testing is really about, and how to formulate a helpful bug report. Most don't even try to write one (as the first post showed). They want the new cool piece of software, and they want it now. Either they are happy with that and just play around a bit, ignoring even grave bugs - or they are shocked to find out that it isn't finished ( it's a beta, so what did they expect??? ) and just moan. Maybe some of them point out some really obvious bugs that the developers probably had already announced as 'known problems' for the beta (but who cares to read release notes anyways ), and then they leave. Or they bomb you with 500 variations of the same problem, but with such a poor description that you have no idea what to look for.

Of course there are exceptions to this, but they are rare and the result probably not worth the huge effort of filtering and analyzing just to find the few useful reports. But - if you go this route of a public beta - then only allowing people with a strong affection to your product is the most stupid thing you could do - fan boys simply are blind to certain problems that other players are confronted with. You want (and need) criticism and feedback, not just happy fanboys who love their treat of candy!

[dm 9]

I'm a software developer myself - we have dedicated inhouse specialists who do nothing but test our software. I have nothing but respect for those people, because they provide us with a first cruel reality check outside our cosy development environment.

Testing and improving software prior to release is an essential part of the development process...

I think its important to make the distinction between "real" testing, done in-house by dedicated and experienced staff, and "public" testing, done by external inexperienced people, usually fans of the product.

The "real" testing is far from a joke, being an important and integral part of the testing process. The "public" testing is indeed a joke.

[suma 8]

I think its important to make the distinction between "real" testing, done in-house by dedicated and experienced staff, and "public" testing, done by external inexperienced people, usually fans of the product.

The "real" testing is far from a joke, being an important and integral part of the testing process. The "public" testing is indeed a joke.

It depends on what is the expected result of the testing. From my point of view, a public beta is a good way to verify internal Q/A process did not miss any serious issue, which perhaps shows itself only with some particular HW / SW combination which is rare, but not that rare so that we can ignore it.

Besides of "public" and "internal" there is also a middle way, which seems to be the case about which this topic was started, and that is "private" (or sometimes called "closed public"). With private beta you usually carefully select participants to make sure their feedback is useful to you, and as a result the process can be really helpful. You should expect to somewhat more "noise" from external testers compared to internal ones, however as already written, this is quite easy to handle using tools suitable for the purpose (the main tool being well defined priority system).

I would really like to say big "Thank you" here to all external testers who helped us during development of the ArmA - they all did very good job - and there was definitely no one dumped out from the beta.