Proposed Experiment to Test the Eternal Question: success vs. Failure

[andyhre]

I think I have come up with a potential experiement to test the hypothesis that the chance fo a skill up is/isn't higher on success than failure.

As we all know there are many issues of causation vs. correlation that cause our anecdotal evidence to fall flat. What we need is the ability to control for the other related explanations and that has proven elusive. I think, however, that I have an easy experiment that many in the community can perform to help us increase our number of observations AND which goes almost all the way to eliminating other explanations. Sadly, it is not perfect, but I think it might be worth trying.


First we will need to create a series of identical level one test alts. I will leave it to someone else to define the ideal candidate. They will need identical stats, equipment, etc. perhaps the Heady Kiola experiment's set up is the best standard.

Next, half of the alts need to make:

Class 1 Wood Point Arrows
Fletching Components: bundled wooden arrow shafts, field point arrowheads, large groove nocks, several round cut fletchings
In: fletching kit, planar fletching kit, collapsible fletching kit

http://www.eqtraders.com/items/show_...r=080060900000


This trivials at 16 and can be failed.


The other half of the alts need to be given a bow of some sort and they simply restring it. For example, they could take a rough hickory recurve bow and change its strings from hemp to linen and back over and over:

Rough Hickory Recurve Bow

Components: linen string, rough hickory recurve bow
In: fletching kit, planar fletching kit, collapsible fletching kit
Yield: 1
(Notes: combine never fails. Restring TRIVIAL IS 15)

http://www.eqtraders.com/items/show_...r=080060025000



So, in one case, the alts will always succeed. In the other case, they will fail about 33%-50% of the time as they go from zero to fifteen skill.

Whiel the trivials are not identical, the impact of performing a slightly easier (15) combine on the restringing shouldn't be too great vs. a 16 for the arrows. And if the success bonus is worth caring about, it ought to dwarf the 1 trivial difference.

If you get a greater chance of skill up on success, the restringers ought to zoom to 15 much faster than the arrow makers. My personal hunch is that they will not, and that even over repeated sampling, the number of combines needed to get to 15 using the arrows will not differ in a meaningful fashion (i.e. if it is a statistically significiant difference, it will nevertheless be de minimus ).


Okay, so is anyone with me on this? How do we start? Who wants to be the central data collector, etc. Who wants to volunteer to do some combines?


Andy

[Malodium]

I have never felt that you get more skill ups on successes. Seems the same to me really.

[Hibashira]

I don't think that's a very good test.

First, we have no way of knowing how skill up is handled on no-fail combines.

Second, trivial of 15/16 will only produce a small sample, not even close to being representative. Skill up at low level is very fast.

If the test is whether skill ups comes more often on success or failure, then you have to compare across a significant sample of:

1) same tradeskill, same combine
2) same starting level, same ending level
3) same race/class/stats

Instead, I think this might be a better test:

Fetid Essence 0 - 122 over 50 identical toons. That way, each skill up will have a sample size of 50. i.e. from skill 0 to 1 you have 50 samples, from skill 1 to 2 you have 50 samples, etc.

In each set, you will tally how many were skill up on a success vs failure. By comparing the results of the 122 skill ups, you'll have a much more representative picture.

[andyhre]

Okay let's see.

The issue of small sample is handled by doing this experiement many, many times. Given Sony's recent admission that it takes long stretches to be truly random, 50 is no where near enough, I think. You are right that we should consider the data for zero-> one as one sample, one->two as another sample, etc., but we migth be able,m as a first try, to skip that and lump all the data. With this, one combine is always 100% and the other is in a fairly tight range 50-66%.

Admittedly, we don't know how no fails are handled, but I have skilled up my bazaar-alt on tailoring no fails just for kicks and ease of handling (buy some spiderling silk and make thread, saves storage) and she did not get a skill up every time. It felt like any other early skill up progression.

I agree that we want the same race, the same class, the same skill, and ideally the same combine, but suppose that the reason the two things are correlated is that the success roll and the skill up roll are the same (or one is a function of the another), then you will get a correlation b/w success and skill up b/c, say, both use the same (high) roll. But it won't tell you whether to equip a Stein of Maggok or a Geerlok when trying to skill up, since the pure correlation would argue for more INT while the causation would say +5% skill. So you need to have some way of separating that.

You are correct that we may only be discovering that its easier to skill up on restringing than on arrows. That, from this, cannot be disentangled. But, I think that if we were to find no difference, it'd be hard to argue that restringing is just that much harder so that the success bonus was cancelled out by the more difficult skill chance.

[Craftah]

I think a far better test would be to have an enchanter start JC after they have JCM III.

With a 50% reduction in failures, the toon should see a significant number of successes.

If the 'better chance on skillup with success' theory holds true, this toon should GM JC substancially quicker than the rest of us didn't.

Maybe someone has a boxed guild enchanter that never did JC...

BTW, I do believe that the theory is true. The biggest thing that makes me believe so it the fact that the question was asked at a Fan Faire and the developers responded that "Yes, you have a better chance of a skillup on succes than failure."

[andyhre]

We'd defintiely have sample size problems on this one, I think.

;-)

[Hibashira]

Hehe, no kidding, hmmm, n chanters with JC3 vs n level 1 anything.

[eniamn]

lots of times you will be working close to trivial. at the high end where you do tha most combines per skill point you get close to a 95% success rate with a triv really close to you. if you fail 1 in 20 and do 200 combines for a skillup then your chance of skilling up on a failure is low and it may seem like you have better skillups on successes than failures. i dont really think its true myself but /shrug.

[Sylphan]

That's just not so, eniamn. For jewelers and brewers maybe, but not everyone.

People skilling up tailoring on solstice robes often go large stretches with 95% failure rate. In that case all you really need to know is whether 5%, or more than 5%, of the skillups came on successes.

When I was raising smithing the failure rate often around 40% over several thousand combines in the 200-225 range. But skillups on failures were closer to 20% of total skillups.

Even if you do get only 5% failure, the question is whether 5% of your skillups come from those failures, or more than 5%, or less than 5%.

[Cynix]

I looked in vain for the previous thread, but I know this has been tested rather conclusively before. The person doing it created two identical alts and skilled them up to over 100 in jewelcraft. One alt used the traditional approach of always using the next closest trivial combination. The other alt only used the high trivial combination at the ultimate skill level he was shooting for.

The alt that always used the next closest trivial combination took significantly (I think about 30%) fewer attempts to reach the target skill level.

[Lothay]

I beleive the thread you reference is not included on these boards due to our move from EzBoard to a self-hosted board.

On a related note, Laurafin did some testing to determine the effects of INT/WIS on the rate at which you skill up.

While this doesn't answer the main question in this thread, I thought someone might get a kick out of it.

For those dying for a the graph:



The graph shows WIS/INT stat along the bottom, and the average attempt per skill-up at the time she conducted these tests. Each skill is represented in a different color.

NOTE: This data was collected over a year ago. It cannot account for anything added to the game since then, and I'm including it here for informational purposes only.

[andyhre]

this has been tested rather conclusively before. The person doing it created two identical alts and skilled them up to over 100 in jewelcraft. One alt used the traditional approach of always using the next closest trivial combination. The other alt only used the high trivial combination at the ultimate skill level he was shooting for.

The problem is, is that this doesn't quite test the cause. We can't untangle whether skill ups and successes are both easier when you are closer to the trivials (we know successes are).

Taking fewer combines to skill up close to trivial gives good advice: stay close to trivial. it does not help answer the question, though, when you WIS/INT is 245 with a geerlok in your primary or 255 with a WIS/INT item in your primary, whether you shoudl aim for more INT or more success.

[Gaspode]

I think the way to test this would be to skill both Char up to maybe 150 JC straight, not following the closest trivial way. One Char should hold a geerlok, the other doesn't. In the stretch from 0-100 you should not see much difference in succes rate, but at 120 JC the geerlok adds 6 Skill points. Maybe at this level you see a difference.
I hadn't had the option to try this experiement, because you would need at least 10+ Runs to get any statistical acceptable data. And i don't have the money.

P.S. I'd prefer JC to fletching, because it takes only 2 Components.

[Chenier]

I remember another old test that was done with fishrolls to 100...just to get a % of how many times one got a skill up on a successful vs non successful combine...

But I can't remember the findings...

[Moraganth]

BTW, I do believe that the theory is true. The biggest thing that makes me believe so it the fact that the question was asked at a Fan Faire and the developers responded that "Yes, you have a better chance of a skillup on succes than failure."

Which fan fair? i don't remember ever seeing this reported.

Ultimately to test this, all anyone needs to do is a bunch of combines.. and record the number of combines, failures, successes, and wether the skillups came on failure or success.

The percentage of points coming on failure or success should match the percentage of fails and successes over the run if it doesn't matter.

What this means is, while we're all working our various skills, if we can post those batch numbers, all we have to do is collate them all together, then figure percentages. Depending on how the percentages are skewed, is the answer.

This makes the assumtion that the answer is universal. If they've coded one tradeskill differently than the others, then this won't work. I would think that the basics for each tradeskill would have been done the same, purely to save time and energy, although it might be a good idea to run 8 sets of numbers.. one for each skill, and a collated batch.