quentron.com

[sarkeizen]

It is a folly of hubris to unilaterally declare that low probability dictates that an experiment should not be conducted.

I'm not sure what "unilateral" means in this context but I will say that data should drive your decisions, all your decisions.  Including the decision to run the experiment.  As soon as you (or someone else says) that we should ignore the data saying that the experiment is stupid and perform the experiment - that is being just as arrogant  as ignoring that true equipoise exists.

It is important to distinguish the differences between:  likely, extremely likely, and absolutely true.

I disagree, those distinctions are both difficult to determine and are subsumed but a much better criterion:  Is the experiment likely to produce useful data?  The rationality of performing an experiment is directly proportional to this. 

But none of us are omniscient.

Precisely! Which is why a validating experiment without having an understanding it's likelihood of success is worthless - as an experiment and performing an experiment which is only going to succeed 1 in 10^1000 times is equally so.

Before I give up an idea I must first have it.

If you're talking about research which is hypothesis generating.  This is fine but you have to let go of the end result being meaningful and you would design your experiment differently than profitis did.

I argue the idea that if someone wishes to expend their resources testing an idea, even if the idea seems silly, that I see no good reason to discourage such an act.

There are no good reasons to discourage bad science? or is something that is far more likely to produce bad data rather than good somehow not "bad science"?

That means that an experiment that produces a false positive has educational value provided the reason for the false positive is tracked down.

Only if it's novel.  I'll also point out that this is moving the goalposts somewhat.  Value to the extant body of work is different than "might possibly be helpful to someone somewhere sometime about something maybe!".

Case in point was the FTL neutrino experiments at CERN.  An unusual result was reported.  It was the result of all things a loose optical connector.  The episode taught many people valuable lessons in conducting experiments

However it's a lesson that is impossible to learn from profitis's experiment. So it's not a very good example.  OPERA had a confidence interval.  Profitis has shit squared.  It was having a probabilistic model that MADE that lesson possible.

A limit approaching zero is distinct from zero.

Irrelevant.  Every day, you personally treat hundreds of thousands of non-zero probabilities as if they were zero (at least).  Thousands of those (at least) are greater than the probability of pomodoros experiment producing useful data.  So to me, this is just cherry picking.

[MarkE]

I'm not sure what "unilateral" means in this context but I will say that data should drive your decisions, all your decisions.  Including the decision to run the experiment.  As soon as you (or someone else says) that we should ignore the data saying that the experiment is stupid and perform the experiment - that is being just as arrogant  as ignoring that true equipoise exists.

In the one case we have an apriori declaration that we should not collect data, because the likelihood that new useful information is low.  On the other hand we have a decision to collect data recognizing that the likelihood that it will yield new useful information is low.  There is no arrogance in acknowledging the data we have collected and the probabilities surrounding any new data that we collect.  I submit that there is great arrogance in declaring that because a likelihood is low that we MUST treat the likelihood as zero, so much so that we must avoid collecting new data.  I am sorry but that is really bad logic.  At what arbitrary confidence level do you declare that all data collection MUST stop?

I disagree, those distinctions are both difficult to determine and are subsumed but a much better criterion:  Is the experiment likely to produce useful data?  The rationality of performing an experiment is directly proportional to this.  Precisely! Which is why a validating experiment without having an understanding it's likelihood of success is worthless - as an experiment and performing an experiment which is only going to succeed 1 in 10^1000 times is equally so.If you're talking about research which is hypothesis generating.  This is fine but you have to let go of the end result being meaningful and you would design your experiment differently than profitis did.

Pomodoro has performed carefully constructed experiments, and there is little surprise that his results are consistent with general understanding and inconsistent with Profitis' extraordinary claims.  How is this not useful?  Why is it that you appear to declare this "bad science"?

There are no good reasons to discourage bad science? or is something that is far more likely to produce bad data rather than good somehow not "bad science"?Only if it's novel.  I'll also point out that this is moving the goalposts somewhat.  Value to the extant body of work is different than "might possibly be helpful to someone somewhere sometime about something maybe!".

I gather that maybe you do not do much lab work.  There is much that goes wrong in a lab that people do not predict or expect.  It takes quite a bit of experience for people to get really rigorous about the way they conduct themselves in a lab.  Experiments with expected outcomes are fundamentally useful to that process, both for the experimenters in the lab and all observers.  Pomodoro is clearly very experienced, and has conducted his work carefully.  He has provided a wonderful example of how to go about setting up an experiment, and conducting verifying experiments to cross-check results.  Those are important skills to share.  Were we to discourage that sort of behavior then I submit we create a vacuum that leaves people more likely to conduct poorly controlled experiments because we never taught them any better.

However it's a lesson that is impossible to learn from profitis's experiment. So it's not a very good example.  OPERA had a confidence interval.  Profitis has shit squared.  It was having a probabilistic model that MADE that lesson possible.Irrelevant. Every day, you personally treat hundreds of thousands of non-zero probabilities as if they were zero (at least).  Thousands of those (at least) are greater than the probability of pomodoros experiment producing useful data.  So to me, this is just cherry picking.

Of course sane people make most of their decisions treating less than absolute certainties as absolute.  There is no other practical way to function.  It should guide people on how to make productive use of their resources.  Should and must are quite distinct.  Take for instance the diversion that you and I afford ourselves by posting here.  It is a safe bet that you assign a very low probability of positively influencing for example:  Profitis.  Yet, you expend much energy engaging him.  Is it productive?  For purposes of changing Profitis' stated views it probably isn't.  But that doesn't matter.  It's a diversion you choose to engage in, and are most certainly free to do so.  You may or may not manage to influence others who are not so dug in as Profitis with some of the valuable mathematical and logical skills that you possess.  Even if you don't, the diversion does something for you. 

[sarkeizen]

In the one case we have an apriori declaration that we should not collect data, because the likelihood that new useful information is low.

I'm not really making an a priori argument or even a declaration in the way you appear to be using it.  Nice attempt to use loaded terms though!  Please be more honest (and less arrogant) in the future.   I'm talking about a simple consequence of math which I assume you agree applies here - that a wrong result was many orders of magnitude higher than new data.  You rather carefully left that out and talked as if the only consequence was "no useful information".   It would be nice if you could be objective about this.

On the other hand we have a moronic decision to collect data recognizing that the likelihood that it will yield new useful information is low. There is no arrogance in acknowledging the data we have collected and the probabilities surrounding any new data that we collect.

Doing something implies that it is worth doing.  To use your terms you are making an arrogant a priori declaration that this was worth doing.  I would suggest a better criterion.  Arrogance is simply how much you are willing to put your views above the evidence.  For the sake of argument lets say that this experiment could be done millions of times (or millions of variants of the same experiment could be done) without getting new data specific to the experiment i.e. that which we are attempting to demonstrate.   The evidence in this case says this experiment is not worth doing.  In fact it will be indistinguishable for the vast majority of people who attempt it - to attempting to trisect an angle with a compass and straightedge.  The data says: This isn't worth doing.   Doing it anyway is putting your views far, far, far above the actual evidence.

So by my more objective definition of arrogance.  Isn't it you who are being arrogant here?

we must avoid collecting new data.

You're being a bit deceptive here.  What I'm saying is it's only "new data" if the result is novel and correct.  An experiment where the unexpected outcome which is far, far, far, far more likely to be bad than good.  You are not actually collecting any new data (in the vast majority of cases).  More accurately I'm saying "stop doing experiments of this kind" or "do better experiments".

I am sorry but that is really bad logic.

Nope but please show me my error by providing a syllogism demonstrating your point using the terms as I have defined them without omitting anything.

At what arbitrary confidence level do you declare that all data collection MUST stop?

"confidence level" isn't really the right word since there are lots of reliability estimates.  CL being only one.  Secondly you appear to be misunderstanding the difference between a reliability estimate and a prior probability.   A better way of putting it is, assuming we are talking about a CL it's the ratio of the prior to it.

Pomodoro has performed carefully constructed experiments, and there is little surprise that his results are consistent with general understanding and inconsistent with Profitis' extraordinary claims.  How is this not useful?

The likely outcome is not novel.  The most likely novel outcome will be wrong.  How is that useful to science?

Why is it that you appear to declare this "bad science"?

See above.   It's interesting that you can't see why this would be true.

Experiments with expected outcomes are fundamentally useful to that process, both for the experimenters in the lab and all observers.

Expected novel outcomes.  Experiments where the expected outcome has already been demonstrated so many times that the most likely case by an enormous margin for a novel outcome is a mistake is so far from the kind of expected outcomes which you want in the lab makes your statement much closer to a lie than an honest ignorant mistake.

He has provided a wonderful example of how to go about setting up an experiment, and conducting verifying experiments to cross-check results.  Those are important skills to share.

As stated before this is you simply moving the goalposts.  I mean I understand that moving from talking about if something was good science to "if there's any possible way this might be beneficial to someone somewhere somehow" helps you avoid being wrong but it's a little less than honest to consider that you're doing anything but reaching here.

[MarkE]

At the third ad hom I just stopped reading.

[sarkeizen]

At the third ad hom I just stopped reading.

For reference an "ad hominem" is technically an attack on you (e.g. your abilities, your honesty) to make an argument.  I haven't done anything like that. :-)  If I have, again I'd be happy to be corrected IF you can provide a more formal argument demonstrating your point.  (You can use the term colloquially to just mean "insult" but if so you might as well use that term instead)

I have stated that you used words in ways that are misleading and that you should be more honest and objective and less arrogant.  My argument does not hinge on you being or not being any of these things.

That said, if you needed an out.  I do understand how stomping off under the guise of being insulted might be a good way to avoid dissonance.   It's a little interesting if this is not the case that you're kind of implicitly claiming that I am so amazing at being insulting in a single post that you couldn't complete reading it.  Yet you have endured many insulting posts from other folk here.

The upshot of my post is:  In order to rationally do something you must have decided or assumed that it is worth doing.  So assuming that somehow you didn't accidentally perform an experiment.  Deciding to do something, like all decisions is either informed by data or it isn't.  If it is and the data says otherwise then that fits my definition of arrogant. 

Your comment about an "arbitrary confidence interval" should convince you of my point.  You appear to be saying there that, to you not only does the amount of data against a useful result not have ANY control over you performing an experiment (assuming unlimited resources) but you don't believe any amount of empirical data would be sufficient to change your mind.  Surely that meets even your own internal definition of arrogant.  Holding on to your own belief without any regard for the data?

Of course you can make up any number of other reasons for doing something that fit your outcome but I'm surprised that they sound reasonable to you.  For example sure you can fabricate that this was done for entertainment purposes, or for the education of profitis or for people who want an education in how to do labwork.  However by the same token you would have to believe that it's okay to say "This is a perfectly useful car" even though it can't move an inch.  When questioned you can then say: "Well it's perfectly useful for teaching someone what an engine block looks like" and expect people to think that the first statement wasn't misleading.  In fact you could point to a cinderblock and say "This is a perfectly useful car" and confabulate some explanation like: "It's perfectly useful for instructing people on how to construct a car from a cinderblock".

Aside: Philip has missed his deadline in Australia.  Let's see if he makes it here.