Version: Firmware 13.1.3 + DOS 2.2.4 Date: 2019-02-12

[David A]

If RAAT makes a difference, and I believe it does, proving that with measurements may not be all that easy.

If it makes an audible difference it will show up in a controlled listening test, and we can go from belief in a difference to knowledge of a difference.  Then it gets really interesting.  Did you say somewhere you are in an audio club, if so getting their help on the double blind part will make things a lot easier.

Putting myself through blind testing was one of the most interesting things I done in audio, and I encourage everyone to have a try at something like flac vs mp3 using a software tool.  I found it a fascinating process being presented with no other input than my ears, and having to trust them alone.

I used to work in health and safety and scientific studies were regular reading for me. In fact, as part of ny tertiary qualifications I was required to design, conduct, and write up a research project of my own and I subsequently presented my findings at a scientific conference and had them published in a peer reviewed journal. Controlled tests are not easy to set up or conduct.

One big problem with A/B listening tests in audio is the fact that audio memory is short and you need to be able to switch between inputs very quickly. That's not possible with a Devialet where you need to cycle between inputs by repeatedly pressing the input button. Even if you only have the ethernet input active you need to cycle through different streaming inputs and it takes more button presses to go from AIR to Roon Ready than from Roon Ready to AIR. You can't do a blind or double blind test that way because the listener can always work out which is A and which is B and actually tell what A and B are if they know how the inputs cycle on a Devialet. The other way to do the switch is to use Roon and switch one zone off then switch the other zone on which also takes time. That rules out a double blind test because the person conducting the test knows what the input is, and it also means you can't give control of the swap to the listener because they will then be able to see which input is in use and you have a sighted rather than blind test. You also can't switch between the same point in the music when switching zones in Roon and that creates another problem. I have no way of conducting a test where the results are going to be any more valid than my conclusions based on my own listening comparisons to date.

People like seeing test results because they can put an end to arguments. They can, but only if there are no arguments about the tests. If there are arguments about the tests then you've got a bigger mess because you have the original arguments and the arguments about the tests and you've actually just made things worse. If you're going to do a test then you need to do the right test and you need to do it extremely well so there's no arguments about what you've done. There are always arguments about amateur listening tests for audible differences in audio and I could not put together a test that isn't amateur standard and that means that whatever results I got aren't going to change anything because all I can do is say that I had so x number of people listen and y of those people heard a difference. That's no better than counting the reports made by those who have reported their listening impressions in this and the couple of other threads here where this is being discussed.

BTW, audible differences don't always show up on controlled listening tests. Some years ago a company claimed to have developed a "watermark" which could be added to the audio signal on CDs to help identify and prevent bootlegging of the music and they claimed their 'watermark' was inaudible. They had conducted controlled tests and they started doing demos at audio shows. The first few demos ended with no evidence that the watermark could be detected but then at one test there was one listener who got it right every time. They asked him what he was hearing and after he told them what he was hearing they themselves could pick it every time. They dropped the "watermark" because it ended up being impossible not to notice despite all the initial controlled listening tests which showed it could not be reliably detected. You can get it wrong even with controlled listening tests.