+- Devialet Chat (https://devialetchat.com)
+-- Forum: Devialet Chat (https://devialetchat.com/Forum-Devialet-Chat)
+--- Forum: Devialet News (https://devialetchat.com/Forum-Devialet-News)
+--- Thread: Version: Firmware 13.1.3 + DOS 2.2.4 Date: 2019-02-12 (/Thread-Version-Firmware-13-1-3-DOS-2-2-4-Date-2019-02-12)
RE: Version: Firmware 13.1.3 + DOS 2.2.4 Date: 2019-02-12 - thumb5 - 17-Feb-2019
It's even simpler than you're making it out to be, @David A. The fact is that if people are hearing a difference in sound quality and report that in good faith, there is a difference. Equally, if people are not hearing a difference, there is no difference. Both statements are equally valid and do not conflict, because we're talking about a subjective experience.
If one wants to determine whether there is an objective cause of any reported sound quality differences, that's another story entirely, and -- as I think you acknowledge -- not something that can be resolved by any amount of uncontrolled, individual listening reports.
RE: Version: Firmware 13.1.3 + DOS 2.2.4 Date: 2019-02-12 - David A - 17-Feb-2019
(17-Feb-2019, 11:44)Axel Wrote: @David A There is a better way to switch zones. In Roon, click on the loudspeaker icon, bottom right. You’ll see two arrows pointing away from each other. Click this, then it’s simply a matter of clicking on one of the zones that are available to switch to. I’ve been doing this just now, it takes maybe a second to switch zones and there is a dropout in sound while it happens, but it’s a whole lot quicker than pressing the input button and having to manually restart the music. I still can find no discernible difference between RAAT and AIR.
That won't do it for me. You can't be streaming the same track at the same point in time by both RAAT and AIR because you can't pair the 2 zones. You can swap the zones quickly but you can't avoid a delay in getting to the same point in the music and you just can't swap from one zone to another and have the music continue playing without a gap.
All I'm saying is that I can't think of a way of conducting a listening test for this which isn't going to be criticised for problems in design and conduct. That means that whatever test I do is going to be considered just as valid to let anyone who's interested conduct their own comparisons whichever way they like, report their results, and seeing how many report hearing a difference and how many don't. If I can't come up with a test design which I think provides greater validity than counting the opinions of people who have done their own comparisons any way they like, I'm not going to try doing a test with a number of people. I know the sorts of criticisms which will be made of my method and honesty compels me to concede that those problems are present in the method.
I used to work in health and safety. As part of my studies for my tertiary qualification I had to design and conduct my own research project and write it up. I subsequently presented that research at a scientific conference and had it published in a peer reviewed professional journal. I know a bit about the standards you have to meet when conducting research, and also when simply running tests for something in normal professional situations. If you can't do it right when it comes to procedural issues your results aren't accepted. If I'm going to try and conduct a test with a number of listeners I would want to conduct one that meets all of the design and procedural standards that critics are going to expect. They'll still criticise it if they don't like the result I get but at least they won't be able to say that the test was worthless because it was done wrong. At present I can't think of a way of doing a test that would be accepted as being done right.
RE: Version: Firmware 13.1.3 + DOS 2.2.4 Date: 2019-02-12 - Soniclife - 17-Feb-2019
(16-Feb-2019, 12:55)GuillaumeB Wrote:A null test would be great, but I don't see a way of doing it without a good ADC that could be attached to the speaker outs to capture what comes out. I don't have anything like that.(16-Feb-2019, 12:40)Soniclife Wrote:(15-Feb-2019, 23:14)David A Wrote: If RAAT makes a difference, and I believe it does, proving that with measurements may not be all that easy.
If it makes an audible difference it will show up in a controlled listening test, and we can go from belief in a difference to knowledge of a difference. Then it gets really interesting. Did you say somewhere you are in an audio club, if so getting their help on the double blind part will make things a lot easier.
Putting myself through blind testing was one of the most interesting things I done in audio, and I encourage everyone to have a try at something like flac vs mp3 using a software tool. I found it a fascinating process being presented with no other input than my ears, and having to trust them alone.
How about performing a null test to compare Roon AIR with RAAT?
http://www.galaxyclassics.com/index.php/en/recordingen/audiostuff/69-nulltest
https://www.youtube.com/watch?v=52kbgFAuu6k
Guillaume
Version: Firmware 13.1.3 + DOS 2.2.4 Date: 2019-02-12 - Soniclife - 17-Feb-2019
When I originally suggested this could be blind tested I thought that using the mute button and transfer zone would all that was required to properly blind the subject. But having played with it today I see the audio goes back about a second on transfer, so mute and pause would be required, with a suitable pause on some switches. This is all getting a bit complicated for a human to do reliably, software could be written to do it, with something like a sub 2 second silence between switches, but it would only work for this scenario so it's very limited in scope.
RE: Version: Firmware 13.1.3 + DOS 2.2.4 Date: 2019-02-12 - Pim - 17-Feb-2019
(17-Feb-2019, 14:40)Soniclife Wrote: This is all getting a bit complicated for a human to do reliably.
No it's not actually. We're not solving the worlds problems here. We're listening to music. So there's either a large enough difference for you (and you only) to enjoy, or there's not. If you need to force yourself to hear a difference then forget about it and just enjoy the tunes. After all, that's what it's all about.
RE: Version: Firmware 13.1.3 + DOS 2.2.4 Date: 2019-02-12 - Soniclife - 17-Feb-2019
(17-Feb-2019, 21:12)Pim Wrote:That's not what I was trying to get at, but rather as David said above designing tests that really prove something and cannot be invalidated is tricky, and what I was proposing was getting too complicated to be practical.(17-Feb-2019, 14:40)Soniclife Wrote: This is all getting a bit complicated for a human to do reliably.
No it's not actually. We're not solving the worlds problems here. We're listening to music. So there's either a large enough difference for you (and you only) to enjoy, or there's not. If you need to force yourself to hear a difference then forget about it and just enjoy the tunes. After all, that's what it's all about.
Your test is great, and good enough for most people, it's all I would do if my RAAT was stable now you have proposed it.
RE: Version: Firmware 13.1.3 + DOS 2.2.4 Date: 2019-02-12 - David A - 17-Feb-2019
(17-Feb-2019, 12:28)thumb5 Wrote: It's even simpler than you're making it out to be, @David A. The fact is that if people are hearing a difference in sound quality and report that in good faith, there is a difference. Equally, if people are not hearing a difference, there is no difference. Both statements are equally valid and do not conflict, because we're talking about a subjective experience.
If one wants to determine whether there is an objective cause of any reported sound quality differences, that's another story entirely, and -- as I think you acknowledge -- not something that can be resolved by any amount of uncontrolled, individual listening reports.
No. If someone hears a difference and reports it in good faith, they're simply reporting that they hear a difference, not that there actually is a difference. Similarly those who don't hear a difference and report that in good faith are reporting what they hear, not that there actually is no difference.
I don't think anyone here has so far suggested that people are not honestly reporting what they hear. No one is disputing the reports. The issue of concern is n to whether people are hearing different things, the issue of concern is whether there is an objective difference which is why the discussion of objective tests of one kind or another..
What kind of listening test would I set up if I were going to try to do a controlled listening test? Well, it would have the following features"
- 1 listener at a time sitting in exactly the same spot so room acoustics had no affect on the listener and there was a completely controlled listening environment.
- 2 identical amps, one running RAAT and the other running AIR. Both stream from identical but different servers so both servers can be independently controlled to start a track playing at as close to the same moment as possible. Volume settings matched on both amps and the display turned off or covered so the user can't tell which stream an amp is getting.
- speaker outputs of each amp connected to a high quality switching box which feeds the speakers. The listener has control of the switching box and can switch between one and and the other at will.
Both servers are running identical playlists of different music with the listener giving the tester a list of tracks beforehand so their personal playlist can be prepared and they can make comparisons based on music they like. Because the user can only control the amp switch, not the playlist which needs to be separately controlled for each amp/streaming method at the start, the listener can't go back to rehear bits of a track, they can only listen to the playlist streaming in real time and switch between amps whenever they want to switch.
It's not ideal because there are limits on the listener's control of the music and the switching box might introduce noise which masks some things but it provides instantaneous switching at matched levels and a blind test. It can be made double blind if the person interfacing with the listener has no knowledge of which amp is getting which stream and is not one of the people involved in starting the streams from the 2 servers running.
It wouldn't be an easy test to set up and since it involves only 1 listener at a time it would take some days to get a reasonable sized sample of subjects through the test. Put those 2 things together and what you've got is a test requiring professional level setup and conduct, several days with several people involved on the test administration side, and it's not going to be cheap to run. Then you've got to design your questionnaire. Do you want a simple yes/no for whether or not there is a difference each time the amp is swapped or do you also want a report on which one the listener prefers, and last of all do you want comments on what differences were heard when a difference is reported. Then you have to collate that data in a spreadsheet or database and do your statistical analysis. That's the kind of listening test which will hold up to criticism reasonably well and might deliver some compelling results, especially if a statistically significant number of people report hearing a difference (the minimum requirement for establishing that a difference exists) and then, if you gather data on preference and on what listeners heard, whether people with different preferences are reporting hearing similar sorts of differences so we can tell that their preferences are based in personal taste.
It would be a hell of a lot easier to establish that there is an objective difference by making measurements of server output and Devialet DAC output (at pre out outputs probably) and looking for differences there in things like package error rate, noise, distortion, jitter and anything else a good researcher could think of. Of course then there's the problem of establishing whether any difference revealed is audible but it's easier to design a test for that if you know what the difference is and under what sort of circumstances that difference might be expected to be audible.
Good tests aren't easy or cheap to design and conduct.
RE: Version: Firmware 13.1.3 + DOS 2.2.4 Date: 2019-02-12 - thumb5 - 17-Feb-2019
@David A , with respect, I think you're missing my point. At the risk of taking this thread further off topic...
It's possible that terminology is the problem here. I thought we were all using the word "difference" to mean "a noticeable change in sound quality"; apologies for seeming obtuse if that's not what you meant.
Since sound quality is purely subjective, any report of sound quality difference is unique to the individual(s) who did the listening. Therefore if they hear a difference, or do not, then the (subjective) difference is unarguably there or not, as the case may be. (Unless they are deliberately mis-reporting, hence the caveat in my previous post -- not that I think there's any of that going on here!)
One person's subjective experience doesn't give any significant information about whether anyone else may or may not hear a difference, or indeed whether there is an objective -- that is, measurable in principle, even if perhaps not in practice -- difference in output from the system (say, at the point where sound waves enters the listener's ear).
Of course if you have enough listeners independently and repeatably reporting a sound quality difference, that starts to build evidence that there may be an objective change in output of the system, but in principle listening tests can never answer that question with certainty. Conversely there may be an objective change in output that no-one ever reports hearing.
In any case, there is no "right" or "wrong" and nothing to "prove" either way about whether or not any given listener hears a difference, since the experience is personal and subjective. In particular there is no conflict or mystery to be resolved if some listeners do and some do not hear a difference, whether or not there is an objective change in output
Version: Firmware 13.1.3 + DOS 2.2.4 Date: 2019-02-12 - Soniclife - 17-Feb-2019
I think we might all be agreeing bitterly.
Music listening time.
RE: Version: Firmware 13.1.3 + DOS 2.2.4 Date: 2019-02-12 - thumb5 - 17-Feb-2019
Yup, I think so!