WEBVTT

1
Rm 330: You're muted.

2
Rm 330: Good morning.

3
Rm 330: Every time you connect, it looks different. Oh, it looks like, your computer needs permission.

4
Rm 330: So we will, we have to, unfortunately, I can't…

5
Rm 330: We have to, unfortunately, exit everything.

6
Rm 330: League meeting?

7
Rm 330: All right, and then if you're… Yep.

8
Rm 330: And then we want to go…

9
Rm 330: I need to go to applications, right? Right, privacy and security? Yeah.

10
Rm 330: And then, we want to enable Zoom.

11
Rm 330: I think we are getting a little bit of a split, but it's just better.

12
Rm 330: Did a pop-up, show before, when you went to share the screen in the case?

13
Rm 330: Oh, no, not that one.

14
Rm 330: Y ayudo a todos a tecnicos de la, todorera, y a fue la deja seguir terlea, seguir ayuda, y ayudo a todorera, y ayuda, y ayuda, y ayuda, y ayuda, y ayuda, y ayuda, y ayuda, y ayuda, y ayuda, y ayuda, y a

15
Rm 330: Yes.

16
Rm 330: Bye!

17
Rm 330: It should be pointed out where to run.

18
Rm 330: Join again.

19
Rm 330: And We don't have to worry.

20
Rm 330: Alright, then go to share… Casey, it should be yellow.

21
Rm 330: Yeah, because usually when… Click that, it should get in, okay. System settings, privacy, security… There we go.

22
Rm 330: Okay.

23
Rm 330: Now we have to reopen.

24
Rm 330: I can go. I can go.

25
Rm 330: Okay.

26
Rm 330: Perfect.

27
Rm 330: Bye!

28
Rm 330: Is that it? Awesome.

29
Rm 330: Sure.

30
Rm 330: Share…

31
Rm 330: We should be all right. And we'll know it'll work, because once you share your screen, you'll see it here.

32
Rm 330: And we do.

33
Rm 330: So that looks very cool.

34
Rm 330: Alright, so audio's working here.

35
Rm 330: We have… Few people in, so if we can do this good… No.

36
Rm 330: To those in the audience, can you hear us well?

37
Rm 330: You can type it or respond if you can.

38
Rm 330: Thank you!

39
Rm 330: They can hear us.

40
Rm 330: perfect.

41
Rm 330: Thank you so much.

42
Rm 330: Thank you for the help. My pleasure.

43
Rm 330: Oh… So we are set.

44
Rm 330: Just a second here… Out of my system.

45
Rm 330: How have you been?

46
Rm 330: Good.

47
Rm 330: I am… I'm teaching this semester. So I teach over on College Ave.

48
Rm 330: Honors College Continuous Learning Center.

49
Rm 330: on the science side. This is reading.

50
Rm 330: So this is a psychotic moment while reading, and every big jump is coming. But how do you… we also have a… This is Noah, Noemi. He's a first-year adopted thesis student.

51
Rm 330: Bilingual Future.

52
Rm 330: How do you know what you're reading? How do you… Keeping it grounded in science. What do you mean? So how do we measure? How do you know exactly where the eye is? Because you're not following the pupil, you're following

53
Rm 330: Yeah, my bad.

54
Rm 330: Not exactly.

55
Rm 330: So every time, basically, you can use equations on them. And I didn't know if that was gibberish. Just by the position, the way the eyes, you have a voltage that represents the position of the eye.

56
Rm 330: And so, every time you move your eye, that, you see a shape.

57
Rm 330: in the alternate voltage.

58
Rm 330: The tricky thing is that, coming to the realization that it's not perfect. I mean, in eye tracking, you always have the problem of the eye drifting out, so you always have to calibrate. So here you have something similar, which is, drifts in the blindness.

59
Rm 330: singularity. So this, this is something that we started calibrating, you know, how accurate can we really… what angle, can we actually, resolve

60
Rm 330: If you wanted to really know, you know, if the eye… in principle, when you see regular reading, you would see just this. They're trying to move steps very, very regularly. You know, if it jumps in the other direction, you really see these changes in direction. You see them every time you cross

61
Rm 330: a line, using this big, big jump. You can really extract the angle.

62
Rm 330: Then it's.

63
Rm 330: And you can identify… But in principle, we could identify, so if we have 10 words, we are able to identify, 10 steps, along

64
Rm 330: a longer range. So, in a tracking, what we do… what's going on? In the visual world paradigm, which is images, we move the data to 100 megabytes, which is… because from the time that you make the decision to, like, I'm going to… any decisions, right?

65
Rm 330: It takes 200 milliseconds to actually move the eye, but are you getting this earlier? The voltage? I assume yes. Or no, or no, because the voltage is because of…

66
Rm 330: It's created with a movement, simultaneously. So it's simultaneous with a movement, and, but, in principle, it's one-to-one, so you don't need.

67
Rm 330: It really depends on the sampling rate, which is really fine. So I don't… I don't think we should issue with… And what about here? From the time that you make a decision to the time that you actually… So here, it's supposed to be even to proceed.

68
Rm 330: the actual — so the action potentials in the muscles precede the mechanical. So there you actually see —

69
Rm 330: It's crazy. We've done what you… well, it's articulation, because they're preparing for… Yeah. No, it's not just in articulation. It's in every motor… in every motor action.

70
Rm 330: the electrical activity, and then you see the actual mechanical… this is the time it takes the machinery to move, like, I like… But it's really subtle, because, it's very complicated. So it depends on which system, on which.

71
Rm 330: One experiment that we did last year was to stimulate the nerve that activates this muscle here in the shoulder.

72
Rm 330: you know exactly when you… when you do the electrical stimulation, and then you can see exactly when the muscles start, showing electrical issues, and when it shows mechanical SDVDSDU. So this is very easy to map. We have done, a similar…

73
Rm 330: It was a pilot experiment, it was just one time that we stimulated the nerve that activates the zygomaticus, and then…

74
Rm 330: AI. Measured.

75
Rm 330: Well, the zygomatic was in the other side. In principle, we could also measure at the same time, so we could, in principle, do this experiment to really see the time between the sexual activation and mechanical

76
Rm 330: It's not aware it's on the clock yet. Right. But it's not… That's interesting. I did hear it was… yeah, this is…

77
Rm 330: Yeah, because…

78
Rm 330: Yeah, but because of that, we move our data. So it's telling you, like, 100 milliseconds, because that's the mean. I'm like, this is so crazy, so we don't know. I'm sure that we could do a test.

79
Rm 330: That's the reason why interaction is so interesting, because all the actions, really, everything has to do with feedback.

80
Rm 330: So it doesn't have to be feedback from another person. But there's certain anticipation for a result.

81
Rm 330: And then, you look, and you kind of evaluate whether you got the right result or not? Yes, yes, this is just what we talked about today, so I have it in two meetings.

82
Rm 330: And a blog post about… Actually, not the first slide. If you haven't read the piece itself, it's fantastic. It's called The Rise and Fall of Conscious Citizenship.

83
Rm 330: It's not agent, but agent. Super clever.

84
Rm 330: Yes.

85
Rm 330: Really readable.

86
Rm 330: Sacrifice puts a really long promotion.

87
Rm 330: The allegedly…

88
Rm 330: Excuse me?

89
Rm 330: Okay.

90
Rm 330: Yeah.

91
Hajer Ali: speaker talking.

92
Rm 330: The speaker is talking, not really… Hello? Can you hear?

93
Hajer Ali: We just hear three different voices at the same time for the pastor.

94
Rm 330: Well, we have a microphone here.

95
Rm 330: Can you hear me okay?

96
Hajer Ali: I can hear you.

97
Rm 330: Okay, it's just because we have some background noise, there's lots of conversation going on, but once the seminar starts, it should just be the speaker talking, and then questions.

98
Hajer Ali: Okay, I'm sorry.

99
Rm 330: Please let us know, let us know if you have any.

100
Rm 330: I'll be honest.

101
Rm 330: Okay, thank you.

102
Rm 330: Yeah, at least.

103
Rm 330: Yeah, I don't know. We'll try.

104
Rm 330: You know, it's a 10-month contract, and I don't think it's ever in my contract. Those 10 months are, like, kind of weird. It's just a reminder email. Yeah. Yeah, the more general, like, what is it?

105
Rm 330: I've never been. I've never been my first child. Even now, I understand it's so different. Yeah. Yeah, I do. I'm happy to… happy to request you.

106
Rm 330: you know, No, no, no question. Yeah. Since special Spain. Which… Congrats to Lisa. That's awesome. Yeah. Who is?

107
Rm 330: It's like, yeah, there's this text, and it links to some website somewhere.

108
Rm 330: Okay, so that's everything.

109
Rm 330: The very simple answer is that dry electrodes can work.

110
Rm 330: Traditionally, many years ago, I think it'd be nice to have a council.

111
Rm 330: Very good, very good.

112
Rm 330: He's right at this workshop.

113
Rm 330: It's not just you. It's just especially right out of sitting here.

114
Rm 330: I know, you said this on your face.

115
Rm 330: I'm published all the time about housework.

116
Rm 330: And there are always a lot of this — It's not just you. It's also very short. Somebody said my hair looks like — Aren't my jails?

117
Rm 330: I did it.

118
Rm 330: Even that doesn't make it.

119
Rm 330: And start talking and explaining in those ways.

120
Rm 330: So it is a stable. So it's only the — Everything else is just medical — The electrodes are very, very thin. So they're like, can we have one? Can we have one?

121
Rm 330: I'm fine, as well.

122
Rm 330: Yes.

123
Rm 330: This is the facial emoji.

124
Rm 330: This is Neil.

125
Rm 330: No, no. Oh, sorry.

126
Rm 330: Yeah, John, I'll take shotgun.

127
Rm 330: Because you have to set your goals, and then around…

128
Rm 330: Yeah, I would appreciate it, just because I'm thinking about people that have… Okay, so, is that one… It's basically possible that…

129
Rm 330: So I would have to sit with him and tell him.

130
Rm 330: He's very strong. He's in the Alzheimer's. But in terms of the diagnosis of the muscles? I'm not sure. I've never seen him be nasty before.

131
Rm 330: Start with paper.

132
Rm 330: Yeah, so then in June, we'll have another meeting.

133
Rm 330: I'm calling, so if I say, I mean, German, Uncle Ben? Yeah. That would be the sister sign. Whatever you say. But I didn't want to. This one is .

134
Rm 330: Tori said, if you can't, we can take that from here.

135
Rm 330: But I just don't like it, because here's my phone. See, she's good at it. But if they didn't do that…

136
Rm 330: Yeah, do you… Are you figuring out the language? Well, both sides, or just one side? So… No, so it depends for which application. Yeah, so for policy…

137
Rm 330: It's almost as how they…

138
Rm 330: I was looking at that and the screen was so and there's a tiny person underneath you like I said come on is your daughter tall

139
Rm 330: Is your daughter taller than you? No, I think the screen gave the scale, you know, that was the issue.

140
Rm 330: You're not tiny with the screen. I mean, everything is…

141
Rm 330: You see a picture. I was, like, starting to laugh at this. Something big flying, and there's a tiny person underneath talking.

142
Rm 330: Who did the movies? Is it professional work or?

143
Rm 330: Because I thought maybe you did this.

144
Rm 330: Is that okay?

145
Rm 330: No, it was very good. He didn't help you either, he didn't even say anything.

146
Rm 330: They took some time.

147
Rm 330: Yeah. You know, like, I talked to Nathan, and I said, you know, tell me, you know, if somebody judged my work by the way I… not by presentation, but the way I… Yeah, yeah, but I don't know which. Technical, right? It's not the end. Then maybe it's the wrong place, you know, like, even talk about it, right?

148
Rm 330: It's not the judgment of presentation, it's the technical glitch, right? And if somebody say, oh, you're not going to get this, because that is wrong.

149
Rm 330: I mean, no, you did it on purpose, you know, just curious. Okay. Before I hand over my…

150
Rm 330: Before I hand over to Jed to present today's speaker, I wanted to alert you for our upcoming schedule of talks.

151
Rm 330: Next week, we don't actually have a Colloquium, we're taking a break, but in two weeks' time, we have Alex Kontorova, who is the… actually the chair of the math department.

152
Rm 330: And he's going to tell us about new developments in geometry and mathematics, but also the effect of AI on mathematics today. He's going to tell us about that, which should be quite interesting. So please make a note of that in your calendar for two weeks from today.

153
Rm 330: And we have others lined up as well, some interesting speakers coming later in the semester. So, with that, I'll hand over to Jed to introduce today's speaker, Lyon's lecture. Because…

154
00:00:00.000 --> 00:00:10.600
Rm 330: Thank you, everybody, for coming to today's Colloquium and a continuation of the Iron Lecture that began yesterday through a couple public lectures by Eva Yale and Naomi Hallis.

155
00:00:10.720 --> 00:00:14.560
Rm 330: It's a real pleasure to have y'all join in today.

156
00:00:14.600 --> 00:00:24.059
Rm 330: She got her undergraduate at Tel Aviv University, and then a PhD at Weissman, working on good old condensed matter physics.

157
00:00:24.060 --> 00:00:26.860
Rm 330: And then moved on in her postdoc.

158
00:00:26.860 --> 00:00:49.199
Rm 330: to transitioning to interacting with humans, postdoc at University of Washington, and then moving back to Tel Aviv University as faculty, where she's now a full professor, and I don't remember the names of the companies that you started, so I have to try to look those up, but, oh, Extrodes is the founder.

159
00:00:49.510 --> 00:01:08.289
Rm 330: Oh, but you were the VP, for Scientific Affairs for Nano Retina, from 2009 to 2023, and so they… that sounds like that was probably more invasive, maybe, harder to get going. So yeah, we're very excited to hear about your work on listening to the body, soft electronics, social physiology, and the physics of human-machine interfaces. Thank you so much for coming.

160
00:01:08.420 --> 00:01:09.210
Rm 330: Okay.

161
00:01:09.889 --> 00:01:19.400
Rm 330: So… Yeah, so thank you very much, and It's, it's always…

162
00:01:19.520 --> 00:01:33.520
Rm 330: a challenge when speaking, to different audiences, what to focus, because, we do, very multidisciplinary work.

163
00:01:33.650 --> 00:01:51.220
Rm 330: But I decided not to try to convince you that I'm doing physics because I'm not doing physics anymore. We are really doing electrophysiology. We do electrical engineering.

164
00:01:51.730 --> 00:01:57.420
Rm 330: This is this is really our main focus. But I think I will be able to

165
00:01:57.480 --> 00:02:16.539
Rm 330: help you appreciate that, this field is actually, really benefits from, physics training. So there are quite few other, people with background in physics that, found themselves in, this really interesting

166
00:02:16.570 --> 00:02:17.860
Rm 330: Field.

167
00:02:18.080 --> 00:02:36.210
Rm 330: Okay, so there are several motivations to the work that we're doing, and the story that I want to describe to you today focuses on the concept of human-machine interfaces. And, you know, just by looking at this combination of

168
00:02:36.210 --> 00:02:42.590
Rm 330: words, you may think that you know what human-machine interfacing is, and

169
00:02:43.030 --> 00:02:59.019
Rm 330: you see the very obvious examples of keyboards and… or a mouse, or joystick, and touch screens. These are all human-machine interfaces. These are tools that allow us to interface

170
00:02:59.020 --> 00:03:03.589
Rm 330: With, say, a computer or a machine.

171
00:03:03.590 --> 00:03:12.390
Rm 330: The point is that these human-machine interfaces, they are indirect interfaces.

172
00:03:12.390 --> 00:03:31.620
Rm 330: In the sense that we need to execute a physical operation in order to do this interaction. You need to press a button on the keyboard to activate the keyboard, and that makes these interfaces indirect.

173
00:03:33.130 --> 00:03:46.549
Rm 330: If you look at the evolution of human-machine interfaces, or human-machine interaction, this is an adaptation of an adaptation of some really nice

174
00:03:46.550 --> 00:03:49.100
Rm 330: figure that I found online.

175
00:03:49.100 --> 00:04:10.780
Rm 330: I think some of you are old enough. I'm not old enough to experience this. Maybe some of you are just too young and never even been exposed to this transition. What you have to appreciate is that the level of complexity that has been introduced into this field is enormous.

176
00:04:10.950 --> 00:04:22.060
Rm 330: So, it's not this story of people from Apple visiting Xerox PARC and seeing the mouse, and then you have a mouse. There's an enormous level

177
00:04:22.060 --> 00:04:31.260
Rm 330: Of, complexity that has been introduced in order to make these, systems, very convenient to use,

178
00:04:31.290 --> 00:04:50.120
Rm 330: almost seamless, in the sense that any kid, probably all of you have seen kids or grandchildren just pick up a touchscreen and, in 5 seconds, operate it better than a grown-up. So, it's the psychology, if you want, that is involved in this.

179
00:04:50.120 --> 00:05:02.459
Rm 330: There's a lot of engineering, a lot of algorithms, to correct, errors, and this is what we've been seeing for, say, the last few decades.

180
00:05:02.490 --> 00:05:04.629
Rm 330: What is going on?

181
00:05:04.970 --> 00:05:16.380
Rm 330: In recent years is also the introduction of, voice detection, so Siri, Alexa, these kind of things. Again, enormous level of

182
00:05:16.380 --> 00:05:28.340
Rm 330: sophistication from the linguistic point of view, the algorithmic point of view, error corrections, and you know that these companies are… a lot of companies are still struggling with these technologies.

183
00:05:28.500 --> 00:05:40.529
Rm 330: What people are really trying very hard to do, and this is still ongoing, this is what is called the context-aware interfaces, we've been talking about that, that when you do

184
00:05:40.530 --> 00:05:56.399
Rm 330: this interaction, when you interact, you activate things, it matters how you do it, and what is the context of what you do. If you're typing when you're angry, or you're typing where you're not angry, whether that would

185
00:05:56.400 --> 00:06:12.399
Rm 330: How do you correct that, in order to make the interaction seamless? And considering the fact that most of us, most of the time, are actually using a lot of devices constantly, together, how does that

186
00:06:15.630 --> 00:06:34.750
Rm 330: How do you optimize that in a way that allows us to, work with a computer, listen to a talk, maybe checking our email, maybe operating our, WhatsApp, all at the same time with a minimum, cognitive… So, this is a very, very challenging

187
00:06:34.770 --> 00:06:36.420
Rm 330: endeavor.

188
00:06:36.750 --> 00:06:49.949
Rm 330: The next level, and this is, what's, what the things that I'm going to, talk to you about are the direct interfaces. So, direct interfaces don't

189
00:06:50.160 --> 00:06:57.329
Rm 330: Need the physical, intermediate between what you want to do and what you do.

190
00:06:57.330 --> 00:07:10.389
Rm 330: Basically, the idea is that you think about it, you move your finger, you move your eyes, you blink, and that actually does the interfacing for you.

191
00:07:10.500 --> 00:07:19.560
Rm 330: This is where the field, this is the roadmap for this field.

192
00:07:19.640 --> 00:07:35.269
Rm 330: What are people doing in this field? What are the technologies? What is the technology that actually allows us to even imagine such indirect interfaces?

193
00:07:35.530 --> 00:07:39.089
Rm 330: So one obvious thing is EEG.

194
00:07:39.460 --> 00:07:45.160
Rm 330: And EEG is a technique where you record the electrical activity of the brain.

195
00:07:45.470 --> 00:07:47.999
Rm 330: And this is the most obvious thing to use.

196
00:07:48.110 --> 00:08:01.490
Rm 330: The trouble with these signals, the trouble with EEG is that typically you work with very, very low signal-to-noise ratio, and typically this technology is very limited in its ability to

197
00:08:01.490 --> 00:08:08.649
Rm 330: Work properly when you move, so any tiny mechanical movement would,

198
00:08:08.750 --> 00:08:20.060
Rm 330: be a very major challenge. So, even though it's very common in academic research, it's extremely common in academic research.

199
00:08:20.060 --> 00:08:32.890
Rm 330: People in psychology love it, people in many different fields love it. Also, people in electrical engineering love it because there's a lot of signal processing to do. It's very challenging technology.

200
00:08:32.890 --> 00:08:40.839
Rm 330: But from a practical point of view, maybe not the best. The other system.

201
00:08:41.090 --> 00:08:52.619
Rm 330: or approach that you can utilize is, recording the electrical activity of your muscles. So, if you move your thumb, your muscles are activated, there's electric field.

202
00:08:52.620 --> 00:09:06.609
Rm 330: involved, and this is something you can pick up with electrodes. So the most advanced system that exists today is by Meta that spends hundreds of millions of dollars to develop this system.

203
00:09:06.650 --> 00:09:11.649
Rm 330: It's already available commercially, it's one of the very, very few systems.

204
00:09:11.700 --> 00:09:18.450
Rm 330: And it's sensitive to a pretty narrow set of gestures.

205
00:09:18.510 --> 00:09:21.500
Rm 330: But it's very, very, advanced.

206
00:09:21.740 --> 00:09:22.569
Rm 330: Another one.

207
00:09:23.860 --> 00:09:35.539
Rm 330: In research, are still systems that look advanced something, like this, okay? So generally when you look at,

208
00:09:35.630 --> 00:09:44.009
Rm 330: Or you think about a neural interface, what you have to take into account is that a system

209
00:09:44.180 --> 00:09:54.300
Rm 330: Actually consists of these five elements. You need a physical interface. You need something that takes the signal and amplifies it.

210
00:09:54.350 --> 00:10:05.830
Rm 330: You need approach how to clean the data so you actually look at what the muscles are doing or what the brain is doing and not all sorts of different.

211
00:10:06.160 --> 00:10:21.090
Rm 330: mechanical or electrical artifact. You need to stream the data, and it's quite a lot of data, so streaming it is not that obvious, at least when you have to stream a lot of it in parallel.

212
00:10:21.090 --> 00:10:29.919
Rm 330: You need to decode it, so AI, of course, becomes very, very handy. In some cases, you may want to close the loop.

213
00:10:30.030 --> 00:10:33.440
Rm 330: And bring the information back to the user.

214
00:10:33.520 --> 00:10:46.699
Rm 330: That means that you need to do all those four operations very quickly because if you go a certain threshold of, say, 100 milliseconds, then things become very, very annoying.

215
00:10:46.700 --> 00:10:54.219
Rm 330: So think about talking when you have very strong echo, you would appreciate how much.

216
00:10:54.220 --> 00:11:06.059
Rm 330: A little bit of latency completely drives your brain nuts. A 100 millisecond or so is a long time, which is way above what you can.

217
00:11:06.360 --> 00:11:13.070
Rm 330: Allow. Okay, so… I'm sorry, how about, implanting things directly into the brain?

218
00:11:13.450 --> 00:11:16.860
Rm 330: So that's another.

219
00:11:17.160 --> 00:11:26.689
Rm 330: Yeah, it is on the timeline, and people like Elon Musk talk about these kind of things, completely unethical, completely unrealistic.

220
00:11:26.900 --> 00:11:38.709
Rm 330: Okay. You're not allowed, at least under the current, laws, to use, implantable devices for enhancement.

221
00:11:39.120 --> 00:11:41.570
Rm 330: Unless he wants to create his own army.

222
00:11:41.670 --> 00:11:46.130
Rm 330: Well, unless you do your clinical studies and, you know.

223
00:11:46.130 --> 00:12:06.470
Rm 330: Yeah. Different countries with different rules, beyond minerals. I mean, I have a longer, less brutal answer, but I kind of felt that in this audience, you would appreciate this answer. So, what we're doing… what we're working on is non-invasive.

224
00:12:06.810 --> 00:12:11.710
Rm 330: Direct, interfaces.

225
00:12:11.850 --> 00:12:30.099
Rm 330: And these interfaces are primarily built on a technology that you see here, and I'll explain to you the origin of this technology in a lot more details, and really try to explain the engineering or the physics of these interfaces.

226
00:12:30.150 --> 00:12:37.660
Rm 330: And what does it allow us to do, and how we can use it? And at the end of the talk, I'll show you some

227
00:12:37.840 --> 00:12:50.769
Rm 330: Some examples. This will be the outline. First, explaining a bit, what are the signals that we're measuring? How do we measure them?

228
00:12:50.770 --> 00:13:04.169
Rm 330: How do we measure them at high density with a wearable system, and again, how do we integrate that into several.

229
00:13:04.200 --> 00:13:08.460
Rm 330: examples of out of many, many, many applications.

230
00:13:08.930 --> 00:13:10.969
Rm 330: So, what are we recording?

231
00:13:11.220 --> 00:13:23.189
Rm 330: So, the fundamental signal that you want to record, I mean, if you want to listen to the body, there are different signals that you can record. You can record mechanical signals, you can

232
00:13:23.190 --> 00:13:32.060
Rm 330: Look at blood pulsation, you could look at all sorts of different things. This is where my physics.

233
00:13:32.060 --> 00:13:47.009
Rm 330: or solid-state physics remains. I love electrical signals, and so I do only, almost exclusively, work with electrical signals. So, what are these electrical signals? So.

234
00:13:47.220 --> 00:14:00.600
Rm 330: We have two primary cell types that generate electrical activity. These are neurons and muscle cells. This is part of their

235
00:14:00.600 --> 00:14:16.229
Rm 330: function, biological function, is that a phenomena that is called action potential. So an action potential is a situation where these cells start from a resting DC potential.

236
00:14:16.230 --> 00:14:21.689
Rm 330: For one reason or the other, this potential goes up.

237
00:14:21.710 --> 00:14:32.769
Rm 330: and then restores back to this DC level. So, neurons do that, muscle cells do that, and if you want to record this electrical activity.

238
00:14:32.770 --> 00:14:51.350
Rm 330: all you need to do is join good enough electrodes at close enough proximity, use good enough amplifiers, low enough noise, and basically you can record the electrical activity of neurons or muscles. Now.

239
00:14:51.640 --> 00:15:09.549
Rm 330: If you do it invasively, if you take electrodes and you push them into the brain, and you come at very, very close proximity to the cells, then, of course, you can get very high resolution, but as we discussed before, under this context, we don't really want to do that.

240
00:15:09.590 --> 00:15:24.819
Rm 330: Muscle cells, on the other hand, are very, or muscle activity is actually very, very strong. So even when we work from the skin, we can actually work with really nice.

241
00:15:24.820 --> 00:15:34.759
Rm 330: Signal to noise ratio. I'll talk about the noise in a minute, but the signal levels could be in the range of even few millivolts.

242
00:15:34.760 --> 00:15:42.790
Rm 330: Okay, so just by placing electrodes, on the skin against muscles, activating,

243
00:15:43.310 --> 00:15:54.939
Rm 330: the muscles, you can get a very decent electrical activity. Now, what makes these systems also very, very interesting and very challenging to

244
00:15:54.940 --> 00:16:03.650
Rm 330: Study is also the fact that signal goes in both directions, so we have signals going from the brain commanding.

245
00:16:03.670 --> 00:16:22.250
Rm 330: the rest of the body, but we also have sensory input that goes back, into the brain. So from, control perspective, these are very, elaborate systems to, to look into. So there's a whole, a very wide

246
00:16:22.250 --> 00:16:27.940
Rm 330: Range of or wealth of phenomena that you can look at.

247
00:16:28.260 --> 00:16:41.210
Rm 330: Which I'll show you later on. What you see on the right, this is the state of the art in electrophysiology, skin electrophysiology.

248
00:16:41.250 --> 00:16:57.919
Rm 330: So, the way that we describe the interface between the electrodes and the skin, so this represents, say, the dermis and the epidermis, which is essentially the dead layer, the upper layer of the skin.

249
00:16:57.940 --> 00:17:12.960
Rm 330: And then, the electrodes. So, each one of these, layers is actually an electrochemical, interface. And it's crudely, described as a parallel capacitor and a resistor.

250
00:17:12.960 --> 00:17:21.349
Rm 330: So the capacitor comes about because the system is fundamentally insulating, right? It's just a biological

251
00:17:21.580 --> 00:17:38.229
Rm 330: dead or alive tissue. And the parallel resistance comes about because we do have sweat glands, we do have things that ultimately conduct ions, not in the most incredible way, but it does conduct ions.

252
00:17:38.520 --> 00:17:42.430
Rm 330: So this is how we crudely describe these systems.

253
00:17:42.510 --> 00:18:00.609
Rm 330: When we take electrode, and we put it against the skin, or against the tissue, or against a conducting medium, it's also described as this parallel resistor, system. This is really the fundamental building block in electrochemistry. All electrochem…

254
00:18:00.610 --> 00:18:05.110
Rm 330: Chemical systems, function one way or the other, as.

255
00:18:05.800 --> 00:18:14.889
Rm 330: Yeah, when you do this empirically, is the frequency dependence consistent with a capacitance, or is there an inductive component?

256
00:18:14.890 --> 00:18:32.189
Rm 330: There's no inductive components, okay? But both these are, frequency dependent. Yes. Okay, so it's not, and it's actually pretty hard to characterize it, and most of the time, people don't really bother, characterizing these systems very, in much details.

257
00:18:32.190 --> 00:18:38.850
Rm 330: Now, going back to the inductance, we work at very low frequencies. Okay, so neurons fire.

258
00:18:38.850 --> 00:18:50.140
Rm 330: At best, at say the highest frequency that you would ever consider is kilohertz, okay? Hundred of hertz.

259
00:18:50.160 --> 00:18:53.310
Rm 330: Okay, so this is the… this is your frequency range.

260
00:18:53.360 --> 00:18:56.360
Rm 330: Okay, so these are the, Thank you.

261
00:18:57.400 --> 00:19:08.070
Rm 330: Okay, yeah. What is your potential range? Is it, like, microvolts? So the potential range goes from the floor of your, your baseline noise.

262
00:19:08.390 --> 00:19:14.070
Rm 330: Okay? And all the way to a few millivolts, depending on how big your muscle is.

263
00:19:14.350 --> 00:19:21.770
Rm 330: So if you have a very big, strong muscle, you can get to, say, 10 millivolts.

264
00:19:22.210 --> 00:19:34.890
Rm 330: Okay? And the lowest for small muscles, or for a very small activity, this could be as low as few microvolts, but then your noise level would be very similar.

265
00:19:35.700 --> 00:19:36.630
Rm 330: Okay.

266
00:19:36.830 --> 00:19:55.649
Rm 330: So when you put electrodes against, the skin, you end up with very, usually very high, input impedance, not input, very high, impedance, and what traditionally people have done to control this impedance was to put a gel

267
00:19:55.670 --> 00:20:08.600
Rm 330: If you ever did an ECG measurement or EEG measurement, you must remember that gel placed underneath the electrodes and this stabilizes the system.

268
00:20:08.970 --> 00:20:11.330
Rm 330: If you think about dry electrodes.

269
00:20:11.350 --> 00:20:17.990
Rm 330: Dry electrodes would be, just the skin, and then the electrode without the gel.

270
00:20:17.990 --> 00:20:33.549
Rm 330: And traditionally, if you would just take a piece of metal and press it against your skin, you would end up with a very bad, contact. So it would be very unstable, very noisy, very sustainable, sustained.

271
00:20:33.800 --> 00:20:47.940
Rm 330: very sensitive to mechanical movement, and generally a very bad system. Now, this is something that all of you know very well. This is how we characterize noise.

272
00:20:47.940 --> 00:21:02.340
Rm 330: in, in a resistance, and this is exactly the type of noise that we expect from such an impedance, okay? So it would not be a simple resistor, but otherwise.

273
00:21:02.630 --> 00:21:12.260
Rm 330: It's a very familiar system. It's really about how we control this noise.

274
00:21:12.260 --> 00:21:23.110
Rm 330: to be low enough, so whatever signal that we're trying to measure would be significantly above this value, okay? So…

275
00:21:24.270 --> 00:21:40.340
Rm 330: this is the state of the art. The state of the art are pieces of metals covered with a conducting gel, pressed against the skin with all these wires. This is how you do conventional skin electrophysiology. Question? Yep.

276
00:21:40.340 --> 00:21:44.689
Rm 330: Sorry. So within the circuit picture, how does the gel help?

277
00:21:45.970 --> 00:21:48.330
Rm 330: So, the gel…

278
00:21:48.590 --> 00:22:07.370
Rm 330: Okay, so you don't see here, values, but what the gel does is to seal the voids between the metal and the skin. The skin is not flat, and the electrode is not flat, so if you just take, in principle, a sheet of metal, and you put it against the skin, most of it would be just void.

279
00:22:07.430 --> 00:22:20.199
Rm 330: Okay? If you take the metal and really press it very, very hard, you would end up with something pretty equivalent. So, dry electrodes have been available

280
00:22:20.310 --> 00:22:24.969
Rm 330: For ages. It's just that they're not very easy or convenient to work with.

281
00:22:25.600 --> 00:22:45.380
Rm 330: Okay? The gel allows you, is the gel conducting, or… It's conducting, yeah. It's conducting, it also buffers, it also damps mechanical, movement, so it's… these are actually pretty fancy gels. Still under research, so people still trying to make them, to improve them.

282
00:22:46.070 --> 00:22:46.910
Rm 330: Okay.

283
00:22:48.220 --> 00:22:51.549
Rm 330: So what… the very first thing that we did.

284
00:22:51.950 --> 00:23:04.079
Rm 330: to improve the system was to develop these electrodes that are actually patches that are very soft. I have some examples here.

285
00:23:04.230 --> 00:23:08.110
Rm 330: See them, and when you put them against the skin.

286
00:23:08.600 --> 00:23:15.520
Rm 330: They're completely dry, the electrodes form the contact against the skin, And the reason why.

287
00:23:16.000 --> 00:23:34.259
Rm 330: you can actually work with these systems is because they are very, very soft, okay? So, if these were rigid, or just, say, flexible, like Kapton, if you would put them against the skin, they would not work, and you would still need to use a gel.

288
00:23:34.300 --> 00:23:46.110
Rm 330: What we managed to do in the very, very early stages of this project was to do the printing to realize these electrodes on very soft materials.

289
00:23:46.110 --> 00:23:56.709
Rm 330: So the material is both soft and very thin, so when the electrodes are actually resting against the skin, they form a very nice, contact.

290
00:23:56.820 --> 00:24:00.139
Rm 330: Okay, so this was all hand waving.

291
00:24:00.510 --> 00:24:12.309
Rm 330: But we really had to prove all that. So the first proof was that it was working, okay? So we started working and collecting data and running more and more and more and more measurements.

292
00:24:12.310 --> 00:24:20.820
Rm 330: And we could see that we have fantastic signal-to-noise ratio, okay? So we already had hundreds of data sets from people sleeping.

293
00:24:21.080 --> 00:24:24.680
Rm 330: What's the material that you use? So we use carbon.

294
00:24:24.890 --> 00:24:29.259
Rm 330: So, the gray traces, these are silver.

295
00:24:29.630 --> 00:24:32.790
Rm 330: So, these are silver traces.

296
00:24:33.050 --> 00:24:38.520
Rm 330: And these are… the black, dots are, carbon ink.

297
00:24:38.850 --> 00:24:42.010
Rm 330: Is it carbon? Carbon ink.

298
00:24:42.100 --> 00:24:59.159
Rm 330: I don't know, the carbon ink is, flakes of, graphite. Graphene. If you want, you can call it graphene. We stopped publishing in, material journals, so we… it's commercial, carbon ink.

299
00:24:59.310 --> 00:25:05.440
Rm 330: But yeah, it could be so so. And what's the solution in which is it?

300
00:25:06.670 --> 00:25:17.120
Rm 330: So it's just the graphene, or the carbon, or what else? So, to produce these electrons? No, no, no, but this is just the carbon, or do you have something else that holds it together?

301
00:25:17.680 --> 00:25:29.490
Rm 330: Well, you do, you center. So after you print the so that when the ink comes with particles suspended in in a solvent.

302
00:25:29.560 --> 00:25:40.159
Rm 330: You print it, and then you sinter it. So the sintering breaks whatever chemical coating that you have on the particles.

303
00:25:40.160 --> 00:25:55.180
Rm 330: And then the heating also, fuse the particles together. Well, enough to, to have electrical contact. And your solvent… And the solvent evaporates partially, also,

304
00:25:55.440 --> 00:26:11.049
Rm 330: penetrating the substrate. So when ultimately the you you finish the the fabrication, the carbon becomes part of the film. So there's no detachment. There's no delamination.

305
00:26:12.230 --> 00:26:19.579
Rm 330: And the film is what? Polyurethane. And it's very soft and thin. It's very, very soft, and it's very thin.

306
00:26:19.980 --> 00:26:21.780
Rm 330: And you buy it commercially.

307
00:26:23.000 --> 00:26:24.799
Rm 330: Yeah, we, we…

308
00:26:25.580 --> 00:26:40.770
Rm 330: Yeah, we're trying to avoid as much as possible, working with, esoteric, materials, although, we do collaborate every now and then with, material scientists, and then we try, new materials, just for

309
00:26:41.670 --> 00:26:53.339
Rm 330: funding, I don't know. Okay, so, so one proof… yeah. So, so this doesn't have a gel? No gel. So, so how is the electrode stuck onto the skin?

310
00:26:53.440 --> 00:27:08.690
Rm 330: So you do have adhesive material around the electrodes. So under the polyurethane, you have adhesive material. It's medical grade. It's adhesive use for medical applications.

311
00:27:09.360 --> 00:27:10.020
Rm 330: Okay.

312
00:27:10.290 --> 00:27:13.159
Rm 330: and that plays the role of your gel.

313
00:27:13.270 --> 00:27:16.799
Rm 330: No, that just holds it against the skin.

314
00:27:17.450 --> 00:27:19.520
Rm 330: There's no jail. No jail.

315
00:27:19.810 --> 00:27:38.630
Rm 330: Okay, so initially, the electrodes are just resting, against the surface. After some time, there might be some sweating, some hydration, and so maybe some capillary, some type of capillary, action, and, you, you end up… so, so.

316
00:27:38.680 --> 00:27:57.860
Rm 330: These are typical data. So this is for making a fist. This is EMG electrodes when you do swallowing. And the green is recorded from the forehead of a person.

317
00:27:58.000 --> 00:28:00.059
Rm 330: With their eyes,

318
00:28:00.080 --> 00:28:09.339
Rm 330: close, actually, just before falling asleep. This is alpha waves. These are, you see, 20. Well, you don't see. Yeah, you can. You do see.

319
00:28:09.340 --> 00:28:20.519
Rm 330: So it's 20-something microvolts. So this, the green is brain activity. This is EOG. So this is the movement of the eyes.

320
00:28:20.520 --> 00:28:34.160
Rm 330: That is picked up because the eyes are, the retina is polarized, it's charged. So every time the eyes are moving, you can see, you basically measure the dipole. It's a very strong dipole.

321
00:28:34.160 --> 00:28:40.620
Rm 330: And these are the action potentials of the muscles here and here.

322
00:28:41.000 --> 00:28:46.979
Rm 330: Okay? So, you do these measurements, and you get really good, signal-to-noise ratio.

323
00:28:47.100 --> 00:28:53.290
Rm 330: Okay? And… but we still didn't, explain or show directly

324
00:28:53.290 --> 00:29:10.009
Rm 330: Why does it work? Why, for centuries, people were working with these, gel electrodes, forcing everybody to use, gel, and we just use, dry electrodes, and it's working, so well.

325
00:29:10.120 --> 00:29:13.419
Rm 330: Sorry, what's the, surface roughness on those?

326
00:29:15.160 --> 00:29:25.880
Rm 330: They're pretty rough, I don't… well, we've done SEM, we've done possibly even AFM, I don't have a number.

327
00:29:26.060 --> 00:29:35.260
Rm 330: It… but they're not particularly rough. It's not carbon nanotube rough, because that could possibly Okay.

328
00:29:35.720 --> 00:29:36.480
Rm 330: Yeah.

329
00:29:37.200 --> 00:29:42.899
Rm 330: Yeah, but this is not. This is not really the answer. The answer is even more.

330
00:29:43.520 --> 00:29:49.980
Rm 330: fundamental. Okay, so you remember the Nyquist, Johnson noise.

331
00:29:49.980 --> 00:30:04.489
Rm 330: This is what we expect. So, we basically, if we measure the impedance of the electrodes, we should be able to figure out or estimate what should be the expected noise.

332
00:30:04.520 --> 00:30:11.450
Rm 330: And so, this is what you see here. So, the blue lines, these are impedance measurements.

333
00:30:11.590 --> 00:30:14.609
Rm 330: So it's 2 terminal impedance measurement.

334
00:30:14.720 --> 00:30:18.800
Rm 330: Of electrodes resting, on the skin.

335
00:30:18.940 --> 00:30:20.080
Rm 330: Umm.

336
00:30:20.490 --> 00:30:40.100
Rm 330: So let's look first at this example. So this is… these are electrodes resting on the arm, and here we calculate how much noise. This is in frequency domain, and this is the power, this is the calculated power.

337
00:30:40.130 --> 00:30:55.590
Rm 330: Now, if the hand is in a relaxed condition, so no muscle activation, you see that the expected, or the measured signal is actually pretty similar to what we calculated from the imbalance.

338
00:30:55.620 --> 00:31:14.859
Rm 330: If we activate the muscle, you see this content here. This is really the contribution of the muscle activity, and you see that this jumps up, but roughly what we expect to see kind of works out.

339
00:31:15.400 --> 00:31:31.729
Rm 330: Now, we do something similar in the face, okay? So, one thing that you have to appreciate is that the impedance in different body parts is very different. Actually, it's two order… at least two orders of magnitude difference between your hand.

340
00:31:31.790 --> 00:31:45.690
Rm 330: And your face. The chest is somewhere in between, and the legs are even more. It has to do not just with the thickness of the skin, not just with the concentration of sweat glands.

341
00:31:45.690 --> 00:31:52.719
Rm 330: But also with the amount of blood hydration that you have underneath the tissue.

342
00:31:52.990 --> 00:31:53.960
Rm 330: Okay.

343
00:31:54.020 --> 00:32:00.040
Rm 330: And so, if you look at now what you expect the noise to be in the face.

344
00:32:00.040 --> 00:32:14.179
Rm 330: It's actually very low, okay, so it's much lower than in the hand. It's so low that it's even below the baseline noise of the short circuit of the amplifier, basically the noise of the amplifier.

345
00:32:14.180 --> 00:32:18.140
Rm 330: And… If you look at a person,

346
00:32:18.290 --> 00:32:20.120
Rm 330: A wake person

347
00:32:20.260 --> 00:32:36.869
Rm 330: So you see, this is the, the FFT, so this is alpha, so this is around, 10 Hz, and this is some residual muscle activity. Okay? And now, the question is, how do we turn a person off?

348
00:32:37.080 --> 00:32:47.629
Rm 330: You remember here, I turned the muscle activity off because we said, okay, relax the hand. How do we do it in the face?

349
00:32:48.600 --> 00:32:59.520
Rm 330: So, okay, so you look at a person, we said, okay, let's look at a person when they're relaxed, okay? Relax your face, okay? So you can really try very, very hard to relax their face, but

350
00:32:59.850 --> 00:33:18.329
Rm 330: I'll show you later, well, if we have the time to get to later, but… Sleeping is actually a condition where we do relax our muscles, and actually during REM sleep, we… I mean, well, even without controlling it, we're in full atonia.

351
00:33:18.620 --> 00:33:36.989
Rm 330: Okay, so this is, data. We have a lot of sleep data, so this is data from a person, during REM. So, no alpha, no muscle activity, but you see that the noise is much, much higher than the Nyquist.

352
00:33:37.280 --> 00:33:50.720
Rm 330: Okay? Much more than what we expect from the interface between the electrode and the tissue. So, what is this… where is this noise coming from? What is the noise that we have here?

353
00:33:51.820 --> 00:34:06.510
Rm 330: So the simple answer is that humans are alive, and we can't shut them down, okay? There's no such thing… well, unless we measure… well, we won't go to that.

354
00:34:06.850 --> 00:34:14.789
Rm 330: So So, as long as we work with, people, live people.

355
00:34:14.790 --> 00:34:32.760
Rm 330: even if they're asleep, even if they're in REM sleep, there's still a lot of noise which comes from the biological system. So the answer is, and we proved it with a lot more statistics, later on finding a paper from some years back.

356
00:34:32.770 --> 00:34:44.169
Rm 330: from a Finnish group that did a very, very similar analysis, basically demonstrating exactly the same point. At least in the face, the noise

357
00:34:44.240 --> 00:34:57.809
Rm 330: is, or the contribution of the impedance to the noise is negligible. So, most of the noise comes about from, residual biological, activity. So, breathing.

358
00:34:57.810 --> 00:35:17.130
Rm 330: eye movements, brain activity, all that, is established. This is the floor of the… of the noise. Okay? Yeah? Basic question. So, you have one electrode, and so are you measuring, like, sort of, like, voltages with respect to some ground? Yes, yes, yes, yes, yes. And what is ground?

359
00:35:17.229 --> 00:35:20.450
Rm 330: Okay, so, ground…

360
00:35:21.550 --> 00:35:25.840
Rm 330: You see here, this is our ground. So it's a big electrode.

361
00:35:27.529 --> 00:35:47.499
Rm 330: Okay, now, in electrophysiology, you put ground or reference electrodes on bony areas that you don't expect to see much activity. Okay, so in the face, you would put it either here or on the nose, okay? On the hands, you…

362
00:35:47.689 --> 00:35:51.970
Rm 330: Conventionally, you would put it, against, here.

363
00:35:52.890 --> 00:36:04.640
Rm 330: Okay, so you need to look for a region where you don't expect noise, because otherwise your ground electrode would be also measuring some signal, so it's…

364
00:36:04.750 --> 00:36:14.339
Rm 330: also something that for your impedance measurements. You're just looking. You're measuring the current and voltage.

365
00:36:14.490 --> 00:36:34.170
Rm 330: So, yeah, yeah, it's just the same way that you would measure a resistor. The difference is that, we measure it as function of frequency, because all of these things are frequency-dependent, and, the best… the impedances are pretty high.

366
00:36:34.180 --> 00:36:36.330
Rm 330: So, you can't use a fluke.

367
00:36:36.680 --> 00:36:51.629
Rm 330: Or a low impedance amplifier. So you need a specialized amplifiers, but otherwise, it's exactly the same measurement. And by the way, for certain applications, we also do four-terminal measurement.

368
00:36:53.500 --> 00:36:56.850
Rm 330: And then you get rid of the contact impedance.

369
00:36:58.430 --> 00:37:11.850
Rm 330: So, do you have an… is that the amplifier? The amplifier? Yeah, yeah, so inside this box, we have the, 16 discrete amplifiers, and the Bluetooth, and everything, everything else.

370
00:37:12.290 --> 00:37:14.390
Rm 330: Yeah, in the previous floor.

371
00:37:15.310 --> 00:37:21.020
Rm 330: from… Even during sleep, the body will have some motion.

372
00:37:21.200 --> 00:37:24.490
Rm 330: So they will… won't there be some contribution from Motion as well?

373
00:37:25.510 --> 00:37:43.330
Rm 330: Yeah. Well, you… Does that then compete with the phase signal? This is part of it. So a healthy person, when a healthy person is asleep, during REM, in principle, you have muscle atonia.

374
00:37:43.760 --> 00:37:48.749
Rm 330: You still have eye movements, you still have breathing, clearly.

375
00:37:48.830 --> 00:38:08.209
Rm 330: We also have now recent data that shows that you have some residual, facial EMG, activity, but yeah, this is exactly that. There's blood pulsation, which brings about a bit of low frequency, activity.

376
00:38:10.900 --> 00:38:23.969
Rm 330: How is the person-to-person variability in terms of level of noise and shape of the torps there? Oh, that's part of the reason why people don't bother doing these kind of measurements. The variability is huge.

377
00:38:26.270 --> 00:38:36.160
Rm 330: person to person, day to day, morning to evening. It has a lot to do with the hydration of the skin, with.

378
00:38:36.450 --> 00:38:47.589
Rm 330: the thickness of the skin, exact region, where you do this type of measurement. Doing these kind of measurements is hellish, and that was the reason why it took us several years before we

379
00:38:47.750 --> 00:38:49.839
Rm 330: We ended up doing it.

380
00:38:49.980 --> 00:39:02.459
Rm 330: What we do today, we actually developed a phantom to mimic the electrical properties of the skin, so we use the phantom to characterize the electrons.

381
00:39:02.640 --> 00:39:08.130
Rm 330: But doing this kind of measurements on on people is really agonizing.

382
00:39:09.050 --> 00:39:23.490
Rm 330: It's very similar to characterizing electrical devices, it's just that you put things on, on people. But eventually, you want to calibrate and standardize so that the device can be used for

383
00:39:23.740 --> 00:39:32.919
Rm 330: You need to be within a certain range, okay? So, for example, if you want to,

384
00:39:33.560 --> 00:39:35.870
Rm 330: You need to be low enough.

385
00:39:36.310 --> 00:39:46.189
Rm 330: Okay, it's… you don't need to standardize it. You need the impedance to be below a certain value, and the determinant value is the input impedance of your amplifier.

386
00:39:46.460 --> 00:39:53.570
Rm 330: So if you have a very high input amplifier actually in the in the paper by Meta.

387
00:39:53.570 --> 00:40:06.619
Rm 330: They cite some really, really high, it's the highest input impedance I've ever seen. It's something above gigaohm input impedance exactly to avoid this.

388
00:40:06.900 --> 00:40:16.250
Rm 330: Variability. We have 10 mega Ohm in, in this frequency range.

389
00:40:17.010 --> 00:40:24.639
Rm 330: Okay, so when you try to work, for example, with hairy legs. This maybe a problem.

390
00:40:26.300 --> 00:40:30.560
Rm 330: Would you expect a signal at the heart rate?

391
00:40:30.790 --> 00:40:41.230
Rm 330: Should there be some signal associated with the heart rate? Yes, in the face, we mean. In any of the measurements, but not say the face. You put it against your chest, of course, this is ECG.

392
00:40:41.750 --> 00:40:46.700
Rm 330: Okay, if you put it in,

393
00:40:47.680 --> 00:40:50.869
Rm 330: Oh, not this computer.

394
00:40:51.660 --> 00:41:09.950
Rm 330: I'll show you some examples later. Yeah, right down here, during REM sleep, you can see the heart, okay? If you put the electrodes a bit lower, then you see it, very, very easily. If you put it here, it's massive.

395
00:41:10.510 --> 00:41:14.059
Rm 330: It's big signal, very, very big signals.

396
00:41:14.510 --> 00:41:18.739
Rm 330: Sorry. What was that? The short, short, 50.

397
00:41:20.740 --> 00:41:22.020
Rm 330: Shows up everywhere.

398
00:41:22.710 --> 00:41:25.139
Rm 330: Is it? Is it the one?

399
00:41:25.620 --> 00:41:27.399
Rm 330: My interference in Israel.

400
00:41:27.730 --> 00:41:30.600
Rm 330: It's your 60 Hertz.

401
00:41:32.310 --> 00:41:33.829
Rm 330: Yeah, so,

402
00:41:34.460 --> 00:41:48.580
Rm 330: Yeah, so now, okay, so I think I should, speed up. So, going beyond the electro technology, the next level of sophistication is to introduce, data cleaning.

403
00:41:48.730 --> 00:42:00.190
Rm 330: And so this is some heavy artillery that we introduce in order to clean the data.

404
00:42:00.600 --> 00:42:16.829
Rm 330: Then the next level of sophistication is when we record from the skin. So this is, for example, the structure of the facial muscles. Okay, so you see, it's quite elaborate.

405
00:42:16.890 --> 00:42:28.490
Rm 330: And one of the challenges in recording electrical activity from the skin is that you're picking up signals from many different sources.

406
00:42:28.610 --> 00:42:40.100
Rm 330: And so, what you need to do, or what we wanted to do, was to do source separation. It's a big thing in electrical engineering, where you pick up a signal

407
00:42:40.520 --> 00:42:58.200
Rm 330: So, for example, if you're electrode here, and you may be picking up signal from these different muscles, how do you go backwards and go back to the original source? So, luckily, they're very well established

408
00:42:58.200 --> 00:43:14.550
Rm 330: tools. One of them is called independent component analysis, and it's one manifestation of that is something which is called FAST ICA, and it's a readily available algorithm, and what it does

409
00:43:14.570 --> 00:43:24.119
Rm 330: is to minimize the statistical dependence between different sources. So you take the data from the 16 channels.

410
00:43:24.120 --> 00:43:41.129
Rm 330: and you, basically extract separated, sources, and what we did on top of that was to take data from more than 30 subjects, I think it was 32 different subjects, and after we extracted the sources.

411
00:43:41.130 --> 00:43:59.440
Rm 330: was to categorize the different sources into an atlas, okay? So, what the algorithm gives us is both the physical location of the sources, and also their time domain behavior, and when we look at

412
00:43:59.810 --> 00:44:16.010
Rm 330: those different sources and categorize them in terms of repetition in different people, we could identify, basically, specific repetitive sources, which we can then

413
00:44:16.020 --> 00:44:23.949
Rm 330: Identify with specific muscles, so this is the basically applying the, the reverse.

414
00:44:24.060 --> 00:44:37.599
Rm 330: Problem, and then we can associate, not just electrical activity with associate… which is associated with a general, region, but to associate it with a specific, muscle.

415
00:44:37.880 --> 00:44:53.460
Rm 330: I will show you later why, it's, it's so, important. So, now, I want to show you, several, examples from, yep. How do you know how many metrics to put on, like, how many…

416
00:44:53.690 --> 00:44:57.900
Rm 330: you know, like what in that picture was taken of the Atlas. You had all these

417
00:44:58.630 --> 00:45:01.210
Rm 330: Divided it up, see if it was naturally.

418
00:45:01.830 --> 00:45:08.790
Rm 330: Okay? So the honest answer of experimentalists is that we put as many as we Ken.

419
00:45:09.450 --> 00:45:21.880
Rm 330: Okay? In, in principle, I would love to have, more, and then go, down. Smaller resolution. Smaller resolution, as.

420
00:45:22.690 --> 00:45:34.250
Rm 330: We know that the number that we use for the face is roughly close to what we would love to have, maybe a bit more.

421
00:45:34.420 --> 00:45:36.630
Rm 330: But it's, it's…

422
00:45:37.870 --> 00:45:47.550
Rm 330: The thing is that from engineering point of view, from data point of view, this is a lot of data, so you don't want to,

423
00:45:49.570 --> 00:45:57.499
Rm 330: Until very recently, these kind of data were, I mean, it's not something that.

424
00:45:57.600 --> 00:46:01.350
Rm 330: To cry about, but we're,

425
00:46:01.850 --> 00:46:14.259
Rm 330: one major technical challenge is the amount of data that we, we collect and how, how to handle this kind of data. So you don't want to, to go overboard.

426
00:46:14.740 --> 00:46:16.939
Rm 330: With too much data.

427
00:46:17.500 --> 00:46:30.279
Rm 330: But it's not that we're at the optimized, necessarily at the optimized level. Okay. So these are the four that I want to show you.

428
00:46:30.320 --> 00:46:43.630
Rm 330: One concerns, facial expressions, voiceless, speech, just, something, very recent that we started, some work that we've done on

429
00:46:43.630 --> 00:46:51.229
Rm 330: Cognitive load and, sleep, and facial expressions during, during sleep.

430
00:46:51.250 --> 00:47:07.699
Rm 330: Okay, so these are four examples of how we can extract data and, basically, infer about the state, of the person. So why do we care about facial expression? So.

431
00:47:08.400 --> 00:47:12.330
Rm 330: You… all of you probably have seen,

432
00:47:12.470 --> 00:47:20.440
Rm 330: ins inside out the movie inside out. So you probably saw the first movie. And you saw the second movie.

433
00:47:20.440 --> 00:47:32.519
Rm 330: The first movie actually builds a lot on a theory that was holding, still hanging there, that argues it's a very, very,

434
00:47:32.740 --> 00:47:40.430
Rm 330: Well-respected by some circuits, theory that argues that we have

435
00:47:40.540 --> 00:47:58.099
Rm 330: Basically universal expressions which are associated with basic emotions. Now, for… especially in this kind of crowd, you know, with people coming from different parts of the world, speaking different languages, we know how

436
00:47:58.350 --> 00:48:14.189
Rm 330: emotion is such a complex thing, and how difficult it is to explain emotion, and what does sadness or happiness mean in one language, and what does it mean in another. Still, the argument was that.

437
00:48:14.320 --> 00:48:28.749
Rm 330: Or the argument is that we all share, as humans, actually it was Darwin that started this hypothesis, basic emotions, a very narrow set of emotions, and these emotions have

438
00:48:28.750 --> 00:48:45.159
Rm 330: facial manifestation in facial expressions. So, you smile, you're happy, you're disgusted, you do this, kind of thing. There are a lot of research, a lot of work that was, done into it.

439
00:48:45.160 --> 00:49:00.649
Rm 330: Let's just say that there's a lot of criticism against it. When we just started, we came across this work, and so this was part of the motivation. Full disclosure, we're not really

440
00:49:00.650 --> 00:49:11.030
Rm 330: We're actually busying ourselves trying to revoke much or help others revoke some of these ideas.

441
00:49:11.090 --> 00:49:12.060
Rm 330: Oh.

442
00:49:12.830 --> 00:49:32.010
Rm 330: How do you ensure that the person under the experiment agrees with this definition? Exactly. This is the exact problem that how do you the methodology behind these experiments. The guy behind this theory is Paul Ekman. He died very recently.

443
00:49:32.010 --> 00:49:39.049
Rm 330: What he did originally was to travel to Papua New Guinea. He was in the '60s or '60s.

444
00:49:39.060 --> 00:49:48.079
Rm 330: It needed two layers of translation. So the idea was that these were people that were completely isolated from the Western world.

445
00:49:48.090 --> 00:50:01.930
Rm 330: Sorry. Yeah, yeah. And he he needed 2 levels of translations. He was. It took him some time to figure out how to do this communication, how to

446
00:50:02.230 --> 00:50:19.389
Rm 330: questioned them, but his final… he was using these pictures, showing the pictures, basically asked them to describe the emotions that they could infer, ultimately reaching this conclusion.

447
00:50:19.600 --> 00:50:39.879
Rm 330: Okay, the current understanding in psychology is much, much, much more complex than that. And that's the reason why the second movie already incorporates. He was a consultant on the first movie. In the second movie they brought in other people. So actually introducing much more sophisticated

448
00:50:39.880 --> 00:50:41.030
Rm 330: View.

449
00:50:41.030 --> 00:50:41.830
Rm 330: Okay.

450
00:50:42.570 --> 00:50:45.609
Rm 330: ml So what you see here.

451
00:50:45.740 --> 00:50:50.829
Rm 330: On the left, you see… okay, I'll let you watch.

452
00:50:51.190 --> 00:50:55.289
Rm 330: It's nice to watch, it also brings emotions.

453
00:51:00.470 --> 00:51:10.330
Rm 330: Okay, so this is a real-time avatar, and what the avatar, what we did in this avatar was to control it with the EMG.

454
00:51:10.530 --> 00:51:25.399
Rm 330: Okay? So, there's no camera, that, uses to control the avatar. The avatar is controlled entirely by, the EMG data. And the way you do that is by building a model

455
00:51:25.570 --> 00:51:41.879
Rm 330: That is person-to-person, varies from person to person, but what the model does is to take the computer vision representation of the facial expressions, okay, so the deformation, of the face.

456
00:51:41.990 --> 00:51:54.860
Rm 330: fitted to the EMG data, and essentially extract a model, and then you can control the avatar in real time from EMG data.

457
00:51:54.890 --> 00:52:13.930
Rm 330: Now you have a human machine interface that builds entirely on EMG data. You don't need a camera, so a person can be sitting here communicating with somebody else. In principle, keeping your privacy.

458
00:52:13.930 --> 00:52:22.379
Rm 330: Because the only thing that you deliver to the system is just your EMG data. You don't really have to

459
00:52:22.650 --> 00:52:30.599
Rm 330: be exposed to, the camera. You can pick up, these signals in a completely dark room.

460
00:52:30.600 --> 00:52:48.789
Rm 330: And for research purposes, people can be interacting one with the other, directly, not through, computers or cameras. So this is, what we had in mind when developing these systems. There are a few other, reasons why

461
00:52:48.790 --> 00:52:52.430
Rm 330: Yeah, sure.

462
00:52:56.690 --> 00:52:57.490
Rm 330: God!

463
00:53:07.400 --> 00:53:08.070
Rm 330: Okay.

464
00:53:09.630 --> 00:53:12.100
Rm 330: Very cute.

465
00:53:12.680 --> 00:53:31.360
Rm 330: Okay, so one of the other reasons why we developed it is to increase the engagement of, I'll show you at the very end, people suffering from palsy that need to go through rehabilitation, and we want them to have engagement with interactive system, rather than just seeing

466
00:53:31.360 --> 00:53:36.880
Rm 330: You know, data or bars, on the… on the screen.

467
00:53:37.620 --> 00:53:38.540
Rm 330: Okay.

468
00:53:39.300 --> 00:53:48.979
Rm 330: Okay, so I'll skip that. I don't know. I'm out of battery, and I don't see any clock, so I don't know if I'm way way way 5 min left. Okay.

469
00:53:49.050 --> 00:54:04.059
Rm 330: The other example is, this is really something that we just started, but I really love this example. Voiceless speech is a huge thing now in human machine interfacing.

470
00:54:04.210 --> 00:54:18.169
Rm 330: Part of the reason might be that just like the revolution that we had a few years back with earbuds, imagine a situation that now we can speak.

471
00:54:18.270 --> 00:54:25.820
Rm 330: without… Okay? So, if you can imagine something like that, then…

472
00:54:26.250 --> 00:54:35.020
Rm 330: People can work on the trains. You can squeeze more people into open workspace.

473
00:54:35.140 --> 00:54:39.479
Rm 330: I don't know, but big companies are investing.

474
00:54:40.250 --> 00:54:52.530
Rm 330: Stupendous amount of money. Google was just buying an Israeli startup for $300 million, very recently, completely…

475
00:54:52.830 --> 00:55:11.379
Rm 330: Completely crazy. But the interesting thing, so you could use in principle since I showed you that we can see the facial muscles. In principle, that should be entirely straightforward. If we can know what the expression is, we can do a lip reading.

476
00:55:11.470 --> 00:55:15.569
Rm 330: The thing is that, and this is, even us

477
00:55:15.610 --> 00:55:24.969
Rm 330: Simple engineer, we could very quickly realize that speech is not a simple human behavior.

478
00:55:24.970 --> 00:55:40.280
Rm 330: Not by a long shot. And so, the two models, or the two decoders that you see here, so this is a confusion matrix, this is a standard way of showing, how, how you, what you predicted and what is the true.

479
00:55:40.280 --> 00:55:53.310
Rm 330: measure. And so you see up here, we get really nice results. So these are 22 words, these are… it's actually the phonetic alphabet, so alpha.

480
00:55:53.330 --> 00:56:01.959
Rm 330: What do you have? Alpha Bravo, Charlie. Okay, so this is in English Romeo. Whatever.

481
00:56:01.960 --> 00:56:13.819
Rm 330: In Hebrew, we have our own version, every language has its own. This is what we did, and you get these beautiful decoding performances.

482
00:56:13.890 --> 00:56:27.060
Rm 330: And the trick, though, was that the way that we did it here was repeat the same word 10 times, so every word was repeated 10 times. So, Aleph.

483
00:56:27.540 --> 00:56:29.530
Rm 330: It… Oh.

484
00:56:30.180 --> 00:56:43.000
Rm 330: I don't remember the rest, David, something, and then, you take 8, the first 8, push it into the model, you take the two, left, and put it into the task.

485
00:56:43.000 --> 00:56:50.300
Rm 330: And voila, you have this wonderful AI predictive model. Now you do.

486
00:56:50.510 --> 00:56:52.299
Rm 330: Send by Carly.

487
00:56:52.980 --> 00:56:55.660
Rm 330: No, it's… it's one, two… well, it's…

488
00:56:56.120 --> 00:57:02.269
Rm 330: Yeah, it's, it's… no, it's 20… 22 against 22. Yeah.

489
00:57:02.410 --> 00:57:13.080
Rm 330: Well, it's… A psychology student. Okay, now we did almost the same thing, we just randomized the word.

490
00:57:13.390 --> 00:57:23.589
Rm 330: the order of the work. So, you still do 10 repetitions, but you don't do them sequentially. You randomize it. And then everything just falls apart.

491
00:57:24.020 --> 00:57:27.530
Rm 330: Okay? And maybe if you want to learn more about why

492
00:57:27.600 --> 00:57:39.129
Rm 330: You can go, now we have connections with the linguistic department. The point is that the way that you pronounce or articulate a word.

493
00:57:39.130 --> 00:57:56.419
Rm 330: it really matters on context, it matters on what was before, what's coming next, and when you do the same thing again, again, again, again, then you do the same thing, then your predictive models work so well. And so this is the reason why

494
00:57:56.670 --> 00:58:00.320
Rm 330: Things become very interesting and challenging and fun.

495
00:58:00.890 --> 00:58:17.110
Rm 330: Okay, cognitive load. So, cognitive load is a very big, concept, a huge thing in human-machine interfacing, a holy grail in understanding human performance. It's, it's actually a very, important.

496
00:58:17.480 --> 00:58:36.649
Rm 330: concept in cognitive neuroscience and psychology, it's the amount of mental effort required to process information and to perform a task. And in principle, when you build devices, when you build technology, when you teach, many, many things that you do.

497
00:58:36.650 --> 00:58:44.940
Rm 330: You want to reduce the cognitive load. You want it to be easy for people to understand what you're talking about.

498
00:58:45.030 --> 00:58:57.709
Rm 330: And now the question is, how do you measure that, right? So in physics, when we define a concept, we always know, or most of the time, we know how to measure it. In psychology.

499
00:58:57.960 --> 00:59:14.599
Rm 330: It's not always the case. So they have a lot of concepts, but there's no clear definition of how you measure that. So cognitive load is a wonderful example. There are many ways of trying to estimate cognitive load.

500
00:59:14.600 --> 00:59:19.549
Rm 330: There are ways to induce cognitive load. And so what we

501
00:59:19.950 --> 00:59:38.960
Rm 330: What we did was to build a paradigm that is known from the literature to introduce cognitive load, and so you actually make people play a game of maze, and you make the maze more and more and more complex, so you make it more difficult.

502
00:59:38.960 --> 00:59:44.989
Rm 330: And you look at what happens to the physiology of the person as you do that.

503
00:59:45.010 --> 00:59:56.210
Rm 330: And to establish a ground truth, you need some measure of reporting. So, NASA TLX, that's a standardized measure used

504
00:59:56.940 --> 01:00:04.319
Rm 330: It's a self-report developed by NASA. It was developed, actually, for pilots.

505
01:00:04.490 --> 01:00:18.999
Rm 330: Okay? But this is… this is the standard. This is how you, try and estimate, cognitive load. So, what we did was to look at all sorts of different, physiological measures from the face

506
01:00:19.000 --> 01:00:26.399
Rm 330: Looking at blinking and eye movements and EMG and EEG, different bandwidth, different

507
01:00:26.400 --> 01:00:38.109
Rm 330: statistical parameters, and try to correlate them with the cognitive load that we got from the self-reporting, and, observe

508
01:00:38.170 --> 01:00:55.409
Rm 330: what are the best values, that we can, measure. The really interesting thing, again, that once you leave the lab, so this was done in the lab, the person is sitting in front of a computer, everything is controlled, everything is.

509
01:00:55.530 --> 01:00:56.690
Rm 330: is set.

510
01:00:57.000 --> 01:01:01.020
Rm 330: We have the technology to take people out.

511
01:01:01.110 --> 01:01:20.620
Rm 330: Okay, so this experiment was actually very similar to this experiment. She could hear the commands, she could hear, what to do. We were nagging her with, challenges that she had to… it's called ANBAC, she had to, listen to specific

512
01:01:20.620 --> 01:01:24.999
Rm 330: Auditory cues and try to report on them.

513
01:01:25.040 --> 01:01:39.119
Rm 330: And all of that is sent to the computer, and we have the data tagged or annotated with all this information, but now she's not playing a maze, she's actually walking, okay? When a person is walking.

514
01:01:39.120 --> 01:01:51.580
Rm 330: they have to pay attention that they don't step on things, okay? That people don't get into their way, and then suddenly, things that work so beautifully in the lab

515
01:01:51.580 --> 01:02:01.520
Rm 330: becomes so, different, outside. And this is one of the things that we're, trying to do now, is really to extend

516
01:02:01.610 --> 01:02:08.869
Rm 330: The know-how from the experiments in the lab into the real world.

517
01:02:09.190 --> 01:02:13.389
Rm 330: Okay, so I, I think I, I, I don't like,

518
01:02:13.660 --> 01:02:22.029
Rm 330: going too much overtime. We have been doing a lot of work, recording people during, sleep.

519
01:02:22.110 --> 01:02:38.970
Rm 330: So this is, a validation that we've done. These are, more than 70, here it was 50 people, but then, we measured more than 150 people, I think, by now. Actually more. People that went to sleep with our system.

520
01:02:38.970 --> 01:02:50.330
Rm 330: With a conventional system in the hospital, and then we can, compare, again, this is a confusion matrix, between what our system sees, what

521
01:02:50.330 --> 01:02:59.489
Rm 330: A conventional system sees, and we can see that we can really faithfully reconstruct the full, night, architecture.

522
01:03:00.360 --> 01:03:16.319
Rm 330: Just one really, nice thing, to see, these are two, records of, the same person sleeping first in the lab. You see, this is A, verbal in the lab.

523
01:03:16.320 --> 01:03:32.539
Rm 330: and B, this is wearable at home. And you can see that during the lab session, the person ends up falling asleep very, very late, continuously waking up, and actually having very little amount of RAM during the night.

524
01:03:32.680 --> 01:03:39.720
Rm 330: When the same person is going to sleep at home, you see these really beautiful stretches of REM sleep.

525
01:03:40.020 --> 01:03:49.630
Rm 330: Okay, so this is one of those things that we're aiming for, to take the technology of sleep staging, of sleep.

526
01:03:49.860 --> 01:04:00.249
Rm 330: research from research labs to the home environment. This was again repeated many, many, many times.

527
01:04:01.160 --> 01:04:15.139
Rm 330: Now, just the very, very, final thing. So, I'm going back to the facial expressions. So, just as we discussed before, it's very likely that facial expressions, when we're.

528
01:04:15.240 --> 01:04:33.929
Rm 330: awake and communicating has a lot to do with variability that has to do with our social status, our culture, whether we are interacting with people that we're familiar with, people that we want to impress. It's a very complicated scenario.

529
01:04:33.970 --> 01:04:40.610
Rm 330: And then the question is, What happens if we try to do these recordings at night?

530
01:04:40.840 --> 01:04:54.429
Rm 330: when people are asleep. And so, technologically, we can do that, and this is something that we started, doing, very, very recently. By the way, if you want to convince… so, inside out, this is Ekman.

531
01:04:54.780 --> 01:04:58.180
Rm 330: And if you like this kind of…

532
01:04:58.330 --> 01:05:17.750
Rm 330: literature. So, Malcolm Gladwell, has a relatively recent book in Talking to Strangers, and he has a full chapter just bashing, this, this concept, which, by the way, has been adopted in many, many, many fields. Judges here in the U.S. use

533
01:05:17.890 --> 01:05:23.490
Rm 330: facial expression to as a testimony to

534
01:05:23.630 --> 01:05:29.970
Rm 330: To the nature of, you know, the reliability of testimonies of people. It's, it's really sad because it's

535
01:05:30.950 --> 01:05:32.870
Rm 330: very problematic theory.

536
01:05:33.210 --> 01:05:50.390
Rm 330: Okay, so this is what we can do. We can actually record facial expressions during night, we can do the night staging, we can look at the heart rate, we can look at the entire physiology of people, and then we can look at the signatures

537
01:05:50.390 --> 01:06:00.600
Rm 330: of the expression, so the purpose was to see, you know, it's not that Ekman was making up those facial expressions. They do exist, but

538
01:06:00.640 --> 01:06:16.009
Rm 330: In normal circumstances, in wakefulness, there's a lot of noise. There's a lot of extra things, and so what we're trying to see, whether we can see these expressions actually showing up at night.

539
01:06:16.010 --> 01:06:23.069
Rm 330: In, in a much more, in a cleaner, in a cleaner way, and we think we…

540
01:06:23.310 --> 01:06:28.859
Rm 330: Can. So, it's preliminary, it's not published yet, but this is, one…

541
01:06:29.200 --> 01:06:45.030
Rm 330: direction, and the very, very, very final thing, this is not human-machine interface, but these are recordings that now we can do from the two sides, so not just the single side, and for people suffering from facial palsy.

542
01:06:45.030 --> 01:06:57.639
Rm 330: This is really an ability to quantify which muscles are active, and when, and also quantify, the asymmetry, and.

543
01:06:57.810 --> 01:07:10.610
Rm 330: We are already working. We've been working for a couple of years with healthy individuals, but recently started working also with patients, and it's really rewarding.

544
01:07:10.920 --> 01:07:20.130
Rm 330: Because it's really give a wonderful edge to their ability to really see their

545
01:07:20.160 --> 01:07:32.660
Rm 330: their problem because otherwise they just have a very crude manifestation of the problem and also to help focus the rehabilitation process.

546
01:07:33.140 --> 01:07:39.899
Rm 330: So, yeah, this is what I wanted to, share with you.

547
01:07:43.850 --> 01:07:54.239
Rm 330: This is showing you that we can do this avatar work, not just in the lab. This is what I've been promising that we can all these things work.

548
01:07:54.370 --> 01:07:58.460
Rm 330: Also, when We are out.

549
01:07:58.600 --> 01:08:09.039
Rm 330: In the wild, so WILD is a conference in Boston, I think? You need a computer, what can you do?

550
01:08:09.110 --> 01:08:25.729
Rm 330: So yeah, and and this is this is my my group. And a lot of amazing collaborators that have been helping us steer our research. And so we're always very eager to

551
01:08:25.860 --> 01:08:31.719
Rm 330: to collaborate or help other people in these fields. So, thank you very much.

552
01:08:37.210 --> 01:08:41.040
Rm 330: Yeah, very inspiring talk. Any questions?

553
01:08:43.840 --> 01:08:44.540
Rm 330: Okay.

554
01:08:45.490 --> 01:08:50.249
Rm 330: During REM sleep, the motion is inhibited, right? So what are you measuring?

555
01:08:51.690 --> 01:08:54.690
Rm 330: Emotion is inhibited? Motion. Motion!

556
01:08:54.850 --> 01:09:06.739
Rm 330: Well, in principle, okay, so, I… we spent a lot of time working on, a phenomenon called REM sleep without atonia.

557
01:09:06.890 --> 01:09:14.770
Rm 330: REM sleep without atonia is an abnormal situation where people actually have

558
01:09:14.830 --> 01:09:29.959
Rm 330: muscle activity during REM, okay? So the classical textbook is healthy people have atonia during REM, only during REM, okay? And, people,

559
01:09:29.960 --> 01:09:46.120
Rm 330: It's a characteristic of onset of some neurodegenerative conditions. They have some muscle activity during their REM sleep.

560
01:09:46.290 --> 01:10:01.340
Rm 330: But it turns out, when you actually look at the data, so this is how, you know, you… this is the textbook, but when you actually look at the data and you start quantifying REM sleep without atonia, you understand that there… it's… it's not, like.

561
01:10:01.510 --> 01:10:05.910
Rm 330: There's a lot of muscle activity.

562
01:10:06.050 --> 01:10:15.689
Rm 330: people with a lot of muscle activity, the threshold between people that have high REM sleep without the tone and people that have

563
01:10:15.710 --> 01:10:30.549
Rm 330: no REM sleep without atonia, it's not discrete, it's not quantized, you know, very strong REM sleep without atonia, and zero REM sleep without atonia. So, perfectly healthy people would have, some

564
01:10:30.870 --> 01:10:44.360
Rm 330: Some muscle activity, and the pilot study that we did was 9 people, looking at their facial expressions, during the night, and there are facial expressions.

565
01:10:46.080 --> 01:10:52.150
Rm 330: So the muscle activity is just in the face during REM, or is it every… somewhere else?

566
01:10:53.220 --> 01:11:00.519
Rm 330: Okay, so this is a good question. I don't know, don't know the answer.

567
01:11:00.840 --> 01:11:11.490
Rm 330: In neurological patients, it's everywhere. So, actually, the problem is…

568
01:11:11.700 --> 01:11:20.040
Rm 330: It's called Rbd, it's Rm behavior disorder. And in those conditions, people basically they live the dream.

569
01:11:20.350 --> 01:11:26.799
Rm 330: Okay, so part of the reason why we have this atonia is so we will not

570
01:11:26.960 --> 01:11:28.899
Rm 330: Act the dream.

571
01:11:29.010 --> 01:11:52.950
Rm 330: And people with RBD, they actually have dream enactment. So you could see people — there's one reported case where a person does this. So it's like smoking. He quit smoking, so in his dream, he's smoking. But this is very abnormal behavior, and you can see it in the video.

572
01:11:53.170 --> 01:12:03.149
Rm 330: So sleepwalking is part of this? No, no, no, sleepwalking is a different condition. So there are a lot of abnormalities. There are sleep tremors, there are sleep,

573
01:12:03.370 --> 01:12:15.599
Rm 330: people suffering from post-trauma, the line between them is, is not always very, very clear. So RBD is a very well-defined

574
01:12:15.620 --> 01:12:35.059
Rm 330: Neurological thing, it's categorized by excessive muscle tone and dream enactment, so you really need to see incidents of dream enactment.

575
01:12:35.240 --> 01:12:43.380
Rm 330: All the rest of, you know, tremors and nightmares and, It's kind of blurry.

576
01:12:44.930 --> 01:12:45.720
Rm 330: Okay.

577
01:12:46.340 --> 01:12:51.529
Rm 330: So there are certain diseases that can't be diagnosed until after, say, death.

578
01:12:51.720 --> 01:12:53.600
Rm 330: And a lot of them are neurological.

579
01:12:54.060 --> 01:12:55.630
Rm 330: like dementia.

580
01:12:56.010 --> 01:13:04.540
Rm 330: I hope I'm not showing it, but are there… are there ways that you might think of the very sophisticated and accurate

581
01:13:04.770 --> 01:13:11.559
Rm 330: Sensitive measurements you're making off base that might actually, as time progresses.

582
01:13:11.810 --> 01:13:24.599
Rm 330: find a connection, a correlation with dementia? Yeah, so… Okay, this is a big, big, big field. It's called, biomarkers, or early biomarkers.

583
01:13:24.890 --> 01:13:33.470
Rm 330: And there's another, if you want to be aware of foundation models.

584
01:13:33.610 --> 01:13:48.719
Rm 330: in neural networks. So what you can, what people have been doing. It's very, very recent papers coming out really, in the last year or 2. You can take massive amounts.

585
01:13:48.720 --> 01:13:49.420
kloet: you.

586
01:13:49.420 --> 01:14:04.930
Rm 330: say, sleep data, tens of thousands of patients and patients that you know what their what was their history, or what was their evolution. So you can take the the say, sleep data.

587
01:14:04.980 --> 01:14:09.640
Rm 330: and extract from and using the demographic of these patients.

588
01:14:09.640 --> 01:14:10.520
kloet: Yeah.

589
01:14:10.520 --> 01:14:25.569
Rm 330: correlate or build predictions for exactly for that. And not just for dementia. So there's a big thing now what's called SleepFM, so Sleep Foundational Model.

590
01:14:25.690 --> 01:14:31.680
Rm 330: It's a model that, Supposed to, predict.

591
01:14:31.820 --> 01:14:38.290
Rm 330: Actually… Any condition from sleep data.

592
01:14:39.370 --> 01:14:53.010
Rm 330: It's a couple of papers that came out from Stanford, and there are more and more such papers. Basically, you need a lot of annotated data.

593
01:14:53.230 --> 01:15:01.020
Rm 330: EEG data, sleep data, and, knowledge about, the patients.

594
01:15:01.230 --> 01:15:16.840
Rm 330: and basically develop a model that would give you the prediction. So, what we've started doing was to take the data that we have. We don't have access to tens of thousands of data, but we do have hundreds of

595
01:15:16.890 --> 01:15:22.559
Rm 330: patients. So, for example, in dementia.

596
01:15:22.630 --> 01:15:27.650
Rm 330: Before dementia, you have something which is called mild cognitive decline.

597
01:15:27.750 --> 01:15:31.910
Rm 330: This is a very strong predictor of Alzheimer.

598
01:15:32.010 --> 01:15:41.389
Rm 330: And so, among the population of subjects that we measured, there are patients with mild cognitive decline, and so.

599
01:15:41.390 --> 01:15:55.470
Rm 330: In principle, you can observe that phenomena, so we can take the sleep data, you can put it into the model, this foundational model, and then, get the probability for.

600
01:15:55.600 --> 01:16:01.250
Rm 330: For Alzheimer's, it's a bit problematic.

601
01:16:01.950 --> 01:16:17.789
Rm 330: I mean, I think it would still take time to see how, you know, how robust these foundational models are, because I think part of the problem is that they're still statistical in nature.

602
01:16:17.970 --> 01:16:21.059
Rm 330: Okay? So if you would start measuring everybody.

603
01:16:21.500 --> 01:16:32.829
Rm 330: And you have some, You know, tail, around, You know, around your prediction.

604
01:16:33.910 --> 01:16:37.470
Rm 330: I don't know if you, if if any one of us want to.

605
01:16:39.280 --> 01:16:41.220
Rm 330: There's no pill you think you can use.

606
01:16:42.600 --> 01:16:46.330
Rm 330: Well, the in in many of the neurologic. So so in.

607
01:16:46.760 --> 01:17:05.680
Rm 330: For the pharma companies, actually, this may be a very strong tool because this can allow them to focus their clinical studies. You as an individual can benefit because it may give you better motivation for changes in lifestyle.

608
01:17:05.760 --> 01:17:09.980
Rm 330: which are very, very important in neurodegenerative diseases.

609
01:17:10.430 --> 01:17:13.189
Rm 330: unless you have a genetic.

610
01:17:13.720 --> 01:17:14.660
Rm 330: Oh.

611
01:17:16.980 --> 01:17:21.370
Rm 330: There's a big article exactly about that in Rutgers today.

612
01:17:22.780 --> 01:17:24.420
Rm 330: Sorry you can.

613
01:17:25.250 --> 01:17:33.470
Rm 330: From your eyes, they can, research can predict signs of cognitive decline.

614
01:17:34.410 --> 01:17:48.240
Rm 330: I was curious about, so in the… in the policy case, so you were using it kind of as a diagnostic for… I was wondering, is it… is it the case that, like, let's say you have some kind of

615
01:17:48.240 --> 01:18:02.500
Rm 330: muscular degeneration that makes it so that your muscles cannot trigger, but you can still see it in your signal that this is the muscle that wants to be fired. Can the avatar then have the correct

616
01:18:02.500 --> 01:18:08.880
Rm 330: patient expression to… Right, so this is exactly what we, what we want to do, so…

617
01:18:08.930 --> 01:18:19.520
Rm 330: What, the sad thing about, Facial palsy is that it's a very… it's a relatively rare condition.

618
01:18:19.780 --> 01:18:38.109
Rm 330: So, there's not much investment in this, in this field, and the technologies are very, very, there's not enough knowledge in, among physicians, not enough clinics that actually, specialize in.

619
01:18:38.320 --> 01:18:51.709
Rm 330: In Vienna specifically, they do have a rehabilitation center, and it's one of the very few, even in Europe. So you can do rehabilitation, but you need feedback.

620
01:18:52.100 --> 01:19:08.539
Rm 330: And so people come there, and they put, the electrodes, and then they… they get the guidelines what to do, in order to, relax. Actually, what they need to do is… is work on how to relax

621
01:19:08.580 --> 01:19:15.629
Rm 330: the muscles. It's not really so much that the muscles don't work, it's that they don't work when they should work.

622
01:19:16.590 --> 01:19:27.770
Rm 330: So, yeah, it's not just… I mean, the visualization is just one element. The other… so there are two options for patients, so…

623
01:19:27.980 --> 01:19:30.209
Rm 330: One is the rehabilitation.

624
01:19:30.640 --> 01:19:41.590
Rm 330: Which some patients are not very eager to do, because it's a lot of work, and they don't always have access to.

625
01:19:41.730 --> 01:19:45.599
Rm 330: To clinics to to train them how to do it properly.

626
01:19:45.790 --> 01:20:04.730
Rm 330: What they can also do is to, they can benefit from Botox injections, but then the challenge is to, steer the injection. And this is also something that, so this is a work that we're, part of it, we're finalizing now.

627
01:20:04.730 --> 01:20:14.670
Rm 330: With our colleagues, and ultimately, we would like to build the avatar, in a way that we, we can

628
01:20:14.770 --> 01:20:24.749
Rm 330: interface it… well, use it for, for functions to, to interface directly with the avatar. The nice thing about the avatar is that we can tweak it.

629
01:20:25.010 --> 01:20:36.289
Rm 330: So we can… this is the hope, that we can amplify, for example, so if the person is doing a very little, say, asymmetric smile, we, we, we…

630
01:20:36.290 --> 01:20:45.880
Rm 330: artificially change the action. We did the very first test last week with a patient.

631
01:20:45.930 --> 01:21:02.390
Rm 330: It was very, you know, very emotional, both for her and for us. The first few minutes, she was ecstatic, she was so happy, and 3 minutes later, she completely lost interest, and she was super tired, and she said, oh, I don't need that, I just need 2 bars.

632
01:21:02.390 --> 01:21:08.740
Rm 330: So, you know, really building a system for patients with,

633
01:21:09.030 --> 01:21:10.370
Rm 330: It's… it's…

634
01:21:10.600 --> 01:21:21.580
Rm 330: there's still a long way, because it's… it's how to engage them, to keep on… you know, it's just like going to the gym. It's not how you do it 5 minutes once a month, it's…

635
01:21:21.750 --> 01:21:27.039
Rm 330: How to make it, engaging and effective.

636
01:21:28.930 --> 01:21:30.060
Rm 330: Last question?

637
01:21:30.360 --> 01:21:31.970
Rm 330: Yeah, so…

638
01:21:31.990 --> 01:21:49.460
Rm 330: your data is, like, a time series data that combines, the signal coming from multiple different parts of the data. They're all combined in the timelines, right? But what you say… so what… I was wondering whether, you know, the signal from different parts of your face come

639
01:21:49.460 --> 01:21:54.070
Rm 330: A different time, so there's probably the time correlation.

640
01:21:54.180 --> 01:22:09.670
Rm 330: So the way you kind of identify the different, you know, the signal source is based on that kind of correlation? Yeah, so there are two related ways of doing that.

641
01:22:10.500 --> 01:22:18.560
Rm 330: One, so the ICA, the independent component analysis, assumes that you have relatively high frequency.

642
01:22:18.630 --> 01:22:34.879
Rm 330: basically a superposition of several sources that you see, with different amplitude in different channels, and so you reduce, you minimize… what the algorithm does is to minimize the.

643
01:22:35.040 --> 01:22:44.869
Rm 330: the correlation or the statistical dependence between these sources. And this is how you find the cleanest sources.

644
01:22:44.920 --> 01:23:01.770
Rm 330: If you work only with EMG data, and you have some time separation, then there are simpler tools, similar to what you do in spectroscopy, right? When you have different sources, so you assume that you have several, say.

645
01:23:01.770 --> 01:23:09.320
Rm 330: Gaussians, and so you can just find the right superposition. So there are few

646
01:23:09.390 --> 01:23:12.790
Rm 330: related approaches. How to how to do that.

647
01:23:13.440 --> 01:23:32.269
Rm 330: All right, last question. Actually, on that same note, I was reading about ICA quirks, like PCA, but it does not… it only gives you components, but then how do you map that component to a specific muscle? Okay. It took us only one year to figure that out.

648
01:23:32.270 --> 01:23:37.090
Rm 330: Yeah. So the the trouble with Ica is that it it just

649
01:23:37.620 --> 01:23:50.799
Rm 330: It does the separation, but then it gives you random, sources which you don't know, what to do with. But the thing is that it also gives you the.

650
01:23:50.880 --> 01:23:57.840
Rm 330: The weight of each source, and its location, basically the electrode.

651
01:23:58.520 --> 01:24:08.040
Rm 330: Right? Because it gives you the full matrix of each source, so how was it constructed from each

652
01:24:08.280 --> 01:24:19.050
Rm 330: you know, what was the original? So you have the sources, and you have the original signal, and you have a matrix that transforms between these two vectors, right?

653
01:24:20.080 --> 01:24:39.190
Rm 330: Now, so that gives you a physical mapping of where that source is, okay? And so, the way that we take the output of the ICA and convert that into physical, consistent physical, location.

654
01:24:39.190 --> 01:24:49.349
Rm 330: is by looking at similarity. So, the first thing was to build an atlas. So, we took data from 32 people, and that built

655
01:24:49.450 --> 01:25:04.129
Rm 330: From that, we built an atlas of those consistent sources that I showed you. And later on, any other person that we record, we look for similarity against the atlas.

656
01:25:05.100 --> 01:25:09.699
Rm 330: I hope that makes sense, but we have, we have it in, in,

657
01:25:09.850 --> 01:25:18.629
Rm 330: in a paper. It wasn't… it wasn't August, so we were doing it manually for a very long time. It was very agonizing.

658
01:25:20.480 --> 01:25:23.340
Rm 330: Thank you for 2 talks.