YouTube Video
Episode Summary
In this episode of Speak with Tyler Bryden, discover the cautious yet optimistic perspective on AI and research from product marketer Katya Ryabova in this insightful podcast conversation.
Katya Ryabova (She/Her) is a Customer researcher and Product Marketer and is recognized as a Top Product Marketing Voice on LinkedIn.
Katya owns, SMM Headquarters, a registered solo consultancy providing customer-centric research, growth strategy and advisory services to scaling tech, SaaS and B2C subscription businesses.
If you’re interested in qualitative research, consumer insights, and exploring the human side of emerging technologies, don’t miss this nuanced discussion!
Gain a balanced understanding of the cautions and possibilities for integrating AI into the research process. Both Katya and host Tyler Bryden provide insightful perspectives from their work in this space.
You can find Katya on LinkedIn here: https://www.linkedin.com/in/katyar/
Interested in connecting with Katya?
Visit this link and book a time for her Research Power Hour: https://smmhq.ca/power-hour/
Transcript (Unedited)
Tyler Bryden: Hello, everyone. It is Tyler Briden here. I hope everything is going well. Very excited to have a guest recording and creating with me today. I’ve been creating on YouTube for a little bit over a year now, and I’ve enjoyed it very much. But I’m lonely and I need some people to talk to. And I found a wonderful person to talk to not.
A guinea pig in a bad way, but she’s coming on to this channel and she’s coming into this weird domain and audience that I’ve built. So I appreciate you being welcoming checking this out and I hope to have some more people on in the future. Without further ado, I want to welcome Katya. I’ll just give a little bit of context on Katya, at least from my perspective, she was nice enough through the web works of Toronto and AI and all the hype around this to invite me to a talk and I showed up. I did my best. I felt like I was rambling and ranting, but Katya was nice enough to say, Hey, you did a good job and reconnect with me here today.
So I’ll take that as a sign Katya, thank you so much for joining here.
Katya: No, it’s my pleasure. I love guinea pigs.
Tyler Bryden: Exactly. You’re a researcher. So maybe tell a little bit about that.
Would love to hear it in your own words, but you’re a researcher at heart and you bring a lot of great experience in that some perspective. And I think we want to address a little bit about this. Crazy question that research teams or just research individuals are asking today about, like, how can they embed technology into their workflow?
Should they embed this kind of technology into their workflows?
Katya: I like introducing myself normally as a product marketer and researcher because I, I have a lot of background in research, and this is something I actively do day to day, specifically for product marketing for B2B tech and B2C subscription businesses mostly, and I focus on customer led, it’s a buzzword a little bit, but customer led Strategies for growth, which obviously come from understanding customers for your product.
And I work with product teams the most. So product marketer slash researcher. And AI has been a big part of the conversation lately, and there’s lots to talk about, so I’m happy to dig into that
Tyler Bryden: and maybe just I’m out of curiosity. So you talk about this sort of dual title or role that you’re in, but obviously they’re pretty intimately linked when you engage with an organization.
What does that look like? Is that focus groups? Is it 1 on 1 interviews? Is it survey data? What is a sort of engagement look like the type of data that you’re collecting and
Katya: analyzing? Yes, it’s normally one on one interviews and surveys. If we can, we would do a mix of both. It really depends on what the client is looking for, because the idea is that I come in to do research, design research to extract insight that will empower the teams to be better decision makers.
To get them unstuck to get them moving into the direction of the strategy for their product. That actually would make sense, which is impossible without understanding how customers actually use the product and how they came to use the product. I bring the most value when when the company already has a solid base of customers that are paying for the product and are happy with it.
And then we learn from them directly, either through interviews or surveys, and then analyze all of that to get to the insight that they need
Tyler Bryden: to move forward. We’re taking a little bit of some side here about should AI or what should we think about before we put AI into research?
I have a couple, quick questions just out of curiosity for you there, though, 1st, before we delve too deep into that, when you work with the customer and you say, hey, they’ve got a better. Bigger customer base or at least some sort of active people and they’re using. So then you are you’re responsible for reaching out, incentivizing these users to actually spend some time with you jumping on a call or submitting some form or something like that.
Katya: So normally, actually, I recommend that the client is the one who initiates. That first contact. So I work with the teams, obviously I help them to develop the language for that email or that ask whether that be an app or, normally it’s an email because there is an existing relationship between the customer and and someone from the company at least they recognize the name because they don’t know who I am.
I jump in when when they say, yes, I will take a survey or yes, I am excited to do an interview. Then I reach out directly and introduce myself and take over the more logistical side of things, nailing down the time, making sure that they show up. If they don’t show up, I follow up we reschedule if needed.
And then I take it from there. But it’s it’s yeah. It’s always better for response rates, honestly it’s better that, that the client initiates that that ask, but I’m holding their hand through it in a good way to minimize their workload on that
Tyler Bryden: part. And then you and maybe it changes a little bit per customer, but you come in and, this actually has an impact I find on how you interact with technology, at AI, where um,
Are you coming in and the customers are giving you, for example, a preset of themes or things that they’re looking at learning more about? Maybe it’s why things are churning or feedback on specific product feature. Or is it maybe a combination of both, or is it a little bit more of an exploratory where we’re going to go into these interviews and question and interrogate.
And then at the end, we’re actually going to find what emerges. Or maybe it’s a combination of both. We just love to hear. You know how that sort of comes goes through it normally
Katya: is so it can be. Anything really, it really depends on what kind of insight the client is looking for so I never go into an interview without having a script prepared, which I write myself based on the project that we’re doing based on the scope based on what we need to find out.
If we’re looking to figure out, how the customer came to use the product, what, why they’re sticking around, why they’re happy to pay, why they’re. why their lives are so wonderful because the product is in their lives. I recommend using the jobs to be done approach. And this is my preferred interviewing way.
I think coming at it from the point of view of, understanding the struggle before product, the motivation to find that product and the desired, the better life that they get with the product leads to uncovering, first of all, some functional benefits. It’s unavoidable that features that they’ve come up in the conversation, but it also goes deeper than that.
It it gives you a bit of a more of an emotional angle that is useful at a later stage when you developing messaging, or if you’re looking to refine your positioning and understand where you actually stand. So that is one thing, but I’ve worked on projects where the client had specific things that they wanted to find out, and I would develop a script based on that.
And the questions, that would get at that. But every interview participant, if it’s interviews, obviously gets the same set of questions. So once, once we agree on the script I asked the same questions, whether sometimes in different ways, but I try to get at every single thing so that the data set I get after is , uniform in a way that they, that I get the most answers to the same questions to look at themes.
Which, which will crystallize
Tyler Bryden: later and it must be, just props to you because I feel like if I was going through that process, I would struggle to keep that script the same. You just learn so much throughout each interview where you start to want to iterate on those sort of responses and things.
Katya: Honestly iteration is actually part of the process for sure, because you learn. You learn where questions don’t make sense to people, because sometimes you phrase it a certain way that seems clear to you, but you but people just react to it, not the way you would expect. And you think, okay, how can I phrase it better?
So that they understand what I’m, what I am asking, but they don’t feel like I’m leading them to an answer because, leading questions are no. Yeah. So you definitely learn. And, if. If I compare the script that I start with, the way questions are phrased, the way they’re ordered, and then, because I rewrite it as I go along, and then if I compare that to the end result, so to speak, the essence of the question stays the same, we still get that information that we need, but they’re very different just because the participants do inform how I lead that conversation and sometimes, I get, Answers in different order, but that’s just, that’s my problem.
After I get to
Tyler Bryden: it, and maybe, that’s something I’d be really interested in. Maybe brings us closer to this discussion around sort of and research and the integration of that. But 1 of the experiences we’re seeing right now, so we work with. Both academic research teams and market research teams, and then we’ve seen product and user and all different types of interview.
Basically, focus groups, interviews, survey data, dump it into speak and try to run some analysis. And I would say we had used some more at the time. They were cutting edge techniques to run that analysis named entity recognition, sentiment analysis, all that. And quickly they were. Transformed into legacy techniques, almost in a way through large language models and open AI and anthropic and chat and all of these things.
But 1 of the things that I found super fascinating is. In the end, a researcher or research team often needs to bring their insights into some sort of template or framework to create a valuable end state. So you mentioned the jobs to be done framework. There’s some argument that these large language models have ingested lots of information about jobs to be done framework.
So they can understand it and help put at this high level. But I guess I’d love to hear your thoughts on that about Thank you. Manually synthesizing data and outputting it into some sort of valuable format that you deliver versus the idea of automating that with AI. And then I’d just be happy to hear any of your other thoughts on your stance, around AI and research and sort of your thoughts before you maybe use this kind of technology for the work that you’re doing.
I’m firmly in
Katya: the camp of manually. Uh, Synthesizing for now, because I am really wary of feeding the data. Into AI as it is right now, I have not done it yet because I do have a lot of concerns about what happens to that data, how that data is analyzed and fed into the models, what happens to it, which I don’t, I frankly don’t claim to understand what what happens behind the scenes.
But as it is right now I’m really aware of how sensitive the information that I hold is. And I can only justify using only my own brain power to process it, even though it’s probably much more limited than to what dbt can do, for example, so I mentioned to you before we started recording that I have not had.
An experience of feeding the actual data into, say, chat GPT and seeing what it comes up with. I have seen it try and distill jobs, isn’t jobs to be done statements from data, but I don’t, I cannot speak to its accuracy because I don’t have access to the raw data, so to speak.
So it is a complex thing and on the one hand, I really would love to be able to take that shortcut in a good way shortcut because manual processing is probably the most time and intensive part of research and it takes the most brainpower and it’s, for every hour of an interview that I do, I need three more to.
To make it at least a little bit readable for anyone, kind of processing that that afterwards but I’m not sure if AI in its current form is not, it’s not capable. It probably is capable because all it is looking for specific. Nuance in the language used in how, in, in how the person is describing their experience.
And that’s I’m fairly certain that, language models are capable of caching that and putting that in the format that I would need for the jobs to be done statement, for example. But I have other concerns about AI that just to me make it incompatible with the research process as it is right now.
On the other hand, I’m one person, I, my consultancy is just me. It’s I’m a solo consultant. So if I can make unilateral decisions about how I work I’m sure for some companies it is an advantage to work with someone who can say outright, okay, no way I. The data is very much contained to my computer to to this head that the, it will get deleted when when we’re done and then it’s, whatever you do with what you retain that’s up to you for other teams where.
Efficiency is important where budgets are much, much larger than say, even I am accustomed to working with. That’s a different question. And I would really love to hear your experience, about speaking customers, how they tackle that and what sort of challenges they have, because I’m sure they’re different
Tyler Bryden: than what I have.
Yeah, so many, almost too many interesting things that you said there. My head’s exploding. So I’m going to try to route this into something digestible. The one thing that so we come across one of the core concepts, I think, or concerns for anyone is definitely security. Where am I putting my data? What is that data?
What is being done with that data once it’s there? And I think there are, messages from some of these companies that, hey, we’re especially it’s. Especially through specific interaction. For example, if you’re using the open API, apparently, even Jack pro, their statement is around not retaining your data or using it for any other training.
And I think that in some ways you could trust that in other ways, I would be more skeptical about that statement or. Like any sort of cloud environment, as much as security has been done for it, there’s always risk of that data going somewhere. I think the 2nd biggest issue or question around it is often about bias and explain ability, which is okay.
I just put in a transcript into whatever chat or speak or whatever system that’s allowing you to. Put it into a large language model and then query that information. The question is, how did I arrive at that answer? How did this answer come? And, you’re putting yourself at risk if you are taking an answer that came from that engine, sharing that as an insight to a client or whatever.
And then when they ask you, how do you got that? You don’t have the answer for it. So I think there’s, that’s the other element to it. There’s a super interesting technical. Implementation I’ll talk about maybe at a later point. That’s fascinating. That allows you to understand what source material was pulled from that data.
And I think some at a surface level that satisfies some people, but I actually think , it doesn’t serve the actual desire there as much as it should. Or if you look at a deeper level, just out of curiosity. What you have these foundational principles that you think about security and privacy.
And that’s why you’re not uploading data. How much of you, what percentage of you wants to how much is that urge right now, as you’re seeing all this stuff happen?
Katya: So what you mentioned about not knowing how how GPT came up with the answer that actually is a big one that.
Ends up not taking so much off my plate that I can actually justify it, right? Because I need to figure out, okay, I can see what the machine told me. And I can think, okay, probably if I were the client, if I asked how that insight came to be. I, cause I could probably see it in the data and I’m the one who obtained the data.
So I would have some understanding of where that answer came from. So I’ll be like, okay, so that probably is because of. X or Y because in other interviews, we see this and this is the theme, but I’m essentially doing the work that I would normally be doing with even without GPT giving me that answer.
So part of me would like to check my assumptions or check my solutions. Like having chat GPT as an answer key, so to speak, to compare findings and see, okay, because the machine might be off base too, because I have some more nuanced understanding maybe about something the client shared in the discovery call that would not come up in interviews, but maybe they have some insight about how the specific feature was developed or how customers requested it in the past, for example.
So I do have some insight that. That the machine I keep calling it the machine but that it wouldn’t have so comparing notes. I think that would be exciting. And I also to the point of data security. I actually recognize that me not trusting open AI, for example. I actually, I, to a degree, I actually trust them.
If they say, okay, if you disable your chat history, we will not retain the data. I trust that actually, because I trust it as much as I trust any other privacy policy, any product that I’m using, because I’m, I’m on Google. I’m on LinkedIn. I’m on zoom. I’m on all the tools and they all claim to process data in a certain way.
And I, yeah. I implicitly trust those claims because I’m using the products because
Tyler Bryden: I can’t, we have to. Otherwise we wouldn’t be able to do anything .
Katya: Exactly. So part of me realizes that it’s a bit maybe hypocritical to, to hold open AI to higher standard than say Google, because Google has access to so much of my information that that they definitely use for advertising, in.
And changing my experience, which is I think this is just something that we accept as part of where we are and how we work in the 21st century. When 21st. Yeah.
I would actually but I would be really curious to, to see how that answer came to be part, like that you said, doesn’t go far enough, but I would be really curious to to see it because I do think that maybe it would even put me more at ease, even though I do think that there are additional concerns about, Okay.
Giving data to a language model where the data is not strictly mine to give.
Tyler Bryden: Yeah, because it’s not exactly. Yeah. And, there is an interesting trend that has happened though, which is open source releases of large language models that you can also mount locally on your computer without any interaction with the cloud environment.
Basically, that interaction is only happening at the local level on your computer, and that information is being sent nowhere. And so I think. To me, what sticks out is, and it’s crazy to say that, but meta with this system llama, which became one of the most popular large language model continues to be iterated on and basically can be run with a license locally on the computer.
And I hope I’m not getting that wrong. And there was also a company named Mistral. Who started a company last year, got about 113 million euros in funding starting the company three weeks earlier, but they just released their first model locally that you can interact with and download and is performing in some cases like outperforming chat GPT or like GPT 3.
5. So what’s interesting to me, the two things is that you haven’t, you could have an opportunity. If you locally mount and install those to test what the interaction layer is with a large language model locally without those concerns. So I’m wondering do researchers move to some sort of environment like that, where there literally is no connection, but the benefits of the models are so valuable or timing, and time enhancing or whatever value proposition there is there that it’s worth at least making that.
Jumping in the other one, just out of curiosity, I would be like, for example, we do demos and speak sometimes and we use like public focus groups that are on YouTube as a demo and as an example, you could take the transcript of that, plug it into chat GPT because that’s already on the Internet. You could interact with that and do some sort of sample analysis just to have that interaction without worrying about client and sensitivity there.
Anyway, those are just two things that I’m thinking of because, and I’m interested to hear your thoughts about because. Some of the, we talked to research teams, and there’s 2 pressures putting on put on them to try to adopt into their work. 1 of them is internal. Where they’re just seeing these huge advancements in and so they’re like, whoa, what’s happening here?
We need to use this. I’ll actually say there’s 3 a 2nd is it’s actually coming down from a top level where it’s executives at that company, or maybe that firm is owned by a private equity company or whatever. And so there’s a top level pressure, but then we’re now recently having. Research firms who are working and saying their clients are pushing them to deliver insights through some and maybe they’re just small sample trials, or they’re getting them to do a split test of manually synthesize insights versus but I thought that was super interesting.
And as a recent trend that we’ve just started to see. So that just wanted to share that with you, but I’m interested to. To me, almost in a way, the fact that you are staying on this manual analysis and some of the synthesis that you’re doing and using your experience is almost a differentiator because I think there is going to be a sub segment of this market who wants to work with people who are doing this work themselves and have built up that acumen.
And, for whatever very logical reasons don’t trust. The engines to do that. So I’m not sure if you have any thoughts on that. And is that a lane you’re picking or continuing to pick? Or do you see yourselves not relapsing? Because you haven’t done it before, but breaking down as you
Katya: move forward.
So much to unpack because you gave me like 3 things to talk
Tyler Bryden: about. Sorry, we’re doing it to each other. So I’m going to, I’m sorry.
Katya: Totally fine. First of all, the locally, I think the locally mounted. A model would be something if I ever use a I, that would be something that I’ll be using 100%.
And that is interesting to me, which brings me to the 2nd part, which is exactly how much more efficient or helpful. Would that be for me? Yeah, and I cannot answer that. It’s probably possibly would be helpful. And this is something like, I’m making a note because I do want to test some of the things out.
And depending on how cost prohibitive they are at the moment, whether I’m able to dip my toes into it and see how it works, because I would I would like to take a look. And the 3rd part is that how much does it matter to the clients? I’m not even talking about, the executives pushing the use of AI because it’s trendy, because I see that a lot for many reasons, not some of them are shallow, some of them are okay, maybe justified their demands on teams.
Tyler Bryden: But I’m losing my train of thought. That’s okay. We’ve gone deep here. You
Katya: mentioned that, manual processing being a differentiator. If I’m honest, that definitely crossed my mind. And if I’m even more honest, it is a pretty scary. Differentiator to put out there right now, because I in deep inside, I’m thinking, yes, I probably it would probably attract companies that I would be interested in working with, whether it’s companies who share my the same concerns I do about privacy or how data is processed, or for other reasons, because, to be honest, other reasons don’t come to mind.
But yeah I am considering coming out more strongly against AI, but at the same time, I don’t feel like I want to because there is a lot of potential there. And it’s, there is a lot of possibility to for it to be off assistance. I just don’t see a valid viable path towards it yet. And, the local ones is one thing.
The I guess trusting the data processing claims is another. So I’m not against it full stop but actually would be interesting to explore whether there would be clients who would want. Me to do just manually because I’m happy to just I’m a manual practitioner in the in that way.
And whether it is a bit hard to admit and being in tech and working with a lot of tech companies, but I’m a bit of a late adopter when it comes to a lot of things, they’re. Early adopters, early majority, I’m never one of those from iPads to from iPods, actually iPods, iPads, Apple
Tyler Bryden: Zoom, CD ROM player is what you’re saying.
I’m late to the party. You know what, I think there’s benefits to being late to the party. I think, we’re, there’s that idea of the cutting edge, and then there’s the idea of the bleeding edge, and I actually think there’s a lot of bleeding edge work going on right now. There was an example, and we’re dating ourselves.
But I think it was Microsoft even just yesterday. So they’re losing money for. People using basically their co pilot or their GitHub because the amount of usage and the cost and the compute costs and what people expect it to do for pennies is not comparable to like literally the fundamental needs on a electrical or computational level.
And I think we’re seeing that and, I guess I’ll touch into just something that I think you’ll, you’ll find fascinating. One of the challenges in, okay, so chat will just start there. A researcher will try to upload a transcript. What they very quickly realize is that if you try to upload that transcript you will run into a character limitation where I can’t digest that entire transcript.
And so then you are chunking those pieces of transcripts together and you can say, hey, I’ve got 5 pieces of text. I’m going to copy into you. And then that’s all 1 transcript. So treated as such. And then I can query from there. It’s that same problem that happens with a research team who, instead of 1 transcript has 50 transcripts from all the focus groups and interviews.
And then that, that’s solved by times 50. And then you you don’t want to be the 1. they’re sitting, copying and pasting into chat. So you could have the actual conversation. And so there was this concept I call it 1 shot. I don’t know if that’s actually what it’s called.
But say, you take that transcript, you upload it into a large language model, and you chunk it into those little pieces. And then that becomes the actual full document that it’s working off. But all these systems today, just as an example, their biggest model has, I think, a 16, 000 token. Limitation.
, which token is very abstract, but it’s a certain number of characters basically that it can process. A company called Philanthropic has a hundred thousand token model limit, which is a huge, that’s a huge thing, but it still has its own limitations, especially in focus groups or interviews where people are talking.
And I’m gonna try to shorten this down here, but what happens is if you’re trying to interact with one of those files, you’d be uploading it into a large language model and at some point that upload process could break and it’s not going to be able to contain all that information, or
it works, but you’ve got these chunks and then you’re querying the chunks. And so this process emerged. People call it rag retrieval, augmented generation. And so what happens there instead is you actually take that document. You upload it 1 time and you numerically represent that data.
And it changes the fundamental way that you’re interacting with the text, or the transcript or the focus group, because instead of re, uploading every time, you’re actually pulling from that database, which is a bunch of numbers to get the answer that you’re looking at. And how does that work? You ask a question and it’s trying to map the question that you ask.
To the information in that database, and then prioritize that information to fit it into that token limit to then spit out the answer. And so what’s really fascinating is the research or the prompting changes if you’re a researcher. If you can handle a short document, you can really ask whatever you want, because the system’s intelligent enough to do it.
But if you ask a question to this embedded model, or this reg model. It’s only going to prioritize information that is related to your question. And so questions that had previously worked on that might not work on this because it might not match. Properly and what this happens. I’ll just give you a more concrete example.
If you were interviewing a theme in a focus group and you say, hey, with this theme is about vaping and children. I don’t know why my mind went there, but let’s go. Let’s go there. And you said, if you’re a parent, what was your perception of vaping and children and you type that as the question into the large language model and you kept the moderators questions in those focus groups, then it would only it might only pull out.
The questions the moderator asked because it was a one to one match based on where you are. So there’s this really interesting component. This is where, to me, the bias or the challenge emerges is I’m now trusting this engine to prioritize the information that it will pull out to synthesize the analysis and that the only way it’s prioritizing that information is a semantic mapping based on the question that I asked in the information that it pulled out.
And I just think, holy, the amount of. Thank you. Abstraction we’ve done away from the original data source is super challenging. I think philosophically as a researcher who, values that original data source as the truth for presenting insight. So I don’t I know. I just went way too long on that.
I’m not sure if you have any sort of thoughts on that process or just that concept.
Katya: You just gave me another limitation to, right? Because, yeah, that that would make sense that challenge would come up because if you’re asking a question and the model is looking for the one one on one match.
Then of course it will prioritize the questions. Yeah, that’s the example is just fascinating to me because it’s honestly, like control F in a Google Doc, finding themes about, children vaping and be like here’s the question. Yes. How about the other 50 pages of the transcript where the answer actually is.
So yeah,
Tyler Bryden: And I’ll add, so we’ve tried to do another function on top of that, which is to remove the moderator from the focus groups from a high, let’s say, you’ve got 20 focus groups. Let’s remove the moderator. Now, again, that moderator needs to be properly labeled and all of that to make this happen.
But in that ideal world, you’ve now removed the moderator and, in some case, you. Might have removed out the context necessary for the engine to even understand what the response was because it needed to know the question to then know the response. And to me, what I’m thinking as almost any of these interactions that people are having with larger documents through these large language models.
There is. Yeah, just an abstraction away from the original data source and in both methods you’re going to run into that challenge. But then there’s also a bias of this engine trying to semantically map this, which looks at lower core level. It’s just a matrix basically for these. And we then need to understand how were these original numerical representations.
Created where those created accurately otherwise that prioritization of what information to pull out could be completely wrong too. And so I think these are the challenges that. Researchers are going to, need to come up. Those are the ones that we’re bumping into when we work with research teams today, or the questions, or when they say, they’ll ask to ask as an example, a research team or go in and they would say.
What did the respondents think of this concept, but instead of typing what the concept was, they would just type this concept. And now the engine truly has no understanding of what the concept is. You’re, you’re telling it to figure out the concept. Plus then summarize it. Whereas if they said, head, can you tell us what children think about.
Different flavors of vaping and the problems associated with it. Then all of a sudden, at least there’s some semantic understanding and it can start to do that matching framework. But I maybe have come on your side and supported too many sides on your one there, but I just, it’s so omnipresent right now.
And I can’t, at least in my current right, even with all the references and some of these things I’ve showed you figure out yeah. Exactly what the solution is. So I’m going to take a step back because I just put myself into a corner. I want to talk about 1 thing that maybe is interesting, or that I’d be interested to know is there’s also an evolution of that’s happened 1 of the original ones.
That 1st, got a lot of pushback in research, and then seems to be more slowly adopted is transcription and people don’t think about speech recognition as an AI that is at a core level. That was a huge series of, innovations that got us there. I’m wondering, you’ve talked about a little bit of hesitation around adoption of large language models, but I’m wondering where your mindset is around.
AI induced or AI or automatic transcription and how you use that in your workflow, or do you transcribe it yourself professionally, maybe a little bit of insight on that would be super fascinating.
Katya: So when I’m just going to share what I share with the client, when we talk about the specifics of the project and how it’s going to run, I bring up transcripts because obviously any interview, we will need a transcript because we, I with written text when I analyze.
And the first thing I say is that. The transcript, I ask you to source the transcribing service because it’s your data. If you want me to do it, I can also do it. As a source the service. And all I ask is that the service you choose, it’s not AI transcribed. Got it. Yeah. And but the reason for it is actually not even because of my concerns about AI and I mean that as well.
But the experience I had with AI generated transcripts. Which granted is outdated because I haven’t done it in a while. It was not as accurate as I needed it to be.
Tyler Bryden: What was that accuracy? What do you think when you were when did you last experience it? And what do you think the accuracy probably was when you
Katya: were doing it?
Last time was 2021. And that is like three, three decades ago. It’s
Tyler Bryden: 25 years.
Katya: And this is actually an interesting topic of discussion that actually really feeds into what you were talking about how the models interpret the input what you asked them to do whether the question is there, what context is around there is that specific project I was working with interviews that I did not run myself.
So for me, it was. Triple quadruple important that the transcript was as accurate as possible so that I could get as much context as I could. And when I got, and when I got the transcripts back. I read them and half the time I couldn’t understand, what terms were being used because it was highly technical also.
And there was a lot of terms used were understood by both both the interviewee and the interviewer. They weren’t really understood by me because I had very limited context at that time. So what I had to do I really, I still had to go and watch the video recordings. Of the interviews to understand what was going on and I ended up actually sitting down and correcting the transcripts myself, which was really labor intensive.
And it, it was honestly like pulling hair out for me because it was, I know it was wasting my time. Frankly, I felt like I was wasting my time and I definitely wasn’t spending the time on analysis. I was wasting the time when trying to understand why there was some sort of Yeah, Some sort of first name kept popping up and then it took me a while to understand that was actually an acronym that in context was pretty obviously understood, but not by the AI generated transcript and not by me until I looked it up.
That was a while ago. So I haven’t had a chance to actually see. What a transcripts look like right now. But right now, since I have a bit more understanding and a bit more knowledge around a I in general, I honestly I share the same concerns.
Tyler Bryden: Yeah, and I’m hesitant to say that since 2021, there’s been a couple of breakthroughs, but I still, we’ve actually embedded professional transcription into our offering at speak because there’s no other real option.
And we don’t foresee that still changing for the next. At least few years, because I think, like you said, there’s acronyms, there’s specific terminology. There’s just these, contextually relevant things in a conversation that are not understood by an engine and that engine needs to be whether it’s Microsoft’s or Google’s or whisper by open AI.
All of these have these fundamental limitations. I do think. We’ve heard one of the feedback is that editing an AI automated transcript is worse than just doing it from scratch. So I think that’s one
Katya: thing. For some reason, so much more frustrating. And I don’t, so I don’t transcribe myself because I’m a very slow typer.
So I don’t do that. So I I delegate that usually. But even then the service I’ve been using they actually added AI tools into their into their stack. So I’m not, I need to read their policy now because I know they offered AI trans transcripts for much lower fees than human transcribed audio and video.
But I don’t know how much more they will lean towards AI fully. But their differentiator has always been using actual people listening or watching the video and typing it up. So that is amusing. Also, there’s some amusing things that they hear there, but they are very easily understood because, they might mistype my name or they might mishear something that the interviewee says.
But again, very easily corrected because it makes sense. Yeah, my,
Tyler Bryden: My favorite one was we had one that was the automated AI transcripts said eagles testicles. And it was egotistical which was the original word. That was you. So you see some pretty outlandish things.
Sometimes I will say that there’s this interesting element that we’ve tried to figure out through this is if you can get someone to suffer through editing and AI. transcript, then the professional transcripts gets remerged with the software. And there’s a lot of advantages of that.
So I think we start to see more software enabled transcription than just going into a raw word doc. And then that starts to see, I’ve seen a lot of softwares that then whether it’s more like ones that have been around like a long time in vivo where I help you, then you can manually code through that or Atlas.
Some of these platforms are, and then now I’m starting to see even some of these technologies start to offer a I assisted coding which to me, coding had been such a sort of personal labor, intensive process. And I still, I question, and I’m not sure, about you, but like the accuracy of this automated coding, or does it go to the depths that are required?
And what’s fascinating is, even in these tools. They’re not really assigning the coding automatically. They’re more recommendations that then you formalize because I think what we’ve seen if we’ve ever tried to help automate coding completely that a researcher will look back and say, I haven’t done my work.
If I’m not going through and creating these codes by myself, I’m not sure if you have any sort of, thoughts on that.
Katya: I love how you called coding personal, because to me, research analysis is also personal. It’s, it’s not free of bias because the way I look at themes and the way I see analyze themes is it’s different If I, if we, you and I, for example, if we sat down looked at an excerpt from the transcript and try to distill a theme, we would come up with a slightly different interpretation.
Yeah, which is fine because that’s just how research, especially qualitative research works. So in, in that way, actually, I think of research analysis that part a little bit like coding because you’re, it’s technical and not technical at the same time. And the question is, how much can you trust the machine to, to be as accurate as you want it to be to the angle that you want to take.
Because with that with my 2021 example of having that AI transcript is that, if I just skimmed it I’ll, I would think, okay, that’s pretty okay. It’s, I can work with this, but then as you, as I stumble across some inaccuracies, and I have to go back and verify, and then I have to go back and verify, and then find that, that part in the video again, and see what it’s talking about.
I cannot trust the rest. If I verified the first page, I need to go through the rest of the other nine pages to make sure that I’m actually getting the transcript that I need to to to distill the insight. And to me, it goes back to, what’s the point of using AI is saving time and saving brain power.
How much time and power is it actually saving me? Because it’s also adding that frustration because I expect it to be helpful. I want it to work. I’m actually, I’m rooting for it to work in my favor, but then it’s, the output that it gives me is imperfect. And I have to go and fix it. So I’d rather, as you said, I’d rather do it myself from scratch.
At least to save myself that frustration and just get into that flow of, those eight hours of organizing whatever it is that I need to organize to to map up themes.
Tyler Bryden: Yeah, no, I feel like I’ve betrayed my side in this conversation here, but I’ll add just a couple quick points. I think of why we’ve seen we’ve seen demand, even with all those shortcomings that we both shared here together.
The demand does not seem to be going away. It’s like more and more at least experimentation and testing. I think 1 of them is. The speed of insights which in some cases we’ll see actually an interesting nuance here. An academic research team who’s maybe doing a thesis or not, or like a PhD or a funded project, they’ve got a 3 year timeline and they need their data.
They’re the ones doing the professional transcription, the manual coding and just, fully executing on that. Whereas we start to see maybe more in the private or in the enterprise. But they’re more interested in is if we were to take on this big process ourselves, we might get some really incredible insights at the end of 3 months.
But by then all that data is almost old because in 3 months, we’ve collected so much new relevant data. So how can we capture information in real time and produce insights in real time? Even if they’re only 80 percent will dive deeper if something gets flagged, but at least it’s. Keeping us maybe more on I or on the eyeball of things that are going really well or things that are going bad that then we need to intervene with deeper research it.
So I think that’s one of the other reasons why I can at least be a valuable asset. And I will say that when you get the data structured properly and the prompt structured properly, and there’s these great ways that you can now add context to these engines to say who you are and what you’re trying to achieve with this process, with this project, even before you write the prompt, you can get a response.
Pretty magical. And we’ve seen that live on calls with researchers with teams where they say, holy crap. And so for every frustrated person who’s hung up or maybe not hung up, but temporarily hung up the laces to wait for this thing to evolve. There’s been another 1 who’s whether it was luck or the right timing of the engine and all this got.
An incredible response, and that what, kept them going on this process and kept them trying to figure this out. And I think we’ll continue to see sort of case studies, both positive or good, where teams say their researchers say, hey, we adopted this. Here’s what we’ve achieved 80 percent reduction, 90 percent and for some reasons that will be good for maybe business outcomes, maybe for.
Honoring or thinking about the process of research and what can be delivered or uncovered. If you truly adhere to some of the more principles that have been rooted for so long. Maybe those might suffer. I actually don’t know the full outcome of that. I
Katya: think there’s a, there’s place for both and honestly, using AI to keep your.
I wouldn’t call it quick and dirty research, but
Tyler Bryden: yeah,
Katya: I like it better than 80 percent up to date and they’ll give us the broader picture or, the high level of what’s working, what’s not, I’m all for it and the way the, as you were describing it, I thought, you know what, it’s just I’m just doing different work.
I’m more in the camp of deep research, not three year.
Tyler Bryden: There are long hauls are going at you.
Katya: I don’t have that luxury, neither do my clients, but we execute brilliant research projects in six to eight weeks. Yeah. That, that are enough to move the needle for them, which aren’t AI powered, not yet, but it’s just, I think there are different things.
And I think there’s place for both. And I’m honestly, I’m rooting for a good setup for the former. I’m rooting for AI to work. Honestly, if AI were sentient, I would, and if I was the sentient AI, I would be terrified of how much pressure there is on me to deliver beyond what’s currently possible.
Tyler Bryden: Yeah, it’s funny. Some people will they’ll say, Hey, can you please help me with this? And other people will just bully you. You’re the sentient being and someone’s, being so nice and polite saying, Oh, you got this a little bit wrong. Can you give it me the other person? Hey, now give me I think I don’t think any of us, neither you and I are fully capable of writing, arriving on a conclusion today.
I just think the space is moving so much. People are learning so quickly. But. I guess if you were to have, if say we’re to formalize what we just talked about and do a couple takeaways, do you have any in mind that anyone who made it through this conversation might be able to walk away with from our conversation today?
Yeah,
Katya: I still feel like it’s okay to be cautious. And I think it’s always worth investigating what you’re using AI for and how it’s how how it is using what you’re giving it. I think it’s worth knowing at least you don’t have to, be like me and be like, no, that’s a no. And I’m actually not thought of that stance anymore.
Not really. I think it’s always worth investigating a bit deeper instead of just mindlessly going. Yeah. Here’s my everything. Let me tell you all about my research participants as well. I think that’s worth thinking about, but on the other hand, it’s always, it’s also worth. Trying to see if that new technology can work for you in a way that it’s a capable off at this moment.
Maybe tempering your expectations, but also keeping an open mind. I think I’m getting a bit preachy. No. I think there’s a lot more to know about it than most of us realize. Because I feel like we’re all sitting in our bubbles, like on LinkedIn or Twitter slash X. And, it’s fun to see your point of view echoed but there’s usually more to it. So you taught me a lot that I actually did, that I didn’t know and gave me some ideas how I could try and explore while still thinking that it’s not that I think that my way is better. It’s just, I think that my way is different and hopefully there will be place for both.
And demand for both. Whichever one is bigger, that’s, that remains to be seen.
Tyler Bryden: And and I think that was it when you spoke to me on a level of around, I think for very logical reasons why you haven’t necessarily done a bunch of testing. And even if you do a bunch of testing and things go wrong, I, please don’t blame me.
That’s I would appreciate that. But just that there are maybe, I think increasingly we’ll see this open source movement contribute and that there will be more private local installations of interacting with this that you can do without the concerns of sharing that data. So I’m fascinated by that.
I wanted to make sure that we’re, we’re closing out this conversation, but you’re doing something really interesting right now, which is the research power hour. I’d love to hear maybe a little bit more about that. I think, you’ve beyond just some of the points you shared about AI and technology, I think you’ve demonstrated, and it really, exciting acumen around the research work that you’ve been doing for a long time. So we’d love to hear a little bit more about that. And then any way that, if people were interested in this conversation or by you that they could get in contact with you. Yeah, sure.
Katya: Wanted to add to, I would never experiment with my client’s data, by the way, future clients, data safe, AI in any way, shape or form.
Unless we need to have that conversation if that ever happens. Yeah, because again, I feel strongly about the fact that the data I work with is not my data. Yes. Yeah. Full stop. And it’s not just my client’s data. It’s also my participants data. Ultimately, they also need to be aware that there’s a potential that, that their, what they share might be fed into AI.
And I think they have the right to know and to opt out. And frankly, I believe a lot of people would actually not participate. Yeah, if they were specifically told, and I think that this would be a fair thing to mention not to mention that in some future state, it will probably be required by law
Tyler Bryden: and I’ll just add 1 element to it.
I think we, we get these, all these blasts of, Hey, this call might be recorded for quality assurance and things like that. But I think we don’t truly understand the level of analysis that’s happening on the back end when you hear that message. So even the idea of a voice recording and that being transcribed and all that stuff definitely, I think we under assume what is capable of being done with that.
So
Katya: going back to, us, trusting all of the product Transcribed Products that we’re using for work or in personal and our personal lives, and they all have vast access to data that we give them and whatever they do with it. Yeah, but back to research power hour. Yep.
That’s a new service that I launched, which is a consulting call 60 minutes that is designed for probably non researchers. Marketers founders to get any sort of help they need with any aspect of research. They’re either running or want to run and I throw in product marketing as well, because I do have quite a bit of background in that and working with multiple companies, but it’s basically.
60 minutes of my time where we could actually hands like roll up our sleeves and work on something tangible, like designing your survey questions or rewriting your interview script, or I could coach you on how to run an interview, because, as much as it’s art and science it’s mostly something that you get good at with practice.
And it’s, it is a consulting call, but I like to say that it’s not 60 minutes of blah, blah, blah, because we can talk in theory, all we want. But for that, I don’t need to charge you. We can just get a virtual coffee. It is us getting together to do work that will get you unstuck. And it’s a middle ground between either being stuck and not and just not moving forward with research or having to hire me or.
Thank you. Or develop like a full blown research project. It’s okay, it might be closer to, not doing research at all. But basically, there’s some problems that you might have that don’t require a full on research project. But also, you can’t really maybe Chajipati can help with that for whatever reason, even if you tried.
I’m happy to be that human research person to put you on the right track. It is for a fee. But it’s mostly just to make sure that we don’t waste our time and that you show up and actually get value out of it.
Tyler Bryden: And I think consultants sometimes rightfully so get a bad rap for using a bunch of buzzwords and not getting that much done, so I like the sort of tactical approach that you’re bringing where it’s bring, let’s bring a task into this or something. And I feel like any sort of, even a fee, something you can save people a lot of time or pain or effort or money by optimizing some things before they dive into it.
So I think, yeah, this is
Katya: specifically for tactical things. If you need full on Yeah. Research design and, organizing a team around it and looking more long term. That could be a first step towards working together on a larger project, test the water, so to speak, or wrap your mind around what would be required for the larger project to succeed.
But at its core, I think about as that tactical thing where, I want to solve your problem that’s keeping you blocked this week. Not something that you’re facing Q1 2024. I want to solve the problem that you are having now that will help
Tyler Bryden: you down the line. Beautiful. And I’ll share that in below the the YouTube in the description.
Anything else before we close this out? Or will AI, so researchers still seem like we’ve got some time ahead of us here. I think.
Katya: Yeah, I agree. I think no matter how much I hate saying it, I think we will all be using AI in some capacity in future. Some of us already do with tasks that are not related to specifically to the data analysis.
And so do I. Little things. I use Chachaputika for little things. I’m not completely… Haha, she admits it. No, I do. I do think it’s pretty stupid, but that’s probably because my prompts aren’t up to snuff. Because again I use it for fun because why not? But I, again, I’m very cautious about what I ask
Tyler Bryden: off it.
I think we heard from a true researcher today. I think Cathy, I appreciate you spending some time with me. I don’t think we violently debated as much as as we can. I think there’s a lot of knowledge sharing and some perspectives that we both agree with here. So either way, I appreciate you doing this very much and thank you for everyone who tuned in and watch this or listen to this.
I hope to have some more great episodes to come and if there’s anyone you think that is worthwhile having a call who for some reason wants to talk to me, please feel encouraged to send them my way. I enjoy this very much. Thank you. Thanks, Tyler.
