Governing AI for Real

Transcript generated automatically from the episode audio and lightly cleaned up (filler words removed, misheard names corrected where clear). Speaker labels were added afterward and may contain errors.

Rob May: Hi everyone. I'm Rob May, the co-founder and CEO at Neurometric.

Ryan Eppley: And I'm Ryan Eppley, co-founder and CEO of RootAccess.

Anna Kirk: And I'm Anna Kirk, sales lead at Thread AI.

Rob May: And this is the AI in NYC show. The next stop is All right, so we'll kick things off again today with a note from our sponsor, BePresent. BePresent is an app started by two brothers who spent too much time on social media, and so they use the same techniques that social media uses to keep you hooked. They use those techniques to keep you off your phone. So, go check it out. We'll put a little, code up on the screen that you can use to sign up. But, check out BePresent and, you know, you'll be a better person for it.

Anna Kirk: Great. And today we have a very special guest who I'm really excited to introduce. Alayna Kennedy is director of AI governance at Mastercard. But is also very accomplished and I would say a prolific thought leader in her own right. So Alayna, do you want to give a quick introduction on yourself and maybe some of your past works you'd like to plug before we get into the meat of today's conversation?

Rob May: It's a nice way of saying Alayna's opinionated. Yeah. Right.

Anna Kirk: Yes. Well, she should be. Which is why we had her on the show because we love it. But it's like she's opinionated and there's substance behind it. Yeah, that's true.

Alayna Kennedy: Thank you. Yeah, it's great to be here with you guys. So yes, I'm director of AI governance at Mastercard. I'm on the AI governance team. And in my past I was an academic. So I have an academic background in science and technology studies. So some of my past work has been at the ACM backed conference fairness accountability and transparency in machine learning. Also in the AIES conference, AI ethics and society. These conferences all love a three to five letter like acronym. And so just love writing about some of the ways that you can really operationalize governance and ethics into an industry setting. I will caveat though that I am not here officially on behalf of Mastercard but here to just share my professional opinions on AI governance. But happy to chat about like what I've seen in space and I think the big takeaway for me in the governance world is like how do you make things real? When you take principles and turn them into practice, something like transparency, right, from a word on a page or a word on a web page. I don't know if you guys remember the like 2018 era where everyone was like, it's our principles document. We have like our our five-page principles and it's like we're being fair and transparent. It's like, okay, that's great. And I actually am not opposed. I think that's really great.

You need to get a lot of organizational alignment to even get something like that out there. But then actually how do you operationalize that into products into developing your pipelines into your data into like all of the things that actually make AI real much trickier problem. So that's a lot of what we do at Mastercard.

Rob May: Tell us a little bit about what was the academic work that you did. Was it related to governance and ethics and things like that?

Alayna Kennedy: Yeah, so I actually started my career at IBM. I actually did model development and like AI model development for several years for the US federal government. So, I was like with you know, it's funny because I like did my first podcast when I was at IBM and they like gave me media training and they're like you can't say what client you work for, so you should just say that you work for like a federal tax agency and they'll that'll be like enough.

But yeah, I worked for a federal tax agency building like fraud detection models and then ended up at the AI ethics board on on IBM. So started governance work there and so when I went to my do my master's degree after I was at IBM I was focused on like how do you put principles into practice actually my master's thesis was like a comprehensive qualitative and quantitative analysis of okay so you're a company who has this principles document on your website let me like go interview people on your like data science team and on your development team and on your governance team and say like anonymously like what's going on and get a real picture of okay how many companies are actually operationalizing all of this and what does that look like? And back then in like that was in like 2021, a lot of operationalization was still pretty new. A lot of people were still not really putting things in place and then of course you get like LLM a year later and it totally changes things in terms of safety.

Anna Kirk: So it sounds like there's kind of if I were to like distill down some of what you just said, it sounds like there's kind of like the algorithmic element of operationalizing ethics. There's the people management operationalizing ethics and then there's also like what you explicitly state as your principles and what you as a company or as an organization have decided is important and what you'd like to publicly present on that topic.

Alayna Kennedy: Totally.

Anna Kirk: Where do you think we are at just based on your kind of personal experience, your personal opinion in terms of like the evolution of that in this kind of new age of AI?

Alayna Kennedy: I think it's well so this is very science and technology studies right because the whole field is really about the combination of social sci like a rigorous social sciences research with the technological field so like a boundary object for example is like an STS term of like what is the actual thing that is translating between the technical yeah your model deployment your Python script whatever and the social which is the organization your infra your like your infrastructure how many bonus bonuses you get at the end of the year and what do you get a bonus for and what's your company culture right like what are the things that are tying those together I think for governance and AI and for well for LLM specifically it's I think a whole new world but I do think there's a lot of stuff like that is not new right a lot of stuff that is just about good risk management good product development good organizational principles and of course LLM are I think newly emergent in both their capability ities and their impact on the economy and there's a lot more like macro risk to that. But in terms of like product level stuff, it starts to look a lot more similar.

Like it's not changing any of the underlying principles. Yeah. Or I mean Yeah. It's not really changing the structure of like what do we care about? We like care about making things work well. We care about communicating how those things work to people. We care about being fair and making sure that everyone gets treated the same way. And how do we do that through like a fundamentally predictive technology changes a little bit between LLMs and also I think one big thing with LLMs is it sort of changes the deployer developer landscape quite a lot. So you have these companies that are really wellversed in doing predictive AI and analytics that are now just fully only buying Gen AI products. Right. So it's like you don't have you're you're not a foundation model company. Yeah. Right. Maybe you're downloading like a local copy of Mixtral or something and you're building stuff on top of it or you know you're doing local instances of cloud and you're building sort of on top of other things but there's this base layer that is not controlled by you and so the administration and the organization of all of that it just changes developer versus deployer I think how how that that looks.

Ryan Eppley: Yeah. I'm like kind of curious because you mentioned a lot of different technical pieces that you've been researching and you've kind of built a career around, but I have to imagine you're also working with people with maybe different humanities degrees, right? Like how does how does your team composition look like?

Alayna Kennedy: Totally. We have an incredible team. We have I think like half of our team has PhDs. So a lot of the work that we do you know specifically if you think about research stuff there's a lot of work we do with external academics but within our team we have a lot of people who are you know PhDs in chemistry and math and psychology and philosophy but it really is more of a handson you know again operational shop not a research shop but some of the work that we do with academ mics are like the paper that I got published in AIS. I was in Spain last year was on looking at fraud detection systems in Latin America, right? And so how do fraud detection systems start failing in unique ways when they're put into different socioeconomic and socioultural contexts? That's sort of like the STS piece of this, right? Where it's like partially a technical question because you have questions about data quality and data formats and things that are different like name structures are different in different parts of the world, right? And so that can your data just can be like this isn't a name but it's like oh it is a name because it's too long or too short or whatever.

But then there's these social issues of okay now we're seeing different things that we wouldn't expect. The actual source of truth is going to be different. How are we grounding that? So that's like an example of some of the research that we would do. So you can actually look up that paper. My what's it what's it called? It's a operationalizing critical data approaches in Latin American fraud detection.

Rob May: Ah rolls off the tongue. Yeah. So what do you we hear a lot when we talk about AI ethics, we hear a lot of people talk about some of these same examples over and over. The AI that manufactures paper clips that wants to destroy us, right, from from super intelligence.

Alayna Kennedy: That would be I would classify that as AI safety, not AI governance.

Rob May: Okay.

Alayna Kennedy: I actually think are pretty fundamentally different fields. I don't know if you want me that's break it down.

Rob May: What I Well, what I was going to ask you and you tell me is like what are the things related if you put all these under AI ethics, right? What we hear a lot about the same things over and over and over. What are the things people don't think about that they should be thinking more about? Like if there's a thing you're like, "Oh, your average person doesn't realize this is a concern for AI." Yeah. On the governance side or the ethics side or something like that.

Alayna Kennedy: So to answer your question, I'm going to do like the annoying academic thing where I actually have to like take a step back where AI ethics, AI governance, and AI safety I think are all very different things and fields. The thing I work on, so AI governance to me is really how do you govern risks?

How do you apply good controls? And how do you develop good controls within a setting where you're doing deployments? AI ethics is a little bit more the philosophical side of things of what are the right choices that need to be made what are the ethical what are the ethical considerations that is more to me academic civil society thought leadership great work coming out of there but to me it's not as applied and it's not really kind of an industry problem and then AI safety is the world of like Stuart Russell I think is the guy who did the paperclip clip problem right this idea of catastrophic risks from AI catastrophic macro level economy smashing like worldending type risks.

How do you think about those really macro level things? So I think those are all like slightly and importantly different. So to answer your question in governance specifically, I think some of the things that often people don't think about is really like just robustness, right? Like you just want your system to work well. You want to make sure that it's working appropriately. You want to make sure that you have good KPIs. And I actually think a huge risk that we're seeing now with the LLM era is an efficacy risk, right, of like AI has been so hyped and especially if you're starting to buy AI systems versus build them yourselves, you can get like what I've started calling like people get like charisma sniped by like some startup with like the really good pitch and they're like I'm going to deploy this in my system and it's going to like I'm just going to like put this into my data lakeink infrastructure and then it's going to solve all my problems.

It's going to like normalize all of my data and everything's going to be great and then like can I swear on this podcast? Yeah. Like everything up, right? It just like everything up. It doesn't work, right? So, I think there's a lot of risk to me that like it just like doesn't work well. Bad KPIs. You're not tracking KPIs. You have no idea what it's actually doing. You're not measuring right. There's no metrics and evaluation of the system. So, sort of just like pretty basic of does it work well? Does it work as intended? And are you tracking those success metrics?

Anna Kirk: You've lot of really interesting stuff to unpack there that I'm excited to unpack. You mentioned a couple times this element of fairness which maybe you would put in I guess kind of could fall either in the AI ethics bucket or in the AI governance bucket to some extent in terms of like how you actually measure and operationalize. Any like metrics around like demographic trends like bias and algorithms and how enterprises might think about that. How do you measure like I guess one one kind of more direct or explicit metric might be around like fraud detection and maybe like certain biases that an algorithm might have with regard to like demographic trends. When you think about like the unintended biases like as an example of that I was talking with some of my team members today around like like if you're an LLM and you're displaying leading information that might be misleading to a human interpreting that in terms of like the level of confidence or fidelity of the response like how do you capture the implicit or like maybe like second order effects of that bias?

Alayna Kennedy: Yeah, this is such a good question and I kind of speak to this a little bit more from like just an academic perspective and more on my personal opinion of when you talk about fairness in predictive systems, you're you're mostly talking about like categorization and like traditional machine learning that's making some type of decision or categorization task. And when you go into the world of LLMs, it becomes much less clear. And to me there's like a couple different things that matter for like, you know, the real core question here is are people being treated in a way that is fair, right? And fair is such a tricky word to unpack.

And sometimes you actually want things to like be like quote unquote like unfair, right? Like if you actually if you're a marketer for example or if you're like the Google ads people, they're like definitely unfair. They're like, I want to show men and women different things. Like, I want them to get different results in my ads because they're probably going to buy different things. But that's not like a negative impact, right? It's not like an impactful decision necessarily on someone's like like and then you see like things like the Facebook where Facebook was able to say we are not going to show ads for houses to like certain demographic groups to like black people and that that's illegal and they got in a lot of trouble for that. But like that's an impactful decision, right? You can make an argument for that. So I think there's like the element of is it an impactful decision?

Is there any recourse that is given to you, right? Like are you able to reverse the decision? And it's a really tricky problem. Honestly, I think even with traditional AI systems, it's not very solved. Like the bias efficacy tradeoffs are kind of I think a really interesting research topic where if you want a system to work well let's say like a traffic system or something you're like you have a traffic AI that makes everything work better for people and maybe it discriminates against blue cars a little bit like doesn't let blue cars go through but it's like okay a 1% reduction in accuracy of this system maybe causes like six people to die and then like blue cars get to go a little bit slower right those sorts of questions are are really tricky anyway, those are some just of my personal musings on on fairness is that there's a sort of a bunch of elements to it where is like what's the actual decision being made? What's the impact? Is there recourse?

And when with LLMs, yeah, it's just really tough because it becomes more like macro level things about disinformation and bias and toxicity, right? And even things like, you know, image generation of, you know, Robin a bikini.

Rob May: Yeah, we did that a couple episodes.

Alayna Kennedy: Make a nurse, right, or a doctor. It'll be like men versus women often. And that's not necessarily like impactful, but then you start thinking about how that like really snakes into other decisions being made by the LLM and can get kind of thorny.

Rob May: Yeah. Well, I just read the first economics textbook on AI came out, Microeconomics of Artificial Intelligence by Yeah. By Joshua Gans. And I got really excited and I got it and lots of math in there, but there was an interesting section at the end about bias and sort of like market solutions to bias and you know the economics of bias and all that. And he raised some really interesting points that like sometimes people just try to use blunt tools like well don't ask for race and then it can't creep in. And he shows a bunch of examples like it still can creep in positive and negative ways. And he's like, let's say you had two kids who had similar SAT scores, similar grades. One grew up in like ghetto southside Chicago, one grew up in like suburbs of Palazzo. He's like, you actually might want to let that first kid in of like, wow, that's really impressive that you could accomplish that in like a sort of chaotic environment versus somebody who had all the tools and nurturing at their disposal. And if you couldn't take zip code into account or whatever, like you might not you might not know that. So, yeah. So there there's a lot of challenges on on both sides of this stuff by sometimes the data can lead to bias one way or the other and I think it's difficult to resolve.

Alayna Kennedy: Yeah. I mean this is like what what's it called Goodhart’s law right like you start aiming for a metric and then the metric becomes the thing and that's a really tricky problem in all of society. So especially when you're starting to use things that are just data driven.

Ryan Eppley: Yeah agreed. So I think you tried to break it down for me but help me where does governance stop and ethics start because I'm trying to really understand like what questions are you able to tackle within the sphere of governance that doesn't creep into ethics because it feels so entwined.

Alayna Kennedy: Yeah, definitely. I think it really is a question of risk management versus philosophical questions, right? Where what are the risks that we're trying to manage based on these systems and then what are controls that we can put in place to make sure those risks are mitigated and with ethics there's really important discussions around you know what kind of should these systems be doing what are some of the bigger questions things like you know is AI conscious or not right is an AI system actually conscious and then what does that mean ethically if we find that the answer is yes to that that has a lot of implications to policy and to probably like how we would want to behave in the world. Or you know things like the cyborg manifesto, right? Like at what point do we become human versus computer? And then again, what does that mean for our humanity and our consciousness and our worth? These are all really interesting and cool questions that are hard to operationalize.

And so governance, I think, is more about okay, how can I measure risk in a system and then mitigate that risk through controls, right? And to the extent that we can do that, that becomes governable. And governance I think is different difficult because you talk about like governance in the government sense. So people are using words that start with govern a lot to mean multiple different things. But that's just kind of my perspective on it.

Anna Kirk: Okay. Super helpful. And I'm curious. So when you think about like and again this is me asking you this is like Alayna Kennedy friend, not Alayna Kennedy representative of Mastercard. Obviously I got to throw that disclaimer in there. But when you think about managing that risk, like the one we talked about how like yeah, there are a lot of common threads and policies that have really been best practice for decades. But the one like novelty here is that for maybe the first time ever, you have people who don't know how to code who are building software.

Alayna Kennedy: Totally.

Anna Kirk: And there may have been elements of that. There may have been precursors to that in like past iterations of this cycle, but like how do you think about that in an enterprise setting where everyone can vibe code everyone probably has a personal OpenAI account account account that they're not disclosing? Like how do you think about quantify quantifying and putting guard rails around that risk at work?

Alayna Kennedy: Yeah, I mean this is like what I was talking about with efficacy. If you want to make anything into like a product that's going to be actually touching customers, you need to know does it work? And you need to measure that. You need to measure and test it and show that it does work. So you have clear success metrics and you can prove that those success metrics are met. It doesn't really matter if you viodated it or not at that point, but it's like okay, does it, you know, you know, for example, I want to make a chatbot is going to save me so much time and all my customers are going to love it. Like do they does it save time?

Like and it's annoying but like actually then the task becomes it's sort of an offloading of work because the task then becomes like and I actually think this is really good like you vibe code something you save all the time it took to make that software development and then now you actually have to set up a good system of like okay you need to find some way to get data about how this is working in the world right and I think a lot of software developers were just like ship this ship this ship this and I was like okay you developed it in a week like go find some people to either talk to get qualitative data back or set up a quantitative data gathering scheme for it works. I can tell you it works. I have metrics. I've thought of good metrics. And so I think that's much more the question of is it doing the thing that you want it to do? And is and is the thing that you want it to do actually a problem that needed to be solved in the first place.

Anna Kirk: Yeah. And then like how do you track that? And I imagine like some of that is related to like governance and policy, but some of that is related to like engineering and like can you embed like the metrics tracking within the actual workflow that you're building. And there's a visual that I'm sure we can like flash up on the screen that I think about a lot in my work which is like it's an iceberg and it's like the top of the iceberg is like the demo or like the thing that you see and then underneath that is like all these like off controls, all the auditability, all of the like traceability of different like logs and like who's accessing them and what it's doing and like how you kind of benchmark those KPIs. Yeah. So it doesn't seem like there's a super great answer.

Alayna Kennedy: Totally. And I think like a lot of a lot of stuff is just like so complex. Like a lot of business problems are really really complex. And so it's actually a really great opportunity to figure out how do I solve the actual problem, right? And then actually figure out that I'm solving that problem. Yeah. Get data on it, do some actual data gathering, and then move from there. Right. And if you actually just shorten your development time, that's great. Then maybe you can spend more time figuring out that it actually works. It's doing the thing that you're you want it to do.

Ryan Eppley: Yeah, like starting with the business problem and like the business process. I would what would be your recommendation to probably a bunch of startup founders are are following the podcast and things like that about setting up their first efficacy kind of benchmarking like what are there some like tips that you'd want to share?

Alayna Kennedy: Yeah, actually so we have I can share this about our Mastercard program. We have a three-stage process. We do risk assessment, controls, and monitoring. So what are the risks of this product? And there's a million great risk assessment. Actually, I think this is also a good chronology of like the maturity of the AI governance space because there's tons of great risk assessment tools. There's some good risk assessment tools for startups. There's good risk assessment tools like the Canada AI impact assessment, right?

There's there was one from the US technology office that I think has been taken down but might still be up. But like there's tons of these, right? What are AI risks that you need to look at, right? So you have to look at the context of your specific product and figure out okay like if I am making a product that is directing directly interacting with the consumer that might have some risks like some different risks than I'm building just like an autom backend automation type thing. And again, we could kind of go through a taxonomy of those, but there's like a million things on Google. There usually like taxonomy of AI risks, AI risk assessment setup, right? So, it's like what in your specific thing is risky?

And you can kind of think, right, if I'm like making a, you know, direct to consumer chatbot that's going to help talk to teenagers about their mental health that you can already think about some risks from that versus I'm actually going to build a way better way to use ChatGPT with Excel spreadsheets and like scale that on a B2B enterprise level. Different sets of risk for those two things and then think about okay what are controls for this. So there's controls like we talked about fairness, bias testing, you know, tabular data to figure out if you have any type of disparate impact. You can do good, there's great LLM evals that are out there, right? Like in terms of toxicity, bias, hallucination, all that stuff. Like there's a lot of other stuff you can do, but you kind of have to map it out. You're like, okay, my risk here is that it I could be like telling a teenager something wrong or something harmful. Like I should be doing bias and toxicity tests on my LLM, tons of open source ones, and getting those results. And then you like monitor where it's like okay every couple of months or based on how risky I think this is every couple of weeks checking some of those things again as it goes out into the world.

Ryan Eppley: Yeah. It kind of starts like creeping into we had a guest on here that is more in like the insurance space not too long ago talking about like the future is coming quick where you're going to have a company have to get their agent like insured.

Alayna Kennedy: I'm so interested in AI insurance stuff.

Ryan Eppley: Like this is right on the cusp of kind of what you're saying. Like you know we already have these assessments set up like are companies going to have to go through this almost like it's a standard sock 2 kind of protocol

Alayna Kennedy: now yeah sock 2 seems to be the gold standard I think in a lot of discussions around AI governance and AI standards as like the new thing that people really care about and people really care about this and especially in an enterprise setting saying like how are things how do we set things up one of the big things you know that I think people think about is like how do I know I can trust this system and that's true for consumer level, right? People really want to think that they can trust a system that they're interacting with in some way. And then also equally so on an enterprise level, enterprises want to say like, how do I how do I know this is going to work? How do I trust this?

Rob May: You know, you know what's interesting about that point is like do you think like when you think about insurance for humans, it's hard to predict what any given human is going to do. We can predict human behavior in aggregate. Will it be the same or different for agents? It's like are the agents all going to be driven by similar models where they're all going to follow like here's the probability distribution of the output of a given agent and it's the same as the probability of output for the distribution of agents.

Ryan Eppley: Super challenging because people change the underpinnings of their agent all the time even as simple as changing what model you're you're calling.

Anna Kirk: You know, I'm going to get super nerdy for a second, but I feel like there's a really cool way to like securitize this of like how can you like arb out the difference of like, oh, I think like net net like Gemini will actually hallucinate less than this insurance product or like this certain pricing it in. Yeah, you hear first.

Ryan Eppley: Yes. Should we move into some of the articles?

Anna Kirk: Let's do it. We got some good ones this week. See us up, Rob.

Rob May: So, so, so our first article for this week is one that it's a post that Citrini Research put out that sent the markets, the financial markets, you know, yesterday, the day before we're shooting this this podcast, into a tail spin because they took the position of let's say we're right about AI, but the outcome is really bad. Let's say we should be bullish, but the outcome of this bullishness is bad. And they wrote this thing from the perspective of sort of late 2028 investment memo that said, "Wow, you know, like at first we thought it was going to be great and then this happened and now unemployment is another 10% and all these bad things happen as a result of AI."

And I think it's really hard because I don't know how you guys feel like I am technical. I've worked in the AI space for 10 years and I find it hard to keep up with everything that's going on even at like one layer of the tech stack, right? And so I cannot imagine being a non-technical person trying to allocate capital against this. It's got to be crazy. And so I think people, you know, really had a big problem with it yesterday and it was a big riskoff day in the market. But like what did you guys think about that sort of point of view of the world or that article if you read it?

Anna Kirk: Well, you guys are looking Tell me what you think. I'll take either side. I can argue either side. So, I'll I'll take the I think like okay to your point which is like I couldn't imagine being a non-technical person trying to allocate capital into this but like I almost think that like in some respects that article although a really interesting like thought experiment scenario was underestimating like the long tale of imagine how many non-technical people are out there like I was on the phone with my dad last night for 45 minutes trying to place a single online order like I think they're maybe underestimating the behavioral momentum to how quickly do we actually get this adopted. And I think a big part of that article was around like you are now reducing the friction to engagement in a way that means that like there's no consumer loyalty to Uber Eats anymore.

There's no consumer loyalty to DoorDash because the agents can just go off and like price ARB the entire thing. And so like in removing that friction, you're actually removing a lot of like value added services from the labor economy. I think that there's still a lot of that friction left. Like I actually think we have a long way to go in terms of like education having this like technology become much more universal. What are your guys' thoughts?

Ryan Eppley: One I thought it was a cool article mostly because it it was a different dichotomy. Like a lot of people have said like, "Oh, it's either a bubble or it's not." So, it's been like such a nearshore kind of idea of like the market response to AI, but this was much more like societal impact going back to AI ethics in a lot of ways. I actually thought it it was it was on to something and I gleaned from it that there's probably going to be a need for us to really understand like how does the free market put guard rails up against technology because it's how to do this with telecommunications, right? Like you got to like, you know, do a lot of work to get rights to a band, right? Like telecommunications was the wild wild west for a long time and then they put a ton of

Rob May: have a auction for

Ryan Eppley: Exactly. So, I think that's going to happen and that took a long time because there was only a handful of players that could really operate in the telecommunications. It's not so dissimilar to the frontier labs, but I think the deployment is a little different. So, the government's governance of it might be a little different. But I do think that this article is assuming that that that guard rail won't be put up in some type of lzair. It's a total free market and this is the response of the free market. There's going to be some type of regulations on on how deployment works probably by before this article really comes out.

Anna Kirk: Oh, you think so?

Ryan Eppley: There there has to be. There's just no way people are going to be wild wild westing the whole time. Otherwise, telecommunications would have ended up very different. And we kind of know the story that that got tamed pretty pretty fast when things got off the rails. I mean, for God's sake, they even broke up AT&T, right? Like that was a huge moment where they're like, "Oh, yeah. Like, oh, we don't like that. Will like open AAI get too young to remember that. Hey, I was I was 19 I was two years old when that merger took place. But you know the point is like will we see before this article in 2028 you know if OpenAI and Anthropic public later this year like there's been like you know dabblings of that potential will like they get broken up even before this article's timeline. I don't know. It's a weird thing to think about for a company to IPO and then less than 18 months later to get broken up, but based off of the pace, you know, I couldn't I could imagine that being a case that people bring up make a lot of sense.

Anna Kirk: Yeah, I kind of feel like and like this is one part of the article I did really resonate with around this like this like massive sectoral shift around like all of the sudden you have all of these white collar jobs which made up the majority of the US economy now getting put out of work because you have like 180k product manager salary that can now be done by an LLM in like for like $200 an hour. Mhm. And one of the points that it made kind of at the end was that like the trickle or like the trickle down effects of that are your like the way in which we underwrote individuals income, individuals ability to pay individuals creditworthiness is now like fundamentally shifted and now the economy isn't ready to adapt for that in terms of like how we're under underwriting different credit scores like consumers abilities to Hey, like how we're lending against like mortgages. And so I think that like I think from that standpoint it was really interesting to hear like yeah like this is a good problem but also like there like you are it's not that you are not like underwriting consumers like creditworthiness in good faith. It's that you're underwriting it in good faith and everything is shifting underneath our feet.

Rob May: Yeah. I would say I mean I think there's two points. One is some people would argue massive productivity like that if it happens is going to be economically deflationary. It's not a type of environment we have a lot of experience with. And like maybe slightly Japan in the 90s and early 2000s, but even then it wasn't massive deflate massively deflationary. And we don't know exactly what that be. I'm not convinced that deflation is going to happen at that scale, but if it did, then you know, somebody who makes $130,000 a year after five years of that could make $35,000 a year and have the same purchasing power that they had before. So that could change things. But I kind of agree with the point you were making earlier, which is like say intelligence was like a car and I come in and I'm like I can make a car that goes 800 miles an hour.

Mhm. And just uses regular gas and gets 200 miles to the gallon. People be like, "Oh my god." And the first thing he would do is like, "Well, we don't need pl short airports, short airlines, short Amtrak, like 800 miles an hour, you'll just drive to California in 3 hours or 5 hours or whatever." Like, like no, no, no. Because like you can't take it on the roads. The system wasn't built to accommodate it yet. So like you still like I mean think how fast you could you could drive from the you know from from Harlem to Fi in 45 seconds, but like on what street? There's a lot of underlying assumptions that have to challenge. You have to configure society and the economic channels and supply chains and just like all these things around these capabilities. So I think even as these capabilities grow I think they're going to put pressure up against those things but like integrating them and p pushing them through is going to be it's going to be different.

Ryan Eppley: So we have a guesta has a ton of governance expertise. I know this is a new long article that you still have to chew on and sit on, but like when you start hearing the implications of what this article is implying, you know, how do you how do you think like governance will be at least attempted?

Alayna Kennedy: Yeah. So, I think this is again kind of a macro problem, right? That is very much more economic, I think, and macro level in its scale. A lot of what we do on the governance team, we look at products, right? We look we look at products, we make sure those products work, we make sure those products work well. So, it's a little beyond the scope of what I can speak to just from a governance perspective. But I can say I mean I think about this a lot personally and I do think that you know I don't know if you guys remember AI 2027 which do you guys do you guys read that came out like this time last year's book? No, it wasn't. It was the prediction article from like Slate Star Codex and a bunch of like effective altruist thinkers.

Rob May: Yeah, I remember it talking about but I forget what the title was.

Alayna Kennedy: It was it's already wrong. Like it's already like it's like and I think it's so cool to be doing long-term predictions like that, but I think it's really hard in this environment. I do think that all my p my prediction if I had to make one is that what ultimately comes down to AI is that it's much more of a productivity like let's say 8 to 10% productivity increase technology across the board which is huge and again probably causes deflationary pressures across the economy but not in a way that's like we're all dead and we go to the moon you know and like yeah not in a way like we don't spend because your money is just worth so much more tomorrow if you wait like it's

Yeah definitely and I think I think more concerning ing to me is this idea that there's a, you know, knowledge gap emerging in the economy between lower like younger workers, entry- level jobs and then sort of people who are higher level and that consumer spending is already just rapidly shrinking to like the upper 5% of the income band and that that is not seeming to reverse anytime soon. So if you continue to consolidate the gains for a lot of product, right? You have these huge productivity gains that don't get distributed beyond the like very highest, you know, classes, then you actually like really undercut consumer spending, which this article talked about a lot and I was really glad it did because I often don't see that of that's a huge driver for our economy. People spend money, people go places, people are doing things. And if you start creating a world where it's like everyone's losing their job and I probably gonna have to drive Uber next year and make $30,000, you're not gonna spend any more money.

Rob May: And I drive the Uber, there's no Uber jobs.

Alayna Kennedy: Yeah. So, Exactly. So, so, so I think that actually is a real concern of like what is the actual, you know, emotional tenor of the spending economy? And I think this applies to the housing market too. But I will say I am not an economist. I think you guys are all much more economically versed than I am. In terms of actual governance to try and take that perspective of it. I think that again this involves like how do you make what are the actual capabilities and do these work well for people right do they work well on an enterprise level do they work well for consumers there's actually like few research that's come out recently about how a lot of people just don't really like AI a lot of people don't like AI in their products a lot of people don't like interacting with an AI when they should they want to talk to a person and this actually causes like sort of attacks on products even if you are like decreasing your actual labor costs by implementing AI systems

So I think really strong use cases in like software engineer white collar jobs automation of some level of productivity but I don't think that totally wipes these jobs off the market like I was talking about in an example earlier about okay I can vibe code a software in a week now and now I can spend now I have to figure out like how does does it work well is it okay to use can I actually get better outcome data how do I figure that out and you can use AI to help you with that as well but now your task becomes kind of different so all that to say is that I think it's probably closer to reality than a lot of the more more apocalyptic takes. But I do think I do think that the idea of having just like 8 to 10% productivity growth probably in terms of some of the valuation of these AI companies is actually not going to be good news because they're trying to do like a lot more a lot more ambitious work. So, I'm not going to make any predictions, but I'm going to just leave you all with some of those some of those thoughts.

And again, really macro and I think this actually points to the difference too pretty cleanly of like AI ethics and academic level, you know, science and technology studies of like what is this? What are our policies? What are we thinking about high level versus like governance risk assessment on a product level?

Anna Kirk: I think the like cyclicality element that it kept going back to around like what are the macro like trickle down effects of this. I don't I don't know about you guys. I don't see a world in which regulation acts as a backs stop to whatever effects like trickle down from like these productivity losses or like cons like deflationary trends or whatever. But like I do see this just being kind of like a aftermath like after the fact you like enact some regulation that maybe kind of post 2008 is more retroactive but I don't see regulation like being a proactive measure in this case because like who the hell can predict it?

Rob May: Yeah, you have to be so I'm very interested in systems design, right? Because I'm engineering major and as an electrical engineer, I started working on chips and then moved up to boards and systems like that. And so one of the things you think about a lot of times with systems design and I use this in the businesses that I've run is you talked about good law earlier, right? Which is when when a measure becomes a target, it ceases to be a good measure because people game it. So the way that you get around that is you try to design a system where if you try to game this one, you have a competing metric that kind of pulls against it, right? So to give you a simple business example, if you just if you just incentivize sales people on the amount of business they close, they'll close bad deals, people that might cancel, people that are unhappy, whatever, like you they don't care because they just want that in their next pay paycheck. So if you also comp them on churn, well, now they're like, well crap, if I close bad deals and my churn goes up and I get that taken away and you know, whatever.

So, so, so you have metrics that pull against each other. And so I think the way that you have to do regulation in this sense, which I would not expect our politicians to do because they're not engineers or scientists, they're mostly lawyers and don't think this way. But like I think you have to design these kinds of systems that are sort of like, oh, as things grow faster in this way, these things pull it like we put things in place to pull again. So, you know, you could look at something like, well, we take the unemployment measure every quarter and if it's going up too fast, this thing happens to try to slow it down or be a breaking effect, but you see this in a lot of natural systems, right? And that's these sort of positive and negative feedback loops. And I think it's not normally how we design laws in most cases. Maybe the Federal Reserve sort of operates that way, but that's but there's always like a secondary effect.

Ryan Eppley: So there's a great graph about how bookkeepers got absolutely decimated when Excel came out, but CPAs just grew crazy, right? And there was just so much new labor for a different use of time in the very similar industry. So one would say like, oh, like are bookkeepers losing their job? I think a lot of them found a way to get upskilled. I don't know if it's like to like in this scenario, but there is something to be said about there's precedence in technology completely rechanging how industries work. Is it too rapid though? Because that took years to roll out and it took, you know, well well over a decade to see the crazy curve. We're seeing that in and sometimes it feels like weeks and months rather than quarters and years. So I don't know if that's going to be too fast of a change and then there's going to be these guard rails that have to be put in place. But you're right, incentives drive almost all all outcomes.

Alayna Kennedy: Yeah, I have two great book recommendations about this. One is you may have heard of, it's called Power and Progress by Daron Acemoglu. It's about the ways that, again, this is all this is great. This is a very science and technology studies conversation about when you put technological progress in play in the economy, you also have to think about, you know, policy, social services, and infrastructure for people. So power and progress talks about different examples of you know the railroad being invented the industrial revolution you know how technology gets invented when the railroad gets deployed in the US like long distance horse and cart drivers are like go completely out of business but for like 10 and 20 20 years actually all of those people start now becoming short distance because the train depot is 2 miles out of town and all of those people with their Horton carts take everything from the train depot into town and start delivering and then you actually have this huge urban growth because of this and then this is before the automobile obviously but like you actually have a redistribution of people's both assets like the hort and the cars and their skills to fit this new technological system.

And then the other book I'd recommend is called Blood in the Machine, which is about the Luddite Revolution in the 1810s in England when the automated looms were invented that kicked off the industrial revolution. And it's about the ways that technology was supported by people, but that the things that a lot of the Luddites were arguing for were actually just, hey, can we get like better pay?

Like, can you not put like orphans in the mills and chop their hands off and like pollute the rivers and all of this stuff? And like it's actually like you need the automated loom, but then you also need there's like a civil reformer in England in the early 1800s who I don't remember his name, but he like he comes in and he's like actually we need to make sure that you can't be polluting the rivers. You can't be using orphans in your mills. Like actually this is really bad for society in a bunch of other ways. We have a bunch of secondary effects. Like there's a bunch of like 28-year-olds with no hands because you put them in like the orphan mill and now they got their hands cut off and now they can't do anything. And it's like this is terrible. Like this is not how we want to run our society.

And so you need the automated loom, but then you also need all of the stuff that's like we need cleaner water, we need better health policy, we need to like set up labor laws, right? And then you actually create this world where it's like now we have this huge economic boom in England for many many years. And so blood in the machine is kind of about that process and like a really detailed history of like the lite revolution in a way that's very interesting. So these two histo there's like a lot of historical parallels to this I think that often I draw with

Ryan Eppley: so are they going to come out with a you know upgraded version of blood in like the model blood in the model blood in the model

Alayna Kennedy: like we need a new Ned Ludd we need to be smashing data centers that is under like there are so many historical precedents in this

Anna Kirk: totally like assuming that and I guess like that sectoral shift is assuming that like you can re-educate a portion of the workforce who may have like lost their jobs in a like quick enough way to account for that productivity loss.

Alayna Kennedy: Yeah. And also are you like how much vert vertical integration are you doing? Yeah. Right. Like and I think one of the dangers we see now is like there's so much vertical integration in the economy where like in the railroad example you're like I care about the railroad. I got the railroad and then once I get to town all the horse and cart drivers can have all the stuff that's great for me. I don't care. Right? Not I own the railroad and I own the horse and cart drivers and I own the store, right? Like and I'm making the chips, right? Like Yeah. Exactly. So I mean of course you saw some of that in that era as well, right? That's obviously something that has been again this tension of like vertical integration is really good if you can if you got it and it's not so good for the whole macro economy. So it's this like you know tension of policy.

Rob May: There there have got to be some AI leites here in New York that we could have on the show.

Alayna Kennedy: Oh there's a lot.

Rob May: Yeah, let's find we're going to Madi. Find somebody else.

Alayna Kennedy: There's a there's like a yearly this this other podcast that I like to listen to. There's like a yearly AI Let meetup that I really wanted to go to this year, but I was out of town. So, they mail letters to all the attendees and then they come in and no one gets them and so no one shows up.

Ryan Eppley: Well, you've been having the I guess like you call it a prediction that there's going to be a politician that runs on like an anti- AI policy, right? And like at some point I think that's going to be like an agenda. Yeah, there's enough people that are against it or concerned. I mean, if this article gets even close to you know the job I guess like you know situation as they predict like you know as Bill Clinton says the it's the economy stupid right? Everyone has a job and everyone's identity is usually tied to that job. That's how it's been for a long long time now. If people start losing their jobs like there will be a lot of pressure on politicians to figure out how to rein that in. And to your point, a lot of the people that own shares in these companies are already wealthy people and so their spending is going to continue to increase and there is a lot of dynamics on on course. But all that to say, it's an exciting time to figure it out.

Rob May: I think it's I think culturally we're different now and I think it's going to make it hard, right? If you if you think about I don't know if people who go to business school today study sort of the leadership groups that I studied 25 years ago, but like we read a lot of Peter Drucker and Peter Drucker had this point of view. He was probably the most influential management theorist at that point in time. And he believed that like as a business like look to have customers you have to have a relatively fair and just society, right? And like yes, people should be able to make profits and do whatever, but like if you go polluting the river and then you poison people and then they die and they're not customers. So it's like in your best interest to try to be as a business be a good citizen. I don't think we feel that way anymore, right? I think u I don't think we have leaders who value like wisdom and patience and all those kinds of things. I think we feel like, you know, if you if you look at the people at the very top that you Elon and Bezos and all, they just want to compete against each other and take and do and own everything and like I just don't think it's good. And I so I think it's harder to craft solutions maybe today.

Ryan Eppley: To your point like that happened all throughout different you know ages if you will like the monopolies of JP Morgan.

Alayna Kennedy: Yeah. Weren't they like they didn't weren't didn't they like kill like a hundred miners in Colorado or something? I don't not to be too controversial, but anyway, you're probably you're right. Like this has been happening forever.

Ryan Eppley: I mean, it's a phase. It'll it'll swing back. It's this is what happens.

Anna Kirk: I mean, I'm curious. Last question before maybe we wrap it up for the New York question of the week. What do you think is a counterbalance to that? Like what is a trigger event? Is it cultural backlash and like this AI lite movement? Like what what is the natural kind of like force that swings the pendulum back in the other direction?

Ryan Eppley: It's got to it's got to be the jobs. It's got to be the jobs. It's always tied back to the jobs. I'm thinking of that one like South Park clip where not to make light of a serious issue, but the one that's like they took our jobs. Does anyone know what of course we'll find that? Okay. Thank God. Hell yeah. Yeah. Right there. Drop it there. Oh, they took our jib. They took your jab. They took your jib. Damn. But no, it's going to be people are really feeling it and it's probably what we saw with like the 1% and like Occupy Wall Street but on a whole new level and it's going to be like occupy like these frontier lab situations that would have to happen.

Alayna Kennedy: Yeah. I also think AI systems need to like work well in the way that they have promised, right? And I think that there is still quite a lot of work to be done in the Gen AI product space. I think there's a lot of really strong use cases around like genai coding specifically that a lot of people are you know in the article there's a lot of examples of coding being you know one of the jobs that gets transformed but in a lot of other use cases I think there needs to actually be a follow through of this is good business this is actually making us money this is actually increasing our productivity by a measurable amount right and this actually makes sense for for our business and I think that there actually is still some work to be done in a lot of use cases that are like non-coding for that and be curious to see how that unfolds throughout the year.

Rob May: Yeah, I think people I think generational change will take care of some of this because I think people in a generation like teenagers and people in their early 20s always think whatever the people above them do is stupid and they want to revolt against it. Doesn't matter what it is. And this is why you've seen like you see it most commonly with like like music and sexual behavior and stuff like that where you know it it cycles through and you do the opposite of what the group above you did in terms of what you like. And I've seen this in my, you know, I have two teenagers and I've seen in them this idea that they're sort of like they don't doom scroll and they don't stay on social media as much because they're kind of like there's too much of it and they come up with it and like to them I mean you've seen you've seen cameras come back like Polaroid cameras and real things, right? Like they they they they're sort of like we have so much of this this being out in the world and being off our phone is like novel and cool and the way that I think that translates over is I think it'll rebuild some of those social bonds that we've lost.

And I think it'll take the focus off like like part of the me me I get I win I take everything I think is driven by social media and the fact that like you want to show in front of everybody that you win everything and when people stop giving a about that and people like yeah it's fine.

Alayna Kennedy: I mean, that's already kind of coming through with like Gen Z versus Millennial culture of like it is now really cheoogy to like have a like, if you don't know what that means, Rob, we'll catch up about it afterwards. It is very cheooy to have like a curated post like they're almost like embracing more cringe culture. And last book club of or book plug of the day, if you haven't read The Fourth Turning by I can't remember their first names, but like Strauss and how it came out in like the late 90s. It's really interesting and it's about this exact topic which is like each era is a like immediate counter counter reaction to the one before it. You should read it.

Rob May: I get it. Illegally downloaded it. I've just observed that from my life. So I've lived long enough to see it. Well, that's you know I think Marks wrote about this of the pendulum swinging of history. So for another day but we got a question teed up for us. What's what's our question of the week here?

Anna Kirk: So yes. So, we always ask the guests something about New York and I think we decided this morning, this before the show, we decided what is our your favorite way to spend a Saturday morning in New York. Saturday morning New York. What do you got?

Alayna Kennedy: I Yeah, I'm like back This is appropriate because we're talking about like, you know, getting offline. People like doing things outside. I actually see this a lot in like New York hobbies. So, my favorite Saturday morning activity is the Prospect Park Bird Club. A.m. The big white building by the lake. You just go look at birds. It's really It's actually like really delightful. Like someone brings binoculars for you and there's always someone there who like knows all the birds. So you literally just like walk around and there is something nice I will I don't know how long this section is supposed to go or I could just talk about birding forever where like there's something else. There's I have one more book club. It's like Chris Hayes's new book about attention. One of the nice things about birding is that like you are paying attention. You're paying peripheral attention to the whole world around you. Like you're not just walking around. And so your attention is sort of gently and broadly distributed at all moments in a way that feels engaging while also not being overwhelming. So then you finish and you feel like you've actually like spent time like engaging your attention almost meditatively on the world around you and you're not looking at your phone. You're not doing anything else. You're not even really chatting that much. You're just like engaged in the world.

Anna Kirk: That's a good one. On a Saturday morning. Mine is going to be much more grandmotherly and much well maybe equally grandmotherly, but I'm a big like I guess workout first then like cafe and either do the crossword or read a book or sketch like just like something that I can do on paper right in front of me and just like sit in a cafe and have all the like ambient like maybe like less so observing but kind of just like basking in the white noise. I really like Yeah.

Ryan Eppley: I mean, I'm a huge soccer fan, so my girlfriend and I will wake up. We'll get a little Premier League morning going on. Feeling ambitious. We'll go down to the local pub and watch that with with a bunch of hooligans. But yeah, it's really really nice to have a city with such good pubs to go to watch Premier League. It's a blessing.

Alayna Kennedy: I feel it's so like luxurious to like watch TV in the morning. I feel like you're like you're like I'm this is like I'm doing it.

Ryan Eppley: Especially Premier League cuz like I there is no better like wake up than like I'm kind of hung over from the day before. I want to watch something that like requires some attention but like not a ton. Like it's a super like smooth way to start. Very very fun.

Rob May: I tend to take the subway to parts of the city I never go to and then walk around or occasionally if I'm training for a race like run back like I'll take it way up into Harlem and then run all the way back down. I you know I live in Chelsea. Or I'll take it over to Queens or Brooklyn or some place or even Staten Island and just walk around and see what's around there and stop and get breakfast or something and yeah, I just like to explore the city. Cool. So, well, Alayna, thanks for being on today.

Alayna Kennedy: Thank you for having me. This was really fun.

Rob May: Great conversation. Yeah. And everybody watching, please you know, like us, subscribe, all that kind of stuff. A bunch of stuff coming out. Check out the AI aiinnycshow.com website. We have a brand new website up. If you run a company, you can put job listings there. You can post your events. We got a newsletter now that's going out that you can subscribe to. And speaking of vibe coding, you should check out new yorkgpt.com, which I vibecoded last weekend, two weekends ago, and which is a chatbot for the city. Can answer a bunch of your questions. Check it out. Let us know what we should change to it. And check out BePresent, our sponsor. And we'll see you next week. The next stopage.