Informing Algorithmic Literacy Through User Folk Theories

Michael Ridley*

As part of a broader information literacy agenda, academic libraries are interested in advancing algorithmic literacy. Folk theories of algorithmic decision-making systems, such as recommender systems, can provide insights into designing and delivering enhanced algorithmic literacy initiatives. Users of the Spotify music recommendation systems were surveyed and interviewed to elicit their folk theories about how music recommendations are made. Seven folk theories emerged from this study and are grouped into four themes: agency, context, trust, and feelings. These four themes are used to illustrate how folk theories can inform algorithmic literacy programming and curricula.

Introduction

Folk theories of algorithmic decision-making systems tell us what people believe about how these systems work and how users should interact with them. Consumer-facing recommender systems using advanced machine learning techniques—such as Amazon, Facebook, and TikTok—are the “public face” of artificial intelligence. Equally ubiquitous are academic tools and resources using machine learning that are now essential for scholarship across all disciplines.

These systems underscore that “our entanglement with algorithmic personalization is non-negotiable: it is a market driven pre-condition of the digital everyday” (Kant, 2020, p. 214). Despite their ubiquity in the digital marketplace, most people continue to have concerns about their use and influence (Bao et al., 2022; Pew Research Center, 2018; Sartori & Bocca, 2022). The insights provided by folk theories can be used to focus and enhance strategies towards algorithmic literacy, enabling users to mitigate harm and risk while advancing the effective and productive use of these systems. The results presented here are part of a larger study of the folk theories of the Spotify music recommendation system as well as how those theories could facilitate the development of more transparent and explainable recommender systems (Ridley, 2022). A key question in that research was the relationship between folk theories and algorithmic literacy. Can folk theories inform algorithmic literacy?

Our lives are now “algorithmically mediated” (Anderson, 2020). Students, staff, and faculty encounter tools and services that rely on machine learning in virtually all aspects of their academic and personal lives. However, “the danger is not so much in delegating cognitive tasks, but in distancing ourselves from–or in not knowing about–the nature and precise mechanisms of that delegation” (de Mul & van den Berg, 2011, p. 59). The pervasiveness of algorithms highlights “issues of social justice, inequality, and social exclusion, which left unexamined, can result in positions of precarity and information poverty. Herein lies a role for information literacy, which in turn provides the warrant for the interest of librarians and educators” (Lloyd, 2019, p. 1480). Southworth et al. position the challenge of algorithmic literacy in the specific context of higher education curriculum with libraries and librarians as key participants (2023). In response, academic libraries are beginning to play key roles in advancing algorithmic literacy (Ridley & Pawlick-Potts, 2021) with relevant learning initiatives already in place or emerging (Gasparini & Kautonen, 2022; Hervieux & Wheatley, 2022; Kim, 2019; Upshall, 2022; Weintrop et al., 2021). However, the gap between what users believe about algorithms (i.e., their folk theories) and how to use algorithmic systems effectively remains an area that is both problematic in terms of user understanding and fruitful in terms of pedagogical strategies.

Literature Review

What is Algorithmic Literacy?

While algorithmic literacy is related to information literacy and other “digital” literacies such as computational literacy and data literacy, it also represents a unique area of interest that requires its own attention. As this is an emerging area, multiple definitions are presented. Finn defines algorithmic literacy as a capacity “that builds from a basic understanding of computational systems, their potential and their limitations, to offer us intellectual tools for interpreting the algorithms shaping and producing knowledge” (2017a, p. 25). It provides “a way to contend with both the inherent complexity of computation and the ambiguity that ensues when that complexity intersects with human culture” (Finn, 2017b, p. 2). A more operational definition views algorithmic literacy as “a set of competencies that enables individuals to critically evaluate AI technologies; communicate and collaborate effectively with AI; and use AI as a tool online, at home, and in the workplace” (Long & Magerko, 2020, p. 27). Ridley and Pawlick-Potts provide an action-oriented, pedagogically informed definition:

Algorithmic literacy is the skill, expertise, and awareness to

• Understand and reason about algorithms and their processes,
• Recognize and interpret their use in systems (whether embedded or overt),
• Create and apply algorithmic techniques and tools to problems in a variety of domains,
• Assess the influence and effect of algorithms in social, cultural, economic, and political contexts, and
• Position the individual as a co-constituent in algorithmic decision making (Ridley & Pawlick-Potts, 2021, p. 4).

The importance of algorithmic literacy to academic libraries specifically and the academy more generally is documented in the recent Project Information Literacy report on student attitudes towards algorithmic systems. It found that: students have “ambivalent attitudes” towards algorithmic systems; they use “defensive strategies” to protect their privacy; trust in these systems is “dead”; and that “skepticism lives.” The report’s conclusion is that, “the age of algorithms demands that teaching strategies be reconsidered as we redefine information literacy” (Head et al., 2020, p. 28).

What are Folk Theories?

Folk theories, also known as mental models, are “the mental representations that humans use to structure experience” (Gelman & Legare, 2011, p. 380). They allow people to “systematically investigate what [they] believe to be true about particular domains” (Payne, 2003, p. 152). Folk theories are “surprisingly meager, imprecisely specified, and full of inconsistencies, gaps, and idiosyncratic quirks” (Norman, 1983, p. 8) and yet they are also “causal and explanatory” (Gelman & Legare, 2011, p. 380). Crucial to using folk theories as insights into algorithmic literacy is the understanding that they are “not neutral or passive snapshots of experience; they embody cognitive biases that influence thought and action” (Gelman & Legare, 2011, p. 380). In the context of algorithmic systems, Bucher calls folk theories the “algorithmic imaginary” explaining, “the algorithmic imaginary is not to be understood as a false belief or fetish of sorts but, rather, as the way in which people imagine, perceive and experience algorithms and what these imaginations make possible” (2017, p. 31).

This study builds on prior research that investigates the folk theories of algorithmic systems (French & Hancock, 2017; Martens et al., 2022; Siles et al., 2020; Ytre-Arne & Moe, 2021). Uniquely, this study applies elicited folk theories as insights into how algorithmic literacy can be advanced.

Methodology

Nineteen users of Spotify, recruited using Twitter, were surveyed and individually interviewed to elicit their folk theories about how the system makes personalized music recommendations. All the participants were 18+ years old and from Canada or the United States. Participants were drawn from the general population, not specific groups (e.g., faculty or students). This was done purposefully to not only capture the zeitgeist of the emerging era of algorithms but to recognize that academic libraries serve broad and diverse communities both from within and beyond academia. Spotify was selected as a representative recommender system because of its size, reach, experience, and relative transparency about its algorithms. Available in 184 countries, Spotify has ~400 million monthly users offering over 82 million songs and ~4 billion playlists (Spotify, 2021). Spotify uses a variety of machine learning algorithms including simple heuristics, matrix factorization and collaborative filtering, and state-of-the-art deep learning neural networks and reinforcement learning (Eriksson et al., 2019; Stål, 2021; Whitman, 2012). Machine learning is “the heart of everything we do at Spotify” (Jebara, 2020). Spotify was also selected because of its broad appeal to and use by faculty, students, and staff.

Survey Results

A statistical analysis and factor analysis were conducted on the online Qualtrics survey to determine key background data and beliefs central to Spotify as an algorithmic system (see Appendix A for the Spotify User Survey). Most participants described themselves as “passionate” or “keen” about music. They were avid Spotify users, with most listening every day or most days. Many of the participants had used the system for over five years. Most participants (81%) were satisfied with the recommendations they receive from Spotify. One of the key questions asked participants how they believed Spotify makes its personalized recommendations. The five options—with results in parentheses—are: solely by algorithms (57.9%), primarily by algorithms and partly by humans (36.8%), primarily by humans and partly by algorithms (0%), solely by humans (0%), and don’t know (5.3%). In fact, Spotify makes its recommendations primarily by algorithms and partly by humans (Fleischer & Snickars, 2017; Goldschmitt & Seaver, 2019; Pichl et al., 2017; Popper, 2015) indicating that most participants hold an incorrect belief about how Spotify works. The results also indicate that all participants prioritize the role of algorithms over humans whether solely or primarily.

Another key question asked participants to rate the influence of 22 different factors that Spotify uses in its music recommendation process. The most common responses identified a similar cluster of actions: what users were listening to (i.e., songs, artists, and genres), frequency of listening, skipping songs, “liking” (i.e., “hearting”) songs or playlists, creating playlists, and adding songs to their library. The following categories were all rated “very important” by participants: “What I listen to” (95%), “How many times I listen” (89%), and “Marking something a ‘like’ (i.e., ‘heart’) (68%). Factors representing explicit actions by participants were consistently rated more highly than the actions of other users and inferences made by the system. See Appendix A for a complete list of the items rated. These findings were used to inform the subsequent interviews.

Interviews

The interviews, conducted over Zoom, recorded, and lasting approximately 60 minutes, were thematically analyzed using NVivo. Thematic analysis attempts “to identify or examine the underlying ideas, assumptions, and conceptualizations—and ideologies—that are theorized as shaping or informing the semantic content of the data” (Braun & Clarke, 2006, p. 84). Individual and collective responses from the survey formed the basis of the initial questions posed during the interviews. To focus participant responses on the effect of algorithms, follow-up questions directly or indirectly referenced the three key machine learning functions: representation, evaluation, and optimization (Domingos, 2015). Moving from the general to the specific, questioning sought a deeper understanding of concepts raised by the participant. Counterfactual or contrastive questions broadened the conversation by probing areas unexplored by the participant.

Limitations

This study has several limitations which restrict the generalizability of the findings. The sample size is small and not random. As such, it is not necessarily representative of Spotify users. Selecting Spotify as the single example of recommender systems allowed for specific details and experiences to emerge from users. However, investigating multiple systems might have resulted in a broader set of folk theories that would be more generalizable. Finally, the research methodology used to elicit the folk theories has known weaknesses (Doherty & Doherty, 2018; Norman, 1983). Surveys and interviews rely on reflective experience which may not correlate to actual experience.

Findings and Discussion

The analysis of the survey and interviews elicited seven folk theories. They are grouped here by themes and expressed as verbs (i.e., “Spotify Complies,” “Spotify Decides”):

Agency: Compiles, Decides, Dialogues

Context: Surveils, Exploits

Trust: Withholds & Conceals

Feeling: Empathizes

It is important to remember that individual users will hold some but not all these folk theories and some users may hold contradictory beliefs depending on the context.

Agency: Spotify Complies, Decides, and Dialogues

Some users believe Spotify “Complies” based on the users’ specific directions and actions; for example, User 3 stated, “The only cues that it’s getting are the ones that I’m feeding it.” In this view, the user is in control and the algorithm responds to their signals (e.g., what they listen to, how many times they listen, what songs they “like” or include in personal playlists). The factor analysis revealed overwhelmingly that the algorithm is viewed as “about me.” User 19 was clear about user agency explaining, “Spotify only works because they [listeners] are teaching it to work.”

The folk theory that Spotify “Decides” places agency solely with the algorithm. Spotify’s recommendations are made based on its own objectives and not that of users. For some users, this is acceptable. They put Spotify “on cruise control” and let the system “take the wheel” (User 5). For others, this is problematic: Spotify “silos me into a particular style” (User 16) and when “in doubt” Spotify will “give me the thing they’re being paid to promote” (User 18). With sole algorithmic agency, users believe they have no control. As User 13 said, “It’s all this giant black box, I don’t know anything and there’s nothing I can do about it either.”

The Spotify “Dialogues” folk theory is about shared agency where the user and the algorithm are in a cooperative relationship. As User 19 describes it as: “I’m feeding it, it feeds me.” In this belief, Spotify is a “feedback loop” (User 16). Users believe the algorithm does “a good job of matching my music tastes” (User 12) and is “good at anticipating what kind of music I would be into” (User 14). Some users perceive the dialogue with the algorithm is insufficient, they want a more informed exchange, such as User 10 who said, “Give me a bigger vocabulary and then make it meaningful. Then prove to me that you’ve heard me.”

Context: Spotify Surveils and Exploits

The two folk theories, Spotify “Surveils” and Spotify “Exploits,” reflect beliefs that are both negative and positive indicating that perceptions are contextual. While User 2 said “I don’t like that they’re collecting data … I don’t like that they know so much about me” (User 2), users also understand that data tracking and capture (sometimes experienced as surveillance) is part of the “surrender of personal information that it needs in order to make recommendations that you want. I think that’s part of the deal” (User 20). Similarly, the belief that Spotify “Exploits,” reflected in the observation “my choices, my preferences, are being harvested for their algorithm … [and this is] the product people are paying for” (User 15), is tempered by the perception that this a necessary part of the “bargain” to ensure satisfactory recommendations (User 3). Important here is the recognition that some folk theories contain apparent contradictions unless the specific context and the use case are understood.

Trust: Spotify Withholds and Conceals

The folk theory Spotify “Withholds and Conceals” reflects a breakdown in trust. Spotify is “a complete black box” (User 3) where users are “not exactly fully cooperating here because Spotify is still doing a lot that we don’t necessarily know” (User 13). As a result, users believe Spotify limits the effect of their actions since none “seem to influence algorithms too much” (User 20). Users perceive Spotify as operating “behind the curtain” (User 11) deliberately beyond their scrutiny and influence.

Feeling: Empathizes

While the personification or anthropomorphization of information systems is common, Spotify users had a more specific belief: Spotify “Empathizes.” In the survey, the importance of “what I’m feeling while I’m listening” as a data signal that influences the recommendations the Spotify algorithm provides was rated “very important” or “important” by 32% of the participants. User 14 believes Spotify infers user feelings to make recommendations (“Yeah, I think so”) and another user, although skeptical, “wouldn’t be surprised if I’m wrong” (User 11). Whether, and if so how, algorithmic systems infer and use emotional states is highly controversial (Crawford, 2021; Stark & Hoey, 2021) The “Empathizes” folk theory indicates that, debates and critics aside, this perception is part of many user experiences.

Folk Theories and Algorithmic Literacy Programming and Curricula

The themes arising from the folk theories (agency, context, trust, and feeling) illustrate how folk theories can be utilized to enhance algorithmic literacy programming and curricula.

Agency

The diverse folk theories about agency (Complies, Decides, Dialogues) suggests that this is a key issue in leveraging folk theories to enhance algorithmic literacy. A 2016 study of the folk theories of Facebook’s News Feed found two surprising results regarding agency (Eslami et al., 2016). First, at the beginning of the study, 62% of the participants were unaware that any algorithm at all was involved and believed the user was in full control. Second, even following interventions that described the algorithm and how it worked, 12% of the participants believed the News Feed was completely random and that there was no control.

In fact, recommender systems are built on shared agency (“I’m feeding it, it feeds me”), although the balance of power can vary greatly from one algorithmic system to another. As Lomborg & Kapsch note, “while algorithms do things to people, people also do things to algorithms” (2020, p. 755). While acquiescing to the system by letting it “take the wheel” is a user choice (a form of Spotify “Decides”), it can obscure that a user’s behaviour, whether conscious or not, always influences the recommendations of the algorithm.

Users should be encouraged and supported to explore the range of their agency. What tools and choices are available to influence the algorithms? What impact do they have (if any)? Can the user recognize when system or user objectives are prioritized?

Spotify “Decides,” or even aspects of Spotify “Dialogues,” can lead to explorations of resistance that can ameliorate user concerns while still benefiting from using the system. This is a form of contested agency where the user deliberately attempts to “confound” the algorithm (User 4) to exert greater influence. In a similar manner User 10, in requesting a “better vocabulary,” was asking for a rebalancing of the shared agency. Discussing and exploring agency promotes user empowerment regarding algorithmic systems. Learning strategies that emphasize shared agency open a dialogue about an issue central to algorithmic literacy.

Context

Context matters in all human-machine interactions. However, in the case of the folk theories, Spotify “Surveils” and Spotify “Exploits,” context highlights a key dilemma. Surveillance, which can be described as the tracking and collecting as much user data as possible or allowable, and exploitation, which can be described as the sharing one user’s data to enhance the recommendations of another user, are core processes of any recommender system. Surveillance and exploitation are perceived through two different but simultaneously occurring lenses. These perceptions reflect conditions that are undesirable and unwanted but also necessary and an accepted part of the “bargain.” The conditions of a recommender system make both possible and both necessary.

Algorithmic literacy recognizes surveillance and exploitation as perceptions best treated as a continuums not as unconditional problems or an either/or choice. The context of these beliefs is critical in discussing how to minimize risks while maximizing the value of the system.

Trust

The Spotify “Withholds and Conceals” folk theory is a belief that the algorithmic system is not fully forthcoming about its operations and motivations. In this belief, the system is not merely opaque (i.e., a “black box” because of the complexities of machine learning) it is deliberately so to preserve the “curtain” that hides the system from scrutiny (Pasquale, 2015).

While establishing trust is important for the effective use of any system, unwarranted trust can limit critical engagement and assessment and lead to acceptance of malicious and deceptive practices (Pawlick-Potts, 2022). In that sense, Spotify “Withholds and Conceals” is a belief that maintains a skeptical and a critical stance.

Recommender systems often are secretive or unforthcoming to protect intellectual property, trade secrets, or other competitive advantages. This folk theory opens a discussion about the limits of transparency, the rights and obligations of users and corporate entities, the role of consumer protection, and the possibility of government regulation in this area.

Feeling

While critics are concerned that user personification or anthropomorphization of algorithmic systems harm critical appraisal and trustworthiness (Glikson & Woolley, 2020; Ngo & Krämer, 2021; Watson, 2019), users do it anyway. Spotify “Empathizes” is a belief that the algorithm understands a user’s emotional state and responds accordingly. While Spotify insists that it does not collect or infer emotions (Gutierrez, 2021), users believe otherwise.

It seems unlikely that algorithmic systems will discourage this form of bonding and that users will not continue to form such bonds. As a result, a key question is what constitutes a healthy relationship with an algorithmic system? While this obviously starts with the recognition that a system is not a person, systems are social actors and need to be understood in that context (Nass et al., 1997).

Enhancing Algorithmic Literacy

DeVito classifies the complexity of folk theories regarding algorithms as a hierarchy moving from basic awareness and causal effects (characteristics of DeVito’s functional theories) to the identification of “mechanistic fragments” (e.g., factors or data signals) and finally to the aggregation of these factors into more complex interrelationships indicating “mechanistic ordering” (both latter are characteristics of DeVito’s structural theories) (2021).

Moving users from functional to structural theories is the objective of algorithmic literacy (DeVito, 2021). However, “knowledge itself does not seem to prompt more critical engagement with and valuation of algorithms” (Lomborg & Kapsch, 2020, p. 757). Although the folk theories discussed above are primarily conceptual, the recommended approach to algorithmic literacy is not through theoretical methods but through “real life examples of algorithmic work in different contexts, relatable to the life of ordinary people” (Lomborg & Kapsch, 2020, p. 759). Understanding the way some of the multiple data elements combine and interact (i.e., “the mechanistic ordering” of “fragments”) is sufficient to trigger the transition in theories. The deeper conceptual issues, important aspects characteristic of structural theories, can be layered on as personal experiences (i.e., a user’s folk theories) are explored and perhaps challenged.

In offering a set of principles for algorithmic literacy training, Dasgupta and Hill include “respect community values about technology that may differ” (2020, p. 1-2). This is an important observation for academic libraries given the diverse community they engage with. In this context, “communities” might be students or faculty, humanists or scientists, and technology experts or technology neophytes. Groups and individuals will bring to algorithmic literacy programming or curricula their own perspectives about technology, a point central to the idea of understanding folk theories.

Conclusion

Calling the prevalence and opacity of algorithms “a wicked problem for librarians and archivists” engaged in information literacy, Lloyd situates algorithmic literacy in a sociotechnical context that highlights the co-constituency of people and technology (2019). Folk theories form a bridge that allows us to “meet the user where they are in terms of understanding and literacy, regardless of how contradictory, sparse, or fragmented these understandings may be” (DeVito, 2021, p. 4). They tell us not only how users perceive algorithmic systems but also how they believe they should interact with them. As algorithmic literacy becomes increasingly important for the effective use of research and discovery tools and services fueled by machine learning, academic libraries are well positioned to provide leadership through relevant programming and curricula. Applying the insights of folk theories about algorithms can enhance those algorithm literacy initiatives.

Acknowledgements

The author would like to thank the reviewers for their consideration, advice, and recommendations as well as Dr. Jacquie Burkell for her guidance and support throughout this research.

Appendix A: Spotify User Survey

Which of the following best describes your interest in music?

• □ Passionate about music with extensive knowledge
• □ Keen about music but balanced with other interests
• □ Music is important but other things are far more important
• □ Engage with music but are generally indifferent

Would you describe yourself as a “specialist” (listens to mostly the same artists and genres) or “generalist” (listens to a wide variety of artists and genres)?

• □ Specialist
• □ Generalist

Do you subscribe to Spotify (pay version) or use the free (ad-supported) version?

• □ Paid (subscription) version
• □ Free (ad-supported) version

How long have you been using Spotify?

• □ Less than 1 year
• □ 1 to 5 years
• □ More than 5 years

How often do you listen to Spotify?

• □ Every day
• □ Most days
• □ At least weekly
• □ Less often than weekly

How do you primarily listen to Spotify?

• □ On a laptop or desktop computer
• □ On a smartphone or mobile device?
• □ On a smart assistant (e.g., Alexa, Google Home)
• □ Other

Are you generally satisfied with Spotify’s personalized music recommendations to you?

• □ Yes
• □ No

How do you think Spotify’s personalized music recommendations are made?

• □ Solely by algorithms
• □ Primarily by algorithms and partly by humans
• □ Primarily by humans and partly by algorithms
• □ Solely by humans
• □ Don’t Know

How does Spotify use information to determine the personalized music recommendations for you?

[Open ended question]

What could you do to shape the personalized music recommendations you receive from Spotify?

[Open ended question]

To what extent do you think the following influence Spotify’s music recommendations for you?

(very important=1; important=2, somewhat important=3; not important=4)

_____ Marking something a “like” (i.e., “heart”) What I listen to

_____ How long I listen to a song or playlist

_____ How many times I listen to a song, artist or playlist

_____ What other people are listening to

_____ What my friends are listening to

_____ Songs that are similar to other songs I “liked” or listened to

_____ Playlists I’ve created

_____ Playlists other users have created

_____ What people my age listen to

_____ What people in my location (city/country) listen to

_____ What people with my level of education listen to

_____ Where I am while listening

_____ What I’m doing while listening

_____ What I’m feeling while listening

_____ The time of day I’m listening

_____ The day of the week I’m listening

_____ The season of the year I’m listening

_____ Songs or artists that Spotify is promoting

_____ Posts about Spotify I make on social media

_____ Comments from other people about music on social media

_____ Reviews of music in magazines, blogs, videos, news sources

References

Anderson, J. (2020). Understanding and interpreting algorithms: Toward a hermeneutics of algorithms. Media, Culture & Society, 42(7–8), 1479–1494. https://doi.org/10.1177/0163443720919373

Bao, L., Krause, N. M., Calice, M. N., Scheufele, D. A., Wirz, C. D., Brossard, D., Newman, T. P., & Xenos, M. A. (2022). Whose AI? How different publics think about AI and its social impacts. Computers in Human Behavior, 130. https://doi.org/10.1016/j.chb.2022.107182

Braun, V., & Clarke, V. (2006). Using thematic analysis in psychology. Qualitative Research in Psychology, 3(2), 77–101. https://doi.org/10.1191/1478088706qp063oa

Bucher, T. (2017). The algorithmic imaginary: Exploring the ordinary affects of Facebook algorithms. Information, Communication & Society, 20(1), 30–44. https://doi.org/10.1080/1369118X.2016.1154086

Crawford, K. (2021). Time to regulate AI that interprets human emotions. Nature, 592(7853), 167. https://doi.org/10.1038/d41586-021-00868-5

Dasgupta, S., & Hill, B. M. (2020). Designing for critical algorithmic literacies. ArXiv:2008.01719 [Cs]. http://arxiv.org/abs/2008.01719

de Mul, J., & van den Berg, B. (2011). Remote control: Human autonomy in the age of computer-mediated agency. In M. Hildebrandt & A. Rouvroy (Eds.), Law, human agency, and autonomic computing (pp. 46–63). Routledge.

DeVito, M. A. (2021). Adaptive folk theorization as a path to algorithmic literacy on changing platforms. Proceedings of the ACM on Human-Computer Interaction, 5, 1–38. https://doi.org/10.1145/3476080

Doherty, K., & Doherty, G. (2018). The construal of experience in HCI: Understanding self-reports. International Journal of Human - Computer Studies, 110, 63–74. https://doi.org/10.1016/j.ijhcs.2017.10.006

Domingos, P. (2015). The master algorithm: How the quest for the ultimate learning machine will remake our world. Basic Books.

Eriksson, M., Fleischer, R., Joansson, A., Snickars, P., & Vonderau, P. (2019). Spotify teardown: Inside the black box of streaming music. MIT Press.

Eslami, M., Karahalios, K., Sandvig, C., Vaccaro, K., Rickman, A., Hamilton, K., & Kirlik, A. (2016). First I “like” it, then I hide it: Folk theories of social feeds. Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, 2371–2382. https://doi.org/10.1145/2858036.2858494

Finn, E. (2017a). Algorithm of the enlightenment. Issues in Science and Technology, 33(3), 21–25.

Finn, E. (2017b). What algorithms want: Imagination in the age of computing. The MIT Press.

Fleischer, R., & Snickars, P. (2017). Discovering Spotify. Culture Unbound, 9(2), 130–145. https://doi.org/10.3384/cu.2000.1525.1792

French, M., & Hancock, J. (2017). What’s the folk theory? Reasoning about cyber-social systems. 67th Annual Conference of the International Communication Association. 67th Annual Conference of the International Communication Association, San Diego, CA. https://doi.org/10.2139/ssrn.2910571

Gasparini, A., & Kautonen, H. (2022). Understanding artificial intelligence in research libraries: An extensive literature review. LIBER Quarterly: The Journal of the Association of European Research Libraries, 32(1), Article 1. https://doi.org/10.53377/lq.10934

Gelman, S. A., & Legare, C. H. (2011). Concepts and folk theories. Annual Review of Anthropology, 40, 379–398. https://doi.org/10.1146/annurev-anthro-081309-145822

Glikson, E., & Woolley, A. W. (2020). Human trust in artificial intelligence: Review of empirical research. Academy of Management Annals, 14(2), 627–660. https://doi.org/10.5465/annals.2018.0057

Goldschmitt, K. E., & Seaver, N. (2019). Shaping the stream: Techniques and troubles of algorithmic recommendation. In N. Cook, M. Ingalls, & D. Trippett (Eds.), The Cambridge Companion to Music in Digital Culture (pp. 63–81). Cambridge University Press.

Gutierrez, H. (2021, April 15). Horacio Gutierrez (Spotify) to Isedua Orbhabor (Access Now). https://www.accessnow.org/cms/assets/uploads/2021/04/Spotify-Letter-to-Access-Now-04-15-2021-.pdf

Head, A. J., Fister, B., & MacMillan, M. (2020). Information literacy in the age of algorithms: Student experiences with news and information, and the need for change. Project Information Literacy. https://www.projectinfolit.org/uploads/2/7/5/4/27541717/algoreport.pdf

Hervieux, S., & Wheatley, A. (Eds.). (2022). The rise of AI: Implications and applications of artificial intelligence in academic libraries. Association of College and University Libraries.

Jebara, T. (2020, January 16). For your ears only: Personalizing Spotify Home with machine learning. Spotify Labs. https://labs.spotify.com/2020/01/16/for-your-ears-only-personalizing-spotify-home-with-machine-learning/

Kant, T. (2020). Making it personal: Algorithmic personalization, identify, and everyday life. Oxford University Press.

Kim, B. (2019). AI and creating the first multidisciplinary AI lab. Library Technology Reports, 55(1), 16–20. https://doi.org/10.5860/ltr.55n1

Lloyd, A. (2019). Chasing Frankenstein’s monster: Information literacy in the black box society. Journal of Documentation, 75(6), 1475–1485. https://doi.org/10.1108/JD-02-2019-0035

Lomborg, S., & Kapsch, P. H. (2020). Decoding algorithms. Media, Culture & Society, 42(5), 745–761. https://doi.org/10.1177/0163443719855301

Long, D., & Magerko, B. (2020). What is AI literacy? Competencies and design considerations. Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, 1–16. https://doi.org/10.1145/3313831.3376727

Martens, M., De Wolf, R., Berendt, B., & De Marez, L. (2022). Decoding algorithms: Exploring end-users’ mental models of the inner workings of algorithmic news recommenders. Digital Journalism, 1–23. https://doi.org/10.1080/21670811.2022.2129402

Nass, C. I., Moon, Y., Morkes, J., Kim, E.-Y., & Fogg, B. J. (1997). Computers are social actors: A review of current research. In B. Friedman (Ed.), Human values and the design of computer technology (pp. 137–162). Cambridge University Press.

Ngo, T., & Krämer, N. (2021). Exploring folk theories of algorithmic news curation for explainable design. Behaviour & Information Technology. https://doi.org/10.1080/0144929X.2021.1987522

Norman, D. A. (1983). Some observations on mental models. In D. Gentner & A. L. Stevens (Eds.), Mental models (pp. 7–14). Psychology Press.

Pasquale, F. (2015). The black box society: The secret algorithms that control money and information. Harvard University Press.

Pawlick-Potts, D. (2022). Is anybody in there?: Towards a model of affect and trust in human – AI information interactions. Information Seeking in Context (ISIC) Conference. https://doi.org/10.18452/25258

Payne, S. J. (2003). Users’ mental models: The very ideas. In J. M. Carroll (Ed.), HCI models, theories, and frameworks toward a multidisciplinary science (pp. 135–156). Morgan Kaufmann.

Pew Research Center. (2018). Public attitudes toward computer algorithms. Pew. http://www.pewinternet.org/2018/11/16/public-attitudes-toward-computer-algorithms/

Pichl, M., Zangerle, E., & Specht, G. (2017). Understanding user-curated playlists on Spotify: A machine learning approach. International Journal of Multimedia Data Engineering and Management, 8(4), 44–59. https://doi.org/10.4018/IJMDEM.2017100103

Popper, B. (2015). Tastemaker: How Spotify’s Discover Weekly cracked human curation at internet scale. The Verge. https://www.theverge.com/2015/9/30/9416579/spotify-discover-weekly-online-music-curation-interview

Ridley, M. (2022). Folk theories, recommender systems, and human-centered explainable artificial intelligence (HCXAI) [PhD Thesis, Western University]. https://ir.lib.uwo.ca/etd/9039/

Ridley, M., & Pawlick-Potts, D. (2021). Algorithmic literacy and the role for libraries. Information Technology and Libraries, 40(2). https://doi.org/doi.org/10.6017/ital.v40i2.12963

Sartori, L., & Bocca, G. (2022). Minding the gap(s): Public perceptions of AI and socio-technical imaginaries. AI & Society. https://doi.org/10.1007/s00146-022-01422-1

Siles, I., Segura-Castillo, A., Solís, R., & Sancho, M. (2020). Folk theories of algorithmic recommendations on Spotify: Enacting data assemblages in the global South: Big Data & Society. https://doi.org/10.1177/2053951720923377

Southworth, J., Migliaccio, K., Glover, J., Glover, J., Reed, D., McCarty, C., Brendemuhl, J., & Thomas, A. (2023). Developing a model for AI Across the curriculum: Transforming the higher education landscape via innovation in AI literacy. Computers and Education: Artificial Intelligence, 4. https://doi.org/10.1016/j.caeai.2023.100127

Spotify. (2021). Annual report. https://s22.q4cdn.com/540910603/files/doc_financials/2021/q4/0307a021-254e-43c5-aeac-8242b0ea3ade.pdf

Stål, O. (2021). How Spotify uses ML to create the future of personalization. TransformX. TransformX. https://youtu.be/n16LOyba-SE

Stark, L., & Hoey, J. (2021). The ethics of emotion in artificial intelligence systems. Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, 782–793. https://doi.org/10.1145/3442188.3445939

Upshall, M. (2022). An AI toolkit for libraries. Insights, 35(0), Article 0. https://doi.org/10.1629/uksg.592

Watson, D. (2019). The rhetoric and reality of anthropomorphism in artificial intelligence. Minds and Machines, 29, 417–440. https://doi.org/doi.org/10.1007/s11023-019-09506-6

Weintrop, D., Morehouse, S., & Subramaniam, M. (2021). Assessing computational thinking in libraries. Computer Science Education, 1–22. https://doi.org/10.1080/08993408.2021.1874229

Whitman, B. (2012, December 11). How music recommendation works—And doesn’t work. Variogram. https://notes.variogram.com/2012/12/11/how-music-recommendation-works-and-doesnt-work/

Ytre-Arne, B., & Moe, H. (2021). Folk theories of algorithms: Understanding digital irritation. Media, Culture & Society, 43(5), 807–824. https://doi.org/10.1177/0163443720972314