Advanced literacy – Basic education for the digital age? – EDU 02 2019

From EuroDIG Wiki
(Redirected from EDU 02 2019)
Jump to: navigation, search

19 June 2019 | 16:30-18:00 | AMAZON | Video recording | Transcription
Consolidated programme 2019 overview

Proposals assigned to this session: ID 23, 117 – list of all proposals as pdf

You are invited to become a member of the session Org Team! By joining an Org Team you agree to that your name and affiliation will be published at the respective wiki page of the session for transparency reasons. Please subscribe to the session mailing list and answer the email that will be send to you requesting your confirmation of subscription.

Session teaser

The focus of this session is on practical tools and methods for providing Algorithmic Literacy to citizens in order for people to be able to assess and interact with online algorithmic systems on the basis of informed judgement.

Session description

This session will follow an interactive hands-on format. After a short introduction to the topic Algorithmic Literacy, the session facilitators will guide the participants through the use of some Algorithmic Literacy teaching tools. The final third of the session will be dedicated to feedback, reflection and discussion regarding strengths, weaknesses and ways of using Algorithmic Literacy tools beyond traditional education settings.

10 minutes setting the scene - why algorithmic literacy is necessary
15 minutes reviewing current "digital literacy" teaching practices in school / adult-education curricula
10 minutes introducing some practical tools and methods for Algorithmic Literacy learning
30 minutes hands-on with algorithmic literacy toolkits
25 minutes discussion


Facilitated hands-on interaction with Algorithmic Literacy tools, followed by open discussion.

Further reading


Focal Point

  • Ansgar Koene, University of Nottingham

Organising Team (Org Team) List them here as they sign up.

  • Amali De Silva-Mitchell
  • Yohko Hatada, EMLS RI (Evolution of Mind Life Society Research Institute)
  • Ana Jorge, Catholic University of Portugal
  • Narine Khachatryan, STEM Society
  • Oksana Prykhodko
  • Liu Yong

Key Participants

Key Participants are experts willing to provide their knowledge during a session – not necessarily on stage. Key Participants should contribute to the session planning process and keep statements short and punchy during the session. They will be selected and assigned by the Org Team, ensuring a stakeholder balanced dialogue also considering gender and geographical balance. Please provide short CV’s of the Key Participants involved in your session at the Wiki or link to another source.


Remote Moderator

Trained remote moderators will be assigned on the spot by the EuroDIG secretariat to each session.

Video record


Provided by: Caption First, Inc., P.O. Box 3066, Monument, CO 80132, Phone: +001-800-825-5234,

This text, document, or file is based on live transcription. Communication Access Realtime Translation (CART), captioning, and/or live transcription are provided in order to facilitate communication accessibility and may not be a totally verbatim record of the proceedings. This text, document, or file is not to be distributed or used in any way that may violate copyright law.

>> This one is obviously working.

Welcome. I don't know to what extent people are over in the main Plenary session listening to the U.N. presentation, that was just when I came in, they just started the Q&A bit. So they may be a little bit late. In the interest of not keeping everybody here until very late, I propose that we get started.

So this is meant to be an interactive session to do with some questions around how to teach basic information, how algorithmic systems that are increasingly effecting the way in which we interact with the world online, how those work, getting a basic understanding around that.

If I could have the next slide..

Yes. Great.

So to start things off, I wanted to do a little question near and question ear -- we should switch to the other thing? Is Wagner in the booth? This is the tool that was suggested to use. We can scroll up so people can see the question.

The question is really, -- you also need the address to use. Can we have the scroll a bit so we can see the top.

That's it. Scroll up again so we can see the address. And the previous slide..

We did test this. During lunch..

Can we go to one earlier.

Just talking about the ideas behind this panel while we're waiting for the technical solutions and things to be solved..

What you see, we have distributed around the room already a couple of these boxes that contain a deck of cards basically that are one kind of tool that we have developed as part of the research project that was an engagement with young people, 13 to 17 year olds in which we discussed with them their experiences of interacting with online systems and their concerns around how algorithmic information is being mediated and they don't have control over that, potential of ways in which this could be biased and they may be nudged into certain directions through that.

So through their interaction with them, and hearing from them the concerns around how the teachers, for instance, they have little material, clear knowledge for communicating around how the systems work, we developed a toolkit, and this is an important part of it around how to communicate the basic understandings of how the algorithmic ecosystem that you are interacting with is working.

It is not about the technical details of how to program or how to -- how deep learning works, things like that. It is about the fundamental questions around why is data collected for you in order to do algorithmic content moderation, what kind of values go into that. How is it that even that it is an algorithmic system that in a sense runs on mathematical equations and computations, it is not a neutral thing and there are values built into this. How are we doing on the technical support.

I'll just go on to the next slide. It is more of a general question to the audience, why is algorithmic literacy necessary? Would anyone like to offer suggestions? I did speak around it a little bit already. Any -- yes, in the back.

I don't -- do we have -- does this thing work?

You need to be on the mic, otherwise it is not recorded, it is an unknown question from an unknown person.

>> Sorry. Just repeating myself.

I'm a researcher in London. So when we do a search, we see something on our Facebook feed we know how it came to be there, who decided that it would be presented to us, and also who and what was excluded as well in its place.

>> Thank you. That's a very new theory. Any other things, any other reasons why -- why it may be important to have a basic algorithmic literacy for everybody as opposed to just the children or just people in a particular technical sphere.

Here we have got some -- somehow we jumped to the next question point already.

>> So you need to look at the first question for --

>> Yes.

I have no requested where we're going with the questions? Maybe we should forget about the questions..

Nobody actually has the address to submit the responses.


Does everybody in the room at least see the address? If you have a tool, smartphone or something, go there and you can find the answers there.

So the idea, it is just to get some polling from the room as to where you are. Most people rate themselves, this may be just one person, it is 100%, probably only one person replied. I think we're probably burning time.

I suggest that we actually skip this and go on to the next slide. If I can go to the next slide.

Thank you.

Let's move on to here.

Okay. So why is algorithmic mediation of content an important part of life on the Internet? One way to think of it is a little cartoon that I did of our experience when we interact with the social media platform, for instance. This goes for many things. If you join the platform, you know nobody, you're very sad because it is not really a good experience. As you are on the platform a bit, you connect to more people, it is more interesting. It is more enjoyable to use it, more successful to use it. You interact with even more people and perhaps some non-people issues and sites on the network and it is better. As we have experienced, Internet is vast, there are very many people and vertical gaze different kinds of sites on the Internet, as you connect to very many of them, if you just receive unfiltered input from all of them, it becomes chaotic and it overwhelms you. Therefore, you become unhappy again with your life connecting to the network. Which is where filters come in. You need a way in which to pick out the important bits as opposed to the bits that are just noise for you from your perspective. That's a key rule that algorithmic -- key role that algorithmic systems play in our interaction with the Internet.

One way of thinking about this, it is in a space such as the Internet when the presence of information is really the major stumbling block usually but really the ability to find that information is the core thing, good filtering is what is what determines good service, however, the way in which the filtering is done can have significant impacts on not just people, but also society. This was a study that was done in any we manipulated the way in which a search engine would present the rankings of certain political parties as a result if you search for them.

And basically, because it is known people tend to only look at the first couple of results in a search engine results page moving certain political parties, the response further down in the page, it will have a significant impact on how much is seen, so they actually showed that for undecided, it has a significant impact on how they were going to vote so this is one example of the way in which the underlying algorithm works has a significant impact on society as well. Of course, we're quite familiar with this from recent years regarding our political specs. What is the basic underlying current going on? It is a trade-off between convenience and the level of control that you have. If you think of search engine, for instance, the act of searching for content, we could theoretically contain complete -- maintain complete control if we're using detailed search queries using, for instance, SQL databases, and everybody would have to learn to program the search in a detailed way. You have full control but it is highly inconvenient. It is much more convenient to rely on an algorithm created as a company, such as Google -- Google being the majority provided in the world and typing your question, you get a response, it is very convenient and you don't actually know what the response is based on. You don't know all of the parameters that go into it.

This kind of trade-off between control and convenience, it increases as we go to other kinds of platforms so on a PC, you still have a large screen, a large interface where you get many responses as you move to the mobile device the number of responses you're actually seeing decreases, if you go to a voice activated device, you get one, two responses and the result you're going to will be completely determined by the algorithm.

Even going back to relatively recent times, very huge amount of users of social media platforms were complete lit unaware of the fact that algorithms were playing an important part in the information that they got to see. A lack of understanding of how the systems work, it is large and it is concerning.

And on the other side, there is a lot of hype around the capabilities of the systems, including the use of luring the lines, understanding when it is actually algorithmic versus a human, that's being presented. As I said, it is an algorithm and so people have a blurry understanding of how the world that they're interacting with actually functioning around this which is one of the things that need to be addressed.

This was supposed to be another question here but I think we're going to skip those. That bit of technology didn't work so well.

So there is now a growing movement towards developing some general literacy courses around how these online requirements are working, this is a commonsense education, where they focus on a particular type of question such as how does the tracking technology work, why are you being tracked online, how does that work, how does that implicate -- what kind of implications does that have for the services that you receive. On a general level, we have a thing like the Finnish government's push to provide a general course for a wide part of the population, I believe they're targeting something like 10% of the population to teach them fundamentals of how AI works, now the kinds of -- the way in which they have broken down the course, they're focused on the technological side to give people a core grasp of how the technology works so they're looking at how does AI problem solving work and what is -- how does it do ranking for instance of particular questions such as the technical level, what is the statistical rule that tells you about how to take in to account prior probabilities, et cetera, in making decisions. As well as questions around neural networks, everybody is hearing about what is the neural networks to demystify those kinds of questions. Now, from our point of view, this is very valuable, however, it doesn't address the kind of issue that came out of our discussion with young people, it is the core concerns because they were not really acting -- asking about the technical, how does this work, but more about the social technical side, what goes into the making up of the algorithm systems, how do they impact and getting a basic understanding of how information and values and decision-making processes play a role.

The project, this stack of cards, which is distributed in the room, it comes from, it is the unbiased project that ran for two years in the UK, it was a collaboration between three Universities, and very much was focused around the coproduction approach. Working with young people through stakeholder engagement workshops, hearing from them their core concerns and working with stakeholders from industry, from Civil Society groups and from government to hear how they're approaching the questions and how they might respond to the questions being raised by the young people. One of the outcomes that came out of this project, it was the development of a fairness toolkit which consists of three parts. The first part are these awareness cards which are about raising awareness of how algorithms work, where he this work, how the dynamics are in this space.

The second part, it is this kind of graphical mapping which is basically asking users to refer to a particular service that they're interacting with, and indicating their concerns regarding the service, how they would like to see issues being addressed, and that is something that they're encouraged to up load to us so that we can then share that with the designers in the system, which is the third diagram there, the designer of the system, they'll provide a response to tell the users you communicated, for this system, this is how we address them and why we feel this is an important issue to address.

>> It looks like we have more details on the awareness cards. The deck, it comes with eight suits, one is just a glossary card that explains the basics of what is an algorithm. Then there is a deck of cards -- sets of cards that go around the rights that we have. A lot of people, they don't realize how many rights they do have and how they could potentially challenge the kinds of responses that they're getting. There is values cards. This is to give a -- this is underlying the construction of the system that you are building. This is highlighting the point that yes, it is a technology that you're interacting with, but technology is not neutral. There are values built into it.

There are process cards, process cards are really about a way of making you think about the system, but basically saying you be the algorithm, if you're an algorithm that's supposed to do a certain kind of task, what kind of data would you need? What kind of values might you put into that? Et cetera.

It is to generate thinking around how do these systems work by basically trying to simulate being one yourself.

There is data cards. Fairly self-explanatory, example, cards, temperature a number of samples of cases that are biased that have been shown to arise in algorithmic systems. Factor cards, these are factor that go into decision making, such as what kind of fairness measure are you going to use in your particular context. Exercise cards, they're some queues, some ideas for how to interact with these systems.

An example, a card around analyzing a decision so using the exercise card, it prompted to choose an example card for instance can an algorithm be racist, an example, describing particular case where doing searches on Google here showed up -- ended up showing results that tended to be mostly African or African being here as well as a search for three black men, you tend to get images indicating something like a gang culture or crime whereas if you do a search for three white men, you end up getting the pictures of successful business people or something like that. Indications of some kind of racism that appears to come out of the search engine.

Then the question is, what kind of data types do you think goes into this kind of a search engine algorithm that starts generating the racist result? Start thinking about official identity data, is that something that may play a role, how about Internet search data by the Internet search, is that something that may be used by the search engine to guide the direction in which these results come out? How about financial records? Is that something. If an organization, it is a private sector organization, so they may have a value of power that goes into it, they want to be bigger than their competitors. They may have certain economic values on this particular set.

Perhaps there is also a sense in the company, is that something, yes, no, start thinking about debating with the colleagues, to what extent may these values is, are the values, is it something that went into the way in which the algorithm was built and therefore the way in which produces results.

Then factors, for instance, it shows a kind of racism apparently so a bias and prejudice may go into this kind of system, where may that come from. Again, sort of queues for thinking about how the system is billing period and what it is that may lead it to generating these kinds of results and then discuss would you have trust in this kind of system? What kind of changes do you think would lead to more trustworthy kind of systems? That's really the basic idea, it is that the cards provide a guidance to start thinking about and discussing with the group around questions of how do the systems actually lead to the kind of results that we're seeing.

The over two sets of cards, an example of those, for instance, in the process cards, you may find things like algorithms for political campaigning, what kind of data would you want to do if you were doing political campaigning algorithm, et cetera, and again, you can go through that with your group and use the cards as a reference to start thinking about and discussing, and then this is an example of a consumer rights card.

So what I would like to do now, it is to start engaging with you around exploring the use of these and so we have got 7 decks of the cards in the rooms I suggest, you get together around one of these decks and start taking them south so that we can try running one of these example trials together.

Can we -- can everybody who has a deck of cards with them just hold it up so that everybody else can see that there is one there.

One here, one towards the back, one in that corner, one over here, another one there.

So perhaps people can try to come together around one of these sets of cards so that you can start using them together, having a look at them together and trying to run one of these.

If we have too many people, I have one more deck here that I can be happy to join with.

You can start by having a look at it and we'll try some simple starter games together.

(Group activity)

If you have a quick look at the cards, you can see, they're all based on providing some -- there is potentially a -- they're text heavy if you think of the target audience being children, that's a comment that was given to us. I'm assuming that you're all fine with a bit of text on a card.

So as you can see, we have various types of cards here. Some of them giving more of an explanation around a particular thing, like the exercise cards, the example cards, some of them just being headings like within the different types of data that may be collected and within that.

So the first game, for instance, introductory game that one could try playing with this, it is focused around just the data cards.

Really -- so each group should have a facilitator who will ask the people in your group what kind of a platform do you tend to use a lot when you go online? What do you tend to use that for.

If you as a group decide which platform it is that you want to be thinking about for this exercise, so I'll give you a minute, and then basically the idea is to start -- you take just the data cards for this one, you share them within your group and then basically you put out the data cards and discuss how this data, this kind of data play as role on the platform you tend to interact with and what you're doing with that.

Just as a way to start you thinking about what kind of data goes into this..

I'll give everybody a couple of minutes to try this out.

Then I'll ask just what do you think.

For this one, it really depends how competitive you want to play it! I would say within the group you can share the data cards and collectively talk about this data card, how do we think it relates to the platform that we're using, does it or doesn't it.


>> (Audience member speaking without a microphone)

Elect someone in the group to be a facilitator.

Each group can elect a person to facilitate things.

>> (Audience speaking without a microphone)

>> AUDIENCE: I would like to ask the question -- a question, which Internet webpage do you use most? I'm the facilitator for all or for this small group only.

For all! That's sort of the way -- you have chosen a facilitator in the group, you're facilitator of the group, you ask the group which kind of website, platform do we want to run this task around. You get -- as a group you can decide or if there is no consensus coming forward, the facilitator takes the lead on that.

>> Okay.

>> AUDIENCE: Do I need the microphone or --

>> No. No. It’s up to you whether you want to --

>> The point is, you can take any platform, you can say okay, we're going to use Google, Facebook, we can use -- yeah.

>> You choose what platform you're -- or Amazon, whatever..

To give you an idea, an example, let's say you pick the platform, it is okay.

I'll use Amazon.

Now you start looking at the data cards. You say, okay, what data would the Amazon want to know about me.

That's what was provided.

>> For this one, yeah. Yeah.

Can I borrow data cards? Just one? Sure.

So if we take the example that we sort of suggested around amazon as the platform for instance, how may the official -- what data would the amazon want from you? Why would they want official identity data or not may be a question.

Can I pull your attention back out from the engaging discussions that you all are having. This is a simple kind of introductory thing using only the data cards. As one of the groups already mentioned, one of the things, for instance, you can think of if this is being done in a classroom setting, my -- it might be that you have gone through this, you have decided, okay, this platform that we're all using, amazon, it was the example that some of you work with now, and as a group, as we played this game, we didn't really understand -- we have this idea that maybe they're using this kind of data, we don't really understand why or how they're doing this and as a class, you may collect these and say, okay, this is something for the next project that we'll try to do. Try to investigate, how does -- what kind of information can we find online around why Amazon uses X and Y data around us so it is the trigger of the next piece that the class will do.

Let's ask you perhaps if you have a particular kind of question from this initial exercise that we did, a question about what it meant, something that it raised in you, a complete confusion around what the whole point is of these cards.

Please, if anyone has -- otherwise, I'll move on to the next game.

>> Or a comment about something that surprised you.

>> AUDIENCE: Should I stand?

>> Whatever you like.

>> AUDIENCE: Maybe I'll stand. Too much sitting is not good.

When we were -- let's say playing this game, there was a gradual let's say sensation -- I would say sensation, but a realization, that all of this kind of information, when you really contextualize it, you don't really think that much about it. We have had a Facebook as an example, probably some others, they may have had that one too. We realized that the stand of the information that they really have, usually you realize that some of the information is there, but for some reason, maybe because of the lack of time, maybe because you don't think about it, you don't really realize just how much information Facebook really has on you, how much data they really have on you, and especially concerning that you are also kind of facilitating that if you're active on Facebook, pretty much you can have a conclusion, you can conclude that you pretty much don't have a private life. There is not much left unless your deepest secrets. This was one of the -- one of my takeaways from that little game that we had. Thank you.

>> Thank you very much.

I think that would be a nice example of something that could come out of the fact that you're doing it as a group exercise helps to -- it brings up new questions that you really didn't know to ask yourself first.

To a large extent, I think from our perspective, literacy in in kind of space is about taking a critical approach to it. To asking what is this actually that I'm doing instead of taking a non-critical, whatever is convenient, I'll run with that, it is just magic any way kind of approach.

Any other particular things that people would like to bring up now?

>> Can we take some of these cards.

>> Yes. Actually.

How would you -- are they available somewhere? Yes, the cards -- they'll have a slide at the end with a link to the website where you will find a link to a place where you can buy them at cost. Also the ones that were shared out today, they are for people to get people to take with them.

Don't fight over them.

I really like the idea of bringing cards.

I see from an educational background, it is more of a collaborative approach of using this critical lenses, but also you sometimes need a push, you get the push from the partner or teammate to think or to come up with these things.

We were discussing Amazon and also there was a bit, very interesting, the politics, the beliefs, the criminal records, so we were trying to think, okay, where do these things happen? It is like when you talk about the data privacy, the data of -- the use of the Internet and as a digital citizen, we produce a lot of data. Also we're from an educational context, we're the producers, also the learner ourselves, but a lot of people, I mean, don't even know or even don't care about their data itself, when you put this political belief, which is mentioned in this card, so we had thought about the academia and also the generalism, that they produce these contents, but these contents, they're sometimes used against them. I mean, when you go, you challenge a powerful government, institution, also they have your records and everything, so they put that on you again. It is a kind of conflict -- they provision the data and protect the data and then giving the data in other hands to be used against you.

>> Anything you say may be used against you in a court of law. I heard that somewhere.

>> Let's have a look at this -- this game comes under the header of a starter class, what he just said. If we look at an intermediate one for instance which starts using more of the actual deck of cards, so this is one where you start off using one of the example cards, blue -- are they blue? Yes. One of the blue ones.

No, purple. Yeah. It actually has a color code here on the slide.

So you choose one of the example cards, and you read it out within your group so that everybody knows what the example is. You shuffle the other cards, and you deal them between one another.

Then basically you take turns to put one of the cards and try to explain to the group how this card, how this value or data kind of card relates to the example that was being shown.

So the cards that you need to be shuffling, dealing out between each other, they're the data cards, the factor cards, the rights cards, the value cards.

So to reiterate, this game starts within the group, you pick one example card which you're going to put out and read out to the group so that everybody knows what the example is that you're going to use as a reference within this game.


So it looks like every group has chosen an example that you want to be working with. Clearly just reading the example itself has already led to quite a bit of discussion, which is good.

So then moving beyond just looking at the example, the way to -- the way this kind of tool may help to facilitate further thinking around it, it is that you then take the data, the factors, rights, values, decks, that's the dark blue, light bluish, green, yellow decks, you would shuffle them and deal between each of you in the group and then you go one by one and every person whose turn it is, you have to take a card and explain to the rest of the group why this particular card, how this one relates to the particular example and then the group can decide -- you have this explanation, it makes sense, and if you decide to make it competitive, you can give points on how good the explanation was or you can say we're just here to learn so the explanation, it is the only thing we need to say.

(Group activity)

>> I think this particular discussion and game could probably go on for a while. We only have a limited amount of time today. I'm going to move on.

Does anybody have any observations they would like to share from having gone through this for a little bit now, played with this -- with other parts of the decks of the cards and this kind of an interaction, the particular example with a particular sample or issues, do you have any observations regarding your interaction that you feel you would like to share throughout this.

>> AUDIENCE: Are you asking for --

>> I'm asking the group. Any observations regarding this game for instance that you would like to share something that came up unexpected or otherwise. I have an observation that basically there is a lot of -- we had an example of -- the observation was, basically there could be so many data, but the developer, of the algorithm, they're basically the one that takes what the data will present. I think there should be -- I don't know, maybe there is. An ethical code of which data can be used and how it is valued. In the end, for the results, not to become discriminate or negative. Sometimes I don't think people really see the results that are more meg active is the response. Yes.

>> Thank you. Yes. That's one of the things that can come out of the askings that you say we actually feel there should be some kind of a global national, regional, whatever, kind of guidelines, consensus around what should be done, how things should be done, and that then connects to, as we know, the ongoing discussions around the governance of AI systems, such as the European Commission's high-level Expert Group, the OECD round on discussing around these questions. Various different ones that are currently ongoing.

Yes, please.

>> AUDIENCE: (Speaking too far from microphone)

How much does my data cost, I work with young people. I would play this game with them. This is a question I would like to put them to answer themselves. Why do they want to have my data? How much is the experiment, how much does that cost? How much am I worth? How much is my email worth? I used to work with creating a database and email and it cost up to 20 euro, so I used to run out of money.

>> That's very interesting.

I wonder if that's easily doable coming up with some kind of a price list to share -- it is something that would be a nice project to ask people, how might one go about finding what the cost is or you can have a discussion about how the cost is actually -- the value is being generated. You know, what is this bidding process for data that starts to arise. Those kinds of questions.

We do, I think, have a process card around the algorithm which goes a bit in that direction of be the algorithm that's trying to do some marketing to people, and then it asks you to look at the data cards, what kind of data would you want to have in order to do marketing, the values that go into -- if you are somebody who is building this, what kind of values do you have when you're building that kind of an algorithm.

It doesn't directly answer the interesting questions that you raise. It actually probably points towards raising that same question.

We are actually already in the last half hour of the session.


>> AUDIENCE: I don't know if this works or not. I was going to say that whatever is the price that you cost, it is a heck of a lot. When you look at those companies that actually make money out of that data, it is a multibillion dollar worldwide business and they're among the most powerful companies in the world now. You're obviously worth a lot.

>> So yes. One of the other kinds of games, this is something that focuses around the process cards is this kind of set of the algorithms. You start trying to think if I was an algorithm that was tasked with doing a certain kind of thing, what would I need, what kind of values would I be built on, how would I go about this.

The idea behind this game, it is in the advanced class, it is that you start off with the values, with the process card, the values cards, so you read out the process card amongst each other, for instance, you could have the process like spreading of a rumor, and values, such as tradition and commerce and then you say if this is the process, these are the values, what might be the issue here. You could say, okay, the theme of this session is going to be millennials are killing the music industry. That’s going to be the theme for this particular round that you're going to play. You then start looking at creating your own algorithm around this theme. You choose data cards and lay them out in front of you and then you take turns to choose a data card and think about them, how they may help to create an algorithm for the theme and then lay that out in front of the selected cards. Once you have gone through the data cards, that create sort of the theme -- around the theme of the process then you start looking at right cards, what kind of rights might actually play a role in this kind of a theme. If you're being this algorithm, what are the rights that you need to be careful not to violate or are the rights that you may actually be able to boost to reinforce with your algorithm.

What kind of value Ma I you go into in building this kind of system.

It is really about thinking around an algorithm within a kind of theme and then how might we build that if we think of ourselves as either being the algorithm yourself or people building the algorithm. It may be about getting a feeling as to the fact that these are not constructs that came out of nowhere, they're things that have been built by people within organizations and it is really the human, social construct is the creation of the algorithms we're interacting with.

So trying to get a feel for understanding when you're building this, what are the things that go into that. The idea is to start with a process card and two value cards in order to generate the theme of what this particular algorithm that you're going to build is all about.

>> AUDIENCE: I have a question actually.

Would this be the sort of exercise that in advertising -- an advertising agency may go through if it says we're going to be selling for whatever customer, we're going to promote this product, would they be going through an exercise like this, saying how do we now build the image of this product?

>> Yes. I would imagine so that they may be doing this kind of this inning with slightly different mindsets as to they're just trying to find out how do I maximize the selling, whereas we're in the mindset of how do I maximize my understanding of how these things work.

>> Hello.

We're actually running a bit low on time. Hello.

May I have your attention, please.

Hello. We're almost running out of time. You have not had time to get into the third game and explore it in depth, I would like to ask again whether you have any observations on this or because we're in the final Q&A about it, we only have 5 minutes for that, any other observations you may have suggestions for changing it, you can do that yourself or you can try to work on that or other kinds of ideas around gym literacy raising, this doesn't have to be related to this particular tool. Anything I want to bring tore ward for discussion. You can start with an observation of the last game you played, for instance.

>> AUDIENCE: Thank you. Very interesting game. Very good. At the same time, people we here, we're more or less informed, we're the audience who actually either knew it or identified novelties maybe.

How do you reach to wider audiences, say school children, elderly people that do use Facebook maybe, amazon as well and they don't really know how it would, they're surprised that all of a sudden Google suggestion suggests advertising or Facebook -- for them to have informed choices. It is very important. How do you reach out with this game or with any other resources that would be interesting to you.

>> Thank you. Yes. That's very important. That's one of the things that's very close to our heart as well.

So one of the reasons why we chose this kind of a medium for the tool is because it is low cost. We tried this out with some school teaches and they said we need a bit more of guidance to help you get going with these things and that's why we developed the additional booklet around that.

The hope is that this is a low threshold kind of tool that can be picked up by schools and that teachers can use when they're being asked to do something on digital literacy but they don't have any good tools around that and starting from scratch is very difficult. There are a lot of other populations that are not in this institutional setting like schools, so we're also trying to bring this to groups and organizations that can do this.

One of the things that we're very conscience of, this being an outcome of a research project, the risk dying as soon as the funding for the research project is gone. That's a reason why we're currently engaging with other organizations to try to see who can pick this up and you can basically use it from creative comments, use it as you wish.

Our current research project, which is more on trust in the algorithmic systems, we're engaging with 65 plus group to see what their particular concerns are and potentially do something similar for that.

>> How did you decide the values, the factors, is there a reference framework that you decided to include these values and factors?

>> The factors, they're very much around the questions around bias and fairness, that's a particular theme of the research project.

The values, this came to a large extent out of the discussion also with the young people, the coproduction process, obviously some of them were sort of injected from us saying these are some key things you need to think about, others came out of that as well.

And beyond that, the details of this, I would have to check with the particular team that actually worked on that..

Any other questions, perhaps you have some other educational tool around in this space that you would like to bring to the attention of people?

>> AUDIENCE: Thank you.

Basically I'm working on two projects right now currently, the one deals with the digital competence, the E.U. framework that's been in -- digital competence, it is actually about -- we have this tool, digital competency wheel, which measures the level of digital competence, of how competent you are and those are -- the framework describes certain competencies which are made of sub competencies including everything, it gives you the whole result.

So the other project, it is about -- it is with the volunteer organizations, we're working on the digital literacy program and we have this Digital Grassroots, plus the digital leadership program in which we are -- we are focusing on the African countries and the North American countries to engage youth actually to take part in that program, and then they read and they go through all of these things. Actually that's kind of when you talk about the Internet, all of these things, right now, in the Global South, we have different focus and into the rest of the world we have a different focus because of the accessibility and the use of these kinds ever things.

What actually we're now trying to do, it is maybe in this next month, it is that we're trying to merge those together and I found your idea very interesting that -- of how we can make it. So to combine the platforms together, that on one side you educate people, you bring people on the platform, you educate them, they learn, and then also you evaluate or assess their competence level on a large scale.

That's the basic idea.

Also you mentioned the course of Finland, I come from Finland.

Finland has the -- the design would be the same AI program that we have, and also ICANN and other organizations have ISOC, also they have designed those platforms or programs where you can go, you can take the course, you can also assess yourself and also you can come to these things. Maybe I'll approach you in the future forgetting some suggestions and recommendations from the work as well and how we can work those together.

That's the basic idea that we want to merge -- we want to bridge the gap between the digital device that's been created. We don't talk about the accessibility, we talk about all the things together so that's the basic idea of our project. Yeah.

>> Thank you for sharing. Sounds like a very interesting, it worthwhile project and I would be more than happy to talk with you about how perhaps we can share some of our experiences to help along with that.

>> AUDIENCE: First of all, thank you very much for this, it is really useful. We also are running workshops in schools from 12 it to 19-year-olds and this would be an excellent tool when it comes to bias and prejudice and stereotypes to sort of get them thinking about it, it really sort of enhances critical thinking which I think is one of the difficulties. I'm -- that's really good. I wanted to ask you something and now I have forgotten and I think we're also running out of time.

Thank you.

>> Thank you.

I'm sure the question will come back to you later!.

>> One more thing, can we have -- about the card games, I want to give the floor to people, how -- do we have -- you can build it on yourself as well, right now, it is a specific thing that you can just build the project on in a quick way when you have a good team.

But if you have to design it for ourselves, like for our different projects, is that also possible to make such a deep kind of thing? If you see the cards, there are a lot of things on there.

>> So we're happy to share with you our experiences of how we went through this process and actually -- yeah.

It was a two-stage process. One part of it, it was more of a youth engagement kind of workshops, keeping young people in the room, presenting them with some initial scenarios to start discussions, and really try to have a youth-led discussion going on to hear from them what their views are as opposed to telling them what they should be concerned about. We do provide on our website also links to the way in which we ran those sessions if that's something that you're interested in.

Then doing a qualitative data analysis of the recordings of the discussions during that, which can be a lengthy intensive process of work.

From that, identify the key themes that arose it and use those themes then to drive the construction of this particular toolkit, yes, of course, was targeted by systems in our case.

So you generate the unique sessions, et cetera, around different kinds of trigger examples that are fitting with the particular theme that you want to be going with.

Is there anybody else? Questions, issues?

We are actually already 5 minutes over time.

Thank you very much, everyone! .

The decks of cards that are out there, they're there for you to take with you. Please negotiate among each other who gets to have the deck of cards. Here is the website on how to get -- order some.

Thank you.

This text, document, or file is based on live transcription. Communication Access Realtime Translation (CART), captioning, and/or live transcription are provided in order to facilitate communication accessibility and may not be a totally verbatim record of the proceedings. This text, document, or file is not to be distributed or used in any way that may violate copyright law.