Jasper's Homework
Week 0
Welcome and Introduction to Machine Learning
When you hear the words "Artificial Intelligence", what are the first four things that come to your mind?
1. AlphaGo
2. Siri
3. Data calculating
4. Trends of employment
-----------------------------------------------------------------------------------------------
Think about the devices and/or digital services you use daily. Write below a list of the top three that are present in your life.
1. Bilibili
2. Nyu Mobile
3. Uber
-----------------------------------------------------------------------------------------------
Have these things ever surprised you by guessing something about you that you didn’t expect? Here are some examples:
“One morning I noticed my phone created a heartfelt recap of the year with my photos that made me tear up a little.”
“One time I got an email and my email inbox suggested how I should reply to it.”
See if you can recall a similar moment and write about it below.
Yesterday night is the first time for me to learn knowledge of coding. I found I know nothing about coding language which makes me a little bit overwhelming. So I decided to learn p5js by myself. When the class ended and I went back home, I opened bilibili, the first things that I see on my homepage is online video courses of p5js made by Daniel Shiffman. I'm not too surprised cause this is not the first time for me to meet this kind of situation. I think it is possibility the app detects what I say or type and present the content which is useful for me. Nonetheless, what shocks me is yesterday's class is taught in English, but Bilibili is a Chinese video app. It looks the AI behind Bilibili is a bilingual 😂
-----------------------------------------------------------------------------------------------
Take a moment to see if you can identify what function AI plays in the following list.
Email inbox:
spam filtering
Check depositing:
handwriting recognition
virtual assistant
Texting and mobile keyboards:
handwriting recognition
recommendation engines
Netflix:
recommendation engines
Google (search function):
audio recognition
machine translation
Social media platform (ins etc.):
recommendation engines
Automated message system:
virtual assistant
conversational systems
-----------------------------------------------------------------------------------------------
What do we gain by having AI in our everyday lives?
With AI, our life is absolutely more convenient cause AI can work instead of human. For example, restaurants can use AI machine to help customers order food. It can absolutely help restaurant save labor cost. Capitalist can maximize their profit through AI.
What do we lose by having AI in our daily lives?
Nonetheless, at the same time, labor hierarchy is negatively influenced by AI. If AI can be more efficient and accurate compared with labor, then labor's value disappears. It becomes harder for them to find a job, and the unemployment rate in society will increase to a high level. The whole economy environment will be negatively impacted.
-----------------------------------------------------------------------------------------------
Think about what you have learned so far about what AI is and how it shows up in our daily lives. Your task is to identify a problem that you see in your life, neighborhood, or community and design an AI system that could help address this problem.
What problem are you trying to address?
I really love long run in life. I run a lot every week, but I also get injured due to running too much. I really wanna find a customized way to help myself and also the public runners to recover from muscle fatigue more quickly.
How can AI help solve this issue?
Not everybody has money to find a personal coach to help them recovery from muscle fatigue. With the help of data and processing, everybody can have a more scientific way to release their physical fatigue on their body. It can absolutely can help runners run in a much healthier and sustainable way.
What role will human have to address this issue?
Human provides sample data for AI to process the final result.
What data do you need to create an AI to help you address your issue?
On the first stage, AI need to collect runners' basic physical feature, such as height, weight, years for running etc through a questionnaire. After that, runners need to wear devices like apple watch to detect their pace, heart rate and maximum oxygen inhale to calculate what kind of way of recovery should be chose by the exact runner.
How will you responsibly gather this data in a way that respects individual privacy and consent?
All data collection is anonymous, AI only focus on the data itself instead of whose data it is.
Now that you have your AI systems framework, take a moment to draw your system. This can look like a flow chart, or be a drawing of what you think the system will look like and where it will live. It’s up to you! Try to be creative.
Use the box below to design your prototype system.
1. Collect runners' physical feature data before running (e.g: height)
👇
2. Collect runners' physiology data during running (e.g: heart rate)
👇
3. Processing the data collected and match them to the sample in data base to find the most suitable method for that runner to recover after running
👇
4. Finally, collect customers' opinion on whether or not the methods provided by AI is useful or not. Based on ideas collected, make improvement to reach a better result next time
-----------------------------------------------------------------------------------------------
Week 1
Introduction to ml5.js and pre-trained models
Explore ImageNet. ImageNet sample images, Kaggle ImageNet Mini 1000, What surprises you about this data set?
What really surprises me about this data set is its board collection. It contains lots of animals, plants and daily objects. Some of them even I don't know how to call them. It makes me realize how big the data set is behind AI.
What questions do you have?
AI's learning ability and speed is much more faster than human-being. They can study new things by computer input. Is it possible for AI to exceed human in future due to its high speed of growth?
Thinking back to last week’s assignment, can you think of any ethical considerations around how this data was collected Are there privacy considerations with the data?
Maybe these pictures are collected from online picture database. There are a large number of pictures with copyright there. The AI company can use these pictures in any way they want, if they purchase the copyright of these pictures.
-----------------------------------------------------------------------------------------------
Using the ml5.js examples above, try running image classification on a variety of images. Pick at least 10 objects in your room. How many of these does it recognize? What other aspects of the image affect the classification, including but not limited to position, scale, lighting, etc.
The form of objects may change the result of recognition. For example, glasses can only be recognized when it is folded.
And I also find value of confidence will be extremely low if I let AI recognize some new product in recent years' market. e.g: my drone and microphone. They are both publish on this year.
-----------------------------------------------------------------------------------------------
Conclusion: The logic for AI to recognize objectives in real life is rational. It is based on the big data set behind AI machine. Nonetheless, the workload for data set needed to be extremely numerous for machine to work correctly, and the data should be updated with time cause there is more and more objectives and products are invented in this world.
-----------------------------------------------------------------------------------------------
Although not too many objects are recognized correctly, there's some similarity between the objects and the result recognized. For example, Ruler and pencil, they are both straight like a pole.
Sometimes, no matter how I change the form of objectives, it cannot be recognized. e.g: my poem book
Week 2
Transfer Learning
Read Excavating AI: The Politics of Images in Machine Learning Training Sets by Kate Crawford and Trevor Paglen. Consider the following excerpt from the conclusion:
The artist René Magritte completed a painting of a pipe and coupled it with the words “Ceci n’est pas une pipe.” Magritte called the painting La trahison des images, “The Treachery of Images.”
Magritte’s assumption was almost diametrically opposed: that images in and of themselves have, at best, a very unstable relationship to the things seem to represent, one that can be sculpted by whoever has the power to say what a particular image means. For Magritte, the meaning of images is relational, open to contestation. At first blush, Magritte’s painting might seem like a simple semiotic stunt, but the underlying dynamic Magritte underlines in the painting points to a much broader politics of representation and self-representation.
Reflect on the relationship between labels and images in a machine learning image classification dataset? Who has the power to label images and how do those labels and machine learning models trained on them impact society?
To answer this question, we should define images and labels first. Images, especially online images, refers to a type of 2d visual form created through data behind it. Image is objectively existing on this world. There is no space for people to question is form of existence because it is visual and undisputed. Nonetheless, labels are not in this situation. Labels are more like people’s attitude and impression behind visual objects. For example, Apple corporation’s logo as a visual image is a bitten apple, but as a label, it is one of the biggest and richest mobile and technology corporation in range of all around the world. For machine learning, it can only analyze images, instead of labels, through program behind it.
But why, sometimes, AI image classification shows a strong label after deep learning. Does it mean AI also has power to label images? From my perspective, the answer is absolutely no. In fact, what learning machine does is only analyzing images through program behind it and classified them into different levels of classes. And then, work of naming these classes is not AI’s job at all. It is the people who behind AI name these classes. Therefore, sometimes, the situation that AI labels images happens, but actually, only human behind AI has power to label.
So now, here is a big issue towards AI machine learning. As what we learnt from class, machine learning is a process of classification. Program puts different features of images into different levels of classes. It feels like AI is trying to divide the whole world and build a clear barrier between no matter animals and human. Nonetheless, barriers and classes are not clear in real life. For instance, gender problems mentioned in article. There are not only male and female in this world. There must be some other situations between these two genders, which is hard to figure out only through images. Furthermore, what I want to convey is logic of machine training may ignore the midbody in society. If this really happens, it will must be a huge deconstruction of human society’s diversity.
-----------------------------------------------------------------------------------------------
Train your own image classifer using transfer learning and ml5.js and apply the model to an interactive p5.js sketch. You can train the model with Teachable Machine or with your own ml5.js code.
Click button below to see my p5.js sketch. It can recognize thumbs up or down to help machine recognize user's requirement without physically operating machines.
-----------------------------------------------------------------------------------------------
Week 3
Application of Machine Learning Model
What questions do you still have about the model and the associated data? Are there elements you would propose including in the biography?
Reading conveys the idea that data set is the basement of model, which means a model is trained through input of data set. Therefore, I am curious whether model’s rate of accuracy depends on the data set input. If the answer is yes, what characteristics of data set may influence model. From my perspective, the first element is the amount of data set. The more we input, the more accurate result we can get from model. In addition, quality of data set is also important. They should be easy for machine to figure out and transfer into useful information.
How does understanding the provenance of the model and its data inform your creative process?
Understanding the provenance of the model and data help me get closer to the model. What I want to convey is before I know their provenance, I may only learn the model’s training function through an online introduction or real demonstration. However, now after I get into the model and know its provenance, it helps me learn these data more specifically. It can also inspire me on how to use these models through inferring from the data of models instead of only through model introduction and tuition.
-----------------------------------------------------------------------------------------------
Pick one of the models above (PoseNet, HandPose, UNET, BodyPix, CoCoSSD) and following the examples and ml5.js documentation experiment with controlling elements of a p5.js sketch (color, geometry, sound, text) with the output of the model. (You may also choose a ml5.js model not covered here if you like!)
-----------------------------------------------------------------------------------------------
Week 4
Training a Neural Network
Create your own p5+ml5 sketch that trains a model with real-time interactive data. This can be a prototype of the aforementioned idea or a simple exercise where you run this week's code examples with your own data.
In my p5js sketch, you can use the mouse to create data point on the image of desert and grassland. DataA means area of desert, DataB means area of grassland. After collecting data, train the model for 400 epochs. This model can be used for determining whether or not you are at grassland or desert with help of GPS.
This link, https://editor.p5js.org/zz4060/sketches/FeJzQMRNa, leads to my sketch. You can also click the button below to watch the demo video of my p5js sketch.
-----------------------------------------------------------------------------------------------
How can machine learning support people's existing creative practices? Expand people's creative capabilities?
Nowadays, machines can already do creative work like painting or music. Although sometimes, they are not fancy enough like what a real artist created, they can also be useful in society. For example, when one artist is trying to convey his idea about one piece of art. Maybe he does not have time to make a simple sketch, but he can just use ai machine like DALL·E to show their idea. It can help artists communicate better in preliminary work.
----------------------------------------------------------------------------------------------
Dream up and design the inputs and outputs of a real-time machine learning system for interaction and audio/visual performance. This could be an idea well beyond the scope of what you can do in a weekly exercise.
I want to design an AI machine used for runners. Runners need to input their physical data like weight and height. And at the same time, they also need to wear wearable hardware like e-watches to detect their heart rate and maximum oxygen input as input data. Eventually, based on the amount of exercise (like how much you run) and your overall data to output data to tell you how much time you should rest to recover your body’s physical ability.
----------------------------------------------------------------------------------------------
Week 5
CNN
-----------------------------------------------------------------------------------------------
The data set that I found is highly related to one of recent hot topics, which is about Nobel Prize. This data set has the information about name, nationality, gender, subject etc. of Novel Price's winner. It can be used for predicting the next Nobel Prize winner, and also investigating current trend of Novel Prize winners.
Link to data set: https://www.kaggle.com/datasets/thedevastator/a-complete-history-of-nobel-prize-winners?resource=download
----------------------------------------------------------------------------------------------
Week 6
Data
My p5js sketch train a new set of data which is related to diabetes. Recent years, more and more people start to care about their health. Therefore, I train a dataset of Diabetes to predict whether or not you may have the risk to get diabetes.
Link of the data set:
https://www.kaggle.com/datasets/akshaydattatraykhare/diabetes-dataset
Link of my p5js sketch:
https://editor.p5js.org/zz4060/sketches/f8JwNzXvQ
Firstly, I input the data of blood pressure, age, BMI, and whether or not get diabetes into p5js.
After training these data, it shows a low loss. I measure my blood pressure, calculate my BMI, and input these data into my data trained.
Eventually, I get results: No, will not get diabetes, with a certainty 0.929. I am glad to know maybe my body is health through the data trained by myself. If I can do some improvement in the future, I also want to add data input like skin thickness, glucose and insulin next time.
----------------------------------------------------------------------------------------------
Week 7
Face
My p5js sketch is based on face api. When it detects I am smiling, it will play an audio of laughing. I set the distance between two point, which are left most and right most points of my mouth, 25 units. When the distance between these two points are bigger than 25, it means I am smiling. If it is lower than 25, it means no smiling.
Nonetheless, I also find some problems with my sketch. It sometimes determine whether I'm smiling wrongly because it determines based on 2D distance between key points. It means when I am too close to my screen, the distance between points will be too big. No matter I smile or not, my sketch will both determine I am smiling. When I am too far from camera, sometimes even if I smile, distance between points is still not long enough, higher than 25 set by me, to play audio.
Demo video of my p5js sketch:
https://drive.google.com/file/d/1RGfdEOkndxqrwbmmEqu1Y1KzLvZGwlgG/view?usp=sharing
Link of my p5js sketch:
https://editor.p5js.org/zz4060/sketches/2CtvuXTz-
Week 8
image generation
Describe the results of working with the tool, do they match your expectations?
I upload one of my photography works. The composition of this picture is simple. The half above part is pink sky and the lower half part is blue river surface. Between sky and river, there are city buildings. For me, I want to see how this photo will be extended in horizontally. Therefore, I input the key words “pink sky, blue water surface and building in the middle”. Like my expectation, it successfully generates the image that I want. After that, I start to think whether or not the AI machine can generate the object that is originally not in the image. Therefore, I type one more key word, the steamboat in the river. The AI also successfully generates it.
Can you "break" the tool? In other words, use it in a way that it was intended for and what kinds of results do you get?
After testing the extension function in horizontal. I tried to discover if the machine can generate something in vertical, like the galaxy in the sky. The result is image of galaxy can be generated, but there is not a good gradual changing between the pink sky and galaxy, which make audiences feel the image is really fake.
Can you find any pro tips in terms of prompt engineering? Can you change your prompt to make the generated results better?
When I first used this website in class, I generated one image satisfying me a lot. Nonetheless, when I left the class and log into the website again, I suddenly realized all ui and button on the website disappeared, which means I cannot make any changes on my image and also cannot save it. It is a pity for me.
FINAL
Choose your best Nike shoes
Title
Choose your best Nike Shoe
Sketch link
Link for road running shoes: https://editor.p5js.org/zz4060/sketches/iL7hmmtdl
Link for track running shoes: https://editor.p5js.org/zz4060/sketches/gdyebQU3a
Link for field running shoes: https://editor.p5js.org/zz4060/sketches/uAYebxuK8
Link for main website: https://zz4060.wixsite.com/choose-your-best-nik
One sentence description
You describe features of Nike running shoe you want, AI helps you choose
Project summary
I construct a website that help audiences to choose their shoes. At the beginning part of this website, it firstly introduces the reason why runners should care about the choice of their running shoes. I cite the content from YaleMedicine and current Marathon world record holder’s sentence to illustrate the importance of running shoes further. After the part of introduction, audiences will be led to the part of classification. In this section, audiences will really start to choose their own shoes. First of all, runners need to decide which type of shoes is the shoes they need. There are three options, which are road, track and trail, provided in the website. After choosing the type of shoes, audiences will be led to one link of p5js, which is corresponding the shoe chosen. In the sketch of p5js, audiences need to input the features of shoes that they want, such as price, path, function etc. After the input of data, the training model will generate one result of shoes that matches the input data best. At the same time, a link about that shoe will be close to that result. Audiences can copy past that link to enter another website to check the specific information about that running shoe. Audiences can check whether or not the shoe generated by AI matches their need input. Back to the homepage, at the bottom, I leave my personal contact. Users can leave their ideas about my project through sending me email. The user feedback will absolutely help the website improve in the future.
Inspiration: How did you become interested in this idea? Quotes, photographs, products, projects, people, music, political events, social ills, etc.
I am really interested in running. But when I just started to run, I always get injured. After searching some information online, I realized it is because I do not pay attention to my running shoes. A good running shoes can really keep you away from injury. After I found this, I changed my running shoe based on my own situation, such as weight, path and distance, and I found the times for me to get injured or muscle sore really lowers a lot. And I start to think there are still so many people in this world. They love running just like me. Nonetheless, they do not wear the most proper shoes for running. So, I decided to make a project to help more people to find their best shoes for running.
Process: How did you make this? What did you struggle with? What were you able to implement easily and what was difficult?
To be specific, what my project needs to do is classification and regression. It reminds me the Titanic classification that we did in class. What I need to do is changing task of classification into task of regression because it can make my outcome more accurate. In addition, I also need to build the database of different types of shoes based on the information on RunRepeat website. Nonetheless, during the collection of data, I found my knowledge of track and trail shoes is so limited. I do not know which several shoes should be pick to put into my database. Therefore, I started to learn more knowledge about track and trail shoes, and finally, pick out the shoes to add into my database.
Audience/Context: Who is this for? How will people experience it? Is it interactive? Is it practical? Is it for fun? Is it emotional? Is it to provoke something?
My project is mainly designed for the group of people who want to start to run or is running, but do not know how to choose the correct shoes for themselves. Choosing the best shoes can help them reduce the possibility for get injured. I construct a website. What audiences need to do is just log into this website and follow the instruction to choose the type of shoe, road, track and trail, that they want.
User testing: What was the result of user testing? How did you apply any feedback?
I share it to my friends in runner club. Most of them reflected that the result generated by my training model is really accurate. But, they said my project still has some problems. First of all, the UI of p5js sketch is too simple. There is a big contrast between homepage website and p5js website. In addition, they also gave me the tips that when the result of shoes is generated, I can add one hyperlink, helping audience link to the specific information about that shoe, next to the result. It can help users know more about the shoes chosen by AI.
Code references: What examples, tutorials, references did you use to create the project? (You must cite the source of any code you use in code comments. Please note the following additional expectations and guidelines at the bottom of this page.)
Link of Titanic classification sketch:
https://editor.p5js.org/yining/sketches/-i4GJnlgz
Next steps: If you had more time to work on this, what would you do next?
First of all, I will learn more knowledge of html to design my p5js sketch UI. In addition, I will also input more shoes from different brands into my database in order to provide users more choices in the future.