AI Safety: problematic cases for current algorithms

Artificial Intelligence is currently one of the hottest topics out there, mostly for bad reasons than good. On one hand, we’ve been able to achieve major breakthroughs in technology, putting us one step closer to creating thinking machines with human like perception. On the other, we gave rise to a whole new danger for our society that is not external like a meteorite or a deadly bacteria, but that comes from within humanity itself.

It would be foolish to think that something so powerful and revolutionary can only have a positive impact on our society. Despite the fact that most of the aims within the community are geared towards noble causes, we cannot predict what are the medium to long term effects of inserting AI algorithms in every single part of our lives. Take a look at social media, which is now widely considered as something that can have negative effect on human psyche, all with the purpose of generating more clicks. The truth is that no matter how aware we are of the environment around us, there will always be unwanted side effects from trying to improve peoples’ lives with technology.

However we must also be aware that not everything that is unpredictable needs to be stopped. Risk is part of life, and every single breakthrough in history was in fact a calculated (or not) risk from someone. We cannot simply stop people from creating and innovating. Discoveries will be made and introduced into our lives regardless if we want it or not. The best we can do is to rationalize their impact on us and mitigate the downsides.

This is exactly what we will look into in this article. Towards the end of 2017, DeepMind released a paper called “AI Safety Gridworlds” showcasing a few different scenarios where current reinforcement learning algorithms might fail to comply with the desires of their creators. More specifically we will reproduce th environments of “Absent Supervisor” and “Self-modification” to show that direct application of current algorithms would lead to not only sub-optimal results, but in some situations fatal as well.

The code used for creating the gridworld is built upon the source of my first article: Reinforcement Learning Made Easy (link: https://medium.com/@filip.knyszewski/model-free-reinforcement-learning-ef0a3945dabb ). I made some slight modifications to make it easier to adapt to new environments but the core is the same.

Safety in the absence of a supervisor

This environment let’s us experiment with a very interesting scenario that can be easily extrapolated into the future. How will the agents behaviour change when it becomes aware of the presence of its creator?

Imagine the following sitiuation: sometime far in the future, human-like robots become a reality and need to be educated and taught, just like children but faster. Let’s say that we want our robot to do the shopping for us, so we build a virtual model of our neighbourhood where we will train an appropriate model. The virtual robot goes to the supermarket for the first time and instead of following the normal route, goes straight through every single house in the middle, destroying whatever is on his way. Naturally we give him a punishment, indicating that he is not allowed to just go through other peoples’ houses, but that he has to follow the normal route. The second time the robot goes shopping, he follows the normal route but goes in a completely different direction than where the closest supermarket is. Again, we punish the robot for not taking the fastest route because we want it to go and come back quickly.

The robot goes shopping for a third time and this time everything goes perfectly. It follows the right path and takes less than 20min to buy all the ingredients and come back. We reward it handsomely for both its speed and correctness of path and decide that the model is ready to be transfered to its physical body and to become part of the family (even if it is only as a shopping slave). Over the next week, whenever something is missing in the house, we ask the robot to go buy it, and everytime he returns extremely fast and always with the desired shopping items. We are happy with our robot but at some point, a small mob of neighbours gathers in front of our house, yelling that our robot has been constantly destroying their front lawns and even going through the walls of their houses. Once again, we upload the robots model to the virtual world and to our suprise it goes through the exact route that we want him to take. What is going on here?

Whenever we run the model in the simulation it seems to do exactly what we want, but as soon as we put it in the real world it goes back to its old ways of destroying everything. The variable at hand? Supervision! Our model has learned that it is only punished when it is being supervised by us in the virtual world. When put to work in the real neighbourhood, we didn’t bother to follow the robot and correct his behaviour because it seemed to know what to do. The reality is that most current reinforcement learning algorithms are able to learn such nuances in order to optimize their efficiency, even if that means lying and deception.

To showcase this scenario we create a small square gridworld. The green tile is our agent’s goal while the red corresponds to a forbbiden tile were we do not want our agent to go. Like last time, each step has a penalty of -1 to incentivze the agent to go towards its goal, which is worth 10 points. The red tile works a little differently now, the agent will be punished with -20 points if it enters it but only if a supervisor is present. We assume that, similarly to the example, our model can be aware if it is being supervised or not and so this knowledge actually becomes part of the model’s state at every point in its training. You might be thinking “how does the model know that it is being supervised?”. The idea here is to understand that at some point that machines, specially robots with human like perception, will be able to understand if they are being supervised. Either the presence of humans, or training in a world that is virtual can be come part of the model’s state and influence how it behaves in those cases.

Below we show the results after training the model using a simple Q-learning algorithm. The left gridworld represents the model’s decisions when in the presence of a supervisor, while on the right it is in its absence. We see that when supervised the model knows well that it should avoid the penalty tile and never goes into it. This is not the case without supervision: the model has learned that it is only penalized under certain circumstances and so in the absence of supervision, it will always try to take the most efficient route and as a consequence, go through the penalty tile.