So I found the solution. Due to a test during another problem, I put the port to "no route-only". When I turned it to "route-only" all problems were gone. There was no need for l2-switching on that port anymore. I'm still curious why it drops the packets it in layer 2, but at least the problem is solved. Thanks for all the other answers. :)