We're building an application in the google cloud. We're using App Engine as frontend and Compute Engine as backend. On these Compute Engine instances I'm running a TCP server that accepts certain "command" messages. The Compute Engine instances are also connected to the internet behind a firewall and load balancer for incoming https traffic. I would like to bind the TCP server to a local IP addres and allow only App Engine instances (thes are within the same project) to connect to the TCP "commands" port via the socket api.

For now we're limiting the incoming connections to Google Ip's only

The Compute Engine documentation sais the following:

Every instance is a member of a single network. A network performs the same function that a router does in a home network: it describes the network range and gateway IP address, handles communication between instances, and serves as a gateway between instances and callers outside the network. A network is constrained to a single project; it cannot span projects. Any communication between instances in different networks, even within the same project, must be through external IP addresses. In the API, a network is represented by the Network object.

Would it be possible to build a secure communication setup? Perhaps with App Engine backend instances?

There isn't currently a way to setup a private network between GCE and GAE. Limiting to google's IP range isn't secure since anyone on GAE or GCE could connect to your server. So I recommend that you authenticate incoming connections at your TCP server to verify the connection is coming from your GAE instances.