GAE - How to make a Free(Cheap) application

If you are paying huge or frustrating scaling out on your Web Application.
Looking for "Cloud Solution", and your Team wish to give a try on Google App Engine.
Wait... don't. not until you have learn the following things.

What mistakes I made in the past 3 years with GAE (worshiplace, BioCRF) have given me a lot to share. Maybe there are more coming lol.

1. De-normalise datastore (ndb)

"JOIN" is never gonna work in Google NoSQL Datastore - ndb.
In theory, "JOIN" will cost you (in terms of money/quota) a lot. What could you do about it? de-normalise your table. Never do a second lookup will save you a lot(its depends, for BioCRF report generation save ~50% db read).
Or you could choose Google Cloud SQL. It is not free, we will discuss that in the next post on "COST"

2. Take advantage on Backend Instances Hours

I won't use frontend instance for serval things. e.g. report task, email task. Don't forget there are limitation on request timeout (60s) in frontend instance.

3. IDE/Text editor: VSC > Sublime, PyCharm > Eclipse

When dealing with GAE, you cannot debug easily without "plugin". VSC is free and GAE support is good enough. In the past, I did struggle with Sublime.... forget about it.
PyCharm is amazing, no doubt. Spend on it if you have resource.

4. Cursor and pagination

ndb do not support "offset" SQL way to fetch page by page. (It does support, but cost you fortune, forget about offset is my only advice).
You need to use cursor fetch page by page. But then you will have to figure out by yourself how many entries in the table to tell "the number of page".  You can get around it by creating a ndb.model and store that information.

5. Memcache

Cache is a major solider to guard your quota. Don't just keep fetching without reusing them.

6. NDB Transaction limitation

When Transaction come in, you will starting thinking how many request per second. But in GAE, you need to think more than that.

• You are not allow to query inside transaction. It only allows ndb operation when key is provided. e.g. ndb.Key(urlsafe="key").get()
• Maximum touching(read/write) 25 entities when dealing with more than single ndb.Model

7. Maximise your DB read & write quota

You need to enable appstats to look for improvement to save more on DB ops.
Your system design should always consider finding entry by ndb.Key(urlsafe="key") instead of using .filter and .fetch(limit=1). The urlsafed key is always allow to pass between cliendside and serverside. You could do an encryption layer to prevent leaking out unsafe properties.

8. Deployment - UAT & PRD

This is the worst I came across. There are nothing on separating UAT & PRD datastore in a single application. That mean you have to config a UAT on your own. However, you could export the existing data to Google Bucket and restore that manually.
For small project that you don't care polluting production data, you could use "Version" on each "Service" to get yourself a UAT env.

9. Async your RPC

Most RPC such as ndb and memcache are provided with async, I didn't try on it yet, I hope there are resource in future to tackle larger user base.

10. Monitoring

GAE provide stackdriver. I am very happy that I don't have to config a monit.


I will talk about why I choose GAE in the next post.