Quảng Cáo

Cpm Affiliation : the cpm advertising network

MongoDB Datastore Stub for Google App Engine SDK (Python)

Category: Written by Nam Phạm Ngọc Quang / 03:57

mongoDB

AppEngine Datastore Stub using MongoDB database as a backend. This is a replacement for App Engine's default DatastoreFileStub and DatastoreSQLiteStub. Slightly inspired by Mike Dirolf's and Tobias Rodabel's Mongo Appengine Connector (thx).
Features:
  • provides much faster queries than DatastoreFileStub (10-100 times faster filtering, 5x faster ordering, depends on dataset size)
  • provides much faster inserts than DatastoreSqliteStub (2-3 times faster on large datasets)
  • compatibility with SDK 1.7.7
  • support for devappserver2
  • supported features of ndb like structured properties, query projection etc.
  • optimized for highest performance (safe=False for pymongo < 2.4, write concern w=0 for pymongo >= 2.4).
  • tested against behaviour of DatastoreFileStub.
Dependencies:
  • python 2.7 (tested on 2.7.3 [GCC 4.7.0 20120507 (Red Hat 4.7.0-5)])
  • pymongo >= 2.0 (for versions < 2.0 not tested)
  • mongodb >= 2.0

Install (Linux)

This way you can install the stub directly into your SDK. Warning: some of the files of the SDK will be overwriten (patched). Since SDK 1.7.6 introduces devappserver2, which supports DatastoreSqliteStub only, there's no --use_sqlite option in dev_appserver. Patching will overwrite hard-wired sqlite stub into mongodb stub. (For more info, see patch file).
  1. Extract downloaded zip archive (or clone this repo). Enter the appengine-datastore-mongodb-stub dir.
    $ unzip appengine-datastore-mongodb-stub.zip
    $ cd appengine-datastore-mongodb-stub
    
  2. Install the stub.
    $ sh install.sh /PATH/TO/YOUR/APPENGINE/SDK
    

Install (Windows)

For installation on Windows you will have to
  1. Install patch utility (http://gnuwin32.sourceforge.net/packages/patch.htm)
  2. Run -> cmd (win command line)
  3. Add patch to windows PATH, most commonly the path to patch is "C:\Program Files\GnuWin32\bin\", so:
    > PATH=%PATH%;"C:\Program Files\GnuWin32\bin\"
    
  4. Cd to appengine-datastore-mongodb-stub directory. Run install.bat with first param as path to google_appengine SDK, most commonly "C:\Program Files\Google\google_appengine\".
    > install.bat "C:\Program Files\Google\google_appengine\"
    

Usage

devappserver2:
You may start your dev_appserver now. This way your data will be stored in mongodb databse named after your app ID. For example:
$ python ./google_appengine/dev_appserver.py $PROJECT_DIR
old_dev_appserver:
If you are still using an old development server, you may start it now. Using --use_mongodb flag your data will be stored in mongodb databse named after your app ID. For example:
$ python ./google_appengine/old_dev_appserver.py --use_mongodb $PROJECT_DIR

Usage in tests

import unittest
import os

from google.appengine.api import apiproxy_stub_map
from google.appengine.api.memcache import memcache_stub

from datastore_mongodb_stub import DatastoreMongoDBStub

APP_ID = 'test'

class MyTests(unittest.TestCase):
    @classmethod
    def setUpClass(cls):
        """
        Set up SDK testing environment
        """
        os.environ['APPLICATION_ID'] = APP_ID
        apiproxy_stub_map.apiproxy = apiproxy_stub_map.APIProxyStubMap()
        # memcache stub
        cache_stub = memcache_stub.MemcacheServiceStub()
        apiproxy_stub_map.apiproxy.RegisterStub('memcache', cache_stub)
        # datastore stub
        datastore_stub = DatastoreMongoDBStub(APP_ID,
                                              require_indexes=False,
                                              mongodb_host='localhost',
                                              mongodb_port=27017)
        # we can now edit pymongo.MongoClient's write_concern to use journaling
        # this option is only for pymongo version >= 2.4
        datastore_stub._mongods.write_concern['j'] = True
        apiproxy_stub_map.apiproxy.RegisterStub('datastore_v3', datastore_stub)
Maybe someday here will be patch for testbed if needed.

Notes

  • Tested only on ndb (google.appengine.ext.ndb).
  • Missing tests for threaded environment.
  • Query projection on multiple repeated properties not supported.
Thank for sharing!

About The Author


NamPNQ

Không biết viết gì ở đây :D

Quảng cáo