Adventures in an declarative object-y thing backed by Git and using Elasticsearch as a query backend.
Note
Here be massive dragons.
fromelasticgitimportEGfromelasticgit.modelsimportModel, IntegerField, TextFieldworkspace=EG.workspace('/Users/sdehaan/Desktop/test-repo/') workspace.setup('Simon de Haan', '[email protected]') """# The model looks like thisclass Person(Model): age = IntegerField('The Age') name = TextField('The Name')"""fromelasticgit.tests.baseimportTestPersonasPersonperson1=Person({'age': 10, 'name': 'Foo'}) workspace.save(person1, 'Saving Person 1') person2=Person({'age': 20, 'name': 'Bar'}) workspace.save(person2, 'Saving Person 2') person3=Person({'age': 30, 'name': 'Baz'}) workspace.save(person3, 'Saving Person 3')Data is now persisted in a git repository and is queryable via elasticsearch:
>>>fromelasticgitimportEG>>>fromelasticgit.tests.baseimportTestPersonasPerson>>>workspace=EG.workspace('/Users/sdehaan/Desktop/test-repo/') >>>forpersoninworkspace.S(Person).filter(age__gte=20): ... printperson.name, person.age ... Bar20Baz30Check the examples/ directory for some more code samples.
$ python -m examples.basic_usage e6cb25f00870472fa5223d76dc361667 Baz 30 2bd470372243411c9abd8fdcb969dcf5 Bar 20We've followed the example of Apache Avro when it comes to schema evolution. Avro compatible schemas can be generated from the command line.
Model definitions can be rebuilt from Avro JSON schema files.
A sample model file:
classTestFallbackPerson(Model): age=IntegerField('The Age') name=TextField('The name', fallbacks=[ SingleFieldFallback('nick'), SingleFieldFallback('obsolete'), ]) nick=TextField('The nickname', required=False) obsolete=TextField('Some obsolete field', required=False)Generating the Avro spec file
$ python -m elasticgit.tools dump-schema \ > elasticgit.tests.base.TestFallbackPerson > avro.json $ python -m elasticgit.tools load-schema avro.json > models.pyThe generated model file:
# NOTE:## This is an automatically generated Elasticgit Model definition# from an Avro schema. Do not manually edit this file unless you# absolutely know what you are doing.## timestamp: 2014-10-14T18:51:23.916194# namespace: elasticgit.tests.base# type: record# name: TestFallbackPerson#fromelasticgitimportmodelsclassTestFallbackPerson(models.Model): name=models.TextField(u"""The name""", fallbacks=[models.SingleFieldFallback('nick'),models.SingleFieldFallback('obsolete'),]) age=models.IntegerField(u"""The Age""") obsolete=models.TextField(u"""Some obsolete field""") _version=models.ModelVersionField(u"""Model Version Identifier""") nick=models.TextField(u"""The nickname""") uuid=models.TextField(u"""Unique Identifier""")We're using ConfModel's fallbacks feature and encode this in Avro's Schema as aliases. This allows you to fall back to older names for fields:
>>>TestFallbackPerson({'obsolete': 'oldest name', 'age': 10}).name'oldest name'>>>TestFallbackPerson({'nick': 'older name', 'age': 10}).name'older name'>>>TestFallbackPerson({'name': 'current name', 'age': 10}).name'current name'
