Skip to content

NonVolatileComputing/Mnemonic

Repository files navigation

================================ ###This project has become an Apache incubator Project, this repo does not host this project anymore, Please fork Apache Mnemonic from [Apache_Mnemonic](https://github.com/apache/incubator-mnemonic), Thanks!

(This project has been manually tranferred from https://github.com/bigdata-memory/bdmem)

This library comes up with a new programming model we call it non-volatile object programming model, this model directly offloads object graphs into a variety of memory-like devices e.g. SSD, NVMe, Off-heap, in this way, it brings some promising features for massive data processing and high performance computing.

Features:

  • In-place data storage on local non-volatile memory
  • In-place generic Java object persistence
  • Object graphs lazy loading & multi-process sharing
  • Auto-reclaim memory resources and Mnemonic objects
  • Hierarchical cache pool for massive data caching
  • Pluggable allocator services for extension & optimization
  • A set of non-volatile data structures (WIP)
  • Minimize memory footprint on Java heap
  • Reduce GC Overheads as the following chart shown (collected from Apache Spark experiments)
  • [Coming major feature]: Distributed Object Graphs (DOG)
  • [Coming major feature]: Columnar-aware object graphs & collections (Apache Arrow based optimization)

Mnemonic_GC_stats

Mnemonic Way

Mnemonic_Way

Mnemonic_Modes

How to use it ?

Define a Non-Volatile class:

/** * a non-volatile class should be abstract, implement Durable interface and marked with @NonVolatileEntity annotation */@NonVolatileEntitypublicabstractclassPerson<E> implementsDurable, Comparable<Person<E>>{Eelement; // Generic Type/** * callback for this non-volatile object creation */@OverridepublicvoidinitializeAfterCreate(){System.out.println("Initializing After Created")} /** * callback for this non-valatile object recovery */@OverridepublicvoidinitializeAfterRestore(){System.out.println("Initializing After Restored")} /** * setup generic info manually to avoid performance penalty */@OverridepublicvoidsetupGenericInfo(EntityFactoryProxy[] efproxies, GenericField.GType[] gftypes){} @TestpublicvoidtestOutput() throwsRetrieveNonVolatileEntityError{System.out.printf("Person %s, Age: %d ( %s ) \n", getName(), getAge(), null == getMother()? "No Recorded Mother" : "Has Recorded Mother")} publicintcompareTo(Person<E> anotherPerson){intret = 0; if (0 == ret) ret = getAge().compareTo(anotherPerson.getAge()); if (0 == ret) ret = getName().compareTo(anotherPerson.getName()); returnret} /** * Getters and Setters for non-volatile fields marked with @NonVolatileGetter and @NonVolatileSetter */@NonVolatileGetterabstractpublicShortgetAge(); @NonVolatileSetterabstractpublicvoidsetAge(Shortage); @NonVolatileGetterabstractpublicStringgetName() throwsRetrieveNonVolatileEntityError; @NonVolatileSetterabstractpublicvoidsetName(Stringname, booleandestroy) throwsOutOfPersistentMemory, RetrieveNonVolatileEntityError; @NonVolatileGetterabstractpublicPerson<E> getMother() throwsRetrieveNonVolatileEntityError; @NonVolatileSetterabstractpublicvoidsetMother(Person<E> mother, booleandestroy) throwsRetrieveNonVolatileEntityError; @NonVolatileGetterabstractpublicPerson<E> getFather() throwsRetrieveNonVolatileEntityError; @NonVolatileSetterabstractpublicvoidsetFather(Person<E> mother, booleandestroy) throwsRetrieveNonVolatileEntityError}

Use a non-volatile class:

Setup an allocator for non-volatile object graphs.
// create an allocator instanceBigDataPMemAllocatoract = newBigDataPMemAllocator(1024 * 1024 * 8, "./pobj_person.dat", true); // fetch handler store capacity from this non-volatile storage managed by this allocatorKEYCAPACITY = act.handlerCapacity(); .... // close it after useact.close();
Generate structured non-volatile objects.
// create a new non-volatile person object from this specific allocatorperson = PersonFactory.create(act); // set attributesperson.setAge((short)rand.nextInt(50)); person.setName(String.format("Name: [%s]", UUID.randomUUID().toString()), true); // keep this person on non-volatile handler storeact.setHandler(keyidx, person.getNonVolatileHandler()); for (intdeep = 0; deep < rand.nextInt(100); ++deep){// create another person as mothermother = PersonFactory.create(act); mother.setAge((short)(50 + rand.nextInt(50))); mother.setName(String.format("Name: [%s]", UUID.randomUUID().toString()), true); // set the person's motherperson.setMother(mother, true); person = mother}
Use the non-volatile objects
for (longi = 0; i < KEYCAPACITY; ++i){System.out.printf("----------Key %d--------------\n", i); // iterate non-volatile handlers from handler store of this specific allocatorval = act.getHandler(i); if (0L == val){break} // restore person objects from this specific allocatorPerson<Integer> person = PersonFactory.restore(act, val, true); while (null != person){person.testOutput(); // iterate all mother's ancestorsperson = person.getMother()} }

How to build it ?

Please see the file LICENSE for information on how this library is licensed.

  • core -- the submodule project for core
  • collections -- the submodule project for generic collections
  • examples -- the submodule project for examples
  • allocator-services/pmalloc-service -- the submodule project for pmalloc allocator service
  • allocator-services/nvml-vmem-service -- the submodule project for vmem allocator service
  • allocator-services/service-dist -- the location of pluggable allocator services (auto-generated)

To build this library, you may need to install some required packages on the build system:

  • Maven -- the building tool v3.2.1 or above [Required]
  • NVML -- the NVM library (Please compile this library that is tagged with 0.1+b16) (http://pmem.io) [Optional if nvml-vmem-service is excluded]
  • JDK -- the Java Develop Kit 1.6 or above (please properly configure JAVA_HOME) [Required]
  • PMFS -- the PMFS should be properly installed and configured on Linux system if you want to simulate read latency [Optional]
  • PMalloc -- a supported durable memory native library at https://github.com/bigdata-memory/pmalloc.git [Optional if pmalloc-service is excluded]
  • Javapoet -- a dependant project v1.3.1-SNAPSHOT revised for this project at https://github.com/wewela/javapoet.git [Required]

Once the build system is setup, this Library is built using this command at the top level:

 $ mvn clean package

To exclude a customized allocator service for your platform e.g. OSX, note that if you excluded one or both allocator services, some or all testcases/examples will fail since their allocator is unavailable.

 $ mvn -pl '!allocator-services/nvml-vmem-service' clean package

To build and run the unit tests:

 $ mvn clean package -DskipTests=false

To install this package to local repository (required to run examples and testcases):

 $ mvn clean install

To run an example:

 $ mvn exec:exec -Pexample -pl examples # requires 'vmem' allocator service to run, please refer to the code of test cases for more examples.

To run several test cases:

 $ mvn -Dtest=NonVolatilePersonNGTest test -pl core -DskipTests=false # a testcase for module "core" that requires 'pmalloc' allocator service to pass $ mvn -Dtest=BigDataMemAllocatorNGTest test -pl core -DskipTests=false # the second testcase for module "core" that requires 'vmem' allocator service to pass $ mvn -Dtest=MemClusteringNGTest test -pl core -DskipTests=false # the third testcase for module "core" that requires 'vmem allocator service to pass $ mvn -Dtest=NonVolatileNodeValueNGTest test -pl collections -DskipTests=false # a testcase for module "collection" that requires 'pmalloc' allocator service to pass $ mvn -Dtest=NonVolatilePersonNGTest test -pl collections -DskipTests=false # another testcase for module "collection" that requires 'pmalloc' allocator service to pass

Where is the document ?

How to apply it for other projects ?

About

Moved to Apache Mnemonic (Incubator)

Resources

License

Stars

Watchers

Forks

Packages

No packages published