Skip to content

AI-Memory/bdmem

Folders and files

NameName
Last commit message
Last commit date

Latest commit

History

17 Commits

Repository files navigation

This project has been manually tranferred to

bdmem: Big Data Memory Library

A hybrid memory resources management library. It is featured with in-place non-volatile Java object programming model.

Features:

  • In-place data storage on local non-volatile memory
  • In-place generic Java object persistence
  • Lazily data object loading
  • Any map-able device could be used as a non-volatile memory resource
  • Reclaim allocated memory when it is no longer used
  • Hierarchical cache pool for massive data caching
  • A set of persistent data structures is provided by bdmemgeneric project
  • Minimize memory footprint on Java heap
  • Reduce GC Overhead as following data shown (collected from Apache Spark experiments)

BDML_GC_Footprint

How to use it ?

Define a durable class:

/** * a durable class should be abstract and implemented from Durable interface with @PersistentEntity annotation */@PersistentEntitypublicabstractclassPerson<E> implementsDurable, Comparable<Person<E>>{Eelement; // Generic Type/** * callback for brand new durable object creation */@OverridepublicvoidinitializeAfterCreate(){System.out.println("Initializing After Created")} /** * callback for durable object recovery */@OverridepublicvoidinitializeAfterRestore(){System.out.println("Initializing After Restored")} /** * setup generic info manually to avoid performance penalty */@OverridepublicvoidsetupGenericInfo(EntityFactoryProxy[] efproxies, GenericField.GType[] gftypes){} @TestpublicvoidtestOutput() throwsRetrievePersistentEntityError{System.out.printf("Person %s, Age: %d ( %s ) \n", getName(), getAge(), null == getMother()? "No Recorded Mother" : "Has Recorded Mother")} publicintcompareTo(Person<E> anotherPerson){intret = 0; if (0 == ret) ret = getAge().compareTo(anotherPerson.getAge()); if (0 == ret) ret = getName().compareTo(anotherPerson.getName()); returnret} /** * Getters and Setters for persistent fields with @persistentGetter and @PersistentSetter */@PersistentGetterabstractpublicShortgetAge(); @PersistentSetterabstractpublicvoidsetAge(Shortage); @PersistentGetterabstractpublicStringgetName() throwsRetrievePersistentEntityError; @PersistentSetterabstractpublicvoidsetName(Stringname, booleandestroy) throwsOutOfPersistentMemory, RetrievePersistentEntityError; @PersistentGetterabstractpublicPerson<E> getMother() throwsRetrievePersistentEntityError; @PersistentSetterabstractpublicvoidsetMother(Person<E> mother, booleandestroy) throwsRetrievePersistentEntityError; @PersistentGetterabstractpublicPerson<E> getFather() throwsRetrievePersistentEntityError; @PersistentSetterabstractpublicvoidsetFather(Person<E> mother, booleandestroy) throwsRetrievePersistentEntityError}

Use a durable class:

Setup an allocator for durable objects.
// create an allocator object with parameters ie. capacity and uriBigDataPMemAllocatoract = newBigDataPMemAllocator(1024 * 1024 * 8, "./pobj_person.dat", true); // fetch underlying capacity of key-value pair store for persistent handler storageKEYCAPACITY = act.persistKeyCapacity(); .... // close it after useact.close();
Generate structured durable objects.
// create a new durable person object from specific allocatorperson = PersonFactory.create(act); // set attributesperson.setAge((short)rand.nextInt(50)); person.setName(String.format("Name: [%s]", UUID.randomUUID().toString()), true); // keep this person on persistent key-value pair storeact.setPersistKey(keyidx, person.getPersistentHandler()); for (intdeep = 0; deep < rand.nextInt(100); ++deep){// create another person as mothermother = PersonFactory.create(act); mother.setAge((short)(50 + rand.nextInt(50))); mother.setName(String.format("Name: [%s]", UUID.randomUUID().toString()), true); // set the person's motherperson.setMother(mother, true); person = mother}
Use the durable objects
for (longi = 0; i < KEYCAPACITY; ++i){System.out.printf("----------Key %d--------------\n", i); // iterate persistent handlers from key-value store of specific allocatorval = act.getPersistKey(i); if (0L == val){break} // restore person objects from specific allocatorPerson<Integer> person = PersonFactory.restore(act, val, true); while (null != person){person.testOutput(); // iterate all mother's ancestorsperson = person.getMother()} }

How to build it ?

Please see the file LICENSE for information on how this library is licensed.

  • src -- the source for the library
  • src/main/java -- the Java source for the library
  • examples -- Brief examples for this library
  • src/main/native -- the native source for the library
  • src/test/java -- the Java test & example source for the library
  • uml -- modeling documents for the library
  • target -- the generated packages for the library
  • target/apidocs -- the generated API documents for the library

To build this library, you may need to install some required packages on the build system:

  • NVML -- the NVM library (Please compile this library that is tagged with 0.1+b16) (http://pmem.io) [Required]
  • JDK -- the Java Develop Kit 1.6 or above (please properly configure JAVA_HOME) [Required]
  • PMFS -- the PMFS should be properly installed and configured on Linux system if you want to simulate read latency [Optional]
  • PMalloc -- the supported durable memory native library at https://github.com/bigdata-memory/pmalloc.git [Required]
  • Javapoet -- the 1.3.1-SNAPSHOT revised for bdmem at https://github.com/wewela/javapoet.git [Required]

Once the build system is setup, the Big Memory Library is built using this command at the top level:

 $ mvn clean package -DskipTests -Dmaven.javadoc.skip=true -Dmaven.test.skip=true

To build and run the unit tests:

 $ mvn clean package

To install this package to local repository:

 $ mvn clean install

To build examples:
(Note that the Big Data Memory Library should be installed to local repository at first):

 $ cd examples $ mvn clean package

To run an example:

 $ cd examples $ java -jar target/examples-X.X.X(-SSSSS).jar

About

BigData Memory Library

Resources

License

Stars

Watchers

Forks

Packages

No packages published