Skip to content

TeraFoundation server framework for building clustered node.js applications

License

Notifications You must be signed in to change notification settings

jsnoble/terafoundation

Repository files navigation

TeraFoundation

Node.js framework for building clustered servers

Here are a few tasks it can help you with:

  • Managing multiple database connections
  • Framework to create a number of general or special child workers
  • Schema validations for app configurations

Basic Example

//service.js var worker = require('./worker'); var foundation = require('terafoundation')({name: 'testserver', worker: worker }); //worker.js module.exports = function(context) var logger = context.logger var count = 0; while (true){logger.info(count++) } } 

terafoundation call time settings

ConfigurationDescriptionTypeNotes
namename of applicationStringoptional, defaults to terafoundation
ops_directory'path/to/directory', to look for more custom connectors. Usually this is where you place your custom code not part of core, unless you want to leave your code in place.Stringoptional
descriptorsObject listing all the different modules that a child process could run, used when child process will have different behaviorsObjectoptional, default to creation of children running the worker module passed in
cluster_namename of applicationString or Functionoptional, defaults to terafoundation
scriptjavascript execution of codeFunctionoptional
config_schemasystem schema for the top level serviceObject or Functionoptional
plugin_schemasystem schema for plugins associated with the top level serviceObject or Functionoptional
schema_formatsIf you have custom formats used for your schema validations you must pass them downArrayoptional, used to add any custom formats for convict validation library
start_workersBy default, the service will attempt to create as many child process as set in the config, if set to false then the top level application will be in charge of when a child process will be createdBooleanoptional, defaults to true
shutdownMessagingIf set to true then it provides a ipc shutdown message so all child process can hook into for custom shutdown. All child process will receive an ipc message of{message: 'shutdown'}Booleanoptional, defaults to false which in turn cause the main process to call a kill signal "SIGINT"
emitteryou may pass down a node.js event emitter which will emit certain signals to listen onfunctionoptional

Complex example

var foundation = require('terafoundation')({name: 'teraslice', shutdownMessaging: true, worker: worker, master: master, slicer: slicer, assets_loader: assets_loader, assets_service: assets_service, cluster_master: cluster_master, moderator: moderator, descriptors:{slicer: true, worker: true, cluster_master: true, moderator: true, assets_loader: true, assets_service: true }, start_workers: false, config_schema: config_schema, schema_formats: schema_formats, ops_directory: ops_directory, cluster_name: cluster_name }); 

You may pass in multiple different type of child process that behave differently. To use them you must specify them on the descriptors. After this you may create them using context.foundation.startWorkers api to create that specific worker type. The descriptor is primarily needed for child worker restarts

Schemas

We use the convict library for configuration validations. Any top level program can pass down its schema as the config_schema parameter when instantiating terafoundation

You may reference system_schema at the root of terafoundation for references

Context

As noted in the basic example above, all child process will be called with a context which contains configurations and apis

The context is an object with the following keys:

  • sysconfig which contains all the fully validated configurations of your entire app
  • cluster which is the top level node cluster module used for all child processes
  • name which is the top level name of project
  • cluster_name which is name to distinguish the cluster
  • logger a base top level logger for any logging needs, we use bunyan
  • foundation which contains the api to make a logger, create a database connection, or to make a worker

API

The api is located at context.apis.foundation:

  • getConnection which is used to get a database client.
var client = context.apis.foundation.getConnection({type: 'elasticsearch', endpoint: default, cached: true }).client; 

In this example, type references which type of connector you are using. Endpoint references which of the endpoints you will use, and cached determines if the same client instance will be returned on subsequent calls.

  • makeLogger which is used to create a logger
var logger = context.apis.foundation.makeLogger({module: 'worker', slice: '7dj38d'}); 

The object attaches metadata to any of the logs done by the new child logger. This allows more visible logging per action taken or per module

  • startWorkers which is used to create specific child workers
context.apis.foundation.startWorkers(1,{assignment: 'assets_service', port: assets_port, node_id: context.sysconfig._nodeName }); 

The first argument is the number of specific workers you are going to make. The second parameter is an object. The key assignment must match whats listed in the descriptors listed in the complex terafoundation example above. It allows the system to know what type of child process to create. All keys and values in the object will also by stored in the process environment process.env . This allows you to pass a port number, identifier or anything else that the process needs to know before hand to boot up

  • getSystemEvents which returns the system wide event emitter
var events = context.apis.foundation.getSystemEvents(); events.on('event', function(){// do something for this event. }); 

This allows listening to and emitting events on the system events bus.

terafoundation configuration

A configuration file may either be a YAML or JSON file. You can pass in the configuration file at run time: node service.js -c config.yaml or by having the appropriate configuration file located at the root of your project

/root-app /node-modules terafoundation service.js worker.js config.yaml 

or by setting an environmental variable TERAFOUNDATION_CONFIG to the configuration file

Example Config

top_root_level_application: some: configuration terafoundation: environment: development logging: elasticsearch log_path: "/some/path/to/logs" log_level: - console: info - elasticsearch: info connectors: elasticsearch: default: host: - 127.0.0.1:9200 keepAlive: true maxRetries: 5 maxSockets: 20 endpoint2: host: - 127.0.0.1:9215 keepAlive: true maxRetries: 5 maxSockets: 20 mongo: default: host: 127.0.0.1 mock: false 

Connectors

This is where client config used to set up database connections lives. It is an object with keys set to the different clients and must match the names of the clients listed in connectors/node-modules.

The values are once again objects with keys set to different endpoints within each client. For example in the config above, elasticsearch database has two endpoints listed. The first one is called default which listens to an elasticsearch database on port 9200, and by what its name implies is the endpoint that's used when making a client without specifying an endpoint. The other is endpoint2 which connects to an elasticsearch database on port 9215

Each endpoint will have its own configuration which will then be used by the actual client of the library

terafoundation configuration settings

ConfigurationDescriptionTypeNotes
environmentSet for legacy purposes, if set to production, it will also write logs to fileStringoptional, defaults to development
log_pathdirectory where the logs will be stored if logging is set to fileStringoptional, defaults to root of project
logginga list to where logs will be written to, settings available are ['console', 'file', 'elasticsearch']Array of Stringsoptional, defaults to ['console']
log_levelthis determines what level of logs are shown, options: trace, debug, info, warn , error, fatalString or Arrayoptional, defaults to info, if a string is set then all log destinations in logging will use this, you may also customize log levels for each destination: [{console: 'debug'},{file: 'warn},{elasticsearch: 'info'}]
log_buffer_limitthe number of logs stored in the ringbuffer on the logger before sent, logging must have elasticsearch set as a value for this to take effectNumberoptional, defaults to 30
log_buffer_intervalinterval (number in milliseconds) that the log buffer will send up its logs, this is used if logging is set to use elasticsearchNumberoptional, defaults to 60000 ms
log_connectionlogging connection endpoint if logging is saved to elasticsearch, this is only in use if logs are being saved to a DBStringoptional, uses the 'default' endpoint of elasticsearch connectors
connectorsAn object containing database client information for you serviceObjectrequired

About

TeraFoundation server framework for building clustered node.js applications

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages

  • JavaScript100.0%