BinaryDataSaver Class Reference

A class to handle simulation data saving in binary format. More...

#include <binarydatasaver.h>

Inheritance diagram for BinaryDataSaver:

Inheritance graph
[legend]
Collaboration diagram for BinaryDataSaver:

Collaboration graph
[legend]

List of all members.

Public Member Functions

 BinaryDataSaver ()
 ~BinaryDataSaver ()
virtual bool setParameters ()
 Default interface needed to initialize the component's variables from its input parameters value.
virtual void execute ()
 Execute the event on the pop.
virtual BinaryDataSaver * clone ()
 Cloning interface.
virtual void FHwrite ()
 Default behavior of the class, called by Handler::update().
virtual void FHread (string &filename)
 Default input function.
virtual void loadFileServices (FileServices *loader)
 Loads the component's FileHandler onto the FileServices.
virtual void loadStatServices (StatServices *loader)
 Loads the component's StatHandler onto the StatServices.
virtual age_t removeAgeClass ()
 Removes the returned age-class flag(s) from the current Metapop age-class flags.
virtual age_t addAgeClass ()
 Adds the returned age-class flag(s) to the current Metapop age-class flags.
virtual age_t requiredAgeClass ()
 Specifies what age-classes are required by the LCE to execute.

Static Public Attributes

static pid_t PID = 0

Private Member Functions

void printHeader ()
void printData ()
void storeData ()
void store_trait (trait_t type)
void getIndVars (unsigned long *array, Individual *ind)
void finish ()

Private Attributes

BinaryStorageBuffer _buff
std::string _comp_cmd
std::string _comp_ext
std::string _uncomp_cmd
std::string _tar_cmd
std::string _tar_ext
std::string _dir
bool _isPeriodic
unsigned int _generation
std::map< unsigned int,
unsigned int > 		_offset_table

Detailed Description

A class to handle simulation data saving in binary format.

It is both an LCE and a FileHandler but the writing is executed through the LCE interface (i.e. LifeCycleEvent::execute()). Its derivation from the FileHandler class is used to access the basic file services (i.e. basename and current replicate filename, etc.).

Definition at line 46 of file binarydatasaver.h.

Constructor & Destructor Documentation

BinaryDataSaver::BinaryDataSaver (  ) 

Definition at line 54 of file binarydatasaver.cc.

Referenced by clone().

00055 : LifeCycleEvent("store",""), FileHandler(".bin"), _isPeriodic(0)
00056 {
00057   ParamUpdater< BinaryDataSaver > * updater =
00058   new ParamUpdater< BinaryDataSaver > (&BinaryDataSaver::setParameters);
00059   
00060   add_parameter("store_generation",INT,true,false,0,0,updater);
00061   add_parameter("store_recursive",BOOL,false,false,0,0,updater);
00062   add_parameter("store_dir",STR,false,false,0,0,updater);
00063   add_parameter("store_nocompress",BOOL,false,false,0,0,updater);
00064   add_parameter("store_noarchive",BOOL,false,false,0,0,updater);
00065   add_parameter("store_compress_cmde",STR,false,false,0,0,updater);
00066   add_parameter("store_compress_extension",STR,false,false,0,0,updater);
00067   add_parameter("store_uncompress_cmde",STR,false,false,0,0,updater);
00068   add_parameter("store_archive_cmde",STR,false,false,0,0,updater);
00069   add_parameter("store_archive_extension",STR,false,false,0,0,updater);
00070 }

BinaryDataSaver::~BinaryDataSaver (  )  [inline]

Definition at line 68 of file binarydatasaver.h.

00068 {}

Member Function Documentation

virtual age_t BinaryDataSaver::addAgeClass (  )  [inline, virtual]

Adds the returned age-class flag(s) to the current Metapop age-class flags.

Implements LifeCycleEvent.

Definition at line 86 of file binarydatasaver.h.

00086 {return 0;}

virtual BinaryDataSaver* BinaryDataSaver::clone (  )  [inline, virtual]

Cloning interface.

Implements LifeCycleEvent.

Definition at line 75 of file binarydatasaver.h.

00075 {return new BinaryDataSaver();}

void BinaryDataSaver::execute (  )  [virtual]

Execute the event on the pop.

Implements LifeCycleEvent.

Definition at line 136 of file binarydatasaver.cc.

00137 { 
00138   //check if first generation to record -> we have to create the file and store the params
00139   //if the generation to store is > than the total nbr of generations, correct this and store the last one
00140   if(_generation == _popPtr->getCurrentGeneration() || _generation > _popPtr->getGenerations()) {
00141         printHeader();
00142         if(!_isPeriodic) storeData();
00143   }
00144   //if periodic, store the data in the buffer
00145   if( _isPeriodic && !(_popPtr->getCurrentGeneration() % _generation))
00146         storeData();
00147   
00148   //check if this is the last generation of the current replicate, if so, write the data
00149   //and compress/tar the output file
00150   if(_popPtr->getCurrentGeneration() == _popPtr->getGenerations()) {
00151         printData();
00152         finish();
00153   }
00154   
00155 }

virtual void BinaryDataSaver::FHread ( string &  filename  )  [inline, virtual]

Default input function.

Loads a pop from the genotypes read from the input file.

Implements FileHandler.

Definition at line 78 of file binarydatasaver.h.

00078 {}

virtual void BinaryDataSaver::FHwrite (  )  [inline, virtual]

Default behavior of the class, called by Handler::update().

Implements FileHandler.

Definition at line 77 of file binarydatasaver.h.

00077 {}

void BinaryDataSaver::finish (  )  [private]

Definition at line 256 of file binarydatasaver.cc.

Referenced by execute().

00257 {
00258   bool doComp = (_comp_cmd.size() > 0), doTar = (_tar_cmd.size() > 0),
00259   first = (_popPtr->getCurrentReplicate() 
00260            == max((unsigned)1,_myenv->slaveRank()));
00261   if (!(doComp || doTar)) return;
00262   
00263   stringstream sysCmd;
00264   if (doComp) sysCmd << _comp_cmd << " " << get_filename();
00265   if (doTar) {
00266         sysCmd << ";" << _tar_cmd;
00267         if (first) sysCmd << " c"; //first replicate, create the tar archive:
00268         else       sysCmd << " r"; //next replicates, append files to it:
00269         sysCmd << "f " << get_path() << get_service()->getBaseFileName();
00270         if (!_myenv->isMaster()) sysCmd << _myenv->slaveRank();
00271         sysCmd << _tar_ext << " --remove-files " << get_filename() << _comp_ext;
00272   }
00273   //wait for last child:
00274   if (!first) waitpid(PID,NULL,0);
00275   pid_t tPID = fork();
00276   if (tPID == 0) {
00277         if (system(sysCmd.str().c_str()) < 0)
00278       error("BinaryDataSaver::finish system cmd \"%s\" failed: %s\n",
00279             sysCmd.str().c_str(),strerror(errno));
00280         _exit(EXIT_FAILURE);
00281   } else if (tPID < 0) //fork failed :
00282         error("BinaryDataSaver::finish::could not fork new process: %s\n",
00283               strerror(errno));
00284   PID = tPID;
00285 }

void BinaryDataSaver::getIndVars ( unsigned long *  array,
Individual *  ind 
) [private]

virtual void BinaryDataSaver::loadFileServices ( FileServices *  loader  )  [inline, virtual]

Loads the component's FileHandler onto the FileServices.

Parameters:
loader the file service

Implements SimComponent.

Definition at line 80 of file binarydatasaver.h.

00080 {loader->attach(this);}

virtual void BinaryDataSaver::loadStatServices ( StatServices *  loader  )  [inline, virtual]

Loads the component's StatHandler onto the StatServices.

Parameters:
loader the stat service

Implements SimComponent.

Definition at line 81 of file binarydatasaver.h.

00081 {}

void BinaryDataSaver::printData (  )  [private]

Definition at line 212 of file binarydatasaver.cc.

Referenced by execute().

00213 {
00214   
00215   mode_t mode = S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH;
00216   int flag = O_WRONLY | O_APPEND;
00217   //open in write and append mode with permission flag set to rw-r--r--
00218   int fdesc = open(get_filename().c_str(), flag, mode);
00219   
00220   if(fdesc == -1) fatal("BinaryDataSaver::printData::open %s:%s\n",get_filename().c_str(),strerror(errno));
00221   //get the offset position of the EOF, will be the nbr of bytes to add to the generation offsets
00222   int pos = lseek(fdesc,0,SEEK_END);
00223 
00224   if(pos == -1) fatal("BinaryDataSaver::printData::lseek %s\n",strerror(errno));
00225 
00226   //offset of the offset table:
00227   int off_table = _buff.getByteLength() + pos;
00228   //add the offset info at the end of the _buff:
00229   unsigned char separator[3] = {'@','O','T'}; 
00230   _buff.store(&separator, 3);
00231 
00232   std::map<unsigned int, unsigned int>::iterator IT = _offset_table.begin();
00233   while(IT != _offset_table.end()) {
00234         IT->second += pos;
00235         _buff.store((int*)&IT->first, sizeof(int));
00236         _buff.store((int*)&IT->second, sizeof(int));
00237         IT++;
00238   }
00239   //finally, record the number of generations recorded and the offset of start of the offset table
00240   int nb_recgen = _offset_table.size();
00241   _buff.store(&nb_recgen, sizeof(int));
00242   _buff.store(&off_table, sizeof(int));
00243   //now write the data to the file:
00244 #ifdef _DEBUG_
00245   message("BinaryDataSaver::printData:writing %ikB of data",_buff.getByteLength()/1024);
00246 #endif
00247   if((write(fdesc,_buff.getBuffer(),_buff.getByteLength())) == -1) fatal("BinaryDataSaver::printData::write %s\n",strerror(errno));
00248   if((close(fdesc)) == -1) error("BinaryDataSaver::printData::close %s\n",strerror(errno));
00249 #ifdef _DEBUG_
00250   message(" [ok]\n");
00251 #endif  
00252 }

void BinaryDataSaver::printHeader (  )  [private]

Definition at line 159 of file binarydatasaver.cc.

Referenced by execute().

00160 {
00161   //first, record the parameters in text mode:
00162   ofstream FILE (get_filename().c_str(), ios::out);
00163   
00164   if(!FILE) fatal("could not open Binary output file!!\n");
00165   
00166   list< ParamSet* > current_params = get_service()->get_params();
00167   list< ParamSet* >::iterator Pit = current_params.begin();
00168   
00169   FILE<<"#NEMO "<<MAIN_VERSION<<" "<<MINOR_VERSION<<" "<<REVISION<<" "<<RELEASE<<" "<<VERSION_DATE<<endl;
00170   
00171 #ifdef _DEBUG_
00172   message("BinaryDataSaver::printHeader:storing parameters\n");
00173 #endif
00174   while(Pit != current_params.end()) {
00175     (*Pit)->print(FILE);
00176     Pit++;
00177   }
00178   FILE.close();
00179   //set the buffer ready to be filled
00180   _buff.set_buff();
00181 }

virtual age_t BinaryDataSaver::removeAgeClass (  )  [inline, virtual]

Removes the returned age-class flag(s) from the current Metapop age-class flags.

Implements LifeCycleEvent.

Definition at line 85 of file binarydatasaver.h.

00085 {return 0;}

virtual age_t BinaryDataSaver::requiredAgeClass (  )  [inline, virtual]

Specifies what age-classes are required by the LCE to execute.

Implements LifeCycleEvent.

Definition at line 87 of file binarydatasaver.h.

00087 {return 0;}

bool BinaryDataSaver::setParameters (  )  [virtual]

Default interface needed to initialize the component's variables from its input parameters value.

Formerly called 'init'.

Implements SimComponent.

Definition at line 74 of file binarydatasaver.cc.

Referenced by BinaryDataSaver().

00075 {
00076   Param* param;
00077   
00078   _generation = (unsigned int)get_parameter_value("store_generation");
00079   
00080   param = get_parameter("store_dir");
00081   if(param->isSet())
00082         _dir = param->getArg();
00083   else
00084         _dir = "";
00085   
00086   param = get_parameter("store_recursive");
00087   if(param->isSet())
00088         _isPeriodic = true;
00089   else
00090         _isPeriodic = false;
00091   
00092   param = get_parameter("store_compress_cmde");
00093   if(param->isSet())
00094         _comp_cmd = param->getArg();
00095   else
00096         _comp_cmd = "bzip2"; //default compressor used
00097   
00098   param = get_parameter("store_compress_extension");
00099   if(param->isSet())
00100         _comp_ext = param->getArg();
00101   else
00102         _comp_ext = ".bz2"; //default
00103   
00104   param = get_parameter("store_uncompress_cmde");
00105   if(param->isSet())
00106         _uncomp_cmd = param->getArg();
00107   else
00108         _uncomp_cmd = "unbzip2"; //default
00109   
00110   param = get_parameter("store_archive_cmde");
00111   if(param->isSet())
00112         _tar_cmd = param->getArg();
00113   else
00114         _tar_cmd = "tar"; //default
00115   
00116   param = get_parameter("store_archive_extension");
00117   if(param->isSet())
00118         _tar_ext = param->getArg();
00119   else
00120         _tar_ext = ".tar"; //default
00121   
00122   param = get_parameter("store_nocompress");
00123   if(param->isSet()) {_comp_cmd = ""; _comp_ext ="";} //compression process desabled
00124   
00125   param = get_parameter("store_noarchive");
00126   if(param->isSet()) {_tar_cmd = ""; _tar_ext = "";} //archiving processed desabled
00127   
00128   FileHandler::set(true, false, 1, _generation, get_rank(), _dir);
00129   
00130   _offset_table.clear();
00131   return true;
00132 }

void BinaryDataSaver::store_trait ( trait_t  type  )  [private]

void BinaryDataSaver::storeData (  )  [private]

Definition at line 185 of file binarydatasaver.cc.

Referenced by execute().

00186 {
00187 #ifdef _DEBUG_
00188   message("BinaryDataSaver::storeData\n");
00189   unsigned int byte_count = _buff.getByteLength();
00190 #endif
00191   unsigned int  generation = _popPtr->getCurrentGeneration();
00192   unsigned char separator[2] = {'@','G'}; //generation separator = '@G'
00193   //store the position in the buffer to the offset table:
00194   _offset_table[generation] = _buff.getByteLength();
00195   //store the data, begin with generation separator and number:
00196   _buff.store(&separator, 2 * sizeof(unsigned char));
00197   
00198   _buff.store(&generation, sizeof(unsigned int));
00199   
00200   //Metapop is a StorableComponent, call store_data:
00201   //stores all individual info, including trait sequences
00202   _popPtr->store_data(&_buff);
00203   
00204 #ifdef _DEBUG_
00205   message("BinaryDataSaver::storeData::stored %ikB\n",(_buff.getByteLength()-byte_count)/1024);
00206   byte_count = _buff.getByteLength();
00207 #endif
00208 }

Member Data Documentation

BinaryStorageBuffer BinaryDataSaver::_buff [private]

Definition at line 50 of file binarydatasaver.h.

Referenced by printData(), printHeader(), and storeData().

std::string BinaryDataSaver::_comp_cmd [private]

Definition at line 52 of file binarydatasaver.h.

Referenced by finish(), and setParameters().

std::string BinaryDataSaver::_comp_ext [private]

Definition at line 52 of file binarydatasaver.h.

Referenced by finish(), and setParameters().

std::string BinaryDataSaver::_dir [private]

Definition at line 52 of file binarydatasaver.h.

Referenced by setParameters().

unsigned int BinaryDataSaver::_generation [private]

Definition at line 54 of file binarydatasaver.h.

Referenced by execute(), and setParameters().

bool BinaryDataSaver::_isPeriodic [private]

Definition at line 53 of file binarydatasaver.h.

Referenced by execute(), and setParameters().

std::map<unsigned int, unsigned int> BinaryDataSaver::_offset_table [private]

Definition at line 56 of file binarydatasaver.h.

Referenced by printData(), setParameters(), and storeData().

std::string BinaryDataSaver::_tar_cmd [private]

Definition at line 52 of file binarydatasaver.h.

Referenced by finish(), and setParameters().

std::string BinaryDataSaver::_tar_ext [private]

Definition at line 52 of file binarydatasaver.h.

Referenced by finish(), and setParameters().

std::string BinaryDataSaver::_uncomp_cmd [private]

Definition at line 52 of file binarydatasaver.h.

Referenced by setParameters().

pid_t BinaryDataSaver::PID = 0 [static]

Definition at line 83 of file binarydatasaver.h.

Referenced by finish().

The documentation for this class was generated from the following files:
Generated for Nemo v2.1.2 by  doxygen 1.5.8 -- Nemo is hosted by  SourceForge.net Logo