Release History

Overview of the VECMAtk M18 Release

This page described the latest VECMAtk release, and preceding releases. See the Toolkit tab for descriptions of the individual components.

The VECMAtk M18 release contains the following components:

  • EasyVVUQ
  • FabSim3
  • QCG Pilot Job
  • QCG-Client
  • QCG-Now
  • MUSCLE3
  • EasyVVUQ-QCGPJ

The M18 version was released on December 13th, 2019.

FabSim3 M18 Release

This is the month 18 release of FabSim3 as a part of VECMA VVUQ Toolkit

FabSim3 is an automation toolkit for complex simulation tasks. FabSim3 helps users to perform complex remote tasks from a local command-line, and to automatically organise their data and environment variables when they perform these tasks.

FabSim3 v1.5

New Features

  • Added support multi-threaded job management (#158)
  • Vastly improved the performance of run_ensemble (#158).

Updates

  • Revamped the documentation (#133)
  • Defined additional functional tests for FabSim3 and plugins (#125)
  • Fixed fabsim binary file for MacOS systems (#152)

Tutorials

This version of FabSim3 has been used to enable three of the VECMA application tutorials:

 

EasyVVUQ v0.5

New Features

  • MultiEncoder element, to combine one or more encoders into a single encoder
  • DirectoryEncoder element, to build directory hierarchies
  • PCE Analysis element supports calculation of multiple Sobol indices
  • Multi-app (Multi-solver) capabilities implemented
  • Collation table may optionally be cleared using Campaign’s clear_collation() method
  • Added new collater (AggregateByVariables) that groups by output variable rather than run_id

Updates

  • EasyVVUQ version of the Campaign and database are now compared to catch version mismatch errors
  • Uniform integer distribution now available through new versions of Chaospy (thanks to collaboration with Jonathan Feinberg)
  • Fix bug in collation of empty dataframes
  • py reads install requirements directly from requirements.txt
  • Some documentation now also available as Jupyter python notebooks
  • EasyVVUQ version of the Campaign and database are now compared to catch version mismatch errors

EasyVVUQ-QCGPJ M18 release v0.1

A wrapper integrating EasyVVUQ and QCG Pilot Job Manager

URL: https://github.com/vecma-project/EasyVVUQ-QCGPJ/releases/tag/v0.1

This release of the tool is created for M18 release of VECMA VVUQ Toolkit.

This release provides a new API that greatly simplifies execution of EasyVVUQ scenarios using the QCG Pilot Job system. This version introduces also a new mechanism that makes it much easier to use custom encoders.

The tool works with the recent releases of EasyVVUQ and QCG Pilot Job, namely *EasyVVUQ **v0.5*** and *QCG Pilot Job **v0.6***

MUSCLE 3 Release 0.2.0

This is the second release of MUSCLE 3 from the e-MUSC project of the University of Amsterdam and the Netherlands eScience Center comes the second release of MUSCLE 3, the MUltiScale Coupling Library and Environment.

  • Support for submodels (and other components) written in C++. MUSCLE 3 now offers a native C++ API, backed by a C++ implementation.
  • Support for C++ submodels that use MPI to communicate internally.
  • Improved handling of clusters with multiple networks.
  • Improved error handling and shutdown.

QCG-PilotJob v0.6

A lightweight Pilot Job implementation. This is the release targeted for the M18 release of VECMA VVUQ Toolkit. It includes several improvements and bugfixes, in particular:

  • support for including modules and python’s virtual environment in executing jobs,
  • bugfix related to unclosed zmq sockets

QCG-Now v1.4.1

Graphical desktop client to QCG. URL: http://www.qoscosgrid.org/qcg-now/en/download 

Release notes:

Added:

  • Quick templates functionality
  • Integrated QCG-Monitoring view in the details view of the main window. Allows to monitor progress of tasks executions.

Changed:

  • The Oracle JDK 1.8 has been replaced by recent versions of Open JDK and Open JFX.
  • Chanded packaging procedures that may influence on installation, particularly on Mac OS.
  • Refreshed look&feel.

 

 

Previous releases

Month 15 public release

FabSim3 M15 Release

This is the month 15 release of FabSim3 as a part of VECMA VVUQ Toolkit

FabSim3 is an automation toolkit for complex simulation tasks. FabSim3 helps users to perform complex remote tasks from a local command-line, and to automatically organise their data and environment variables when they perform these tasks.

FabSim3 v1.3

New Features

  • Extended documentation on how user/developers can create their own FabSim automations (CreateAutomationScripts.md)
  • Implemented a first Python API for launching FabSim commands from Python scripts directly. (#140 and (#127
  • Added a preliminary FabSim3 plugin for the MUSCLE toolkit – FabMUSCLE (#145
  • Added auto bash-completion for fabsim command (#148
  • Added bash aliasing functionality to shorten commands (#139
  • Added support for replica execution (multiple simulations with identical input data and input parameters). (#138

Updates

  • Updated saving functionality for jobs, when FabSim schedules a job (#78
  • Clarified the installation instructions for Mac users (#120
  • Updated Pilot-Job implementation (#128
  • Improved support for Eagle both for VECMA and HiDALGO users. (#124

Tutorials

This version of FabSim3 has been used to enable three of the VECMA application tutorials:

EasyVVUQ v0.4

New Features

  • Parameter type and physical range checking (verification) implemented using Cerberus.
  • Implemented a Multisampler element, allowing arbitary number of samplers to be chained together, but behave as a single sampler.
  • Added a SweepSampler element, for parameter sweeps.
  • Added sparse grid functionality to the Stochastic Collocation sampler.
  • Added a “Worker” class (a stripped down version of the campaign) and associated tools (such as an external encoder script) to allow non-linear workflows, such as when integrating with pilot job managers.

Updates

  • Improved consistency and breadth of automated testing. Campaign restarts now properly tested too.
  • Added several more tutorials to the documentation.
  • Fixed bug in the CSV reader with respect to column label.

EasyVVUQ-QCGPJ M15 release

The tool has been updated to work with the recent releases of EasyVVUQ and QCG-PilotJob. Thanks to modifications introduced to EasyVVUQ (> v0.4.1dev) it was possible to abandon PJConfigurator class and fully relay on EasyVVUQ internal database, thus simplifying the internal logic and interface of EasyVVUQ-QCGPJ tool. In turn the upgrades introduced in QCG-PilotJob (v0.5), which significantly improved its scalability, allow to execute more demanding EasyVVUQ workflows without the risk of poor performance.

MUSCLE 3 Release 0.1.0

Initial release of MUSCLE 3.

Added

  • Coupling different submodel instances
  • Spatial and temporal scale separation and overlap
  • Settings management
  • Combining features
  • Python support
  • Initial distributed execution capability

QCG-PilotJob v0.5

This is the release targeted for the M15 release of VECMA VVUQ Toolkit
It includes several improvements and bugfixes, in particular:

  • a new launcher service that significantly reduces time needed to start tasks on nodes,
  • support for an allocation of dedicated core for a QCG PJ Manager instance,
  • automatic binding to free TCP ports (from port range),
  • support for Slurm CPU binding.

Month 12 public release

FabSim3 – VECMA M12 Release (1.0)

This is the month 12 release of FabSim3 as a part of VECMA VVUQ Toolkit

FabSim3 is an automation toolkit for complex simulation tasks. FabSim3 helps users to perform complex remote tasks from a local command-line, and to automatically organise their data and environment variables when they perform these tasks.

 

New Features

  • Added the official FabSim3 web page (#114)
  • Added fabsim command, to be run from anywhere (#101#118,#89)
  • supported the submission and execution of single jobs, ensembles of multiple jobs through schedulers such as SLURM, ARCHER , and QCG (#76)
  • Added mechanism to automate the installation of QCG Pilot Job manager on remote machines (#94)

Updates

  • Clarified plugin documentation (#119)
  • Updated FabUQCampaign tutorial (#100)
  • Updated FabSim3 installation guidance (INSTALL.md#89)
  • Updated status functionality for submitted job on the remote machines (#104,#112)
  • Clarified setup_fabsim output command (#115)
  • Updated plugin functionality, in case of nonexistent plugin, to getting error message and how to fix it (#97)
  • Updated LAMMPS exmaple on FabSim3 INSTALL.md (#121,#122)

Tutorials

This version of FabSim3 has been used to enable three of the VECMA application tutorials:

Assets

Source code(zip)

Source code(tar.gz)

 

EasyVVUQ – VECMA M12 Release (0.3)

EasyVVUQ is a library created to facilitate verification, validation and uncertainty quantification (VVUQ) for a wide variety of simulations.
This release is part of the [VECMA VVUQ Toolkit](https://www.vecma-toolkit.eu/).

New Features

*  A website with [documentation](https://easyvvuq.readthedocs.io/) and a basic tutorial
*  Support for multiple backend databases (via SQLAlchemy)
*  Result storage in database (as opposed to in memory pandas dataframe)
*  Restartable campaigns
*  Incremental collation of results

Updates

*  All distributions now in chaospy compatible conformations
*  Varying parameters now specified in sampler rather than campaign

 

QCG – VECMA M12 Release

QCG Pilot Job Manager

A lightweight Pilot Job implementation

Release name:  0.4.1

The VECMA M12 release of QCG Pilot Job Manager, tagged with version number 0.4.1, is the first VECMA release of the tool and introduces two main changes supporting VECMA UQ analysis scenarios:

*   New LocalManager class, which allows to start QCG Pilot Job Manager directly from a python code.

*   The local, testing mode of execution, which allows to define virtual resources and may be useful for tests performed on users’ laptops outside the Slurm queueing system.

QCG-Now

Graphical desktop client to QCG

Release name: 1.2.14

QCG-Now is an easy to use desktop program that allows to submit computing jobs to HPC clusters directly from users’ laptops or PCs. The tool complements the CLI tools already available in VECMAtk with the flexible and intuitive GUI. This first release of QCG-Now for VECMAtk, tagged with 1.2.14 version number and attached to the M12 release of the Toolkit, brings its basic functionality to the VECMA consortium and it was tested to run PilotJob and EasyVVUQ scenarios.

For the M12 release of VECMAtk QCG-Now has been additionally configured for use with the integrated version of EasyVVUQ and QCG Pilot Job Manager using EasyVVUQ-QCGPJ wrapper.

EasyVVUQ-QCGPJ

A wrapper integrating EasyVVUQ and QCG Pilot Job Manager

Release name: M12

This simple wrapper brings basic mechanisms that allow to run EasyVVUQ encoding and application execution steps in parallel, using the QCG Pilot Job Manager functionality.

This is the first regular release of the tool. It is targeted for M12 release of VECMA VVUQ Toolkit.

QCG-Client

Command line client to QCG

There are no updates in QCG-Client for the M12 release of VECMAtk. The QCG-Client has been offered to VECMA users as a standalone, alternative submission tool.

Month 9 public internal release

The month 9 release contains the following components:

The main additions in this release are:

  • containerized versions of VECMAtk (using Docker and Singularity),
  • integration of FabSim3 with QCG-Client, enabling FabSim3 users to use the QCG brokering environment to schedule their jobs,
  • a more robust version of EasyVVUQ, with examples of polynomial chaos and stochastic collocation,
  • introduction of a new FabSim3 plugin: FabUQCampaign, which combines EasyVVUQ with FabSim3.

Month 6 internal release (consortium + collaborators)

The month 6 release consists of two components:

The main additions in this release are:

  • Introduction of the EasyVVUQ component, to create VVUQ procedures.
  • Improved support for ensemble job execution in FabSim3.
  • Improved documentation and refactored code, in preparation for later releases.

Month 3 internal release (consortium + collaborators)

The month 3 internal release consisted of FabSim3 v0.3 (https://github.com/djgroen/FabSim3/releases/tag/v0.3).

The main additions in this release are:

  • Revamped, and more flexible plug-in system (fab localhost install_plugin:FabDummy).
  • Introduction of the FabDummy plugin, to help testing and plugin development for FabSim3.
  • Continuous integration infrastructure.

Month 1 internal release (consortium + collaborators)

The month 1 internal  release consists of FabSim v0.1 (https://github.com/djgroen/FabSim3/releases/tag/v0.1).

This release consisted of a reworked version of the FabSim toolkit. We ported the toolkit to Python3, updated the version of Fabric underneath, and developed an initial plugin system with two plugins: FabMD for molecular dynamics, and FabFlee for agent-based migration simulations.