- Info
CARMEN data conversion
Return to the main index.
Recommendations and Notes
The following recommendations are made:
- Experimental data in the CARMEN system will only be
stored within an NDF format so all services and tools which cannot use
this format will need to use a converter or wrapper / un-wrapper
(depending on the nature of the data) to access and store data in the
NDF format.
- On upload of experimental data the converters will convert it to one
(or more) of the five NDF native formats - assuming data converters are available. Data in its
original format will also be wrapped as external data within the same NDF instance to
maintain
interoperability with third party / legacy tools and ensure that no
original information is lost.
- It
is recommended that newly developed CARMEN tools / services will
consume and produce data in an NDF format – however, the writer of
tools or service will be also free to use bespoke formats, bearing in
mind that if this is a permanent situation than a converter to / from
NDF will be required for every bespoke format to ensure
interoperability with other services. (If the tool or service use
experimatal data then it will have to consume data in an NDF format or
use an existing converter.)
- The CARMEN system will be able
to use newly developed CARMEN tools and services and third party /
legacy tools and services written in R, Java, Matlab, C, C++, modelling
languages. This list may be extended.
Notes:
- It is possible that the conversion "services" may also be available as libraries to for use in the service frameworks.
- As
mentioned prviously the NDF is a container of several data formats so
it will be possible for situations to occur where a tool or service
(which can read input data from a particular NDF content slot) cannot
use an NDF instance because the instance does not contain data in the
appropriate slot. For example, a tool which can read and use time
series data can only use NDF instances containing time series data -
clearly not all NDF instances will contain time series data.
iRODS aspects- conclusions
For upload and download activities the use of iRODS rules and micro-services offers significant advantages since conversion
and
wrapping activities could be performed automatically. There may / may
not be issues if there is required to be user input on conversion /
wrapping and converter / un-wrapping. This requires further
investigation.
For service and data interaction scenarios it is
likely that the use of a work flow engine to invoke data conversion,
wrapping, etc. is preferable to using iRODS rules and micro-services
since it will be more flexible.
jdk 1.6.0_45 and jdk-1.7.0_21 available
