next up previous
Next: Changes in V4.2
Up: CHANGES AND NEW FEATURES
Previous: CHANGES AND NEW FEATURES

Changes in V4.1

The following describes the most significant changes which occurred in HDS between versions V4.0 and V4.1 (not the current version).

1.
The C source code of HDS now complies with the ANSI C standard, and UNIX implementations of HDS use a new "makefile". Both of these changes are designed to make it relatively straightforward to implement HDS on new platforms.
2.
HDS now contains routines which automatically determine the primitive data representation used by common types of computer hardware (e.g. the floating point number format and byte storage order). This also makes it easier to implement HDS on new platforms, since changes to the source code are not normally required.

3.
A DECstation implementation of HDS has been added. An implementation on Silicon Graphics hardware is also in use, although this is not currently supported by Starlink.

4.
The SUN Sparcstation implementation of HDS now uses file mapping as its default mode of file access. This can give appreciable performance gains, largely, it appears, as a result of reduced memory usage which may allow better buffering of file contents.

5.
The directory in which HDS resides now contains a file called hds_datestamp which holds information about how the HDS system was built. This includes details of the primitive data representation used by the host machine which may sometimes be of use to programmers.

6.
A bug has been fixed which could cause the length of new _CHAR objects to exceed the length actually requested. Typically, an extra digit was being appended to the length as a result of an un-terminated internal C string.

7.
A bug has been fixed which caused HDS not to recognise primitive data values written in big-endian IEEE double precision format (i.e. typically written on a SUN) which have the "bad" data value due to a previous format conversion error. The result of this bug was that a data conversion error could result if the data were read on a machine (typically a VAX) with a more restricted double precision number range. The correct behaviour is for the bad SUN value to be converted silently to the corresponding bad VAX value. This now occurs.

8.
A bug has been fixed which could corrupt a mapped slice of a primitive object because only a single byte of data was being transferred for the final element. This could happen both when reading and writing the data, although not in all circumstances.

9.
A problem has been fixed which typically resulted in "bus errors" on SUN systems when attempting to access primitive data as double precision values where format conversion was also required. This resulted from an inability to handle double precision values passed from Fortran to C, where the value may be badly aligned in memory (i.e. on a 4-byte rather than an 8-byte boundary). This problem was not always repeatable, in the sense that it depended on where the Fortran compiler placed the relevant variable in memory.

10.
A workaround has been installed for a problem sometimes encountered when writing to a file served by a VMS machine using the Network File System (NFS) from a SUN. Depending on the file size and the amount of data written, a failure to extend the file can sometimes occur. This seems to happen only when the new file size requested by the SUN lies between the current end of file and the physical file size allocated by VMS as a result of clustering of disk blocks. This problem seems to have arisen since the previous version of HDS due to changes to the VMS UCX software. The workaround involves repeating the file extend call if it fails on the first attempt.

11.
A bug has been fixed in the queue handling facility used internally by HDS. This could cause regions of memory to be over-written. It is not clear what adverse consequences this may have had.

12.
An error in the documentation concerning the order in which character arguments should appear when passing mapped character values using the %VAL facility has been corrected (see §[*]).

13.
The description of the DAT_BASIC routine has been improved to make it clear that this routine accesses bytes of primitive data as written, and does not perform conversion to/from the data representation of the host machine.

14.
Several minor typos in error messages have been corrected.

15.
Use of the routine HDS_START is no longer necessary in order to start up HDS before using it. The system is now self-starting, typically when the first routine which accesses a locator is called. HDS_START has been documented as obsolete, but its use remains optional.

16.
Use of the routine HDS_STOP at the end of a program is now optional, since its action will be performed automatically by an exit handler. Examples showing its use have been modified appropriately. Note that HDS_STOP is not obsolete, as it remains the only method of closing down HDS in the middle of a program. In practice, the need to do this is likely to be limited.

17.
Since HDS_START and HDS_STOP are now both optional, the routine HDS_RUN no longer serves a useful purpose and has been documented as obsolete.

18.
The method by which HDS determines how long a container file should be held open has been rationalised. This no longer depends on the existence of a "top-level" locator. Instead, the concept of "primary" and "secondary" locators has been introduced (see [*]) and a container file remains open for as long as at least one primary locator is associated with it. Any locator may be designated as primary by means of the new routine DAT_PRMRY, thus removing the special status of top-level locators in this context. Routines which previously incremented the container file "reference count" now issue primary locators (all other locators are secondary by default), so that existing behaviour is retained.

This change has been introduced to remove the requirement that all software using HDS maintain its own table of top-level locators in order to prevent container files being closed. It also opens the way for future improvements to the programming interface of HDS, which should allow objects to be identified by their pathname as well as by locator.

19.
All routines which annul locators will now also close the associated container file if there are no longer any primary locators associated with it. Any outstanding secondary locators associated with a closed container file are now correctly annulled (previously they were simply left "dangling"). The main implication of this is that DAT_ANNUL will now close a container file which is no longer in use, removing the need to do this explicitly.

20.
As a result of the above changes, the HDS_CLOSE routine is now redundant, and has been documented as obsolete. Its continued use is not recommended. Its behaviour has always been flawed, since it decrements the reference count for a container file regardless of whether the top-level locator it annuls originally caused this count to be incremented when it was created. Since several methods exist for generating top-level locators without incrementing this count, it is possible for a container file to be prematurely closed if HDS_CLOSE is used.

HDS_CLOSE may still be useful as an emergency measure to close a file in the presence of a programming error which has left it open.

21.
A new routine DAT_REFCT has been introduced to return the current reference count for a container file. This makes it possible to predict when a file will actually be closed.

22.
The temporary limit on the number of simultaneously open container files imposed in V4.0 of HDS has been removed. HDS now imposes no restrictions on the number of open files beyond those set by the host operating system.

23.
On UNIX systems, HDS now reports an error if the name of a container file which is already in use is given as the name of a new container file to be created. This avoids the fairly common problem on UNIX, where the user of an application supplies the same name for both the input file and the output file, and ends up with no file at all because the output over-writes the input before it has been read.

24.
When creating new container files on UNIX systems, HDS will now check before over-writing an existing file to ensure that it is a regular file and not a directory or FIFO, etc. An error results if it is not a regular file.

25.
HDS now consistently ignores all leading and trailing white space on file names passed to it.

26.
On UNIX systems, HDS now uses the full (absolute) pathnames of all files when referring to them in messages and when returning file names via routine arguments.

27.
On UNIX systems, HDS will now pass file names which contain "special" characters (anything except alphanumerics, `.', `/', `-' and `_') to a shell for interpretation. This means that normal UNIX shell syntax may be used to construct file names, thus allowing environment variable expansion, etc. In fact, any shell command can, in principle, be used to construct a file name. Pattern-matching characters are also accepted - if more than one file matches, then the first match is used.

28.
Two new routines HDS_WILD and HDS_EWILD have been introduced to permit "wild-card" searching for HDS container files specified using pattern-matching characters.

29.
A new tuning parameter `SHELL' has been introduced (see [*]) to allow a choice of which UNIX shell is used to interpret file names (this includes wild-carding of file names using HDS_WILD and HDS_EWILD). By default, the "sh" shell is used but, if available, the "csh" and "tcsh" shells may also be selected. It is also possible to turn shell expansion of file names off.

30.
It is now possible to modify the default tuning profile of HDS by means of environment variables, which are read and interpreted by HDS at startup (see §[*]). For instance, the definition:

setenv HDS_MAP 0

could be used to disable file mapping in favour of I/O in any application which uses HDS.

31.
A call to HDS_GTUNE will now only return the value 1 for the `MAP' tuning parameter (corresponding to file mapping being used as the file access mode) if this mode has previously been requested and if it is actually implemented on the host machine. This makes it possible for the caller to determine whether file mapping is implemented or not.

32.
Three new special values (-1, -2 and -3) may now be given for the `MAP' tuning parameter in order to select the file access mode best suited for a given type of access. These options select (in order) the file access method which gives fastest sequential access, fastest random (sparse) access, or minimum usage of memory (see [*]). The value returned by HDS_GTUNE may be used to determine which file access mode was actually selected, as this will depend on the host operating system.

33.
Some of the values returned by the routine DAT_CLEN describing the number of digits required to format floating point numbers have been changed to reflect the recommendations of the IEEE floating point standard.

34.
The HDS_SHOW routine now has a new `DATA' option which displays details of the primitive data representation in use by the host machine.

35.
Internal changes have been made which allow HDS "container records" (which contain information about the components which reside within an HDS structure) to be reduced in size when components are erased. Some hysteresis is allowed in this process. This typically gives a slight performance improvement and saves a little file space.

36.
The NBLOCKS tuning parameter is now actually used at several places within HDS. Previously its value had no effect.

37.
A new script called hds_dev has been provided on UNIX systems to create and remove symbolic links to the HDS public include files within the current directory. This allows the same INCLUDE statements to be used in Fortran code on both UNIX and VMS systems.

38.
The new error code DAT__WLDIN and the symbolic constant DAT__NOWLD have been introduced.

39.
This document (SUN/92) has been updated and produced using a larger font.



next up previous
Next: Changes in V4.2
Up: CHANGES AND NEW FEATURES
Previous: CHANGES AND NEW FEATURES

HDS Hierarchical Data System
Starlink User Note 92
R.F. Warren-Smith & M.D. Lawden
23rd February 1999
E-mail:rfws@star.rl.ac.uk

Copyright (C) 1999 Central Laboratory of the Research Councils