LOCK and UNLOCK in the Wiki/Manuals

[speedfixer]

re: ramdisk
MUCH faster, though did not time anything (working on nanomsg as an all-inclusive solution now.) Almost all modern Linux systems have ram tmp filesystems available, though I make my own. Really simple to do.


For file safety, your critical DATA file would be rewritten with a temp name, then when copy completed, backup killed, orig renamed backup, temp named to original. This is most secure: a backup always is present, no matter what.
If it is a random access file, backup at start and/or end, or timed.

For lock-indicator files, either of two naming conventions are simple.
1 - name, with changing estension, or
2 - changing name, with fixed extension

The changing part is the current status, example: free, held, busy, etc.
If you have multiple files to lock, it is easy to search for the same extension or same name.

You app must do its own management, and shut down or fix locks.
I try twice or run a loop for a specific amount of time, then go to an error routine.
In my case, I write an identifier of who has the lock in the LOCK file, with a timestamp. This way, I can see who the offender is that did not properly unlock the file.

Whether crash or 'freeze' point, you always know who is doing what.

Many apps: only ONE file and access library for that particular file. Think higher level.
Whoever wants something from the file, or wants to put something in the file, think of that file as a seperated service, independent of your app. Your library can take care of itself. If you make a change to the file format, only the library needs to change, the app can stay the same with just a recompile. Just keep the library interface the same.

david

[Boris the Old]

@Mike

And, I also spent some years with IBM (but... way-back-when!), and I understand your comment! For lurkers, the development approach was to FULLY document each new program's tasks first. Also, superb docs were available on the OS environment, compilers, runtime environments, etc. (No "help-desks!).

I worked at the IBM Lab, Toronto, in the late '60s and early '70s - mostly on new telecommunications techniques. My office had two walls full of program logic manuals, OS internals reference manuals, and equipment wiring diagrams. My main project was to develop a real-time banking network simulator so that it could be used to test new banking hardware and software. Yes folks, we did actually test stuff. :-) I enjoyed that project. There were no formal specs and I was given complete freedom to code it any way I chose. So I chose Assembler, and ran it as a dedicated task on the biggest mainframe system they had. In an environment like that it's amazing how much one can learn about programming.

Rod

[MJK]

I worked at the IBM Lab, Toronto, in the late '60s and early '70s - mostly on new telecommunications techniques. My office had two walls full of program logic manuals, OS internals reference manuals, and equipment wiring diagrams. My main project was to develop a real-time banking network simulator so that it could be used to test new banking hardware and software. Yes folks, we did actually test stuff. :-) I enjoyed that project. There were no formal specs and I was given complete freedom to code it any way I chose. So I chose Assembler, and ran it as a dedicated task on the biggest mainframe system they had.

I think you copied that paragraph from MY CV, though with a few tiny changes ;-) Based (but rarely working in) in EXACT SAME site, and only in early 70s. And the main project then was to CHECK if telecomms traffic SIMULATORs (mathematical models) were close to the real world (they were not!). And on-site, we had no local mainframe - we had to send the sources (on half-inch tape, I think) via train, to another city, for compilation!

I'm way off-topic here, so my apologies to all (others). If you (Rod) PM/email me (details should be on my profile here), I'll gladly supply any details I can remember!

- Mike

[MJK]

For file safety, your critical DATA file would be rewritten with a temp name, then when copy completed, backup killed, orig renamed backup, temp named to original. This is most secure: a backup always is present, no matter what.
If it is a random access file, backup at start and/or end, or timed.

Yep - excellent advice.

For lock-indicator files, either of two naming conventions are simple.
1 - name, with changing estension, or
2 - changing name, with fixed extension

The changing part is the current status, example: free, held, busy, etc.
If you have multiple files to lock, it is easy to search for the same extension or same name.

And the approach could be applied to one or more folders (eg, a temp/work folder), or to one or more files, or to one or more records (within one or more files), etc.

You app must do its own management, and shut down or fix locks.
I try twice or run a loop for a specific amount of time, then go to an error routine.
In my case, I write an identifier of who has the lock in the LOCK file, with a timestamp. This way, I can see who the offender is that did not properly unlock the file.

Whether crash or 'freeze' point, you always know who is doing what.

Many apps: only ONE file and access library for that particular file. Think higher level.
Whoever wants something from the file, or wants to put something in the file, think of that file as a seperated service, independent of your app. Your library can take care of itself. If you make a change to the file format, only the library needs to change, the app can stay the same with just a recompile. Just keep the library interface the same.

All excellent - thank you.

Mike

[Boris the Old]

Here are a few items to show how we handle file locking and data integrity. I hope it can be of use:

1) Overview

Our proprietary file access module, DVSAM, (based on the original IBM ISAM) has features that help with reliability and error detection. I wrote it 30 years ago for use with BASIC and COBOL applications running on DOS. The file structure is still the same, so all our current applications can be mixed with the old DOS code, some of which is still in use. DVSAM performs space recovery on the fly, so files never need to be rebuilt.

My aim was to make file I/O as reliable as possible in a shared environment, without using OS file locking features. We do this to overcome differences in OS locking strategies. The files include data, indexes (1 to 7), and various status blocks. The physical block size is 2048 bytes, but logical records are variable in length. All file I/O is handled in a single module (dll/so). The lock files are stored on a hard drive rather than on a ramdisk. This prevents information being lost in a major system crash. Our aim is to retain as much information as possible for data recovery purposes. Since an I/O request may require many physical read/write operations, the overhead of having the lock files on a hard drive is not significant.

2) Typical i/o request

User code: lock the file -- initiate the i/o request -- unlock the file.

I/O module:
note: every logical data, index, and status record has a 2-byte LRC field that is checked for reads and created for writes

lock the file:
attempt to rename the lock file -- if successful then store status info in the lock file

perform the i/o request:
read the file header block -- store status info in the header block -- rewrite the header block
perform various read/write operations to update the data and index blocks
read the file header block -- store status info in the header block -- rewrite the header block

unlock the file:
store status info in the lock file -- attempt to rename the lock file to its original name

3) Data recovery

LRC errors: The I/O module displays a detailed error message, and attempts to write the same info to disk

Lock/Unlock errors: The I/O module displays a timeout message, and attempts to write the same info to disk

System crash: A recovery program is used to scan the data directory, looking for such things as:
- lock files still in use
- file status showing that an i/o request was never completed
- read requests are usually safe, but anything else could mean that the file may be corrupted
- the output of the recovery program describes problem areas
- many situations can be fixed automatically, such as resetting lock files or file status, or re-indexing files
- depending on the application and the nature of the problems, a full restore from backup may be needed