About sharing data between running executables

General FreeBASIC programming questions.
Tourist Trap
Posts: 2958
Joined: Jun 02, 2015 16:24

About sharing data between running executables

Post by Tourist Trap »

Hello,

I've been investigating since many days about how to share easily data between running exe (not using files). What I have found so far is the possibility to use a shared dll segment.

¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨
¨¨¨¨¨¨¨¨¨¨:¨Prog_a(1)¨:¨¨¨¨¨¨:¨Prog_a(2)¨:¨¨¨¨¨¨¨¨
¨¨¨¨¨¨¨¨¨¨¨¨¨¨/¨\¨¨¨¨¨¨¨¨¨¨¨¨¨¨/¨¨\¨¨¨¨¨¨¨¨¨¨¨¨
¨¨¨¨¨¨¨¨¨¨¨¨¨/¨¨¨\¨¨¨¨¨¨¨¨¨¨¨¨/¨¨¨¨\¨¨¨¨¨¨¨¨¨¨¨
¨¨¨¨¨¨¨¨¨¨¨¨/¨¨¨¨¨\¨¨¨¨¨¨¨¨¨¨/¨¨¨¨¨¨\¨¨¨¨¨¨¨¨¨¨
¨¨¨¨¨¨¨¨¨¨¨/¨¨¨¨¨¨¨\¨¨¨¨¨¨¨¨/¨¨¨¨¨¨¨¨\¨¨¨¨¨¨¨¨¨
DLLb¨::¨nonshared¨'Sharedsegment'¨¨nonshared¨::¨¨¨
¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨¨

I join a demo code below. But this is rather an unperfect solution for I haven't found how to get rid of those limitations :
  • It is not 100% fb since it will require a C++ dll -> I've been using ms visual c++ to set this one up
  • The executables instances have to be linked to the same dll, not a copy - even identical - found in another place (unless of course I'm doing wrong)
This all makes the solution look more like a multithreading variation (even easier). So I would like to know if someone knows how to get the possibilities extended?

And also, and it would be my first concern, how doing the job with 100% fb stuff ?

Example code of a 'dll shared segment' allowing data exchange between executables clones:

Code: Select all

="cpp file  compile as dll"

Code: Select all

//vcppproj01.h
#Ifdef VCPPPROJ01_EXPORTS
#define VCPPPROJ01_API __declspec(dllexport)
#else
#define VCPPPROJ01_API __declspec(dllimport)
#endif

// Cette classe est exportée de vcppproj01.dll
class VCPPPROJ01_API Cvcppproj01 {
public:
	Cvcppproj01(void);
	// TODO : ajoutez ici vos méthodes.
};

extern VCPPPROJ01_API int nvcppproj01;

VCPPPROJ01_API int fnvcppproj01(void);

Code: Select all

// vcppproj01.cpp : définit le point d'entrée pour l'application DLL.
//

#include "stdafx.h"
#include "vcppproj01.h"

#ifdef _MANAGED
#pragma managed(push, off)
#endif

#pragma data_seg(".sdata")    // start of section - must initialize! 

   extern "C" __declspec (dllexport) int _sharedData   = 10; 

#pragma data_seg() 

#pragma comment(linker, "/SECTION:.sdata,rws")

extern "C" __declspec (dllexport) INT GetSharedData() 
{
    return _sharedData;
} 

extern "C" __declspec (dllexport) INT SetSharedData(INT setData ) 
{
    _sharedData = setData;
	return _sharedData;
}

BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  ul_reason_for_call,
                       LPVOID lpReserved
					 )
{
	switch (ul_reason_for_call)
	{
	case DLL_PROCESS_ATTACH:
	case DLL_THREAD_ATTACH:
	case DLL_THREAD_DETACH:
	case DLL_PROCESS_DETACH:
		break;
	}
    return TRUE;
}

#ifdef _MANAGED
#pragma managed(pop)
#endif

Code: Select all

="bas file  compile as exe"

Code: Select all

''Program able to share a variable between multiple instances
''Via a c++ dll embedding a shared data section
''
''Usage : 
''compile to an exe, then launch the exe 2 times and more

	''Load library with the 'shared data segment'
	Dim As Any Ptr vcppLib = DyLibLoad( "vcppproj01" )

	''Load the functions that accesses to the shared data
	Dim _GetSharedData As Function () As Integer
	 _GetSharedData = DylibSymbol( vcppLib, "GetSharedData" )
	Dim _SetSharedData As Function (ByVal As Integer) As Integer
	 _SetSharedData = DylibSymbol( vcppLib, "SetSharedData" )

''Some utility to get a random ID to identify the instance
Dim Shared As String currentID 
	 currentID = ""
Dim Shared As Integer colorStamp
Declare Function ReturnCurrentID() As String 
Declare Function ShowID() As Integer

''Variables
Dim As String _str
Dim As Integer _number

''=================================================================MAIN
	''Initialize
	Randomize Timer
	Screen 0 : Cls
	 Locate 13,10 : ? "(long)Press [ESC] to leave"
	 ShowID()
	 

	''Interrogate the shareddata
	Do
		Locate 10,10 : ? "Content of shared data :" ;
							? _GetSharedData()
		Locate 12,10 : ? "Press [SPACE] to modify data"
		
		If InKey = Chr(32) Then
		 	Input "Enter a number to send in the shared pool"; _number
			 _SetSharedData(_number)
			Cls
			 Locate 13,10 : ? "Keep pressing [ESC] to leave"
			 ShowID()
		EndIf
	
	Loop Until InKey = Chr(27)
	
''=====================================================================
Sleep : End 0

Function ReturnCurrentID() As String
	If currentID = "" Then 
		Dim As String voyelTable(5)
		Dim As String consonantTable (10)
		Dim As String signTable(3)
		''Feed all this people
		For _index As Integer = 0 To 5
			 Read voyelTable(_index)
		Next _index
		For _index As Integer = 0 To 10
			 Read consonantTable(_index)
		Next _index
		For _index As Integer = 0 To 3
			 Read signTable(_index)
		Next _index
		For _i As Integer = 1 To 12
			currentID &= voyelTable(Int(Rnd*6))& _
			             consonantTable(Int(Rnd*11))& _ 
			             IIf (_i Mod 4 = 0, signTable(Int(Rnd*3)), "")
		Next _i
		 colorStamp = Int(Rnd*14)+1
	EndIf

	Return currentID
End Function 'STRING := ReturnCurrentID()
Function ShowID() As Integer
	Dim As Integer _initialLine = CsrLin()
	Dim As Integer _intitialColumn = Pos()
 		 Locate 1, 55 : ? "<Instance ID>"
		 Color 11, 3
		 Locate 2, 55 : ? ReturnCurrentID()
		 Color , colorStamp
		For _line As Integer = 1 To 25
		 	Locate _line, 80 : ? Space(1);
		Next _line
		 Color 7, 0
	Locate _initialLine, _intitialColumn
End Function 'INTEGER := ShowID()

voyels:
Data "", "a", "e", "1", "0", "U"
consonants:
Data "", "Z", "b", "c", "d", "Y", "f", "g", "H", "X", "j"
signs:
Data "", "-", "_", "@" 
Last edited by Tourist Trap on Jul 14, 2015 17:11, edited 2 times in total.
vdecampo
Posts: 2992
Joined: Aug 07, 2007 23:20
Location: Maryland, USA
Contact:

Re: About sharing data between running executables

Post by vdecampo »

Under Windows, a shared memory segment may be an easier solution. Search MSDN for memory mapped files.

-Vince
RockTheSchock
Posts: 252
Joined: Mar 12, 2006 16:25

Re: About sharing data between running executables

Post by RockTheSchock »

As alternative you could use a file or several files in a directory. To speed things up you can use a ramdisk.
http://www.ltr-data.se/opencode.html/#ImDisk
on linux systems you can create a directory for your application under /dev/shm/ or you can use the POSIX shared memory API
http://man7.org/linux/man-pages/man7/sh ... iew.7.html

You need to manage the lock mechanism eventually with some additional files. Maybe just empty with the pid as filename.
Last edited by RockTheSchock on Jul 14, 2015 20:39, edited 1 time in total.
MichaelW
Posts: 3500
Joined: May 16, 2006 22:34
Location: USA

Re: About sharing data between running executables

Post by MichaelW »

There is an example that uses memory mapped files here:

http://www.freebasic.net/forum/viewtopi ... 23#p130623

Although I did not use the term, the example uses a "named" file mapping object, so other apps can use the same object by specifying the name.

And note that you can arrange for the file mapping object to be backed by the system paging file, so the sharing is done at memory speeds.
Tourist Trap
Posts: 2958
Joined: Jun 02, 2015 16:24

Re: About sharing data between running executables

Post by Tourist Trap »

For you understand what would be my final goal here, it is about making a sort of tiny database server.
vdecampo wrote:Under Windows, a shared memory segment may be an easier solution. Search MSDN for memory mapped files.
Thanks. Apparently it has been integrated to .net 4 (for the .net version, or of course I could try windows api but it's as usual a mess for the beginer); I'm sticking to version 2 for the moment. Worth looking at this closer however.
RockTheSchock wrote:As alternative you could use a file or several files in a directory. To speed things up you can use a ramdisk.
http://www.ltr-data.se/opencode.html/#ImDisk
or on linux systems you can create a directory for your application under /dev/shm/
You need to manage the lock mechanism eventually with some additional files. Maybe just empty with the pid as filename.
Thanks again. I've been looking around for how doing ram disk, I'll look at this link. I still remember using ram disk feature on Amiga to reduce load time of games running in the workbench. Of course at this time memory was not bunkerized as today.
MichaelW wrote:There is an example that uses memory mapped files here:

http://www.freebasic.net/forum/viewtopi ... 23#p130623

Although I did not use the term, the example uses a "named" file mapping object, so other apps can use the same object by specifying the name.
And note that you can arrange for the file mapping object to be backed by the system paging file, so the sharing is done at memory speeds.
Thank you all. I'll try this since I'm also looking for a solution that if possible wouldn't use external ressources from FB. The dll shared segment trick seems to be specifically C++.
grindstone
Posts: 862
Joined: May 05, 2015 5:35
Location: Germany

Re: About sharing data between running executables

Post by grindstone »

You also could use a pipe:
http://www.freebasic-portal.de/code-bei ... x-299.html
Put the code into a .bi and include it to your programme.

And if you don't mind that the desciption, comments and variable names are in german, you can take a look at this example how to use the windows message system for information interchange:
http://www.freebasic-portal.de/code-bei ... g-292.html

Regards
grindstone
marcov
Posts: 3462
Joined: Jun 16, 2005 9:45
Location: Netherlands
Contact:

Re: About sharing data between running executables

Post by marcov »

Or a more WIndows approach, communicate to an out of process COM server.

Anyway, I assume .NET has ready made solutions for this?
D.J.Peters
Posts: 8586
Joined: May 28, 2005 3:28
Contact:

Re: About sharing data between running executables

Post by D.J.Peters »

You can use Windows or Linux network sockets also.
On a local machine it's a (virtual) network device 127.0.0.1.
You can share/access or transfer GB's of memory per second between two different processes.
The network layers knows that your data never leaves your computer.
That means all data not transferred thru the (slow 10MBit/100MBit/1GBit) network device.
It use only the data bus same as the fast memcpy command.
I used this method at my business projects often in the past 20 years.

Joshy
(sorry about my bad english)
Tourist Trap
Posts: 2958
Joined: Jun 02, 2015 16:24

Re: About sharing data between running executables

Post by Tourist Trap »

So if I'm counting well, there would be 6 alternatives, 3 windows only, 1 linux only, 2 multiplatform.
  • Windows/Linux Sockets on local (virtual) network device 127.0.0.1
  • Windows COM approach
  • Windows/Linux Pipe
  • Windows Memory Mapped Files
  • Linux POSIX shared memory API
  • Windows Files + RamDisk
I may be wrong but they all seem to me more complicated than sharing a data segment of a cpp dll.
I would like to ask to people who have already implemented some of them, left aside any other criterias, which is the easiest at a pure coding point of view?
marcov
Posts: 3462
Joined: Jun 16, 2005 9:45
Location: Netherlands
Contact:

Re: About sharing data between running executables

Post by marcov »

Tourist Trap wrote:So if I'm counting well, there would be 6 alternatives, 3 windows only, 1 linux only, 2 multiplatform.
  • Windows/Linux Sockets on local (virtual) network device 127.0.0.1
  • Windows COM approach
  • Windows/Linux Pipe
  • Windows Memory Mapped Files
  • Linux POSIX shared memory API
  • Windows Files + RamDisk
I may be wrong but they all seem to me more complicated than sharing a data segment of a cpp dll.
There are also various other forms of Windows IPC like mailslots,rpc etc and of course good old clipboard. See e.g. https://msdn.microsoft.com/en-us/librar ... 85%29.aspx

Memory sharing is difficult with respect to atomicity and detecting when an event happened. IOW how do you know the data in the shared memory window has already been completed, and how do you avoid concurency issues.

And how do you detect if the value in the window is the same, if wasn't overwritten with the new data which happens to be the same.

All are solvable, but doesn't mean that shared memory is the best way. To be honest, I only found it beneficial to cheaply keep a copy of state in case a process with a buggy external component (GTK or COM) could die any minute.
Boris the Old
Posts: 139
Joined: Feb 04, 2011 20:34
Location: Ontario, Canada

Re: About sharing data between running executables

Post by Boris the Old »

@Tourist Trap

For situations like this I've always preferred to use disk files for communicating between tasks. It's simple, cross-platform, and works with both local and networked applications. Also, it doesn't require any API programming, and performance usually isn't an issue.

Rod
RockTheSchock
Posts: 252
Joined: Mar 12, 2006 16:25

Re: About sharing data between running executables

Post by RockTheSchock »

The most flexible surley is communication over TCP sockets. Your processes can run on the same computer or on different computers with even different OS. If you are using TCP sockets with localhost there are some performance tweaks for setting up the connection:

On Windows 8
TCP loopback fast path ( SIO_LOOPBACK_FAST_PATH )

On Linux if you are sending a lot of small packets you should disable buffering to reduce latency
tcpNoDelay=1 (TCP_NODELAY)
grindstone
Posts: 862
Joined: May 05, 2015 5:35
Location: Germany

Re: About sharing data between running executables

Post by grindstone »

Tourist Trap wrote:left aside any other criterias, which is the easiest at a pure coding point of view?
Obviously disk files!
marcov
Posts: 3462
Joined: Jun 16, 2005 9:45
Location: Netherlands
Contact:

Re: About sharing data between running executables

Post by marcov »

grindstone wrote:
Tourist Trap wrote:left aside any other criterias, which is the easiest at a pure coding point of view?
Obviously disk files!
I don't agree. The world of sharemodes and locking is a world of pain too, even more so if you try to do it cross computer (iow on network disk).

Detection is also by polling, which is also not nice, specially if there are multiple filenames to be used. Using only one filename makes a request blocking etc.

For small potato stuff, mailslots are the easiest I think. The advantage with sockets is that the experience gained in getting it to work will be beneficial anyway, and there is a sane cross-computer scenario
RockTheSchock
Posts: 252
Joined: Mar 12, 2006 16:25

Re: About sharing data between running executables

Post by RockTheSchock »

What the best depends on what you want to accomplish.

Criterias:
- low latency / very high number of changes / a lot of messages passed around
- big chunk of shared data mostly handled read only
- cross platform
...

the easiest to code means: you are very unflexible. You may run later into problems.
You said you want to write sort of a database engine. Why dont you use an existent one?

It's like you want to write another gui with a 2d drawing graphics lib and ask which library to use: fb-builtin / SDL / allegro ...

The universal answer: "it all depends!" and "Why not use an existing solution?"
Post Reply