[S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post your FreeBASIC source, examples, tips and tricks here. Please don’t post code without including an explanation.
Post Reply
D.J.Peters
Posts: 8586
Joined: May 28, 2005 3:28
Contact:

Re: [S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post by D.J.Peters »

I got it :-)
The optional param maxSize of the GetData() function was missing in "snc.bi".

Download SimpleNetworkConnection.zip again

If you receive any unknow size of data Data
you can fill a buffer while you get any data and interrupt if a timeout happens.

This timeout was the problem every call to GetData() was waiting this timeout delay.

If you receive data and knows it's size
you can fill a buffer while received bytes < dataSize or a timeout happens.

note: Transfering a large block of data in chunks of 8 KB is much faster as a chunk size of for example 256 KB.

You can compare the time if you change DATA_PAKET_SIZE from 1024*8 to 1024*256.

Joshy

Code: Select all

#include once "snc.bi"

const PORT = 12345

const DATA_PAKET_SIZE = 1024*8 ' 8 KB
dim as ubyte ptr pData,pDatapacket = allocate(DATA_PAKET_SIZE)

var t = timer()
var Server = new NetworkServer(PORT)
t=timer()-t
print "time to create a local server: " & t

t=timer()
var Client = new NetworkClient("127.0.0.1",PORT)
t=timer()-t
print "time to create a local client: " & t


t=timer()
var ToClient = Server->getConnection()
t=timer()-t
print "time to get client connection: " & t

t=timer()
var ToServer = Client->getConnection()
t=timer()-t
print "time to get server connection: " & t

print "mensured data tranfer rate"
dim as double tPut,tGet
t=timer()

dim as integer processed,nbytes
while processed < 1024*1024 ' 1 MB
  tPut=timer()
  nBytes = ToServer->PutData(pDatapacket,DATA_PAKET_SIZE)
  tPut=timer()-tPut
  print "time to put: " & tPut
  tGet=timer()
  nBytes=ToClient->GetData(pData,DATA_PAKET_SIZE)
  tGet=timer()-tGet
  print "time to get: " & tGet
  processed+=nBytes
wend
t=timer()-t
print
print "time to send and receive 1 MB in packages of 8 KB : " & t
print "done ..."

delete client
delete server

sleep
Last edited by D.J.Peters on Oct 09, 2016 18:06, edited 1 time in total.
D.J.Peters
Posts: 8586
Joined: May 28, 2005 3:28
Contact:

Re: [S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post by D.J.Peters »

Looks like the buffer size of a network socket from OS is 8 KB. (i tried different sizes in GetData() too)
10 MB = 10,485,760 bytes
10 MB / 256 KB = 40 packages
10 MB / 8 KB = 1280 packages
time to send and receive 1280 packpackages = 10 MB in chunks of 8 KB : 1.15 seconds.
time to send and receive 40 packpackages = 10 MB in chunks of 256 KB : 25.5 seconds.


Joshy

Code: Select all

#include once "snc.bi"

const PORT = 12345
const DATA_PAKET_SIZE = 1024*8  ' KB <-- change it to 1024*256
dim as ubyte ptr pData,pDatapacket = allocate(DATA_PAKET_SIZE)

var t = timer()
var Server = new NetworkServer(PORT)
t=timer()-t
print "time to create a local server: " & t

t=timer()
var Client = new NetworkClient("127.0.0.1",PORT)
t=timer()-t
print "time to create a local client: " & t


t=timer()
var ToClient = Server->getConnection()
t=timer()-t
print "time to get client connection: " & t

t=timer()
var ToServer = Client->getConnection()
t=timer()-t
print "time to get server connection: " & t

print "mensured data tranfer rate"
dim as double tPut,tGet
t=timer()

dim as integer processed,nBytes,npackages
while processed < 1024*1024*10 ' 10 MB
  tPut=timer()
  nBytes = ToServer->PutData(pDatapacket,DATA_PAKET_SIZE)
  tPut=timer()-tPut
  print "time to put: " & tPut
  tGet=timer()
  nBytes=ToClient->GetData(pData,DATA_PAKET_SIZE)
  tGet=timer()-tGet
  print "time to get: " & tGet
  processed+=nBytes
npackages+=1
wend
t=timer()-t
print
print "time to send and receive " & npackages & " packpackages = " & (npackages * DATA_PAKET_SIZE)\(1024*1024) & " MB in chunks of " & DATA_PAKET_SIZE\1024 & " KB : " & t & " seconds."
print "done ..."

delete client
delete server

sleep
Dinosaur
Posts: 1478
Joined: Jul 24, 2005 1:13
Location: Hervey Bay (.au)

Re: [S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post by Dinosaur »

Hi All

Joshy, being in Germany you must like working at night.;)

I know to the byte how many bytes I will receive, and never more then 80 bytes.
I put the Data_Packet_Size to 1024, downloaded the snc.bi
and then only changed the code from above to this:

Code: Select all

var status = Connection->GetData(.MesgPtr,Data_Packet_Size)
both in Client & Server.

However, it still took 2 seconds at the server & longer at the client to get the reply.

Because I KNOW how many bytes, I did not try to understand what you did to interrupt the time out.
Obviously I am now doing something wrong.

Regards
D.J.Peters
Posts: 8586
Joined: May 28, 2005 3:28
Contact:

Re: [S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post by D.J.Peters »

Dinosaur wrote:However, it still took 2 seconds at the server & longer at the client to get the reply.
I don't know why but I found out that between PutData()
and CanGet() must be a smal break like sleep 1. :-(
Dinosaur wrote:Joshy, being in Germany you must like working at night.
Yes I love the night.

The chicks are drunk.

The criminals are on the run.

I`m sitting in front of my monitor with a glass of Johnnie Walker.

What a wonderful life* ...

*(why not for all peoples on our planet what a f***)
D.J.Peters
Posts: 8586
Joined: May 28, 2005 3:28
Contact:

Re: [S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post by D.J.Peters »

A hint:
to find a problem I use:
while XXX->canPut()<>1: print "p"; : sleep 5: wend
and
while XXX->canGet()<>1: print "g"; : sleep 5: wend

If I see no "p" or "g" all is ok but if there are a bunch of "p" or "g" i know where to search ...

Joshy
Dinosaur
Posts: 1478
Joined: Jul 24, 2005 1:13
Location: Hervey Bay (.au)

Re: [S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post by Dinosaur »

Hi All

Joshy I personally prefer Dimple by Haig, no Ice No water.

Running your latest test.

Code: Select all

time to send and receive 1280 packages = 10 MB in chunks of 8 KB : 50.20832301536575 seconds.
done ...
But putting Sleep 100 between PutData & GetData did not make a difference, still 2 seconds,
Using the calculations from above, the time to transmit / Receive 80 Bytes should be about 5mSec ??

It is interesting that a Tx from Client to Server (without a reply) is less then 1 mSec.
As soon as I try to use Connection->CanGet after PutData it all slows down.

Regards

EDIT: Looks like we crossed emails
21 G's at Client CanGet
1 G at server CanGet
Last edited by Dinosaur on Nov 15, 2015 2:28, edited 1 time in total.
D.J.Peters
Posts: 8586
Joined: May 28, 2005 3:28
Contact:

Re: [S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post by D.J.Peters »

Test it self with and without sleep 1

Joshy

Code: Select all

#include once "snc.bi"

const PORT = 12345

const DATA_PAKET_SIZE = 1024*8 ' KB <-- change it to 1024*256
dim as ubyte ptr pData,pDatapacket = allocate(DATA_PAKET_SIZE)

var t = timer()
var aServer = new NetworkServer(PORT)
t=timer()-t
print "time to create a local server: " & t

t=timer()
var aClient = new NetworkClient("127.0.0.1",PORT)
t=timer()-t
print "time to create a local client: " & t

t=timer()
var aClientConnection = aServer->getConnection()
t=timer()-t
print "time to get client connection: " & t

t=timer()
var aServerConnection = aClient->getConnection()
t=timer()-t
print "time to get server connection: " & t

print "mensure data tranfer rate standing by ..."
dim as double tPut,tGet
t=timer()

#define delay scope : var tt=timer() : while timer()-tt<0.0009:wend : end scope

dim as integer processed,nBytes,npackages
while processed < 1024*1024*10 ' 10 MB
  while aServerConnection->canPut()<>1: print "p"; : sleep 5: wend
  nBytes = aServerConnection->PutData(pDatapacket,DATA_PAKET_SIZE)
  sleep 1 ' <--- without you will get many "g" on screen
  while aClientConnection->canGet()<>1: print "g"; : sleep 5: wend
  nBytes=aClientConnection->GetData(pData,DATA_PAKET_SIZE)
  processed+=nBytes
  npackages+=1
wend
t=timer()-t
print
print "time to send and receive " & npackages & " packpackages = " & (npackages * DATA_PAKET_SIZE)\(1024*1024) & " MB in chunks of " & DATA_PAKET_SIZE\1024 & " KB : " & t & " seconds."
print "done ..."

delete aClient
delete aServer

sleep
Dinosaur
Posts: 1478
Joined: Jul 24, 2005 1:13
Location: Hervey Bay (.au)

Re: [S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post by Dinosaur »

Hi All
time to create a local server: 5.696411244571209e-05
time to create a local client: 0.0001189983449876308
time to get client connection: 1.390755642205477e-05
time to get server connection: 1.065782271325588e-06
mensure data tranfer rate standing by ...


------------------
(program exited with code: 141)
Press return to continue
Without Sleep, yes lots of g's, but with it gives error.
EDIT: Test again, lots of g's both ways, so no difference with Sleep or not.
D.J.Peters
Posts: 8586
Joined: May 28, 2005 3:28
Contact:

Re: [S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post by D.J.Peters »

What is code 141 and where it comes from ?

Joshy
with SLEEP 1 wrote:time to create a local server: 0.009311237324896027
time to create a local client: 0.008338607764390105
time to get client connection: 0.0001530639424558444
time to get server connection: 3.210393776598153e-006
mensure data tranfer rate standing by ...

time to send and receive 1280 packpackages = 10 MB in chunks of 8 KB : 2.51... seconds.
done ...
without SLEEP wrote:time to create a local server: 0.007435950290654247
time to create a local client: 0.006591465886669923
time to get client connection: 0.0001134966305524898
time to get server connection: 2.385869009913222e-006
mensure data tranfer rate standing by ...
gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
ggggggggggggggggggggggggggggggggggggggggg
time to send and receive 1280 packpackages = 10 MB in chunks of 8 KB : 7.30...seconds.
done ...
Dinosaur
Posts: 1478
Joined: Jul 24, 2005 1:13
Location: Hervey Bay (.au)

Re: [S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post by Dinosaur »

Hi All

Just copied your test and ran it.
Perhaps I did not compile first time, so ignore error.
BUT, everytime with or without Sleep 1, the g's are filling the screen for 50 sec's until the end of the test, then at the end I got;

Code: Select all

gggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggggg
time to send and receive 1280 packpackages = 10 MB in chunks of 8 KB : 50.9403480385663 seconds.
done ...
EDIT:
Using my program, the 21 g's are still only 105 mSec, so that is not the error.
EDIT Again:
Sorry, my Error the sleep time was 100mSec, so 21 x 100 = 2.1 Sec which IS the amount of error time.
BUT, after the 21 g's there is at least 2 sec's before I get my reply printed.

Another EDIT:
If I "delete Connection" immediately after GetData is confirmed the Time is 25mSec to RX.
If I simply exit the sub, and wait for a new Connection from Client, then Time is 125 mSec to Rx.
Dinosaur
Posts: 1478
Joined: Jul 24, 2005 1:13
Location: Hervey Bay (.au)

Re: [S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post by Dinosaur »

Hi All

Joshy, I found a workaround.

Client
1. Open Connection with Server , wait till CanPut then Tx data.
2. Delete Connection without delay
3. IF a Reply is needed, Wait for a new Connection from Server, wait till CanGet then Rx data.
4. Delete Connection without delay.

Server
1. Repeatedly check if a Connection from Client, wait till CanGet then Rx Data.
2. Delete Connection without delay.
3. IF Rx Data says it expects a Reply; Then Open a new Connection with Client, wait till CanPut then Tx Data.
4. Delete Connection without delay.
5. Repeatedly check if a new Connection from Client.
With this scenario Tx time is always less the 1mSec and Rx time always around 20 to 25mSec.

I think your test works because you use loop back, but when two computers involved, Client may be ready to
Rx data while the Server is still busy reading and gets interupted by Client. (Maybe)

So, if you don't find a solution, don't worry, I am satisfied with what I have.
I will just manage, if another Client asks for Connection before Reply is sent.????

Regards

EDIT: Is there anyway you can make the calling Client IP addr available ?
then I can prevent sending reply to wrong Client.
Dinosaur
Posts: 1478
Joined: Jul 24, 2005 1:13
Location: Hervey Bay (.au)

Re: [S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post by Dinosaur »

Hi All

Joshy I thought you might like to see how I solved the Reply problem.
On each machine I create a Server & Client but on different ports, 12345 & 12346.
Part of the Server RxData has the IP address of the Client.
If it requires a Reply, then I call the Tx routine.

Time to Recv = 20mSec
Time to Tx = 0.4 mSec

Code: Select all

var Shared Server = NetworkServer(Ethernet.RxPort)
Sub EthRx () 
	' .RxPort is .TxPort on other pc's
	' This sub is continously polled, if data includes Reply req.
	' then reply data is formatted and EthTx is called once.
	With Ethernet
		var Connection = Server.GetConnection()
		if Connection > 0 then
			while Connection->CanGet() = 0
				Sleep 5
			Wend
			If Connection->CanGet() > 0 Then		
				var status = Connection->GetData(.RxPtr)
				If .RxPtr then
					.RxData  = *.RxPtr
					.MesgLen = Len(.RxData)
				End if
			End If
			Delete Connection
		End If
	End With
End Sub
Sub EthTx()
	' .TxPort is RxPort on other pc's
	' call this sub with: .TxData .TxIP (.TxIP is part of Data recvd.)
	With Ethernet
		.MesgLen = Len(.TxData) + 1
		If .MesgLen > 1 then
			var Client = NetworkClient(.TxIP,.TxPort)
			var Connection = Client.GetConnection()
			If Connection > 0 then
				While Connection->CanPut = 0
					Sleep 5
				Wend
				If Connection->CanPut() > 0 then
					var Status = Connection->PutData(StrPtr(.TxData),.MesgLen)
					If Status <> Len(.TxData) + 1 then
						var errorcode = Client.GetLastError()
						if errorCode <> ERROR_NO_ERROR then	
							.ErrMessage = GetNetworkErrorText(errorcode)
							.ErrCode = ErrorCode
						end if			
					Endif
				End if
			End If
			Delete Connection
		End if    
	End With
End Sub
Happy little chappy now.

Regards
podvornyak
Posts: 148
Joined: Nov 12, 2007 11:46
Location: Russia

Re: [S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post by podvornyak »

D.J.Peters wrote:Looks like the buffer size of a network socket from OS is 8 KB. (i tried different sizes in GetData() too)
10 MB = 10,485,760 bytes
10 MB / 256 KB = 40 packages
10 MB / 8 KB = 1280 packages
time to send and receive 1280 packpackages = 10 MB in chunks of 8 KB : 1.15 seconds.
time to send and receive 40 packpackages = 10 MB in chunks of 256 KB : 25.5 seconds.
In my case, package of 32kb size is fastest. 19 seconds. 8kb package transfers in 50 seconds. Linux 64.
cerveza
Posts: 8
Joined: Dec 27, 2013 21:07

Re: [S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post by cerveza »

World community grid recently switched all their web servers over to https.
This routine no longer works. If I change the port numbers to 443 I get a can't get " & ServerName & ServerFile
error message.

I'm not sure what needs changed in here.

Help.


#include once "snc.bi"
' test of a client connection
const as string LineEnd = chr(13,10)
const as string HeaderEnd = chr(13,10,13,10)
dim as string ServerName = "www.worldcommunitygrid.org"
dim as string ServerFile = "/team/viewTeamMemberDetail.do?sort=name&teamId=G5XMSGSVR1&pageNum=1&numRecordsPerPage=500&xml=true"
dim as string FileType = "text/html;text/plain"

' connect to web server at port 80
dim as NetworkClient client=type(ServerName,80)
' get a connection from ConnectionFactory
var connection = client.GetConnection()
' build an HTTP GET request
dim as string HTTPGET
HTTPGET = "GET " & ServerFile & " HTTP/1.0" & LineEnd
HTTPGET += "Host: " & ServerName & LineEnd
HTTPGET += "Accept: " & FileType & HeaderEnd

' ready to send ?
while connection->CanPut()<>1
sleep 100
wend
' put data on the connection
connection->PutData(strptr(HTTPGET),len(HTTPGET))
' ready to receive ?
while connection->CanGet()<>1
sleep 100
wend

dim as zstring ptr buffer
var nBytes = connection->GetData(buffer)
'print "number of received bytes " & nBytes


' get last char position of the HTTP asci header
var LastChar=instr(*buffer,HeaderEnd)-1
var Header =left(*buffer,LastChar)
' no OK answer ?
if instr(Header,"200 OK")<1 then
print header
print "can't get " & ServerName & ServerFile & " !"
beep:sleep:end
end if
' get first byte behind the HTTP header
var DataStart=LastChar+4
' save it
open "tdstatdata.xml" for binary access write as #1
dim as ubyte ptr FileBuffer=@buffer[DataStart]
nBytes-=DataStart
put #1,,*FileBuffer,nBytes
close #1
print "Web site data saved."
' free the buffer (allocate by snc.bi)
deallocate buffer
sleep 100
CLOSE
Roland Chastain
Posts: 992
Joined: Nov 24, 2011 19:49
Location: France
Contact:

Re: [S]imple [N]etwork [C]onnection win and lin 32/64-bit.

Post by Roland Chastain »

Hello! Congratulations, Joshy, for that library. Tested successfully under Windows 10.

Is it possible to call a web service which returns JSON data, like in this Pascal example?

Code: Select all

uses
  fphttpclient;
 
const
  URL = 'http://calapi.inadiutorium.cz/api/v0/fr/calendars/general-fr/today';
 
var
  vJSONText: string;
 
begin
  vJSONText := TFPCustomHTTPClient.SimpleGet(URL);
I don't see that operation in the examples.
Post Reply