Slow opening and closing of files

[jj2007]

the CPUID instruction is used to serialize the instruction execution

Again, this matters if you want to measure a handful of cycles. Here we are talking about Millions.

[IchMagBier]

Awesome, finally I can use ATT-syntax. :-)

Sadly this doesn't fix my problem with the slow Open and Close commands.

[dodicat]

Hi srvaldez.
I cannot get your code to run 32 or 64 bit.
Win 10

It compiles OK with -asm att.
But running produces a blank console which fizzles out after a second or so.

[jj2007]

Sadly this doesn't fix my problem with the slow Open and Close commands.

How do you define "slow"? For me, your code runs very fast:

Code: Select all

dim as uinteger mess_time
'Is it possible to use AT&T-syntax instead of Intel?
#macro start_mess()
   asm
      rdtsc
      mov qword ptr[mess_time],rax
   end asm
#endmacro

#macro end_mess(strin)
   asm
      mov rbx, qword ptr[mess_time]
      rdtsc
      sub rax, rbx
      mov qword ptr[mess_time], rax
   end asm
   print strin;
   print mess_time
#endmacro

print "go" : getkey
start_mess()
var of=freefile
end_mess("Freefile: ")
start_mess()
open "test.txt" for output as #of
end_mess("Open:     ")
start_mess()
print #of,!"Test\n"
end_mess("Print:    ")
start_mess()
close #of
end_mess("Close:    ")
sleep

Note the print "go" : getkey

On my Core i5, your code needs 2e6 cycles, i.e. roughly one millisecond. Is that slow?

[IchMagBier]

Well, it takes 198255 cycles for opening and 180859 for closing a file. Compared to only 6300 for actually writing a string to it. And I wonder if there is a faster way?
I have a program which processes some data and writes its output to a file. But the file-output (only Open and Close) is 10 times slower than the actual data processing. The program gets called a lot and I need it to be as fast as possible, because other programs depend on its output.

[srvaldez]

@IchMagBier
this is only my guess, I am no expert, could it be that when you close the file that at point the string is written to the file and then the file is closed ?, because the test string is small and only is printed to the file once, it's likely that the printing is done when you invoke the file close command, that is, it prints the string to the file and then closes the file.
@dodicat
I don't have time to adjust the code to support both 32 and 64-bit, but perhaps it's the RDTSCP instruction, could you test this code?

Code: Select all

dim as ulong time_high, time_low
dim as ulongint start_time, end_time

function if_rdtscp() as long
	dim as ulong reg_edx = 0
	asm
		"movl $0x80000001, %%eax \n" _
		"CPUID\n" _
		"mov %%edx, %0\n" _
		: "=r" (reg_edx) _
		:: "%rax", "%rcx", "%rdx"
	end asm
	function = (reg_edx shr 27) and 1
end function

'Is it possible to use AT&T-syntax instead of Intel?
'answer: yes.

#macro start_mess()
	asm
		"CPUID\n" _
		"RDTSC\n" _
		"mov %%edx, %0\n" _
		"mov %%eax, %1\n\t": "=r" (time_high), "=r" (time_low):: "%rax", "%rbx", "%rcx", "%rdx"
   end asm
   start_time = (culngint(time_high) shl 32) or time_low
#endmacro

#macro end_mess(strin)
	asm
		"RDTSCP\n" _
		"mov %%edx, %0\n" _
		"mov %%eax, %1\n" _
		"CPUID\n\t": "=r" (time_high), "=r" (time_low):: "%rax", "%rbx", "%rcx", "%rdx"
   end asm
   end_time = (culngint(time_high) shl 32) or time_low
   print strin;
   print end_time - start_time
#endmacro

if if_rdtscp() then
	start_mess()
	var of=freefile
	end_mess("Freefile: ")
	start_mess()
	open "test.txt" for output as #of
	end_mess("Open:     ")
	start_mess()
	print #of,!"Test\n"
	end_mess("Print:    ")
	start_mess()
	close #of
	end_mess("Close:    ")
else
	print "Your processor does not support the RDTSCP instruction"
end if

sleep

[jj2007]

Well, it takes 198255 cycles for opening and 180859 for closing a file. Compared to only 6300 for actually writing a string to it. And I wonder if there is a faster way?

No.

I have a program which processes some data and writes its output to a file. But the file-output (only Open and Close) is 10 times slower than the actual data processing. The program gets called a lot and I need it to be as fast as possible, because other programs depend on its output.

Change the program's logic. If one millisecond for opening and closing a file matters in your case, then something is wrong with the design. If other programs depend on its output, this probably means that you are passing data using files. That is a hack that you can occasionally use for testing something, but there are much, much faster ways for passing data. Consider WM_COPYDATA if you are on Windows.

[marcov]

Well, it takes 198255 cycles for opening and 180859 for closing a file. Compared to only 6300 for actually writing a string to it. And I wonder if there is a faster way?

Filesystem metadata (directories etc) often requires seeks. And seeks are still in the millisecond magnitude. Are you trying HDD or SSD?

- Disable antivirus and other programs that listen on filesystem changes. They are often slow.

- Directory processing is (at least on Windows) quite slower in large (thousands+ files) directories. Keep it small.

- You might be able to fiddle with the sharing options to trigger exclusive mode, which might be faster than sharing compatible.

[lassar]

In my dos version of RadioTelphone Tutor for Android. (using a dos emulator),

I notice, when I would open a topic, it took a while.

What my program does when opening a topic, is opening files, and inputting data.

But this is dos.

Maybe I will need to convert more string functions to assembly.

Maybe string comparisons in assembly.

[dodicat]

@IchMagBier
this is only my guess, I am no expert, could it be that when you close the file that at point the string is written to the file and then the file is closed ?, because the test string is small and only is printed to the file once, it's likely that the printing is done when you invoke the file close command, that is, it prints the string to the file and then closes the file.
@dodicat
I don't have time to adjust the code to support both 32 and 64-bit, but perhaps it's the RDTSCP instruction, could you test this code?

Hi srvaldez.
message to me:
Your processor does not support the RDTSCP instruction

That's Ebay for you.

[speedfixer]

I like the idea that metadata AND directory size would be the major reasons for the slow open/close operations.

If the the filesystem is the way you want to manage your data - for whatever reason - perhaps a ramdisk is your answer.

It is trivial in Linux, not hard in Windows.
You can do a mount with very little metadata in Linux if that is that is the actual reason for the slow open/close operations.


david

[jj2007]

Really, if one millisecond for opening + closing matters, then something is wrong in the design. I just made a little test sending repeatedly a 700k string from one program to another, and that takes typically about one millisecond. No big mess with the file system, just plain old InterProcess Communication.