CSV to fixlength data files
Re: CSV to fixlength data files
Ops sorry I keep forgetting when I send code you do not have access to my global files ;(
When I get home I shall upload the source code to my sever. (The only thing is it uses a lcd display on com1 so how would you see what it outputs)
When I get home I shall upload the source code to my sever. (The only thing is it uses a lcd display on com1 so how would you see what it outputs)
Re: CSV to fixlength data files
I agree. I would add a lot of debug info (print each line read to screen or file) to pin down what is going on.grindstone wrote:Sounds rather like a bug than like a bottleneck.Gablea wrote:So far it has been up to a minute to "look" for a item but then errors out and says Item not found BUT when i search for it manually inside the actual data file i can find it.
If my tests under linux are representative for freedos, line input would not speed up things.
Is it one system with performance issues or multiple? To be sure it it not a hardware problem, corrupt file, etc.
Re: CSV to fixlength data files
The development machine is running Windows 2000 512mb ram and 700mhz cpu. The other pc I have is a 3ghz 8GB ram 64bit machine and that is also runs it slowly.
I will upload the program to my server the only problem is it uses a comport for the user display (so how would you display that info on a normal screen?)
I will upload the program to my server the only problem is it uses a comport for the user display (so how would you display that info on a normal screen?)
Re: CSV to fixlength data files
Hmm that might be your problem. Reading in 7000 lines is nothing, but printing them out to a com port is relatively slow, even if it might faster than printing each line to the screen. Maybe not horribly slow at higher baud rates (only about a few MB of data), but it will still skew the timings a lot. I'm not sure this is the entire cause of your slow down (doesn't explain why your query is taking many minutes but not finding anything), but it will be a contributing factor. Reading and parsing 7000 lines of text, even in an interpreted language like Python should be a fraction of a second.
Perhaps you should create a short test program that does nothing but read in the CSV file and then does a hard coded search for something. No need to output to the com port. That will give us something to work with to help you find the best way to do this. Otherwise all we can do is throw out random thoughts.
Perhaps you should create a short test program that does nothing but read in the CSV file and then does a hard coded search for something. No need to output to the com port. That will give us something to work with to help you find the best way to do this. Otherwise all we can do is throw out random thoughts.
Re: CSV to fixlength data files
Just based on what caseih, just wrote:
- which is the search criteria, you are employing?
- the string would tend to be slower, than for example, the "bar-code"
- the location in the line, would also speak for "bar-code", being at
the first position, in the line ... (no need, to process the whole line!)
In a DB, it would be called a "primary key" search ... based on the
assumption, that "bar-code" is a unique entity.
- which is the search criteria, you are employing?
- the string would tend to be slower, than for example, the "bar-code"
- the location in the line, would also speak for "bar-code", being at
the first position, in the line ... (no need, to process the whole line!)
In a DB, it would be called a "primary key" search ... based on the
assumption, that "bar-code" is a unique entity.
Re: CSV to fixlength data files
The barcode is unique as the main database is set not to allow duplicates.
The barcode number is the first one in the field.
I’m reading I. The file and then using
If barcode_scanned = product_barcode then
End if
The barcode number is the first one in the field.
I’m reading I. The file and then using
If barcode_scanned = product_barcode then
End if
Re: CSV to fixlength data files
Okay, but are you keeping "search" appart from "Load_Line", which should
be called only, after the *match* is confirmed, by the "search" routine!
(saving time, wherever possible!)
be called only, after the *match* is confirmed, by the "search" routine!
(saving time, wherever possible!)
Re: CSV to fixlength data files
I tested some samples:
https://www.sample-videos.com/download-sample-csv.php
But using badidea's string, I create a .csv file on the first run.
Line 88 -- create
On subsequent runs comment out create.
This code is simplistic, I only have find and edit.
But creating add/delete would be simple enough.
I use Win 10 here.
I'll test on Linux later, but I have an inkling that (width) won't resize a terminal.
Anyway, and not in any way helpful to Gablea I am sure:
https://www.sample-videos.com/download-sample-csv.php
But using badidea's string, I create a .csv file on the first run.
Line 88 -- create
On subsequent runs comment out create.
This code is simplistic, I only have find and edit.
But creating add/delete would be simple enough.
I use Win 10 here.
I'll test on Linux later, but I have an inkling that (width) won't resize a terminal.
Anyway, and not in any way helpful to Gablea I am sure:
Code: Select all
Function StringSplit(s_in As String,chars As String,result() As String) As Long
Dim As Long ctr,ctr2,k,n,LC=len(chars)
dim As boolean tally(Len(s_in))
#macro check_instring()
n=0
while n<Lc
If chars[n]=s_in[k] Then
tally(k)=true
If (ctr2-1) Then ctr+=1
ctr2=0
exit while
end if
n+=1
wend
#endmacro
#macro split()
If tally(k) Then
If (ctr2-1) Then ctr+=1:result(ctr)=Mid(s_in,k+2-ctr2,ctr2-1)
ctr2=0
End If
#endmacro
'================== LOOP TWICE =======================
For k =0 To Len(s_in)-1
ctr2+=1:check_instring()
Next k
If ctr Then Redim result(1 To ctr): ctr=0:ctr2=0 Else Return 0
For k =0 To Len(s_in)-1
ctr2+=1:split()
Next k
'===================== Last one ========================
If ctr2>0 Then
Redim Preserve result(1 To ctr+1)
result(ctr+1)=Mid(s_in,k+1-ctr2,ctr2)
End If
Return Ubound(result)
End Function
Function TALLY(SomeString As String,PartString As String) As Long
Dim As Long LenP=Len(PartString),count
Dim As Long position=Instr(SomeString,PartString)
If position=0 Then Return 0
While position>0
count+=1
position=Instr(position+LenP,SomeString,PartString)
Wend
Return count
End Function
#Include "file.bi"
Sub savefile(filename As String,p As String)
Dim As Integer n
n=Freefile
If Open (filename For Binary Access Write As #n)=0 Then
Put #n,,p
Close
Else
Print "Unable to save " + filename
End If
End Sub
Function loadfile(file as string) as String
If FileExists(file)=0 Then Print file;" not found":Sleep:end
var f=freefile
Open file For Binary Access Read As #f
Dim As String text
If Lof(f) > 0 Then
text = String(Lof(f), 0)
Get #f, , text
End If
Close #f
return text
end Function
#macro create
dim as string s="018341751055,""BAREFOOT CHARDONY"",0005,1,10,S,699,A,0,-,0,-,0,0,0,-,0,-,0,0,0,0"
dim as string g
for n as long=1 to 1000000-1
mid(s,9)=right("000000"+str(n),4)
g+= s+chr(10)
next
savefile("csv.csv",g)
print "file created, Press any key"
sleep
cls
#endmacro
create '''<---------- for first run only
function find(path as string, num as string) as string
dim as string s=loadfile(path)
var i1= instr(s,num)
if i1=0 then return "not found"
redim as string a()
stringsplit(s,chr(10),a())
dim as long acc,n
for n =lbound(a) to ubound(a)
acc+=len(a(n))+1 'add 1 for missing chr(10)
if acc>=i1 then exit for
next
return a(n)
end function
function merge(byval original as string,byval s as string) as string
dim as string kb
for n as long=33 to 128
kb+=chr(n)
next
dim as string t
dim as long flag
for n as long=0 to len(s)-1
if s[n]<>32 then flag=1
if flag then t+=chr(s[n])
next
t=trim(t)
dim as long i=instr(s,any kb)
var first=mid(original,1,i-1)+ t
var sec= instr(i,original,",")
original=first + mid(original,sec)
return original
end function
sub edit(path as string,num as string,location as long=5)
redim as string a()
dim as string s=loadfile(path)
stringsplit(s,chr(10),a())
dim as string b=string(loword(width)," ")
var i1= instr(s,num)
if i1=0 then print "not found":exit sub
dim as ulong acc,n
for n =lbound(a) to ubound(a)
acc+=len(a(n))+1
if acc>i1 then exit for
next
dim as string st=a(n)
dim as string tmp,char,char2,ret
lbl:
do
locate location
color 15
print st
locate location+1
line input tmp
locate location+3
color 7
ret= merge(st,tmp)
print ret
print !"Accept? y/n \n "
char=input(1)
if lcase(char)<>"y" then
locate location+4
print "Retry edit? y/n "
char2=input(1)
if lcase(char2)<>"y" then exit sub
if lcase(char2)="y" then
char2=""
char=""
locate location
print b
locate location+1
print b
locate location+3
print b
locate location+4
print b
goto lbl
end if
end if
loop until lcase(char)="y"
locate location+5
print "Accepted"
a(n)=ret
dim as string edited
for n=lbound(a) to ubound(a)
edited+=a(n)+chr(10)
next
savefile(path,edited)
end sub
dim as integer x
screeninfo x
width x\8-1
print "Search result:"
locate 2
print find("csv.csv","018341758193")
print
print "To edit use keys: tab,backspace,space,horizontal arrows to move the cursor"
print "print new values at the cursor"
print "Press ENTER to save edits"
print "Press a key"
sleep
'edit("csv.csv","Max Engle")
edit ("csv.csv","018341758193",10)
print "press a key"
sleep
Re: CSV to fixlength data files
Another thing, I've just noticed ...
The file read must be completed, before any "search" is attempted!
The file read must be completed, before any "search" is attempted!
Re: CSV to fixlength data files
I though I was doing this
As I said I need help to make it more smother to use not only on the windows versions but also on the dos version.
I’m open to idea as how to improve the database function (just remember it has to work on the lowest support OS and that’s DOS for now)
Code: Select all
Open the file
Do
Read inline
If barcodenumber = productbarcode then
Founditem = 1
Exit do
End if
Loop
Close file
Select case founditem
Case 0 ‘nothing found
Display error message to the user
Case 1 ‘ item found
Process the item and show in the displays the item description and the price
End select
I’m open to idea as how to improve the database function (just remember it has to work on the lowest support OS and that’s DOS for now)
Re: CSV to fixlength data files
You want top speed, correct?
Then: keep the file "open" from program-start, to program-end ...
(all the "open" - "close" is using up time, which you don't have!)
Then: keep the file "open" from program-start, to program-end ...
(all the "open" - "close" is using up time, which you don't have!)
Re: CSV to fixlength data files
So I would only close the files once the system shutdown or I need to do a data file update that should save a lot of time
but my issue is still it is not finding a product
but my issue is still it is not finding a product
Re: CSV to fixlength data files
Please realize, that this is currently a "secondary" issue.Gablea wrote:but my issue is still it is not finding a product
Meaning: until you've done, all the proposed improvements, it's moot to
discuss that, as a "separated issue" (should solve itself, in the process).
Everything is interconnected, therefore, splitting makes no sense ...
Re: CSV to fixlength data files
Well, if there is a bug in the program, then buying the fastest hardware and installing the latest OS is not likely to fix it.MrSwiss wrote:Please realize, that this is currently a "secondary" issue.Gablea wrote:but my issue is still it is not finding a product
Meaning: until you've done, all the proposed improvements, it's moot
to discuss that, as a "separated issue".
Everything is interconnected, therefore, splitting makes no sense ...
Re: CSV to fixlength data files
I have uploaded the full project to my server you can download it herehttp://www.algpos.co.uk/KPoS.7z
in the pos.ini file please change the PathToServerDatabase=c:\FromServer to the path of where you have saved the Folder DOSData for example
mine might be PathToServerDatabase=c:\DosDrive\KPoS\DOSData
this simpulates the Network shared folder that the PoS application would download from
also I am using a LCD display on Com1 for the user display you may want to reroute the output to the screen somehow
in the pos.ini file please change the PathToServerDatabase=c:\FromServer to the path of where you have saved the Folder DOSData for example
mine might be PathToServerDatabase=c:\DosDrive\KPoS\DOSData
this simpulates the Network shared folder that the PoS application would download from
also I am using a LCD display on Com1 for the user display you may want to reroute the output to the screen somehow