CSV to fixlength data files

General FreeBASIC programming questions.
Gablea
Posts: 1104
Joined: Apr 06, 2010 0:05
Location: Northampton, United Kingdom
Contact:

Re: CSV to fixlength data files

Post by Gablea »

Ops sorry I keep forgetting when I send code you do not have access to my global files ;(

When I get home I shall upload the source code to my sever. (The only thing is it uses a lcd display on com1 so how would you see what it outputs)
badidea
Posts: 2586
Joined: May 24, 2007 22:10
Location: The Netherlands

Re: CSV to fixlength data files

Post by badidea »

grindstone wrote:
Gablea wrote:So far it has been up to a minute to "look" for a item but then errors out and says Item not found BUT when i search for it manually inside the actual data file i can find it.
Sounds rather like a bug than like a bottleneck.
I agree. I would add a lot of debug info (print each line read to screen or file) to pin down what is going on.

If my tests under linux are representative for freedos, line input would not speed up things.

Is it one system with performance issues or multiple? To be sure it it not a hardware problem, corrupt file, etc.
Gablea
Posts: 1104
Joined: Apr 06, 2010 0:05
Location: Northampton, United Kingdom
Contact:

Re: CSV to fixlength data files

Post by Gablea »

The development machine is running Windows 2000 512mb ram and 700mhz cpu. The other pc I have is a 3ghz 8GB ram 64bit machine and that is also runs it slowly.

I will upload the program to my server the only problem is it uses a comport for the user display (so how would you display that info on a normal screen?)
caseih
Posts: 2157
Joined: Feb 26, 2007 5:32

Re: CSV to fixlength data files

Post by caseih »

Hmm that might be your problem. Reading in 7000 lines is nothing, but printing them out to a com port is relatively slow, even if it might faster than printing each line to the screen. Maybe not horribly slow at higher baud rates (only about a few MB of data), but it will still skew the timings a lot. I'm not sure this is the entire cause of your slow down (doesn't explain why your query is taking many minutes but not finding anything), but it will be a contributing factor. Reading and parsing 7000 lines of text, even in an interpreted language like Python should be a fraction of a second.

Perhaps you should create a short test program that does nothing but read in the CSV file and then does a hard coded search for something. No need to output to the com port. That will give us something to work with to help you find the best way to do this. Otherwise all we can do is throw out random thoughts.
MrSwiss
Posts: 3910
Joined: Jun 02, 2013 9:27
Location: Switzerland

Re: CSV to fixlength data files

Post by MrSwiss »

Just based on what caseih, just wrote:
- which is the search criteria, you are employing?
- the string would tend to be slower, than for example, the "bar-code"
- the location in the line, would also speak for "bar-code", being at
the first position, in the line ... (no need, to process the whole line!)

In a DB, it would be called a "primary key" search ... based on the
assumption, that "bar-code" is a unique entity.
Gablea
Posts: 1104
Joined: Apr 06, 2010 0:05
Location: Northampton, United Kingdom
Contact:

Re: CSV to fixlength data files

Post by Gablea »

The barcode is unique as the main database is set not to allow duplicates.

The barcode number is the first one in the field.

I’m reading I. The file and then using

If barcode_scanned = product_barcode then

End if
MrSwiss
Posts: 3910
Joined: Jun 02, 2013 9:27
Location: Switzerland

Re: CSV to fixlength data files

Post by MrSwiss »

Okay, but are you keeping "search" appart from "Load_Line", which should
be called only, after the *match* is confirmed, by the "search" routine!
(saving time, wherever possible!)
dodicat
Posts: 7976
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: CSV to fixlength data files

Post by dodicat »

I tested some samples:
https://www.sample-videos.com/download-sample-csv.php

But using badidea's string, I create a .csv file on the first run.
Line 88 -- create
On subsequent runs comment out create.
This code is simplistic, I only have find and edit.
But creating add/delete would be simple enough.

I use Win 10 here.
I'll test on Linux later, but I have an inkling that (width) won't resize a terminal.
Anyway, and not in any way helpful to Gablea I am sure:

Code: Select all

 
Function StringSplit(s_in As String,chars As String,result() As String) As Long
    Dim As Long ctr,ctr2,k,n,LC=len(chars)
    dim As boolean tally(Len(s_in))
    #macro check_instring()
        n=0
        while n<Lc
        If chars[n]=s_in[k] Then 
        tally(k)=true
        If (ctr2-1) Then ctr+=1
        ctr2=0
        exit while
        end if
        n+=1
       wend
    #endmacro
   
    #macro split()
    If tally(k) Then
        If (ctr2-1) Then ctr+=1:result(ctr)=Mid(s_in,k+2-ctr2,ctr2-1)
        ctr2=0
    End If
    #endmacro
    '==================  LOOP TWICE =======================
    For k  =0 To Len(s_in)-1
        ctr2+=1:check_instring()
    Next k
    If ctr Then Redim result(1 To ctr): ctr=0:ctr2=0 Else  Return 0
    For k  =0 To Len(s_in)-1
        ctr2+=1:split()
    Next k
    '===================== Last one ========================
    If ctr2>0 Then
        Redim Preserve result(1 To ctr+1)
        result(ctr+1)=Mid(s_in,k+1-ctr2,ctr2)
    End If
    Return Ubound(result)
End Function

Function TALLY(SomeString As String,PartString As String) As Long
    Dim As Long LenP=Len(PartString),count
    Dim As Long position=Instr(SomeString,PartString)
    If position=0 Then Return 0
    While position>0
        count+=1
        position=Instr(position+LenP,SomeString,PartString)
    Wend
    Return count
End Function
 #Include "file.bi"
Sub savefile(filename As String,p As String)
    Dim As Integer n
    n=Freefile
    If Open (filename For Binary Access Write As #n)=0 Then
        Put #n,,p
        Close
    Else
        Print "Unable to save " + filename
    End If
End Sub

Function loadfile(file as string) as String
	If FileExists(file)=0 Then Print file;" not found":Sleep:end
   var  f=freefile
    Open file For Binary Access Read As #f
    Dim As String text
    If Lof(f) > 0 Then
      text = String(Lof(f), 0)
      Get #f, , text
    End If
    Close #f
    return text
end Function

#macro create
dim as string s="018341751055,""BAREFOOT CHARDONY"",0005,1,10,S,699,A,0,-,0,-,0,0,0,-,0,-,0,0,0,0"
dim as string g
for n as long=1 to 1000000-1
    mid(s,9)=right("000000"+str(n),4)
   g+= s+chr(10)
next

savefile("csv.csv",g)
print "file created, Press any key"
sleep
cls
#endmacro
create  '''<----------   for first run only

function find(path as string, num as string) as string
    dim as string s=loadfile(path)
     var i1= instr(s,num)
     if i1=0 then return "not found"
     redim as string a()
    stringsplit(s,chr(10),a())
     
    dim as long acc,n
    for n =lbound(a) to ubound(a)
        acc+=len(a(n))+1 'add 1 for missing chr(10)
        if acc>=i1 then exit for
    next
    return a(n)
end function

function merge(byval original as string,byval s as string) as string
    dim as string kb
    for n as long=33 to 128
        kb+=chr(n)
    next
    dim as string t
    dim as long flag
    for n as long=0 to len(s)-1
        if s[n]<>32 then flag=1
        if flag then t+=chr(s[n])
    next
    t=trim(t)
    dim as long i=instr(s,any kb)
    var first=mid(original,1,i-1)+ t
    var sec= instr(i,original,",")
    original=first + mid(original,sec)
    return original
    end function

sub edit(path as string,num as string,location as long=5)
    redim as string a()
    dim as string s=loadfile(path)
    stringsplit(s,chr(10),a())
    dim as string b=string(loword(width)," ")
    var i1= instr(s,num)
    if i1=0 then print "not found":exit sub
    dim as ulong acc,n
    for n =lbound(a) to ubound(a)
        acc+=len(a(n))+1
        if acc>i1 then exit for
    next
  
    dim as string  st=a(n)
    dim as string tmp,char,char2,ret
    lbl:
    do
    locate location
    color 15
    print st
    locate location+1
    line input tmp
    locate location+3
    color 7
    ret= merge(st,tmp)
    print ret
    print !"Accept?  y/n \n "
    char=input(1)
   if lcase(char)<>"y" then
       locate location+4
    print "Retry edit?  y/n  "
   char2=input(1)
   if lcase(char2)<>"y" then exit sub
   if  lcase(char2)="y" then
   char2=""
   char=""
   locate location
   print b
   locate location+1
   print b
   locate location+3
   print b
   locate location+4
   print b
    goto lbl
   end if
   end if
loop until lcase(char)="y"
locate location+5
print "Accepted"
a(n)=ret
dim as string edited
    for n=lbound(a) to ubound(a)
       edited+=a(n)+chr(10) 
   next
   savefile(path,edited)
end sub

dim as integer x
screeninfo x
width x\8-1

print "Search result:"
locate 2
print find("csv.csv","018341758193")
print
print "To edit use keys: tab,backspace,space,horizontal arrows to move the cursor" 
print "print new values at the cursor"
print "Press ENTER to save edits"
print "Press a key"
sleep


 'edit("csv.csv","Max Engle")
 edit ("csv.csv","018341758193",10)
 print "press a key"
 sleep
  
MrSwiss
Posts: 3910
Joined: Jun 02, 2013 9:27
Location: Switzerland

Re: CSV to fixlength data files

Post by MrSwiss »

Another thing, I've just noticed ...
The file read must be completed, before any "search" is attempted!
Gablea
Posts: 1104
Joined: Apr 06, 2010 0:05
Location: Northampton, United Kingdom
Contact:

Re: CSV to fixlength data files

Post by Gablea »

I though I was doing this

Code: Select all

Open the file 

Do
 Read inline
 If barcodenumber = productbarcode then
   Founditem = 1
   Exit do
End if

Loop
Close file

Select case founditem
 Case 0 ‘nothing found
  Display error message to the user

Case 1 ‘ item found
 Process the item and show in the displays the item description and the price 
End select
As I said I need help to make it more smother to use not only on the windows versions but also on the dos version.

I’m open to idea as how to improve the database function (just remember it has to work on the lowest support OS and that’s DOS for now)
MrSwiss
Posts: 3910
Joined: Jun 02, 2013 9:27
Location: Switzerland

Re: CSV to fixlength data files

Post by MrSwiss »

You want top speed, correct?

Then: keep the file "open" from program-start, to program-end ...
(all the "open" - "close" is using up time, which you don't have!)
Gablea
Posts: 1104
Joined: Apr 06, 2010 0:05
Location: Northampton, United Kingdom
Contact:

Re: CSV to fixlength data files

Post by Gablea »

So I would only close the files once the system shutdown or I need to do a data file update that should save a lot of time

but my issue is still it is not finding a product
MrSwiss
Posts: 3910
Joined: Jun 02, 2013 9:27
Location: Switzerland

Re: CSV to fixlength data files

Post by MrSwiss »

Gablea wrote:but my issue is still it is not finding a product
Please realize, that this is currently a "secondary" issue.
Meaning: until you've done, all the proposed improvements, it's moot to
discuss that, as a "separated issue" (should solve itself, in the process).

Everything is interconnected, therefore, splitting makes no sense ...
badidea
Posts: 2586
Joined: May 24, 2007 22:10
Location: The Netherlands

Re: CSV to fixlength data files

Post by badidea »

MrSwiss wrote:
Gablea wrote:but my issue is still it is not finding a product
Please realize, that this is currently a "secondary" issue.
Meaning: until you've done, all the proposed improvements, it's moot
to discuss that, as a "separated issue".

Everything is interconnected, therefore, splitting makes no sense ...
Well, if there is a bug in the program, then buying the fastest hardware and installing the latest OS is not likely to fix it.
Gablea
Posts: 1104
Joined: Apr 06, 2010 0:05
Location: Northampton, United Kingdom
Contact:

Re: CSV to fixlength data files

Post by Gablea »

I have uploaded the full project to my server you can download it herehttp://www.algpos.co.uk/KPoS.7z

in the pos.ini file please change the PathToServerDatabase=c:\FromServer to the path of where you have saved the Folder DOSData for example

mine might be PathToServerDatabase=c:\DosDrive\KPoS\DOSData

this simpulates the Network shared folder that the PoS application would download from

also I am using a LCD display on Com1 for the user display you may want to reroute the output to the screen somehow
Post Reply