LZLE List Engine with user friendly powerfull syntax - BETA 0.997a

[fxm]

sizeof(string) corresponds to the size of var-len string descriptor (3 uinteger variables), no the size of one character !

In all rigor (if you prefer):
Allocate( ( iLen+1 ) * sizeof( zstring ) )

[Lost Zergling]

I'll post a fix later, whole retest highly recommended, .., speed issues also.

[paul doe]

Some remarks:

On my side : W7 32 bits, fbc 0.23 : compile & run : no crash. Accelerate execution : crash. What's important is compile & run.

You were advised before to use a more recent (ie 1.05.0) build of fbc. Many bugs and issues were addressed since the 0.23 version, and the first 'stable' release was considered to be 0.24. As MrSwiss told you, coding new things for old versions of fbc is asking for trouble. The code doesn't even work, much less has an already established user base, for you to worry about making it 'compatible' with older versions of the compiler.

When it was running OK one time, then it becomes ok even though with "Accelerate execution" => Looks like a cache issue on pointers at compile time or somewhat else ? What a puzzle! I don't know what I could do more to fix this issue.

There's no 'cache issues' with pointers, you have them dangling all over the place. You have like 120 functions or so, and pretty much all of them perform allocations and deallocations of memory, so screwing something up is not as difficult as you may imagine.

I'll post a fix later, whole retest highly recommended, .., speed issues also.

Retested, crashes (both on 32 and 64 bits, although it did get a little better). Also, don't worry about 'speed issues' yet, focus on getting it right, you can always optimize it later. As it stands, there's no use, you'll spend more time debugging the thing than recoding it from scratch...

[fxm]

Remark:
I am not sure that use zstring for returning a substring is faster than use string + mid(), because all allocations and deallocations must be taken into account.
On the other hand, it is a more delicate coding because it is necessary to avoid the memory overflows.
(same remark for left() and right())

Code: Select all

Dim As String s1 = "Viva FreeBASIC version 1.06"
Dim As Double t

Sleep 1000
t = Timer
For I As Integer = 1 To 10000000
  Scope
    Dim As String s2 = Mid(s1, 6, 9)
    'result is in s2
  End Scope
Next I
Print Cast(Single, Timer - t)

Sleep 1000
t = Timer
For I As Integer = 1 To 10000000
  Scope
    Dim As Zstring Ptr pz1 = Callocate(Len(s1) + 1)
    *pz1 = s1
    Dim As Zstring Ptr pz2 = pZ1 + 5
    pz2[9] = 0  '' or (*pz2)[9] = 0
    'result is in *pz2
    Deallocate(pz1)
  End Scope
Next I
Print Cast(Single, Timer - t)

Sleep

[dodicat]

Yea fxm.
I allocated sizeof(string) after I tried sizeof(zstring) which crashed of course.
Deep down I knew really that sizeof(string) was too much allocation, but it stopped the crash.
Here is an attempt at speeding up mid

Code: Select all

 #include "crt.bi"

function middle(byref _in as string, byval x as long,byval y as long) byref as const zstring
  static as zstring * 10000 g
  dim as ubyte ptr p=@g
      memcpy(p,Cast(Ubyte Ptr,Strptr(_in)) + x-1, y)
      return g
end function

dim as string s1,s2
dim as double t
dim as string freebasic="Freebasic"
dim as long limit=1000000

for z as long=1 to 10
t=timer
for n as long=1 to limit
    's1=""
    s1=middle(Freebasic,3,4)
next
print s1,timer-t,"middle",freebasic
sleep 50


t=timer
for n as long=1 to limit
    's2=""
    s2=mid(Freebasic,3,4)
next
print s2,timer-t,"mid",freebasic
print
next z
sleep



  

[fxm]

Yes but with a static allocation, this fails:
s1=middle(Freebasic,3,2) & middle(Freebasic,5,2)

[Lost Zergling]

@Paul Doe : I retested it 32 bits just copy-past LZLE and your exemple from web site : no crash on my side, even with uInteger.
I could replicate the random bug, one time, then it vanished. I don't know why, perhaps because I'm on a 0.23 (on this computer).
I do not believe that the software is not usable, I do not share at all your point of view on this subject. I also tested the code on a version 1.05 in 64 bits and I did not note any major differences of behavior of the code (except those related to the use of uInteger and a better speed in 64 bits).
I respect your opinion, but arguments you mentionned to this point did not change my mind.
I will continue to support this project, trying to fix bugs and answering questions. I respect your opinion about it.
Everyone shall make its own opinion about the usefulness of it.
I'm open minded and I accept the critics when they make me progresss. I will be very attentive to any other remarks.
@fxm :
"I am not sure that use zstring for returning a substring is faster than use string + mid(), because all allocations and deallocations must be taken into account."
I already tried to allocate a fixed size in constructor and deallocate in destructor : this was just a thick faster, drawbacks : fixed max size and really unstable(perhaps not only because the missing 0), so I thought it was not worth and I decided not to go further in this way. Nevertheless, my tests on "long keys" and lots of keys show a 20%-30% speed increase in this specific context. The code is just a string split, not a replace for mid, that's why it might be faster. About the bugs you mentionned : first bug : +1 car and end string 0 can be fixed (0 shall be fill when reach end string otherwise it is slow), the second one is more problematic because it is linked to speed. I tried a solution for theses 2 issues, but I need to validate if it still is intersting compared to Mid or not. ps : left & right are "free stuff" because pointers are already allocated/deallocated.

[paul doe]

@Paul Doe : I retested it 32 bits just copy-past LZLE and your exemple from web site : no crash on my side, even with uInteger.
I could replicate the random bug, one time, then it vanished. I don't know why, perhaps because I'm on a 0.23 (on this computer).
I do not believe that the software is not usable, I do not share at all your point of view on this subject. I also tested the code on a version 1.05 in 64 bits and I did not note any major differences of behavior of the code (except those related to the use of uInteger and a better speed in 64 bits).
I respect your opinion, but arguments you mentionned to this point did not change my mind.
I will continue this project, trying to fix bugs and answering questions. I respect your opinion about it.
Everyone shall make its own opinion about the usefulness of it.
I'm open minded and I accept the critics when they make me progresss. I will be very attentive to any other remarks.

That's ok, of course. Do whatever you want, it's your code, not mine. I'm sure it's very interesting and does something very cool, if you can get it off the ground (it still crashes at the end with the latest fix, in case you were wondering). So, I'm off for now. Best of luck.

[Lost Zergling]

@Paul Doe. I'll try to do my best even thought it is not perfect or bugged. Your post was interesting for me because it forces me to improve the level of requirement. Welcome back with suggestions as your choice.

[fxm]

- First bug:
when allocating memory for a zstring, you must allocate an additional ubyte for the terminal character (0).
.....
Dim As zString Ptr Listptemp=Allocate(iLen+1), Listptemp2=Allocate(iLen+1), zp1, zp2, zp3=Allocate(1) ' * Alternative to Mid, left & right ==>+20-25% speed tested
.....
Dim As zString Ptr Listptemp=Allocate(Len(str_Tag)+1), zp1, zp2, zp3=Allocate(1)

- Second bug in 'Property List.HashTag(str_Tag As String) As Byte':
'len(this.HashTag)' may be greater than 'ilen' (up to 8 characters greater), so allocated memory overflows when executing '*Listptemp=Str_tmp'

- Eventual some next bugs because I stopped here my code review.

- Just to check my hypotheses of bug, I modified as following two code lines:

Code: Select all

..........
''''Dim As zString Ptr Listptemp=Allocate(iLen), Listptemp2=Allocate(iLen), zp1, zp2, zp3=Allocate(1) ' * Alternative to Mid, left & right ==>+20-25% speed tested
    Dim As zString Ptr Listptemp=Allocate(iLen+8+1), Listptemp2=Allocate(iLen+1), zp1, zp2, zp3=Allocate(1) ' * Alternative to Mid, left & right ==>+20-25% speed tested
..........
''''Dim As zString Ptr Listptemp=Allocate(Len(str_Tag)), zp1, zp2, zp3=Allocate(1)
    Dim As zString Ptr Listptemp=Allocate(Len(str_Tag)+1), zp1, zp2, zp3=Allocate(1)
..........

and even with the uinteger type, I get no more crash (which often came close to the end and sometimes in the middle of the program).

- Remain all the other problems seen by paul doe.

[StringEpsilon]

Example #1 leaks memory.

Just the leaks valgrind attributed to LZListsEngine.bi:

==15494== 104 bytes in 1 blocks are definitely lost in loss record 237 of 246
==15494== at 0x4C2EEF5: calloc (vg_replace_malloc.c:711)
==15494== by 0x4024AA: LIST::ALLOWPANCAKE__get__() (LZListsEngine.bas:168)
==15494== by 0x403D12: LIST::VALPRIVATE__get__(FBSTRING&) (LZListsEngine.bas:292)
==15494== by 0x4064E1: LIST::VAL__get__(FBSTRING&) (LZListsEngine.bas:443)
==15494== by 0x414F70: main (test__.bas:243)
==15494==
==15494== 141 (104 direct, 37 indirect) bytes in 1 blocks are definitely lost in loss record 238 of 246
==15494== at 0x4C2EEF5: calloc (vg_replace_malloc.c:711)
==15494== by 0x4024AA: LIST::ALLOWPANCAKE__get__() (LZListsEngine.bas:168)
==15494== by 0x403D12: LIST::VALPRIVATE__get__(FBSTRING&) (LZListsEngine.bas:292)
==15494== by 0x4064E1: LIST::VAL__get__(FBSTRING&) (LZListsEngine.bas:443)
==15494== by 0x413074: main (test__.bas:119)
==15494==
==15494== 141 (104 direct, 37 indirect) bytes in 1 blocks are definitely lost in loss record 239 of 246
==15494== at 0x4C2EEF5: calloc (vg_replace_malloc.c:711)
==15494== by 0x4024AA: LIST::ALLOWPANCAKE__get__() (LZListsEngine.bas:168)
==15494== by 0x403D12: LIST::VALPRIVATE__get__(FBSTRING&) (LZListsEngine.bas:292)
==15494== by 0x4064E1: LIST::VAL__get__(FBSTRING&) (LZListsEngine.bas:443)
==15494== by 0x413792: main (test__.bas:140)
==15494==
==15494== 141 (104 direct, 37 indirect) bytes in 1 blocks are definitely lost in loss record 240 of 246
==15494== at 0x4C2EEF5: calloc (vg_replace_malloc.c:711)
==15494== by 0x4024AA: LIST::ALLOWPANCAKE__get__() (LZListsEngine.bas:168)
==15494== by 0x403D12: LIST::VALPRIVATE__get__(FBSTRING&) (LZListsEngine.bas:292)
==15494== by 0x4064E1: LIST::VAL__get__(FBSTRING&) (LZListsEngine.bas:443)
==15494== by 0x402C12: LIST::GARBAGECOLLECTOR__get__() (LZListsEngine.bas:199)
==15494== by 0x40EEAD: LIST::DROPALL__get__() (LZListsEngine.bas:928)
==15494== by 0x40FBDF: LIST::DESTROY__get__() (LZListsEngine.bas:979)
==15494== by 0x416F92: main (test__.bas:312)

Edit: Your code is *really* hard to read. Why is so much code cramped into a single line with ":"?

Edit2: pValTmp and pValTmp->ListData aren't deallocated properly in the destructor.

[Lost Zergling]

Hello String Epsilon.
I am not very used to the logs of the grid of values. About memory leaks, do you know if it is a fixed leak, proportionnal with the size of the lists or both ? A fixed leak of a few kb is not very good and it's even pretty ugly I grant you, however it would not be unacceptable (for not too demanding users) what would not be the case of a significant memory loss proportional to the load. (ps pValTmp are misses)
I'm using ":" to save space on the screen so as to see a maximum code in the same time. Code is hard to read because on my programming style which is not very conventional. Moreover, this project is the result of a progressive evolution and therefore everything is not at the same level because of the evolutionary maintenance. I focused on time constraints and bug fixes. Thus, the software is a bit complex by some aspects.
@fxm : bug 2 is up to my point of view a variable number of caracters depending on the keys structure. to be fixed (I got a solution wich cost is 2-5% speed). Just need few time, working on it but now on holidays.

[fxm]

Just need few time, working on it but now on holidays.

Quelle zone pour les vacances scolaires (quelle région de France) ?

[Lost Zergling]

Nantes ;-)

[fxm]

Merci.
(end of personal message)