LZLE List Engine with user friendly powerfull syntax - BETA 0.997a

User projects written in or related to FreeBASIC.
Post Reply
Lost Zergling
Posts: 534
Joined: Dec 02, 2011 22:51
Location: France

Re: LZLE List Engine with user friendly powerfull syntax

Post by Lost Zergling »

Link to Tutorial : viewtopic.php?f=9&t=26551&p=245161#p245161
I think I'd appreciate some more returns, tests, remarks, questions and so on. Every subjects open.
About licence : coded in pointers, mainly need pointers and datatypes to work. My intention is that the idea benefits the use and the spirit of Basic langage, not benefit to other language () and I find it legitimate that an idea can be protected for what it was made of.
Lost Zergling
Posts: 534
Joined: Dec 02, 2011 22:51
Location: France

Re: LZLE List Engine with user friendly powerfull syntax

Post by Lost Zergling »

Minor fixes
#1 AllOf : Added "If pFirstNode=pFIRSTFIRSTNode Then : If pLastNode<>pWhyteMove Then : this.Root : End If : End If"
to automate the need to call Root after BlindTag in Main Flat List
#2 NodeFlat : Added "If pFirstNode->pBranch=pFlatRoot Then : Return 0 : End If"
to prevent an element already reduce to be reduced twice
#3 NodeFlat : Two Replaced "this.NodeFlat : this.NodeRecycle2" by
"If pNode->Tag(1)="" Or pNode->Tag(1)=LIST_DEL Then : this.NodeFlat : this.NodeRecycle2 : End If"
to stop recursive reduce on parents which are Keys when Tracking

Evolutions
*1 NodeFlat : Added two "Or bPickReduce=1" and NFrecursive Property to enable/force recursive NodeFlat
behaviour outside the tree index parsing scope.
*2 Flat control : Added fMove(Integer) property on flat list : allowing user to code its own sorting features.
MrSwiss
Posts: 3910
Joined: Jun 02, 2013 9:27
Location: Switzerland

Re: LZLE List Engine with user friendly powerfull syntax

Post by MrSwiss »

FYI, the tutorial is currently at the *wrong place*, it belongs here (in project itself).

Reason: even if not explicitly stated, *Documentation* referres to: FB-Documentation.
(not anybody's doc./tutorial/whatever)
Lost Zergling
Posts: 534
Joined: Dec 02, 2011 22:51
Location: France

Re: LZLE List Engine with user friendly powerfull syntax

Post by Lost Zergling »

Hello,
Ok, I take note of it. This mistake was not wanted. I'll move the documentation subject to a new "General topic" and I'll update the links. Topic in documentation will be outdated. I rather prefer a separate topic for project itself and documentation because for me these are different topics to talk about : one (the project) is dedicated to bugs, technical or low level issues, the other one (documentation) is dedicated to how to use, code exemples, and conceptual remarks/requests. I was wondering if I should put this contribution in projects or in libraries, nevertheless as ambition is to propose an implementation overlay with a set of instructions, it seems to me more in the spirit to see it as a project. Welcome using it for some testing in 64 bits, or any suggestions.
Lost Zergling
Posts: 534
Joined: Dec 02, 2011 22:51
Location: France

Re: LZLE List Engine with user friendly powerfull syntax

Post by Lost Zergling »

Evolution
New Property "HashSort" : MyList.HashSort(1) => Every MyList.HashTag("Key") will be sorted (ascending)

For sorting : Just reduce list to protected "flat" list using NodeFlat(1) in a loop, then MyList.HashSort(1) and RestoreHash in a loop will restore sorted list.
Can be combined with other features like Snatching, RevParsing and so on. Few modifications, mainly using reduce & rollbackreduce to sort.

Code exemple :

Code: Select all

#Include once "D:\Basic\LZListsEngine20180301.bi"

Dim MaListe As List
MaListe.BranchCountDown(1)
MaListe.HashSort(1) ' ***************************Sorting here

Dim u as ulongint : Dim i as ulongint
MaListe.HashTag("a") : MaListe.Val("2")
Print "1"
'For u=1100000000 To 1100000030
For u=1100000030 To 1100000000 step -1
    MaListe.HashTag(str(u)) : MaListe.Val(str(u-1100000000))
Next u
For u=28 To 20 Step -1
    MaListe.HashTag(str(u)) : MaListe.Val(str(u))
Next u

Print "node count=" & Str(MaListe.Nodecount) & " count=" & Str(MaListe.count)
Print "++"
MaListe.First
MaListe.HashTag("8")  : MaListe.Val("55")
MaListe.Root 
MaListe.RootNode
Print "* " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count)  
'While MaListe.KeyStep=1
While MaListe.HashStep=1
    Print "* " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count)  
Wend
Print "---------------------"
sleep

Print "*Count=" & MaListe.Count
MaListe.Root
Print "#Count=" & MaListe.Count
While MaListe.HashStep=1
    If  (Cint(MaListe.Val) > 22  And Cint(MaListe.Val) <25)  Or Cint(MaListe.Val) =30  Then
       MaListe.NodeFlat     
    ElseIf Cint(MaListe.Val) =0 Then
       MaListe.NodeFlat     
    End If    
Wend
MaListe.Root
Print "Count=" & MaListe.Count
sleep

MaListe.Root
MaListe.RootNode
While MaListe.HashStep=1
    Print "% " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) '& " suivant=" & MaListe.NextNode  ': sleep
Wend
Print "F 01"
sleep

MaListe.Root
For i=1 To MaListe.AllOf : MaListe.BlindStep
    Print "+ " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count)     
Next i
sleep
 Print "F 02"
sleep
MaListe.Root
While MaListe.HashStep=1
    Print "% " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count)  '& " suivant=" & MaListe.NextNode ': sleep
Wend
Print "F 03"
sleep
MaListe.Root
MaListe.EndNode
Print "# EndNode=" & MaListe.Tag
Print "root1"
MaListe.Root
Print "root2"
MaListe.FlatStack : 
Print "--------------------" 

sleep

MaListe.Root
While MaListe.HashStep=1
    Print "~ " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ' & " suivant=" & MaListe.NextNode
Wend


Print "--"


MaListe.Last
Print MaListe.HashTag & " =Last ?" 
MaListe.Root
For u=1100000040 To 1100000050
    MaListe.HashTag(str(u)) : MaListe.Val(str(u-1100000000))
Next u
MaListe.Root
While MaListe.HashStep=1
     Print "$ " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ' & " suivant=" & MaListe.NextNode
Wend
MaListe.HashTag("52000067") : MaListe.Val("9")
sleep
MaListe.Root
MaListe.HashTag("32000067") : MaListe.Val("98")
MaListe.HashTag("62000067") : MaListe.Val("5")
MaListe.Root

While MaListe.HashStep=1
    Print "# " & MaListe.HashTag & " - " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ' & " suivant=" & MaListe.NextNode
Wend

MaListe.EndNode
Print MaListe.HashTag & " =EndNode ? "  & MaListe.Tag
MaListe.Last
Print MaListe.HashTag & " =Last ? " & MaListe.Tag : 
sleep

MaListe.Root
MaListe.EndNode : 'MaListe.Val("??")  
Print MaListe.HashTag & " =EndNode ?" :
MaListe.Last
Print MaListe.HashTag & " =Last ?" : sleep

MaListe.HashTag("9876") : MaListe.Val("9")
MaListe.Root

While MaListe.HashStep=1
    Print "##? " & MaListe.HashTag & " - " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ' & " suivant=" & MaListe.NextNode
Wend

Print "Fin 1" 
sleep

MaListe.Root

While MaListe.HashStep=1
    If Cint(MaListe.Val) < 18 Or ( Cint(MaListe.Val)<49 And Cint(MaListe.Val)>40 )  Then  ' Or ( Cint(MaListe.Val)<50 And Cint(MaListe.Val)>40 ) 
    '    Print "Flating " & MaListe.HashTag ': sleep
        MaListe.NodeFlat 
    Else
      '  Print "Parsing " & MaListe.HashTag ': sleep
   '     MaListe.NodeRecycle
    End If
Wend
 'MaListe.NodeRecycle
print "FIN"
'sleep
sleep :' system    
' MaListe.HashTag("1100000000") 

'If 1=0 Then
'MaListe.HashTag("32000067") 
MaListe.HashTag("1100000030") 
MaListe.BlindStep(-1) : Print "30->" & MaListe.Tag
MaListe.BlindStep(1) : Print "30->" & MaListe.HashTag
MaListe.Last : Print "30 Last->" & MaListe.HashTag
MaListe.HashTag("110000004") : Print "110000004 is branch=" & MaListe.BranchCount 
MaListe.HashTag("1100000040")
MaListe.BlindStep(-1) : Print "40->" & MaListe.Tag
MaListe.BlindStep(1) : Print "40->" & MaListe.HashTag
MaListe.Last : Print "40 Last->" & MaListe.HashTag
sleep

MaListe.HashTag("1100000041")
MaListe.Root : MaListe.First
Print "++**++"
While MaListe.HashStep=1
    Print "%% " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ' & " suivant=" & MaListe.NextNode
Wend
Print "Pt A01" : sleep
'MaListe.Root
MaListe.HashTag("110000001")
Print "%" & MaListe.HashTag & "%"
Print "Pt A02" : sleep


Print "~01"
MaListe.HashTag("1100000010")
MaListe.Root
Print "++**++"
While MaListe.HashStep=1
    Print "%-% " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count)  ' & " suivant=" & MaListe.NextNode
Wend

Print "~02"
sleep

Print "===================="
MaListe.Root
MaListe.FlatStack ': MaListe.Branch
Print "Count=" & MaListe.Count & " AllOf=" & MaListe.AllOf
Print "~03"
MaListe.RHMethod(-1)
For i=1 To MaListe.AllOf : MaListe.BlindStep
    Print "restoring " & MaListe.Tag ': sleep
    MaListe.RestoreHash
Next i
' MaListe.NodeRecycle
Print "--------------------" 

Print "Restored "
sleep


MaListe.Root
MaListe.RootNode
 MaListe.Val ("totor")
Print "*** " & MaListe.HashTag & " tag=" & MaListe.Tag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ' & " suivant=" & MaListe.NextNode

While MaListe.HashStep=1
  '   Print "% " & MaListe.Tag & " = " & MaListe.Val
    Print "** " & MaListe.HashTag & " tag=" & MaListe.Tag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ' & " suivant=" & MaListe.NextNode '  : sleep
Wend
Print "--------------------"


sleep

 'system


'MaListe.Root
'MaListe.BlindTag("")
Print "* flating" 
MaListe.Root
'While MaListe.HashStep=1 
'     Print "* flating -> " & MaListe.HashTag ' : sleep
'        MaListe.NodeFlat 
'Wend
'sleep
i=0
'dim i as integer=0
MaListe.Root
Print "count=" & MaListe.Count  : sleep
Print MaListe.Tag

MaListe.Root
Print "# " & MaListe.Tag : MaListe.BlindStep
Print "# " & MaListe.Tag : MaListe.BlindStep
Print "# " & MaListe.Tag : MaListe.BlindStep
Print "# " & MaListe.Tag : MaListe.BlindStep
'Print "**********"
'MaListe.BlindStep : Print MaListe.Tag
Print "**********"
Print "value=" & str(u) & " / " & str(i)

Print "FlatCount=" & MaListe.Count
Print "GarbageCount=" & MaListe.GarbageCount
Print "NodeCount=" & MaListe.NodeCount
Print "**********"
sleep
u=1
MaListe.Root 
MaListe.RootNode
Print "% " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ': sleep
While MaListe.HashStep=1
  '   Print "% " & MaListe.Tag & " = " & MaListe.Val
    Print "% " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ': sleep
    u+=1
Wend
sleep
 Print "% " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ': sleep
' MaListe.BlindStep
'Print "?= " & MaListe.tag
'MaListe.BlindStep
'Print "?= " & MaListe.tag
'MaListe.BlindStep
'Print "?= " & MaListe.tag

 Print "Node visibles parcourables = " & str(u) &  " Garbage=" & MaListe.GarbageCount '& " Flat=" & MaListe.FlatCount
 sleep
 
' Print "Node visibles parcourables = " & str(u) & " Garbage=" & MaListe.GarbageCount '& " Flat=" & MaListe.FlatCount
MaListe.First 'GarbagePtr
 Print "?? " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ': sleep
 sleep
Print str(MaListe.NodeCount) & " totaux"
sleep
MaListe.EndNode
Print "*? " & MaListe.hashTag & " / " & MaListe.Tag
'u=0 : MaListe.Root : While MaListe.HashStep=1 : u+=1 : Wend
Print MaListe.DropAll & " / " & MaListe.NodeCount & " / Total parse=" & u
 print "destroy"
MaListe.Destroy
Print "Fin **********"
sleep : system

paul doe
Moderator
Posts: 1730
Joined: Jul 25, 2017 17:22
Location: Argentina

Re: LZLE List Engine with user friendly powerfull syntax

Post by paul doe »

Lost Zergling wrote:I think I'd appreciate some more returns, tests, remarks, questions and so on. Every subjects open.
Ok, since I'm also coding some data structures I need, I decided to give this a go. These are my results (all code compiled with fbc 1.05.0):

32-bit
  • example 1: CRASHES at the end
  • example 2: CRASHES at the end
  • example 3: CRASHES at the end
  • example 4: CRASHES at the end
  • example 5: HANGS at the end
  • example 6: CRASHES at the end
  • example 7: CRASHES at the end, sometimes taking hostages (my code editor)
  • tuto001: OK
  • tuto002: OK
  • tuto003: OK
  • tuto004: OK
  • tuto005: OK
  • tuto006: CRASHES at the end
  • tuto007: has typo, fixed --> CRASHES at the end
  • hashSort: CRASHES at the end
64-bit
  • example 1: CRASHES HARD at the end: took my code editor with it
  • example 2: CRASHES RANDOMLY (sometimes it ends just by blind luck)
  • example 3: CRASHES at the end
  • example 4: CRASHES at the end
  • example 5: CRASHES at the end
  • example 6: CRASHES at the end
  • example 7: CRASHES at the end
  • tuto001: OK
  • tuto002: OK
  • tuto003: OK
  • tuto004: OK
  • tuto005: OK
  • tuto006: CRASHES at the end
  • tuto007: has typo, fixed --> CRASHES at the end
  • hashSort: prints a '1', then CRASHES immediately
Fortunately, the solution for the crashes at exit is somewhat simple: don't call list.destroy() property (which should be a METHOD, btw) directly from outside the object. Either:
  • make the method private and call it from the destructor only
or
  • include a flag member variable to prevent deallocating the list twice (once explicitly, and the other implicitly at object destruction)
But there's also some random crashes here and there, which tells me not all allocations/deallocations are managed correctly and that, unfortunately, is harder to solve. I could tell you more, but your coding conventions are less than ideal, and I don't feel like making a code beautifier just for this.

Another issue arises with shallow copies. This seemingly innocuous code will crash:

Code: Select all

#include once "lzle.bi"

scope
	dim as List myList '' Ok
	dim as List myList2 = mylist '' Not ok, creates a shallow copy of myList
	'' From now on, using either object will wreak havoc and probably crash
end scope
/'
	And here, the program invariably crashes, for destroying either object will cause 
	all the pointers from the other to become invalid since both objects share the
	same explicitly allocated memory
'/
That's because of the way FreeBasic handles objects. Since you're working with lots of pointers and explicitly allocated memory, you need to overload the copy constructor and let operator (and hence copy the object manually), or disable it altogether by making both private (in which case you'll need to provide another means to create a deep copy of the object).
Lost Zergling
Posts: 534
Joined: Dec 02, 2011 22:51
Location: France

Re: LZLE List Engine with user friendly powerfull syntax - Alpha1 Frozen (hopefully)

Post by Lost Zergling »

Hello Paul Doe
Thank you for interest. I'm on hollidays and I do not have a 64 bit machine for testing.
Be sure you test the last updated version (updated on 19/04)(twice the same day) (I just made a test and ex 2 does not crash at end on w7 32 bits, so I couldn't reproduce bugs you mentionned).
Theses end crashes where due to a multiple deallocation attemps on "garbagepointer", fixed : thanks to sancho3 posts.
dim as List myList2 = mylist lead to crash but should not be needed because there are alternate ways handling the object datas wich are Snatch, GarbageSnatch and FlatSnatch.
dim as List myList2 = mylist this syntax is not compatible with the way lzle has been designed, you're thrue.
And also it is not compliant with FBC object implementation, wich is a problem and a serious issue on wich I may investigate.
I think the copy constructor should be disabled.
You should use :
Dim As List MyList1, MyList2
.. working on list1...
myList2.Snatch(myList1) -> transferring datas from List1 to List2 (current node and sub levels)
myList2.GarbageSnatch(myList1) -> transferring dereferenced elements from List1 to List2
myList2.FlatSnatch(myList1) -> transferring protected flat list from List1 to List2

Then comes the random crash you mentionned : this looks like the critical fix on the 13/04 version.
Maybe you can check this ? Are your tests on Windows platforms ?
Do you have some other exemples of random crashes ?
I'm using FBide : sometimes I need to secure my tests by doing a compile job then a launch by double clic from Windows explorer : compile & run is not always 100% secure.
You mentionned : hashSort: prints a '1', then CRASHES immediately (64 bits) : on wich code exemple does this occurs ?

PS : do not forget to deactivate the licence message : sleep might lead to unexpectable results in some times.
paul doe
Moderator
Posts: 1730
Joined: Jul 25, 2017 17:22
Location: Argentina

Re: LZLE List Engine with user friendly powerfull syntax - Alpha1 Frozen (hopefully)

Post by paul doe »

:sigh:

Let's go through this, carefully and slowly:
Lost Zergling wrote:Hello Paul Doe
Thank you for interest. I'm on hollidays and I do not have a 64 bit machine for testing.
Be sure you test the last updated version (updated on 19/04)(twice the same day) (I just made a test and ex 2 does not crash at end on w7 32 bits, so I couldn't reproduce bugs you mentionned).
You're welcome. The version I'm using is the one in the first post, so I assume it's the latest one. The examples crash on both compilers, 32 and 64 bit versions. I'm using 1.05.0 and/or 1.06.0, compiling with the gcc backend. This is the complete command line switches of my usual compile: fbc -s console -mt -gen gcc -Wc -Ofast. I have a Win10 box, latest update (don't remember the number, sorry).
Lost Zergling wrote:Theses end crashes where due to a multiple deallocation attemps on "garbagepointer", fixed : thanks to sancho3 posts.
These end crashes are due to the code calling 'list.destroy' twice in succession:

Code: Select all

Destructor List : this.Destroy : End Destructor
'' Excerpt from exemple 1.bas follow
Print MaListe.DropAll & " / " & MaListe.NodeCount & " / Total parse=" & u
 print "destroy"
MaListe.Destroy
Print "Fin **********"
This is valid but not really needed. However, take for instance the tuto002.bas file (as it stands, it compiles ok): as soon as you add 'MyList.destroy()', the program crashes. This is of course totally previsible (you're deallocating memory twice), but in that case, the destroy() method should not be allowed to be public if it's called from the destructor. A possible solution:

Code: Select all

constructor List()
	m_destroyed = false '' m_destroyed is a private member variable
end constructor

destructor List()
	if( m_destroyed = false ) then
		m_destroyed = true
		destroy()
	end if
end destructor
That way, you'll avoid these issues. And most of the examples you provided call the destroy() method explicitly at the program end.
Lost Zergling wrote:dim as List myList2 = mylist lead to crash but should not be needed because there are alternate ways handling the object datas wich are Snatch, GarbageSnatch and FlatSnatch.
dim as List myList2 = mylist this syntax is not compatible with the way lzle has been designed, you're thrue.
Of course. But the posibility is there, and that amounts to poor design. Fortunately, the solution is also simple:

Code: Select all

type List
	public:
	'' ...
	private:
		'' Put the copy constructor and the let() assignment operators here
		declare constructor( byref as List )
		declare operator let( byref as List )
	'' ...
end type
There. Now, doing this:

Code: Select all

dim as List a, b

b = a
Will issue a compile time error, which is far better than simply crashing the app, or hope that the user of the library don't do something stupid (that's indeed valid and even expected syntax and behavior, unless you have a very good reason to not allow it, which I'll discuss later).
Lost Zergling wrote:And also it is not compliant with FBC object implementation, wich is a problem and a serious issue on wich I may investigate.
I think the copy constructor should be disabled.
Object orientation in FreeBasic has, like almost every other language, it's quirks and traps. Eventually, you'll grasp each and every one of them. For now, those two simple fixes I propose resolve some simple implementation oversights. However...
Lost Zergling wrote:Then comes the random crash you mentionned : this looks like the critical fix on the 13/04 version.
Maybe you can check this ? Are your tests on Windows platforms ?
Do you have some other exemples of random crashes ?
I'm using FBide : sometimes I need to secure my tests by doing a compile job then a launch by double clic from Windows explorer : compile & run is not always 100% secure.
You mentionned : hashSort: prints a '1', then CRASHES immediately (64 bits) : on wich code exemple does this occurs ?
...this is the meat of the question. I already told you my specs, and I use FbEdit (never had any trouble compiling anything with it).

Let's see:
  • the 'hashSort' example is the one you provided immediately above my previous post.
  • I have already detailed my results, in the aforementioned post. The worst offender is 'exemple 1.bas' which seems totally random. Sometimes it stops halfway, sometimes it goes on a little further, but it seems pretty random to me.
  • have a good look at 'exemple 5.bas'. The fact that the behavior of the example changes depending on which compiler you use (32-bit hangs, 64-bit crashes) probably means there's a pointer amidst the mess that has the wrong dimensions. You see, integer ptrs are 4 bytes on 32-bit, and 8 bytes on 64-bit. The integer (and it's uinteger equivalent) are the only ones that change size depending on the compiler used (as some members, like MrSwiss, cheerfully remind us from time to time =D)
  • the random crashes are probably due to off-by-one errors in pointer dereferences. Depending on how much memory it's allocated (and where), the dereference may *just* land in the pad area (normally 1 to 3 bytes) and you may get away with it. That would explain the extremely random nature of the crashes.
  • the 'hashSort' example is another one of the random crashers. Sometimes it prints a '1' and dies, other times it goes on a little longer, and sometimes it even finishes, provided you remembered to comment that 'MaListe.Destroy' call at the end...
So you see, there's definitely something going on buried inside the code. A closer inspection revealed me that the problem is, unsurprisingly, in the 'destroy()' method. The offender is the 'DropAll' property. 'Example 5.bas' crashes somewhere else, didn't see yet what the problem could be.

Does anybody beside me experience these crashes?
Last edited by paul doe on Apr 27, 2018 5:02, edited 2 times in total.
paul doe
Moderator
Posts: 1730
Joined: Jul 25, 2017 17:22
Location: Argentina

Re: LZLE List Engine with user friendly powerfull syntax - Alpha1 Frozen (hopefully)

Post by paul doe »

There seems to be a problem buried in the HashTag() method also. The 'hashSort' example I mentioned before crashes (most of the times) on a hashTag() call in the second loop:

Code: Select all

'' CRASHES randomly
#Include once "lzle.bi"

Dim MaListe As List

MaListe.BranchCountDown(1)
MaListe.HashSort(1)

Dim u as ulongint : Dim i as ulongint
MaListe.HashTag("a") : MaListe.Val("2")

Print "1"

? "Executing first loop..."
For u=1100000030 To 1100000000 step -1
    ? "  Executing HashTag..."
    MaListe.HashTag(str(u))
    ? "  Done"
    ? "  Executing Val..."
    MaListe.Val(str(u-1100000000))
    ? "  Done."
Next u
? "Done."

? "Executing second loop..."
For u=28 To 20 Step -1
  ? "  Executing HashTag..."
  MaListe.HashTag(str(u))
  ? "  Done."
  
  ? "  Executing Val..."
  MaListe.Val(str(u))
  ? "  Done."
Next u
? "Done."

Print "node count=" & Str(MaListe.Nodecount) & " count=" & Str(MaListe.count)
Print "++"
MaListe.First
MaListe.HashTag("8")  : MaListe.Val("55")
MaListe.Root
MaListe.RootNode
Print "* " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) 
'While MaListe.KeyStep=1
While MaListe.HashStep=1
    Print "* " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) 
Wend
Print "---------------------"
'

Print "*Count=" & MaListe.Count
MaListe.Root
Print "#Count=" & MaListe.Count
While MaListe.HashStep=1
    If  (Cint(MaListe.Val) > 22  And Cint(MaListe.Val) <25)  Or Cint(MaListe.Val) =30  Then
       MaListe.NodeFlat     
    ElseIf Cint(MaListe.Val) =0 Then
       MaListe.NodeFlat     
    End If   
Wend
MaListe.Root
Print "Count=" & MaListe.Count
'

MaListe.Root
MaListe.RootNode
While MaListe.HashStep=1
    Print "% " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) '& " suivant=" & MaListe.NextNode  ': '
Wend
Print "F 01"
'

MaListe.Root
For i=1 To MaListe.AllOf : MaListe.BlindStep
    Print "+ " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count)     
Next i
'
 Print "F 02"
'
MaListe.Root
While MaListe.HashStep=1
    Print "% " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count)  '& " suivant=" & MaListe.NextNode ': '
Wend
Print "F 03"
'
MaListe.Root
MaListe.EndNode
Print "# EndNode=" & MaListe.Tag
Print "root1"
MaListe.Root
Print "root2"
MaListe.FlatStack :
Print "--------------------"

'

MaListe.Root
While MaListe.HashStep=1
    Print "~ " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ' & " suivant=" & MaListe.NextNode
Wend


Print "--"


MaListe.Last
Print MaListe.HashTag & " =Last ?"
MaListe.Root
For u=1100000040 To 1100000050
    MaListe.HashTag(str(u)) : MaListe.Val(str(u-1100000000))
Next u
MaListe.Root
While MaListe.HashStep=1
     Print "$ " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ' & " suivant=" & MaListe.NextNode
Wend
MaListe.HashTag("52000067") : MaListe.Val("9")
'
MaListe.Root
MaListe.HashTag("32000067") : MaListe.Val("98")
MaListe.HashTag("62000067") : MaListe.Val("5")
MaListe.Root

While MaListe.HashStep=1
    Print "# " & MaListe.HashTag & " - " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ' & " suivant=" & MaListe.NextNode
Wend

MaListe.EndNode
Print MaListe.HashTag & " =EndNode ? "  & MaListe.Tag
MaListe.Last
Print MaListe.HashTag & " =Last ? " & MaListe.Tag :
'

MaListe.Root
MaListe.EndNode : 'MaListe.Val("??") 
Print MaListe.HashTag & " =EndNode ?" :
MaListe.Last
Print MaListe.HashTag & " =Last ?" : '

MaListe.HashTag("9876") : MaListe.Val("9")
MaListe.Root

While MaListe.HashStep=1
    Print "##? " & MaListe.HashTag & " - " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ' & " suivant=" & MaListe.NextNode
Wend

Print "Fin 1"
'

MaListe.Root

While MaListe.HashStep=1
    If Cint(MaListe.Val) < 18 Or ( Cint(MaListe.Val)<49 And Cint(MaListe.Val)>40 )  Then  ' Or ( Cint(MaListe.Val)<50 And Cint(MaListe.Val)>40 )
    '    Print "Flating " & MaListe.HashTag ': '
        MaListe.NodeFlat
    Else
      '  Print "Parsing " & MaListe.HashTag ': '
   '     MaListe.NodeRecycle
    End If
Wend
 'MaListe.NodeRecycle
print "FIN"
''
' :' system   
' MaListe.HashTag("1100000000")

'If 1=0 Then
'MaListe.HashTag("32000067")
MaListe.HashTag("1100000030")
MaListe.BlindStep(-1) : Print "30->" & MaListe.Tag
MaListe.BlindStep(1) : Print "30->" & MaListe.HashTag
MaListe.Last : Print "30 Last->" & MaListe.HashTag
MaListe.HashTag("110000004") : Print "110000004 is branch=" & MaListe.BranchCount
MaListe.HashTag("1100000040")
MaListe.BlindStep(-1) : Print "40->" & MaListe.Tag
MaListe.BlindStep(1) : Print "40->" & MaListe.HashTag
MaListe.Last : Print "40 Last->" & MaListe.HashTag
'

MaListe.HashTag("1100000041")
MaListe.Root : MaListe.First
Print "++**++"
While MaListe.HashStep=1
    Print "%% " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ' & " suivant=" & MaListe.NextNode
Wend
Print "Pt A01" : '
'MaListe.Root
MaListe.HashTag("110000001")
Print "%" & MaListe.HashTag & "%"
Print "Pt A02" : '


Print "~01"
MaListe.HashTag("1100000010")
MaListe.Root
Print "++**++"
While MaListe.HashStep=1
    Print "%-% " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count)  ' & " suivant=" & MaListe.NextNode
Wend

Print "~02"
'

Print "===================="
MaListe.Root
MaListe.FlatStack ': MaListe.Branch
Print "Count=" & MaListe.Count & " AllOf=" & MaListe.AllOf
Print "~03"
MaListe.RHMethod(-1)
For i=1 To MaListe.AllOf : MaListe.BlindStep
    Print "restoring " & MaListe.Tag ': '
    MaListe.RestoreHash
Next i
' MaListe.NodeRecycle
Print "--------------------"

Print "Restored "
'


MaListe.Root
MaListe.RootNode
 MaListe.Val ("totor")
Print "*** " & MaListe.HashTag & " tag=" & MaListe.Tag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ' & " suivant=" & MaListe.NextNode

While MaListe.HashStep=1
  '   Print "% " & MaListe.Tag & " = " & MaListe.Val
    Print "** " & MaListe.HashTag & " tag=" & MaListe.Tag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ' & " suivant=" & MaListe.NextNode '  : '
Wend
Print "--------------------"


'

 'system


'MaListe.Root
'MaListe.BlindTag("")
Print "* flating"
MaListe.Root
'While MaListe.HashStep=1
'     Print "* flating -> " & MaListe.HashTag ' : '
'        MaListe.NodeFlat
'Wend
''
i=0
'dim i as integer=0
MaListe.Root
Print "count=" & MaListe.Count  : '
Print MaListe.Tag

MaListe.Root
Print "# " & MaListe.Tag : MaListe.BlindStep
Print "# " & MaListe.Tag : MaListe.BlindStep
Print "# " & MaListe.Tag : MaListe.BlindStep
Print "# " & MaListe.Tag : MaListe.BlindStep
'Print "**********"
'MaListe.BlindStep : Print MaListe.Tag
Print "**********"
Print "value=" & str(u) & " / " & str(i)

Print "FlatCount=" & MaListe.Count
Print "GarbageCount=" & MaListe.GarbageCount
Print "NodeCount=" & MaListe.NodeCount
Print "**********"
'
u=1
MaListe.Root
MaListe.RootNode
Print "% " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ': '
While MaListe.HashStep=1
  '   Print "% " & MaListe.Tag & " = " & MaListe.Val
    Print "% " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ': '
    u+=1
Wend
'
 Print "% " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ': '
' MaListe.BlindStep
'Print "?= " & MaListe.tag
'MaListe.BlindStep
'Print "?= " & MaListe.tag
'MaListe.BlindStep
'Print "?= " & MaListe.tag

 Print "Node visibles parcourables = " & str(u) &  " Garbage=" & MaListe.GarbageCount '& " Flat=" & MaListe.FlatCount
 '
 
' Print "Node visibles parcourables = " & str(u) & " Garbage=" & MaListe.GarbageCount '& " Flat=" & MaListe.FlatCount
MaListe.First 'GarbagePtr
 Print "?? " & MaListe.HashTag & " = " & MaListe.Val & " branch " & str(MaListe.BranchCount) & " count=" & str(MaListe.Count) ': '
 '
Print str(MaListe.NodeCount) & " totaux"
'
MaListe.EndNode
Print "*? " & MaListe.hashTag & " / " & MaListe.Tag
'u=0 : MaListe.Root : While MaListe.HashStep=1 : u+=1 : Wend
Print MaListe.DropAll & " / " & MaListe.NodeCount & " / Total parse=" & u

Print "Fin **********"
There seems to be an older version of it, commented at the side. Do you happen to have the older version? Is it working correctly? What does hashTag() do? Traverse the tree until the 'hashtag' is found, and returns whether the tag was found?
Lost Zergling
Posts: 534
Joined: Dec 02, 2011 22:51
Location: France

Re: LZLE List Engine with user friendly powerfull syntax - Alpha1 Frozen (hopefully)

Post by Lost Zergling »

Hello Paul,
I have tested the random crash exemple, wich is the meat of the question. Well, I could reproduce it on W7 32 bits : your latest exemple
'' CRASHES randomly
I did change the following line :
Dim As uInteger uNodeCOUNT, uGarbCt, uCount, uContainerGarbCt, uContainerGivenCt
To
Dim As uLong uNodeCOUNT, uGarbCt, uCount, uContainerGarbCt, uContainerGivenCt
And it does not crash any longer.

Let's have a look to this post : https://freebasic.net/forum/viewtopic.php?f=17&t=26618 (thx to MrSwiss)
What I do not understand is why the bug is in 32 bits also and why it can be corrected also using the uLong instead of uInteger.
It appears that I have assumed my ability to solve this issue the way I had imagine.

My fault, I thought I could use uInteger while uLong is safe. But perhaps or probably is there something else I do not see ?
My fault again, from 19/04 to 21/04 I made slight changes and I didn't report it hoping no one would be impacted.

I thank you again for returns this really helps me for evolutions.
A by pass on Destroy is an idea, I wish to maintain a public access to Destroy() (for fbc 0.20 and above).
Same I wish to maintain the object implementation compliant with older versions.
Currently, I have no time working on further documentation because I'm investigating on random crashes, result sets and ascending compatibility issues on undocumented features. Probably this uLong instead of uInteger will solve some issues.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: LZLE List Engine with user friendly powerfull syntax - Alpha1 Frozen (Latest FIX 27/04)

Post by fxm »

I also obtain crashes with Ulong in 32-bit and 64-bit.
Lost Zergling
Posts: 534
Joined: Dec 02, 2011 22:51
Location: France

Re: LZLE List Engine with user friendly powerfull syntax - Alpha1 Frozen (Latest FIX 27/04)

Post by Lost Zergling »

On my side : W7 32 bits, fbc 0.23 : compile & run : no crash. Accelerate execution : crash. What's important is compile & run.
I continue my investigations,...
When it was running OK one time, then it becomes ok even though with "Accelerate execution" => Looks like a cache issue on pointers at compile time or somewhat else ? What a puzzle! I don't know what I could do more to fix this issue.
Last edited by Lost Zergling on Apr 27, 2018 11:09, edited 1 time in total.
fxm
Moderator
Posts: 12081
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: LZLE List Engine with user friendly powerfull syntax - Alpha1 Frozen (Latest FIX 27/04)

Post by fxm »

- First bug:
when allocating memory for a zstring, you must allocate an additional ubyte for the terminal character (0).
.....
Dim As zString Ptr Listptemp=Allocate(iLen+1), Listptemp2=Allocate(iLen+1), zp1, zp2, zp3=Allocate(1) ' * Alternative to Mid, left & right ==>+20-25% speed tested
.....
Dim As zString Ptr Listptemp=Allocate(Len(str_Tag)+1), zp1, zp2, zp3=Allocate(1)

- Second bug in 'Property List.HashTag(str_Tag As String) As Byte':
'len(this.HashTag)' may be greater than 'ilen' (up to 8 characters greater), so allocated memory overflows when executing '*Listptemp=Str_tmp'

- Eventual some next bugs because I stopped here my code review.
Last edited by fxm on Apr 27, 2018 11:15, edited 1 time in total.
Lost Zergling
Posts: 534
Joined: Dec 02, 2011 22:51
Location: France

Re: LZLE List Engine with user friendly powerfull syntax - Alpha1 Frozen (Latest FIX 27/04)

Post by Lost Zergling »

Ok I take note. Thx fxm.
dodicat
Posts: 7976
Joined: Jan 10, 2006 20:30
Location: Scotland

Re: LZLE List Engine with user friendly powerfull syntax - Alpha1 Frozen (Latest FIX 27/04)

Post by dodicat »

I allocated thus:
Dim As zString Ptr Listptemp=Allocate(iLen*sizeof(string)), Listptemp2=Allocate(iLen*sizeof(string)), zp1, zp2, zp3=Allocate(1*sizeof(string)) ' * Alternative to Mid, left & right ==>+20-25% speed tested

The ofending property is List.HashTag


The other offender is

List.HasHashTag
but is not used in your test, but needs fixed also.
Maybe an overkill to fxm's adding only one byte.
Post Reply