JSON library (0.20.2)

[StringEpsilon]

Hi,

I was in need for a JSON library, so I made one. I tried to make the parser as fast as possible, using as little string-comparisons and copies as I could.

You can find the code on GitHub. The branch "master" should always be stable. Just copy "fbJson.bi" and the "fbJson"-folder to your project. If you separate the headers, you can also make a library.

The current stable release can be found here:

https://github.com/StringEpsilon/fbJson/releases

You can read and write files, generate new Json objects and arrays from scratch and get values as string.

Example 1: Read object from string:

Code: Select all

#include once "fbJson.bi"

dim as jsonItem item = jsonItem("{""Name"": ""fbJson"", ""Url"": ""https://github.com/StringEpsilon/fbJson"", ""Commits"": 21}")

print item["InvalidKey"].Value ' Access to invalid keys or an index out of bounds just returns an empty item.
print item["Name"].value
print item["Url"].Value
print item[2].Value

' Output: 
'
' fbJson
' https://github.com/StringEpsilon/fbJson
' 21

Example 2: Read array from string:

Code: Select all

#include "fbJson.bi"

dim as jsonItem array = jsonItem("[1,2,3,4,5,6,7]")

for i as integer = 0 to array.Count 
	print array[i].value & " ";
next

' Output: 
' 1 2 3 4 5 6 7

Feedback appreciated. You can also report issues on GitHub.

[StringEpsilon]

Small update: Since the initial post, I added methods to add and remove items, added a way to check if a key exists in an object, fixed a couple of bugs in the parser, and made the [] operator return byref which should reduce the number of (de)allocations. I will update the readme with the API-additions later.

[Roland Chastain]

Hello! Interesting project. Thank you for sharing.

I tested successfully all the examples with fbc 1.04.0. I just wonder why I get this warning:

C:\Atelier\Basic\fbjson\fbJson\JsonDocument.bi(50) warning 38(0): Mixing operand data types may have undefined results

Here is the line 50:

Code: Select all

function jsonDocument.SaveFile(path as string, overwrite as boolean = true) as boolean
	dim as string jsonFile = this.ToString()
	dim as integer ff = freefile()
	dim as integer fileError 
	
	if ( len(dir(path)) > 0 and overwrite = false ) then return false ' <----

I looked for a correction but didn't find.

[fxm]

if ( cbool(len(dir(path))) = true and overwrite = false ) then return false
or
if ( cbool(len(dir(path)) > 0) and overwrite = false ) then return false

[fxm]

However I think that that should work without warning:

Code: Select all

dim as string s
dim as boolean b

Print len(s) > 0 and b = true  '' warning 38(0): Mixing operand data types may have undefined results

Also, the warnings shouldn't appear anymore now if mixing booleans with 0/-1 (e.g. true and (-1)), or integers with true/false (e.g. 123 and true). If I remember correctly, that's how it was intended, it was just not working. This gives better backwards compatibility with code which used the old-school integer TRUE/FALSE definitions.

• (http://www.freebasic.net/forum/viewtopi ... 42#p211342)

[StringEpsilon]

I'll fix the warnings later.

I let them there, because I knew what they are about and mostly harmless - and because I thought it was a bug in FBC that will be fixed soon.

BTW: If you have any wishes, just tell me or throw me a pull-request. I'm still not sure if I should create ".GetBool", ".GetInteger", ".GetDouble", etc. - mostly because conversion-errors are hard to give back to the using code.

[AGS]

I tested your library and ran into some issues. I used the following program

Code: Select all

#define fbJson_DEBUG
#include once "../../fbJson.bi"

dim as jsonItem item = jsonItem("{""foo"":bar}")

if item.DataType = malformed then 
  print "problem"
else
  print "ok"
end if

You'd expect the output of the above code to be "problem" but it's actually "ok".
This is the official json grammar (taken from the json.org site)

Code: Select all

object
    {}
    { members } 
members
    pair
    pair , members
pair
    string : value
array
    []
    [ elements ]
elements
    value
    value , elements
value
    string
    number
    object
    array
    true
    false
    null 

As you can see value cannot be an arbitrary sequence of characters. It can be a string or a number or one of a small set of
predefined words (true false null) but not bar.

Once I found the above problem I tried to come up with some input that would crash your library.
And this is the input I came up with (just replace the dim - statement in the example above with the dim statement
given below)

Code: Select all

dim as jsonItem item = jsonItem("{""foo"":}")

Running the compiled example (using -exx) I get a runtime error

Code: Select all

Aborting due to runtime error 7 (null pointer access) at line 308 of D:\ProgramFiles\freebasic_1_0_4\test\fbJson-0.9.2\fbJson\JsonItem.bi::PARSE()

I also tried various badly mutilated floating point values

Code: Select all

5.e
5.
5e

The library accepted all of the above. And all of the above numbers are illegal according to the json grammar. There should be at least 1 digit after an e
and there should be at least one digit after a dot.

I also compiled one of the examples (read_complex.bas). It ran ok when used with the unaltered input file (complex_file.json).
But after removing the quotes of one of the string entries in complex_file.json (removed quotes from "spinnerToggle") the program
would still accept complex_file.json as a valid json file (which it isn't).

What definition of the json format are you using? Are you using the one that can be found at http://www.json.org/ ?

[StringEpsilon]

Hi AGS

I'm well aware that my lib is not 100% compliant with the specs. It does not support utf-8, it does not support string-escaping sequences and so on. I will give the lib version 1.0 if I feel that the specs are met reasonably well.

For the first error your found: I made a shortcut in the value property of jsonItem to deal with strings a bit easier when manipulating items in code (line 196 - 201 in jsonItem.bi), that probably causes the issue. I should think about that particular property a bit more.

What you observe with the malformed attribute is to be expected though. The malformed attribute is limited to the item where the parser found an issue in, it does not bubble up or down (yet). Implementing that should not take long.

The floating number stuff just irks me. I don't want to write a parser for that stuff too, just because the json-spec is less forgiving than the RTlib. I may do, but I'm much more comfortable using existing code for that. If you have a suggestion or a chunk of code, please let me know.

Thank you for the testing and the feedback. And to answer your last question: I consulted the RFCs when I thought I don't understand something, but mostly just wikipedia and example json files I found (for example, on json.org).

Edit: The goal also wasn't to go 100% with the specs, but to have something for other projects that works and is maintainable (by me). For example: I toyed around with using zstrings, but the way I build my parser it's a bit hassle using them. So utf8 support probably has to wait until I really need it (or someone throws me a patch).

[Imortis]

Interesting project. I always like string parsing stuff like this.

I have a question, though. I see a lot of stuff using JSON files lately. Why? What is the benefit over other data storage and retrieval schemes? Is it just preference or is there some tangible benefit to using it?

I ask out of curiosity, because I am pretty unfamiliar with the format as a whole. I know it exists, and have seen the innards of a few JSON files, but that is about it.

Thanks in advance to whoever answers.

[StringEpsilon]

The reasons why I decided to use JSON over XML:

* It's easier to read for humans.
* It has less stuff going on (An XML node has: A name, a value, n attributes. There is also CDATA and html-encoding.)
* Therefore, it's easier (and faster) to parse*
* Less overhead.
* It has some type-information (strings, bools, floating numbers, object, array, null)

XML has some benefits though, if you don't implement your own lib for it. Many libs support XPathes, stylesheet transformations (XSLT) and schema validation (XSD), which all can be pretty handy.

* The parser in fbJson is like 200 lines and avoids string comparisons and copy-operations where it can.

[StringEpsilon]

Since the last post, I changed quite a few bits. Most importantly: The parser is no longer recursive and slightly faster. The remaining bottlenecks are a bit harder to fix (like the single TRIM() statement I have).

* Reworked parser
* Overloaded LET() operator properly
* Stricter (= more standard conform) value parsing.
* Deescaping of stuff like /t and /r/n works now.
* Better debug output.

[rolliebollocks]

I made a JSON parser awhile back. Out of curiosity, can yours parse indefinitely nested arrays and objects? Mine has no XML compatibility.

[StringEpsilon]

I made a JSON parser awhile back. Out of curiosity, can yours parse indefinitely nested arrays and objects?

I have not tested it with absurd amounts of nesting, but the limiting factor should only be the machines memory - or the string size limit in free basic (if you have huge amounts of RAM).

I just tested this string:

Code: Select all

"[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]"

and that seems to work just fine (yes, that is valid JSON).

I also just pushed the latest version to GitHub that fixes all known issues with the parser.

[AGS]

I have downloaded and tested your library (trunk) and found some issues with it.

First up there is the standards-compliance.

The parser rejects a json file that contains a floating point number starting with a sign. Parsing the following input sets the flag _isMalformed to true (indicating parsing failed).

Code: Select all

{"key":+4.44}

The newest json standard permits any single value as the content of a json file. Meaning that the following is legal json.

Code: Select all

"hello world"

Your parser rejects any file that consists of a single occurrence of either a string, a number or any of the literals null true false.

And there are a couple of issues with the parser. I'll illustrate the problems with examples.

Problem 1 A string that ends with a single backslash gets accepted (but should be rejected)

Code: Select all

{"key":"value\n}

Problem 2 Using an empty string without a closing double quote leads to a segfault (using command-line option -exx)

Code: Select all

{"key":"}

The error message I get is

Code: Select all

Aborting due to runtime error 12 ("segmentation violation" signal) in D:\ProgramFiles\freebasic_1_0_4\test\fbjson\StringFunctions.bi::FASTMID()

Last and certainly least an observation and a question.

Objects (key - value pairs) are stored using a dynamic array. Finding a pair is implemented by the [] operator (when inserting a pair similar code gets executed). Code looks like this

Code: Select all

operator jsonItem.[](key as string) byref as jsonItem	
	if ( this._datatype = jsonObject ) then
		for i as integer = 0 to ubound(this._children)
			if ( this._children(i)->key = key ) then
				return *this._children(i)
			end if
		next
	end if

The for loop indicates that lookup time/insertion time of a pair depends upon the number of keys and the key searched for.

If the number of pairs is small looking up/inserting a pair will take little time. As the number of pairs increases so will the average time it takes to look up/insert a pair.

Which leads me to my question: why are the pairs not stored using a balanced search tree, a hash table, a skiplist , a trie etc...?

Code: Select all

compiler: FreeBASIC Compiler - Version 1.04.0 (10-01-2015), built for win32 (32bit) 
OS: windows 7 (64 bit)

[StringEpsilon]

First: Thank you very much for the bug reports. I'll fix them ASAP (today or tomorrow).

Last and certainly least an observation and a question.

Objects (key - value pairs) are stored using a dynamic array. Finding a pair is implemented by the [] operator (when inserting a pair similar code gets executed). Code looks like this

The for loop indicates that lookup time/insertion time of a pair depends upon the number of keys and the key searched for.

If the number of pairs is small looking up/inserting a pair will take little time. As the number of pairs increases so will the average time it takes to look up/insert a pair.

Which leads me to my question: why are the pairs not stored using a balanced search tree, a hash table, a skiplist , a trie etc...?

I thought about that, and I even have a working hashtable, but I have not gotten around implementing it. For small objects, it's not that slow at the moment, so I just put that aside to work on other stuff (like improving performance and standard compliance).

Edit: I already fixed the string parsing and the rejection of signed numbers (github issues #1 and #2). Feedback appreciated.

Edit2: Truncated quote.