JSON library (0.20.2)

User projects written in or related to FreeBASIC.
StringEpsilon
Posts: 41
Joined: Apr 09, 2015 20:49

JSON library (0.20.2)

Postby StringEpsilon » Dec 04, 2015 17:41

Hi,

I was in need for a JSON library, so I made one. I tried to make the parser as fast as possible, using as little string-comparisons and copies as I could.

You can find the code on GitHub. The branch "master" should always be stable. Just copy "fbJson.bi" and the "fbJson"-folder to your project. If you separate the headers, you can also make a library.

The current stable release can be found here:

https://github.com/StringEpsilon/fbJson/releases

You can read and write files, generate new Json objects and arrays from scratch and get values as string.

Example 1: Read object from string:

Code: Select all

#include once "fbJson.bi"

dim as jsonItem item = jsonItem("{""Name"": ""fbJson"", ""Url"": ""https://github.com/StringEpsilon/fbJson"", ""Commits"": 21}")

print item["InvalidKey"].Value ' Access to invalid keys or an index out of bounds just returns an empty item.
print item["Name"].value
print item["Url"].Value
print item[2].Value

' Output:
'
' fbJson
' https://github.com/StringEpsilon/fbJson
' 21


Example 2: Read array from string:

Code: Select all

#include "fbJson.bi"

dim as jsonItem array = jsonItem("[1,2,3,4,5,6,7]")

for i as integer = 0 to array.Count
   print array[i].value & " ";
next

' Output:
' 1 2 3 4 5 6 7


Feedback appreciated. You can also report issues on GitHub.
Last edited by StringEpsilon on Jun 09, 2018 22:53, edited 9 times in total.
StringEpsilon
Posts: 41
Joined: Apr 09, 2015 20:49

Re: JSON library

Postby StringEpsilon » Dec 07, 2015 19:30

Small update: Since the initial post, I added methods to add and remove items, added a way to check if a key exists in an object, fixed a couple of bugs in the parser, and made the [] operator return byref which should reduce the number of (de)allocations. I will update the readme with the API-additions later.
Roland Chastain
Posts: 851
Joined: Nov 24, 2011 19:49
Location: Dakar, Senegal
Contact:

Re: JSON library (0.9.1)

Postby Roland Chastain » Dec 21, 2015 16:16

Hello! Interesting project. Thank you for sharing.

I tested successfully all the examples with fbc 1.04.0. I just wonder why I get this warning:

C:\Atelier\Basic\fbjson\fbJson\JsonDocument.bi(50) warning 38(0): Mixing operand data types may have undefined results


Here is the line 50:

Code: Select all

function jsonDocument.SaveFile(path as string, overwrite as boolean = true) as boolean
   dim as string jsonFile = this.ToString()
   dim as integer ff = freefile()
   dim as integer fileError
   
   if ( len(dir(path)) > 0 and overwrite = false ) then return false ' <----


I looked for a correction but didn't find.
Last edited by Roland Chastain on Dec 23, 2015 9:30, edited 1 time in total.
fxm
Posts: 8905
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: JSON library (0.9.1)

Postby fxm » Dec 21, 2015 16:30

if ( cbool(len(dir(path))) = true and overwrite = false ) then return false
or
if ( cbool(len(dir(path)) > 0) and overwrite = false ) then return false
fxm
Posts: 8905
Joined: Apr 22, 2009 12:46
Location: Paris suburbs, FRANCE

Re: JSON library (0.9.1)

Postby fxm » Dec 21, 2015 17:17

However I think that that should work without warning:

Code: Select all

dim as string s
dim as boolean b

Print len(s) > 0 and b = true  '' warning 38(0): Mixing operand data types may have undefined results

dkl wrote:Also, the warnings shouldn't appear anymore now if mixing booleans with 0/-1 (e.g. true and (-1)), or integers with true/false (e.g. 123 and true). If I remember correctly, that's how it was intended, it was just not working. This gives better backwards compatibility with code which used the old-school integer TRUE/FALSE definitions.
StringEpsilon
Posts: 41
Joined: Apr 09, 2015 20:49

Re: JSON library (0.9.1)

Postby StringEpsilon » Dec 22, 2015 18:51

I'll fix the warnings later.

I let them there, because I knew what they are about and mostly harmless - and because I thought it was a bug in FBC that will be fixed soon.

BTW: If you have any wishes, just tell me or throw me a pull-request. I'm still not sure if I should create ".GetBool", ".GetInteger", ".GetDouble", etc. - mostly because conversion-errors are hard to give back to the using code.
AGS
Posts: 1284
Joined: Sep 25, 2007 0:26
Location: the Netherlands

Re: JSON library (0.9.2)

Postby AGS » Dec 28, 2015 20:09

I tested your library and ran into some issues. I used the following program

Code: Select all

#define fbJson_DEBUG
#include once "../../fbJson.bi"

dim as jsonItem item = jsonItem("{""foo"":bar}")

if item.DataType = malformed then
  print "problem"
else
  print "ok"
end if


You'd expect the output of the above code to be "problem" but it's actually "ok".
This is the official json grammar (taken from the json.org site)

Code: Select all

object
    {}
    { members }
members
    pair
    pair , members
pair
    string : value
array
    []
    [ elements ]
elements
    value
    value , elements
value
    string
    number
    object
    array
    true
    false
    null


As you can see value cannot be an arbitrary sequence of characters. It can be a string or a number or one of a small set of
predefined words (true false null) but not bar.

Once I found the above problem I tried to come up with some input that would crash your library.
And this is the input I came up with (just replace the dim - statement in the example above with the dim statement
given below)

Code: Select all

dim as jsonItem item = jsonItem("{""foo"":}")


Running the compiled example (using -exx) I get a runtime error

Code: Select all

Aborting due to runtime error 7 (null pointer access) at line 308 of D:\ProgramFiles\freebasic_1_0_4\test\fbJson-0.9.2\fbJson\JsonItem.bi::PARSE()


I also tried various badly mutilated floating point values

Code: Select all

5.e
5.
5e


The library accepted all of the above. And all of the above numbers are illegal according to the json grammar. There should be at least 1 digit after an e
and there should be at least one digit after a dot.

I also compiled one of the examples (read_complex.bas). It ran ok when used with the unaltered input file (complex_file.json).
But after removing the quotes of one of the string entries in complex_file.json (removed quotes from "spinnerToggle") the program
would still accept complex_file.json as a valid json file (which it isn't).

What definition of the json format are you using? Are you using the one that can be found at http://www.json.org/ ?
StringEpsilon
Posts: 41
Joined: Apr 09, 2015 20:49

Re: JSON library (0.9.2)

Postby StringEpsilon » Dec 30, 2015 0:10

Hi AGS

I'm well aware that my lib is not 100% compliant with the specs. It does not support utf-8, it does not support string-escaping sequences and so on. I will give the lib version 1.0 if I feel that the specs are met reasonably well.

For the first error your found: I made a shortcut in the value property of jsonItem to deal with strings a bit easier when manipulating items in code (line 196 - 201 in jsonItem.bi), that probably causes the issue. I should think about that particular property a bit more.

What you observe with the malformed attribute is to be expected though. The malformed attribute is limited to the item where the parser found an issue in, it does not bubble up or down (yet). Implementing that should not take long.

The floating number stuff just irks me. I don't want to write a parser for that stuff too, just because the json-spec is less forgiving than the RTlib. I may do, but I'm much more comfortable using existing code for that. If you have a suggestion or a chunk of code, please let me know.

Thank you for the testing and the feedback. And to answer your last question: I consulted the RFCs when I thought I don't understand something, but mostly just wikipedia and example json files I found (for example, on json.org).

Edit: The goal also wasn't to go 100% with the specs, but to have something for other projects that works and is maintainable (by me). For example: I toyed around with using zstrings, but the way I build my parser it's a bit hassle using them. So utf8 support probably has to wait until I really need it (or someone throws me a patch).
Imortis
Moderator
Posts: 1612
Joined: Jun 02, 2005 15:10
Location: USA
Contact:

Re: JSON library (0.9.2)

Postby Imortis » Dec 30, 2015 1:36

Interesting project. I always like string parsing stuff like this.

I have a question, though. I see a lot of stuff using JSON files lately. Why? What is the benefit over other data storage and retrieval schemes? Is it just preference or is there some tangible benefit to using it?

I ask out of curiosity, because I am pretty unfamiliar with the format as a whole. I know it exists, and have seen the innards of a few JSON files, but that is about it.

Thanks in advance to whoever answers.
StringEpsilon
Posts: 41
Joined: Apr 09, 2015 20:49

Re: JSON library (0.9.2)

Postby StringEpsilon » Dec 30, 2015 18:43

The reasons why I decided to use JSON over XML:

* It's easier to read for humans.
* It has less stuff going on (An XML node has: A name, a value, n attributes. There is also CDATA and html-encoding.)
* Therefore, it's easier (and faster) to parse*
* Less overhead.
* It has some type-information (strings, bools, floating numbers, object, array, null)

XML has some benefits though, if you don't implement your own lib for it. Many libs support XPathes, stylesheet transformations (XSLT) and schema validation (XSD), which all can be pretty handy.

* The parser in fbJson is like 200 lines and avoids string comparisons and copy-operations where it can.
StringEpsilon
Posts: 41
Joined: Apr 09, 2015 20:49

Re: JSON library (0.11)

Postby StringEpsilon » Mar 30, 2016 20:33

Since the last post, I changed quite a few bits. Most importantly: The parser is no longer recursive and slightly faster. The remaining bottlenecks are a bit harder to fix (like the single TRIM() statement I have).

* Reworked parser
* Overloaded LET() operator properly
* Stricter (= more standard conform) value parsing.
* Deescaping of stuff like /t and /r/n works now.
* Better debug output.
rolliebollocks
Posts: 2655
Joined: Aug 28, 2008 10:54
Location: new york

Re: JSON library (0.11)

Postby rolliebollocks » Apr 03, 2016 0:24

I made a JSON parser awhile back. Out of curiosity, can yours parse indefinitely nested arrays and objects? Mine has no XML compatibility.
StringEpsilon
Posts: 41
Joined: Apr 09, 2015 20:49

Re: JSON library (0.11)

Postby StringEpsilon » Apr 07, 2016 17:52

rolliebollocks wrote:I made a JSON parser awhile back. Out of curiosity, can yours parse indefinitely nested arrays and objects?


I have not tested it with absurd amounts of nesting, but the limiting factor should only be the machines memory - or the string size limit in free basic (if you have huge amounts of RAM).

I just tested this string:

Code: Select all

"[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[[]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]]"


and that seems to work just fine (yes, that is valid JSON).

I also just pushed the latest version to GitHub that fixes all known issues with the parser.
AGS
Posts: 1284
Joined: Sep 25, 2007 0:26
Location: the Netherlands

Re: JSON library (0.12)

Postby AGS » Apr 16, 2016 15:19

I have downloaded and tested your library (trunk) and found some issues with it.

First up there is the standards-compliance.

The parser rejects a json file that contains a floating point number starting with a sign. Parsing the following input sets the flag _isMalformed to true (indicating parsing failed).

Code: Select all

{"key":+4.44}


The newest json standard permits any single value as the content of a json file. Meaning that the following is legal json.

Code: Select all

"hello world"


Your parser rejects any file that consists of a single occurrence of either a string, a number or any of the literals null true false.

And there are a couple of issues with the parser. I'll illustrate the problems with examples.

Problem 1 A string that ends with a single backslash gets accepted (but should be rejected)

Code: Select all

{"key":"value\n}


Problem 2 Using an empty string without a closing double quote leads to a segfault (using command-line option -exx)

Code: Select all

{"key":"}


The error message I get is

Code: Select all

Aborting due to runtime error 12 ("segmentation violation" signal) in D:\ProgramFiles\freebasic_1_0_4\test\fbjson\StringFunctions.bi::FASTMID()


Last and certainly least an observation and a question.

Objects (key - value pairs) are stored using a dynamic array. Finding a pair is implemented by the [] operator (when inserting a pair similar code gets executed). Code looks like this

Code: Select all

operator jsonItem.[](key as string) byref as jsonItem   
   if ( this._datatype = jsonObject ) then
      for i as integer = 0 to ubound(this._children)
         if ( this._children(i)->key = key ) then
            return *this._children(i)
         end if
      next
   end if


The for loop indicates that lookup time/insertion time of a pair depends upon the number of keys and the key searched for.

If the number of pairs is small looking up/inserting a pair will take little time. As the number of pairs increases so will the average time it takes to look up/insert a pair.

Which leads me to my question: why are the pairs not stored using a balanced search tree, a hash table, a skiplist , a trie etc...?

Code: Select all

compiler: FreeBASIC Compiler - Version 1.04.0 (10-01-2015), built for win32 (32bit)
OS: windows 7 (64 bit)
StringEpsilon
Posts: 41
Joined: Apr 09, 2015 20:49

Re: JSON library (0.12)

Postby StringEpsilon » Apr 16, 2016 16:09

First: Thank you very much for the bug reports. I'll fix them ASAP (today or tomorrow).

AGS wrote:Last and certainly least an observation and a question.

Objects (key - value pairs) are stored using a dynamic array. Finding a pair is implemented by the [] operator (when inserting a pair similar code gets executed). Code looks like this

The for loop indicates that lookup time/insertion time of a pair depends upon the number of keys and the key searched for.

If the number of pairs is small looking up/inserting a pair will take little time. As the number of pairs increases so will the average time it takes to look up/insert a pair.

Which leads me to my question: why are the pairs not stored using a balanced search tree, a hash table, a skiplist , a trie etc...?


I thought about that, and I even have a working hashtable, but I have not gotten around implementing it. For small objects, it's not that slow at the moment, so I just put that aside to work on other stuff (like improving performance and standard compliance).

Edit: I already fixed the string parsing and the rejection of signed numbers (github issues #1 and #2). Feedback appreciated.

Edit2: Truncated quote.
Last edited by StringEpsilon on Apr 16, 2016 16:40, edited 1 time in total.

Return to “Projects”

Who is online

Users browsing this forum: No registered users and 2 guests