Transform JSON String to Dictionary using shell without escape

python json escape backslash
python json dumps escape double quotes
json dumps escape single quotes
python json loader
python json tools
python json bytes
python deserialize json
python custom json encoder

I am calling a python script from the shell, with one input argument.

python main.py """{"key1":"value1", "key2":"value2"}"""

All keys and values are strings. Once in python, I would like to convert the JSON string to a dictionary, so I have acess to the values by using the keys.

I tried the following

import json
import sys
dict_in = json.loads(sys.argv[1])

But dict_in would end up a string like that {key1:value1, key2:value2} So it seems like I need to find a way to pass the string with quotation marks from the shell to python. I can not use escape characters since the string is provided by a different program.

Is there an elegant way to solve this?

I've found a python 2 module which can handle such cases.

Suppose you have this string:

>>> str = '{foo: bar, id: 23}'

Then you can use yaml as follows:

>>> import yaml
>>> dict = yaml.load(str)
>>> dict
{'foo': 'bar', 'id': 23}

>>> dict['foo']
'bar'

Now you have what you needed.

More info (and also python 3 support and etc.) can be found here: https://pyyaml.org/wiki/PyYAMLDocumentation

Transform JSON String to Dictionary using shell without escape, Once in python, I would like to convert the JSON string to a dictionary, so I have acess to the values by using the keys. I tried the following import json import sys  JSON String Escape / Unescape. Escapes or unescapes a JSON string removing traces of offending characters that could prevent parsing. The following characters are reserved in JSON and must be properly escaped to be used in strings:

Not sure if what you passing in is important but you can pass following and get desired output:

"{\"key1\":\"value1\", \"key2\":\"value2\"}"

or

'{"key1":"value1", "key2":"value2"}'

Here is the code and output:

$cat json_convert.py 
import json
import sys
dict_in = json.loads(sys.argv[1])
print (dict_in)
$ python json_convert.py '{"key1":"value1", "key2":"value2"}'
{'key1': 'value1', 'key2': 'value2'}

Also what you are passing """{"key1":"value1", "key2":"value2"}""" translates to "" + "{" + key1 + ":" + value1 + ", " + + key2 + ":" + value2 + "}" + "" if you are asking bash, if you were calling the function with that as a argument from the python itself you would get the desired results.

So really goes down to what you are calling it from.

If you still like quotes go ahead and pass """{"'"key1"'":"'"value1"'", "'"key2"'":"'"value2"'"}""" to get desired result :)

Convert JSON to escaped string in Python, If you have a JSON formatted file, and you want to put it in the form of a string, with double quotes and newlines escaped, it's a pain to do this  import json import sys dict_in = json.loads(sys.argv[1]) But dict_in would end up a string like that {key1:value1, key2:value2} So it seems like I need to find a way to pass the string with quotation marks from the shell to python. I can not use escape characters since the string is provided by a different program.

use either:

$ your_other_program | python main.py

to send the output of the other program to python, or use base64.b64encode(json.dumps(blah)) and you'll get pretty code like

'eyJtQXV0b21hdGljVGVzdExpc3QiOiBbeyJtWSI6IDguMTE0MTA1LCAibU5hbWUiOiAiYWNjZWxlcmF0b3JFbnRpdHkiLCAibVRlc3RTdGF0dXMiOiB0cnVlLCAibVgiOiAzLjgwNDM1MTgsICJtWiI6IC0zLjM4OTU3MjF9LCB7Im1OYW1lIjogImJhcm9tZXRlckVudGl0eSIsICJtVmFsdWUiOiAwLCAibVRlc3RTdGF0dXMiOiBmYWxzZX1dLCAibUF1dG9tYXRpY1Rlc3RDb21wbGV0ZWQiOiB0cnVlfQ=='

to put in the command line, and then decode it back from base64 into JSON.

Or, even better, use:

$ your_other_program >output_file.tmp
$ python main.py < output_file.tmp
$ rm output_file.tmp

19.2. json — JSON encoder and decoder, Using json.tool from the shell to validate and pretty-print: (the default), the output is guaranteed to have all incoming non-ASCII characters escaped. If indent is a non-negative integer or string, then JSON array elements and object object containing a JSON document) to a Python object using this conversion table. The ConvertTo-Json cmdlet converts any object to a string in JavaScript Object Notation (JSON) format. The properties are converted to field names, the field values are converted to property values, and the methods are removed. You can then use the ConvertFrom-Json cmdlet to convert a JSON-formatted string to a JSON object, which is easily managed in PowerShell. Many web sites use JSON instead

Ok so here is what is my test script:

print("original sys.argv output\n" + (sys.argv[1]))
string_temp=(yaml.load(sys.argv[1]))
print ("first transformation\n" +string_temp)
string_temp=string_temp.replace(":",": ")

dict_in=yaml.load(string_temp)
print("This is the dictionary")
print(dict_in)

This is what I type into the console

python test_script.py """{foo:bar, id:23}"""

And This is the output

original sys.argv output
"{foo:bar, id:23}"
first transformation
{foo:bar, id:23}
This is the dictionary
{'foo': 'bar', 'id': 23}

This only workds if I use tripple quotes ("""). If I use (") or (') to define the input string I get an error.

Alternatively one can remove the (") from the sys.argv[1]

print("original sys.argv output\n" + (sys.argv[1]))
string_temp=(sys.argv[1])[1:-1]
print ("first transformation\n" +string_temp)
string_temp=string_temp.replace(":",": ")

dict_in=yaml.load(string_temp)
print("This is the dictionary")
print(dict_in)

Reshaping JSON with jq, In brief, a JSON object is a series of key/value pairs, where keys are the names for Remember, values of an array have no keys - they are all considered to be of jq, which you may use to speedily parse much larger JSON files.) flag -qO- in order to send the output of wget into jq by way of a shell pipe. The ConvertFrom-Json cmdlet converts a JavaScript Object Notation (JSON) formatted string to a custom PSCustomObject object that has a property for each field in the JSON string. JSON is commonly used by web sites to provide a textual representation of objects. The JSON standard does not prohibit usage that is prohibited with a PSCustomObject. For example, if the JSON string contains duplicate

simplejson — JSON encoder and decoder, Using simplejson.tool from the shell to validate and pretty-print: If skipkeys is true (default: False ), then dict keys that are not of a basic type ( str None (the default) selects the most compact representation without any newlines. input bytes objects in Python 3 and 8-bit strings in Python 2 will be transformed into unicode  Json.NET probably serializes C# dictionaries adequately now, but when the OP originally posted this question, many MVC developers may have been using the JavaScriptSerializer class because that was the default option out of the box.

Working with JSON in bash using jq, jq is a powerful tool that lets you read, filter, and write JSON in bash. Bash doesn't understand JSON out of the box, and using the typical text manipulation tools params: $1 = the string to grep for, $2 = directory to grep in [3] -r is for “​raw-output”, so no quotes around values, which makes it suitable for  Converts the String to its JSON string representation. The value to convert. The string delimiter character. Type: Newtonsoft.Json. The string escape handling. A JSON string representation of the String.

ConvertFrom-Json, To generate a JSON string from any object, use the ConvertTo-Json cmdlet. Without the switch, converting the JSON to a PSObject and then converting it back  ConvertFrom-StringData supports escape character sequences that are allowed by conventional machine translation tools. That is, the cmdlet can interpret backslashes (\) as escape characters in the string data by using the Regex.Unescape Method, instead of the PowerShell backtick character (`) that would normally signal the end of a line in a

Comments
  • use base64 encoding -- you won't need even quote marks
  • or pipe stdout from "another program" to stdin of your python script.
  • can you please elaborate a little bit more
  • This seems perfect, but somehow it only works in the Python IDE. I will keep on trying on monday. Will report back.
  • actually I tested outside and worked just fine. I gave that str as the first input, which is the sys.argv[1] and with the above code, you get the answer you want :)
  • Ok I figured it out. 1) You need a space after the : or else you get in error. Any way to avoid that? 2) I need to call yaml.load twice dict_in=yaml.load(yaml.load(sys.argv[1])) for some weird reason. First time it converts it to a string, seconde time into a dictionary
  • 1) Actually this is one of the few known issues that people were arguing. When I was working with this, I wrote a script to check every colon in the string and add a space after that if there wasn't, and this solved my problem. Some people have tried changing the main module too, like the one suggested here and this is another link that reported this issue.
  • 2) How could such thing happen? I mean, what is the initial type of sys.argv[1] that after yaml.load, it turns to string?! It must already be a string. What do you receive after the first yaml.load? The same string that were passed to it?
  • That works, but you are using escape characters, which I am trying to avoid.
  • He isn't in the second version.
  • @valenzio check out the 2nd approach :)
  • oh sorry did not see that, but that also does not work, same result as with the triple quotes.
  • interesting, I will give it a try on Monday, since I am not in charge of the other_program (it is not in python)
  • @valenzio why don't you use a file??
  • The Json string is saved in a seperate Database, so it would be nice if I could just use that string. Worst case my service layer could create a file with the info.