Is there a way to programmatically convert JSON to AVRO Schema?

generate avro schema from json python
json to avro schema converter online
convert json to avro python
json schema to avro schema python
json object to avro schema
json to avro schema java
kafka convert json to avro
failed to convert json to avro

I need to create AVRO file but for that I need 2 things:

1) JSON

2) Avro Schema

From these 2 requirements - I have JSON:

{"web-app": {
  "servlet": [   
    {
      "servlet-name": "cofaxCDS",
      "servlet-class": "org.cofax.cds.CDSServlet",
      "init-param": {
        "configGlossary:installationAt": "Philadelphia, PA",
        "configGlossary:adminEmail": "ksm@pobox.com",
        "configGlossary:poweredBy": "Cofax",
        "configGlossary:poweredByIcon": "/images/cofax.gif",
        "configGlossary:staticPath": "/content/static",
        "templateProcessorClass": "org.cofax.WysiwygTemplate",
        "templateLoaderClass": "org.cofax.FilesTemplateLoader",
        "templatePath": "templates",
        "templateOverridePath": "",
        "defaultListTemplate": "listTemplate.htm",
        "defaultFileTemplate": "articleTemplate.htm",
        "useJSP": false,
        "jspListTemplate": "listTemplate.jsp",
        "jspFileTemplate": "articleTemplate.jsp",
        "cachePackageTagsTrack": 200,
        "cachePackageTagsStore": 200,
        "cachePackageTagsRefresh": 60,
        "cacheTemplatesTrack": 100,
        "cacheTemplatesStore": 50,
        "cacheTemplatesRefresh": 15,
        "cachePagesTrack": 200,
        "cachePagesStore": 100,
        "cachePagesRefresh": 10,
        "cachePagesDirtyRead": 10,
        "searchEngineListTemplate": "forSearchEnginesList.htm",
        "searchEngineFileTemplate": "forSearchEngines.htm",
        "searchEngineRobotsDb": "WEB-INF/robots.db",
        "useDataStore": true,
        "dataStoreClass": "org.cofax.SqlDataStore",
        "redirectionClass": "org.cofax.SqlRedirection",
        "dataStoreName": "cofax",
        "dataStoreDriver": "com.microsoft.jdbc.sqlserver.SQLServerDriver",
        "dataStoreUrl": "jdbc:microsoft:sqlserver://LOCALHOST:1433;DatabaseName=goon",
        "dataStoreUser": "sa",
        "dataStorePassword": "dataStoreTestQuery",
        "dataStoreTestQuery": "SET NOCOUNT ON;select test='test';",
        "dataStoreLogFile": "/usr/local/tomcat/logs/datastore.log",
        "dataStoreInitConns": 10,
        "dataStoreMaxConns": 100,
        "dataStoreConnUsageLimit": 100,
        "dataStoreLogLevel": "debug",
        "maxUrlLength": 500}},
    {
      "servlet-name": "cofaxEmail",
      "servlet-class": "org.cofax.cds.EmailServlet",
      "init-param": {
      "mailHost": "mail1",
      "mailHostOverride": "mail2"}},
    {
      "servlet-name": "cofaxAdmin",
      "servlet-class": "org.cofax.cds.AdminServlet"},

    {
      "servlet-name": "fileServlet",
      "servlet-class": "org.cofax.cds.FileServlet"},
    {
      "servlet-name": "cofaxTools",
      "servlet-class": "org.cofax.cms.CofaxToolsServlet",
      "init-param": {
        "templatePath": "toolstemplates/",
        "log": 1,
        "logLocation": "/usr/local/tomcat/logs/CofaxTools.log",
        "logMaxSize": "",
        "dataLog": 1,
        "dataLogLocation": "/usr/local/tomcat/logs/dataLog.log",
        "dataLogMaxSize": "",
        "removePageCache": "/content/admin/remove?cache=pages&id=",
        "removeTemplateCache": "/content/admin/remove?cache=templates&id=",
        "fileTransferFolder": "/usr/local/tomcat/webapps/content/fileTransferFolder",
        "lookInContext": 1,
        "adminGroupID": 4,
        "betaServer": true}}],
  "servlet-mapping": {
    "cofaxCDS": "/",
    "cofaxEmail": "/cofaxutil/aemail/*",
    "cofaxAdmin": "/admin/*",
    "fileServlet": "/static/*",
    "cofaxTools": "/tools/*"},

  "taglib": {
    "taglib-uri": "cofax.tld",
    "taglib-location": "/WEB-INF/tlds/cofax.tld"}}}

But how to create AVRO Schema based on it?

Looking for programatic way to do that since will have many schemas and can not create Avro Schema manually every time.

I checked 'avro-tools-1.8.1.jar' but that can not create Avro Schema from JSON directly.

Looking for a Jar or Python code that can create JSON -> Avro schema. It is ok if Data Types are not perfect (Strings, Integers and Floats are good enough for start).


you can use Kite SDK util to infer avro schema from a json input.

https://github.com/kite-sdk/kite/blob/master/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/JsonUtil.java#L539

Example:

    String json = "{\n" +
            "    \"id\": 1,\n" +
            "    \"name\": \"A green door\",\n" +
            "    \"price\": 12.50,\n" +
            "    \"tags\": [\"home\", \"green\"]\n" +
            "}\n"
            ;
    String avroSchema = JsonUtil.inferSchema(JsonUtil.parse(json), "myschema").toString();
    System.out.println(avroSchema);

Result:

{  
   "type":"record",
   "name":"myschema",
   "fields":[  
      {  
         "name":"id",
         "type":"int",
         "doc":"Type inferred from '1'"
      },
      {  
         "name":"name",
         "type":"string",
         "doc":"Type inferred from '\"A green door\"'"
      },
      {  
         "name":"price",
         "type":"double",
         "doc":"Type inferred from '12.5'"
      },
      {  
         "name":"tags",
         "type":{  
            "type":"array",
            "items":"string"
         },
         "doc":"Type inferred from '[\"home\",\"green\"]'"
      }
   ]
}

You can find the maven dependency here

Is there a way to programmatically convert JSON to AVRO Schema , Is there a way to programmatically convert JSON to AVRO Schema , Each field is a JSON object with the following attributes: Unions may not contain more than� Stack Overflow Public questions and answers; Teams Private questions and answers for your team; Enterprise Private self-hosted questions and answers for your enterprise; Talent Hire technical talent


This one works cool with a simple copy and paste of avro schema.

https://toolslick.com/generation/metadata/avro-schema-from-json

How to convert a json object into Avro object if avro schema contains , JSON to Avro conversion tool designed to make migration to Avro easier. using JSON to Avro without having to modify JSON data if it conforms to the JSON schema. If you would like to know how messages will look like after encoding and� So we set ourselves the task of making the REST APIs accept Json and convert the Json into the Avro generated object. Generating Avro. So first we need to get the build process to generate the java classes from the Avro schema files, for this we could use the maven Avro plugin. This is a simple process and only requires the plugin to be configured in the pom.xml.


If you want to avoid creating a dedicated AVRO schema for every JSON format, you can use rec-avro package.

It allows you to take any python data structure, including parsed XML or JSON and store it in Avro without a need for a dedicated schema.

I tested it for python 3.

You can install it as pip3 install rec-avro or see the code and docs at https://github.com/bmizhen/rec-avro

I gave a json to avro example code here: https://stackoverflow.com/a/55444481/6654219

allegro/json-avro-converter: JSON to Avro conversion tool , Avro Schema From JSON Generator is an online tool to convert any JSON document or file into an Avro schema. It also has less size than XML because of no closing tags. Interacting with JSON from JavaScript is extremely seamless. JSON format was first specified by Douglas Crockford in the early 2000s. Avro is a compact and efficient binary file format used for serializing data during transmission. The schema of Avro files are specified in JSON.


Give this one a shot. http://www.dataedu.ca/avro

It basically infers the Avro schema that accepts the JSON.

You can even give it a JSON array. What it would do is generating an Avro schema that is compatible with all the JSON documents in your array.

There are other tools that you can verify the result.

Avro Schema From JSON Generator, We have recently started using Apache Avro primarily for use with Apache Kafka, and the Confluent Schema registry. Previously we had been,� JSON numbers, unlike Avro's numeric types, are not limited in precision and/or scale; for integer types, minimum and maximum are used to emulate Avro limtations. Software used: json-schema-avro (conversion), json-schema-validator (post generation JSON Schema syntax checking).


Easy way to convert Json to Avro, The JsonAvroBinding interface represents values as instances of JsonRecord . need not be fixed at build time if the schemas are treated dynamically. To avoid type conversion errors, it is safest to always define integer fields in the Avro Once you have the binding, you need a way for your application to represent the � An Avro schema is created using JSON format. JSON is short for JavaScript Object Notation, and it is a lightweight, text-based data interchange format that is intended to be easy for humans to read and write. JSON is described in a great many places, both on the web and in after-market documentation.


JSON Bindings, Using Converters Programmatically � 9.6. The Avro converter supports parsing whole Avro files, with the schema For IPC messages, specify the schema in one of two ways: to use an inline schema string, set Avro paths are defined similarly to JSONPath or XPath, and allow you to extract specific fields out of an Avro� Avro has a JSON like data model, but can be represented as either JSON or in a compact binary form. It comes with a very sophisticated schema description language that describes data. We think Avro is the best choice for a number of reasons: It has a direct mapping to and from JSON; It has a very compact format.


9.9. Avro Converter — GeoMesa 3.0.0 Manuals, Learn how to use the Apache Avro serialization library. Data serialization is a technique of converting data into binary or text format. One way of creating the schema is to write the JSON representation, Programmatically generating Avro classes: Classes can be generated using SchemaCompiler. Assuming an input schema (in JSON form) that's already UTF-8 text for a valid Avro schema (including all quotes as required by JSON), the following transformations will produce its Parsing Canonical Form: [PRIMITIVES] Convert primitive schemas to their simple form (e.g., int instead of {"type":"int"}).