Apache Avro is a Framework Which Supports RPC and Data Serialization – it uses RPC calls to send data. For this guide, at least you need have Apache Hadoop installed and running on the server. We have guides to install Apache Hadoop on single cloud server. Here is How To Install Apache Avro On Ubuntu Running Apache Hadoop. Another point to mention is that, Apache Avro API’s several languages – Java, C, C++, C#, Python and Ruby. Other two comparable frameworks are Protocol buffers (by Google) and Thrift (by Apache). Schema is not stored in Avro and has rich schema resolution capabilities probably make difference. Here is official website of Apache Avro and GitHub repo :
1 2 | http://avro.apache.org/ https://github.com/apache/avro |
How To Install Apache Avro On Ubuntu
When Apache Hadoop is already installed, the work becomes easy as prerequisites are already present. Avro uses CMake as its build manager. Avro depends on the Jansson JSON parser. So we need to make sure they are installed :
1 2 | apt update -y && apt upgrade -y apt install build-essential libjansson-dev asciidoc source-highlight |
Test version of gcc and make :
---
1 2 | gcc -v make -v |
Please read source-highlight‘s documentation in case you need more advanced usage. Ubuntu has problem with following source-highlight‘s documentation to combine with less. Avro basically needs :
avro-mapred-x.y.z-hadoop2.jar
avro-tools-x.y.z.jar
…. above jar files in the classpath. These jar files contain all the classes for compiler, hadoop, mapred, mapreduce packages. Official repo of US Mirror of Apache is at :
1 | http://www-us.apache.org/dist/avro/ |
Download avro-1.8.1.tar.gz or newer version. Then :
1 2 | tar -xzvf avro-1.8.2.tar.gz cd avro-1.8.2 |
First Way :
1 2 3 4 | sudo python setup.py install $ python >>> import avro # should not raise ImportError |
Second Way :
1 2 3 4 5 6 7 8 9 10 | mkdir build cd build cmake .. \ -DCMAKE_INSTALL_PREFIX=$PREFIX \ -DCMAKE_BUILD_TYPE=RelWithDebInfo make make test make install |
Third Way :
Go to $HADOOP_HOME/share/hadoop/common/lib directory and check whether already same version is present.
Copy avro-mapred-x.y.z-hadoop2.jar into $HADOOP_HOME/share/hadoop/tools/lib and $HADOOP_HOME/share/hadoop/common/lib directories. Add the above directory path to classpath in .bashrc file.
1 2 3 4 5 6 | cp avro-mapred-1.8.2-hadoop2.jar avro-tools-1.8.2.jar $HADOOP_HOME/share/hadoop/tools/lib/ cp avro-mapred-1.8.2-hadoop2.jar $HADOOP_HOME/share/hadoop/common/lib/ nono ~/.bashrc #### export CLASSPATH="$HADOOP_HOME/share/hadoop/tools/lib/*.jar:$CLASSPATH" export PATH="$CLASSPATH:$PATH" |
