TensorFlow library for adding FPGA based layers

subDesTagesMitExtraKaese d839b063df copied json.hpp 4 years ago
c++ d839b063df copied json.hpp 4 years ago
doku a6e49047a8 moved sources 4 years ago
examples a6e49047a8 moved sources 4 years ago
hostLib a6e49047a8 moved sources 4 years ago
tests a6e49047a8 moved sources 4 years ago
.gitignore a6e49047a8 moved sources 4 years ago
.gitmodules a6e49047a8 moved sources 4 years ago
README.md 0459bf28f0 updated readme 4 years ago
config.json d839b063df copied json.hpp 4 years ago

README.md

TensorFlow library for adding FPGA based layers

Components

  • hostLib/ Python wrapper module
    • layers/ Layer definitions
  • c++/ TensorFlow custom operator library
    • lib/mlfpga/ FPGA data transfer library
  • /bachelor/vhdl-modules VHDL implementation

Usage

import tensorflow as tf
from tensorflow.keras import models
from hostLib.layers.conv2d import Conv2D as Conv2DFPGA

model = models.Sequential()
model.add(Conv2DFPGA(1))

Installation

  1. clone repository and init submodules

    git clone <this url>
    cd ./tf-fpga
    git submodule init
    
  2. install dependencies (on Ubuntu Linux for example)

    sudo apt update                           
    sudo apt upgrade -y
    sudo apt autoremove
    sudo apt install python3 python3-pip
    sudo python3 -m pip install --upgrade pip # update pip globally
    python3 -m pip install tensorflow
    
  3. install C++ compiler

    sudo apt install g++
    
  4. compile operator and fpga libraries

    cd ./c++
    ./configure
    make
    
    > /usr/bin/g++ ... -o build/dummyBigOp.o src/dummyBigOp.cpp
    > ...
    > /usr/bin/g++ ... -o build/op_lib.so ...
    
  5. update config.json with your FPGA addresses defined in the VHDL design

    {"fpgas": [
      {
        "ip":   "192.168.1.33",
        "port": 1234
      },
      {
        "ip":   "192.168.1.34",
        "port": 1234
      },
      {
        "ip":   "192.168.1.35",
        "port": 1234
      }
    ]}
    

Adding new custom layers

For more details on how to contribute to git projects see https://gist.github.com/MarcDiethelm/7303312.

  1. create a computation module in the FPGA implementation
  2. add your FPGA module to the list of modules c++/lib/mlfpga/include/modules.hpp

    then the MOD_DEF macro creates these entries automagically:

    moduleIds[Module::myNewModule];
    moduleNames[Module::myNewModule];
    moduleSendPayloadLength[Module::myNewModule];
    moduleRecvPayloadLength[Module::myNewModule];
    
  3. create a TF kernel implementation MyNewOp inherited from AsyncOpKernel, inside these files:

    c++/src/myNewOp.cpp and c++/include/myNewOp.hpp

    define the constructor and overwrite the ComputeAsync method:

    class MyNewOp : public AsyncOpKernel {
      public:
        explicit MyNewOp(OpKernelConstruction* context);
    
        void ComputeAsync(OpKernelContext* context, DoneCallback done) override;
    }
    

    using your FPGA module

    auto worker = connectionManager.createWorker(Module::myNewModule, count);
    
  4. register the the kernel with a custom operator:

    c++/src/entrypoint.cpp

    REGISTER_OP("MyNewOp")
      .Input("input: float")
      .Output("output: float")
      .SetShapeFn([](InferenceContext* c) {
        c->set_output(0, c->input(0));
        return Status::OK();
      });
    ;
    
    REGISTER_KERNEL_BUILDER(Name("MyNewOp").Device(DEVICE_CPU), MyNewOp);
    //                                  the custom kernel class /\ 
    

    c++/include/entrypoint.hpp

    #include "myNewOp.hpp"
    

    More information on creating custom TF kernels can be found here.


  5. compile everything

    cd ./c++
    make clean
    make
    
  6. append a test for your operator

    tests/op_test.py

    def testMyNewOp(self):
      with self.session():
        input = [1,2,3]
        result = load_op.op_lib.MyNewOp(input=input)
        self.assertAllEqual(result, input)
    
  7. add a custom layer that uses the operator

    hostLib/layers/myNewLayer.py

    class MyNewLayer(layers.Layer):
      ...
      def call(self, inputs):
        return load_op.op_lib.MyNewOp(input=inputs)
    
    
  8. add that layer to the python module

    hostLib/layers/__init__.py

    __all__ = ["conv2d", "myNewLayer"]
    

Tests

There are tests for each complexity layer of this project.

  1. loopback test without connected FPGAs c++/tests/main.cpp

    edit config.json:

    {"fpgas": [
      {
        "ip":   "localhost",
        "port": 1234
      }
    ]}
    

    then run UDP echo server in the background:

    python3 tests/echo.py &
    cd ./c++
    make test
    ./build/test
    
  2. FPGA communication test c++/tests/main.cpp

    cd ./c++
    make test
    ./build/test
    
  3. operator validation test, based on TFs test suite tests/op_test.py

    python3 tests/op_test.py
    

Dependencies

C++

Python3

  • tensorflow
  • c++/build/op_lib.so

Used in examples:

  • Pillow
  • CV2
  • mss
  • numpy
  • IPython