Tokyo Institute of Technology, Department of Computer Science, B3
- Interests
- Machine Learning
- Compiler
- Mathematical optimization
- Statistics
License: MIT License
Currently, DeepShiba assumes that only mathematical functions... , can be back-propagated regardless of the values upstream of the graph.
However, this needs to be modified.
Because DeepShiba needs to add functions that require values to propagate from upstream of the graph - for example, the reshape function.
This requires the design of a more flexible function structure.
In the current version of DeepShiba, each function's type is a ShibaObject.Func.
DeepShiba will be change the system of it,
first, functions will be a new type which supertype is ShibaObject.Func.
And that's behaivior controlled by multiple dispach.
This can lead to many improvements.
We need to do some validation, but for now, here's an example
So far, each function on the graph has its own forward and backward.
However, they are now separated from the structure by multiple dispatch. This reduces the size of the structure to a much smaller size.
The current Func has a field -- "name".
This is for debugging purposes, and if it is not set, the output may not be as clear as it should be.
This is especially true for plots.
However, if you define a type for each function, they can easily be done with typeof()!
The current plot function is a single theme and the output value is a single value.
It would be useful to add themes and output values, e.g. grads, to this.
Currently, DeepShiba uses a circular reference internally.
function (f::Func)(vars::Variable...)
f.inputs = vars
xs = [x.data for x in vars]
ys = f.forward(xs...)
ys = as_tuple(ys)
outputs = [Variable(y, f, nothing, f.generation + 1, nothing) for y in ys]
f.generation = maximum([x.generation for x in f.inputs])
f.outputs = outputs
length(outputs) == 1 ? outputs[1] : outputs
end
function backward!(var::Variable)
(isnothing(var.grad)) && (var.grad = ones_like(var.data))
funcs = PriorityQueue{Func, Int}()
seen_set = Set{Func}()
funcs[var.creator] = 1
while !(isempty(funcs))
f = dequeue!(funcs)
gys = [output.grad for output in f.outputs]
inputs = get_data.(f.inputs)
gxs = f.backward.(inputs) .* gys
gxs = tuple(gxs...)
for (x, gx) in zip(f.inputs, gxs)
if isnothing(x.grad)
x.grad = gx
else
x.grad = x.grad + gx
end
if (!(isnothing(x.creator)))
if !(x.creator in seen_set)
funcs[x.creator] = x.generation
end
end
end
end
end
Variable refers to Func and Func refers to Varible.
This is not good for memory efficiency.
Improvement is needed.
It is in the final stages to design the core of DeepShiba
For this reason, I think it is useful to add type annotations for the structures and functions.
This will improve performance and readability of the source code.
The most core feature, back propagation, needs to be implemented.
If you try to plot directly from script, the window disappears instantly.
This is an example of code that does not work properly.
using DeepShiba
x = variable(2.0)
y = variable(3.0)
f(x, y) = 2x^2 + 12y
z = f(x, y)
plot(z)
DeepShiba's functions are currently undocumented.
It would be better if these had a docstring.
The current DeepShiba does not support broadcast.
However, it is a very powerful feature and is needed when defining complicated functions.
It should be added as soon as possible.
Functions that perform numerical differentiation can be used for testing and other purposes.
I feel it's a very funny mistake.
https://travis-ci.org/github/abap34/DeepShiba.jl/builds/703524347?utm_source=github_status&utm_medium=notification
The current Variable in DeepShiba always returns the same string when requested for text output.
However, when an array of Variables is defined, for example, the current polite output is inconvenient.
Deepshiba needs to make the output in REPL and print() more visible.
The current output is not user-friendly.
The current DeepShiba also uses priority queue when doing plots.
But when plotting, just need to scan through all the elements, and the order of the elements does not matter.
Therefore, DeepShiba will use a simpler structure with no order.
This will greatly speed up the plotting process
A Function, visualize graphs is very useful for debugging.
DeepShiba is becoming more complex.
Long source code has a negative impact on readability and it's management.
The current DeepShiba plot only supports exporting to files.
This is unfriendly for users who want to improve their models interactively.
There is a bug in the back propagation.
using DeepShiba
function goldstain(x, y)
z = (1 + (x + y + 1)^2 * (19 - 14x + 3x^2 - 14y + 6x * y + 3y^2)) * (30 + (2x - 3y)^2 * (18 - 32x + 12x^2 + 48y - 36x * y + 27y^2))
return z
end
x = variable(1.0, "x")
y = variable(1.0, "y")
z = goldstain(x, y)
backward!(z)
print(z)
print(x)
print(y)
(
data:1876.0
grad:1
creator:Mul
)
x(
data:1.0
grad:-5376.0
)
y(
data:1.0
grad:8064.0
)
(
data:1876.0
grad:1
creator:Mul
)
(
data:1.0
grad:-84799.0
)
(
data:1.0
grad:6772.0
)
Currently, DeepShiba builds a computational graph when operations are performed on ShibaObject.Variable.
Also, grads are stored by the usual Julia language types; Int and Float, for example.
Therefore, the graph is not built for back propagation.
That means that "back propagation for back propagation", i.e. higher order differentiation, is not possible.
Today, DeepShiba is type-annotated for many functions and structures.
The purpose of this is to prevent errors by deliberately narrowing the scope of operation.
However, in many situations, abstract types are used, which raises concerns from a speed and memory efficiency perspective.
Experimentation and improvement are needed.
Some mathematical functions are missing from the current DeepShiba. For example, sin and cos, they need to be added.
DeepShiba underwent a major and destructive change.
As a result, a lot of test code stopped working and was removed. (#29)
As a result, the current DeepShiba does not have adequate testing, which leads to bugs.
DeepShiba needs a better testing framework and useful functions.
Need to implement a core file for hypothetical use.
The current DeepShiba requires the use of redundant expressions. For example, when adding, you need to use Add(), which is not user friendly.
We should be able to use + in a more intuitive way.
Users can more easily take advantage of DeepShiba by overloading existing Julia operators to support DeepShiba objects.
The current DeepShiba does not support GPUs.
With GPU support, DeepShiba will run faster.
DeepShiba has added a lot of features, and the number of functions in a single file is enormous.
This should be subdivided and organized by directory.
Currently, DeepShiba does not support branching graphs that use functions with multiple arguments, such as addition.
Also, their reverse propagation will not work properly.
These need to be implemented.
The new functions in functions.jl are not type-annotated.
The current DeepShiba has grown through a series of disruptive changes.
Therefore, the example needs to be modified repeatedly, which is inefficient.
This issue is for discussing how to make DeepShiba run faster.
The current back propagation is implemented using a recursive structure.
However, this is a concern in terms of scalability and speed of operation, and needs to be changed to a loop-based implementation.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.