adamse / bachelor-report Goto Github PK

"was first introduced in section 1.3" Not really. You just dropped
the word. It was introduced in 4.2/4.3.

"source node. (Heckerman 1998)." No "." before citation!

Put your figures into floats otherwise they mess up the vertical
spacing completely.
\begin{figure}[htbp]
...
\caption{Title.\label{fig:bla}}
\end{figure}

"status of adjacent reaches" What is a "reach"?
What a reach is becomes actually clear in Section 4.2. see #27

l 5 "connect four" The reader does probably not know this game, so
there you should explain it in one sentence if you want to make your
point.

Explain the 'one policy per state variable' properly.
How does $\pi(a \mid s)$ look like?

Ah, you did a horrible job explaining this. Trash the paragraph
(which is quite comical to read) and try again. I like the part:

When the time steps ... are divided into subsequences in this
manner, the problem is called episodic.
...
If the problem is not episodic, it is called non-episodic, which
means, consequently, that the interaction is not divided into
subsequences.

:)

But seriously, this is badly explained. A game is a single episode,
so how can its time steps be divided into episodes??

I do not think anyone will read these tables. Nor type them in.
You should provide this data, but on a web page, not in printed form.
Rather, you could submit a set of spreadsheets with your thesis (as
well as your source code).

Save our forests and remove these 25 pages!

line 1 "real-world behavior optimization problems" is too long. Try to
split a part of into a relative sentence, like you do in the Swedish
abstract.

l3 "In cases like these" It is unclear what "these" refers to here.

l9 "in the environment invasive species"
"invasive species" is a proper noun, so it should be separated
in one of the following forms:

1. capitalize: Invasive Species,
2. quote: "invasive species", or
3. emphasize (put in italics): \emph{invasive species}

Also, you monotonously repeat "environment invasive species" three
times. Try to eliminate this redundancy.

l 2 There is a space before the dot in "interaction ."

Continuing with 3.2

N(s,a)
\omega
|| ||_1 The norm you use here is what?

Equation 2.2 is a bit less non-sensical, but still suffers from undefined parts.

p16 "There are four possible actions"

I find these actions underspecified. E.g., do they apply to all
habitats of a reach, or can habitats be targeted individually?

Also, does the agent have to move or can he target all reaches at
the same time?

Describing the "game" "invasive species", you should not
assume any preknowledge from the reader. I should be possible to
"play" the game from your descriptions.

I do not understand what purpose this section serves here. Maybe
you were told that you have to write something about ethics.
But Google's experiment of autonomous cars has little to do with
your investigation. Unless the actually do use reinforcement
learning. However, I doubt it, so you should come up with a
reference for this.

If you want to say something about ethics, you could discuss whether
it is morally correct to let agents that are trained by
reinforcement learning act autonomically in the physical word in
potentially dangerous situations (like traffic).

"the driver properly do [sic!] not have the time to think about the
situation leaving the outcome to coincidence while a computer makes
an active choice".

This sounds very wrong. It seems like you have a very naive
conception of the human thought process. Of course the human (and
lesser being down to insects or even bacteria) "makes an active
choice", certainly in comparison with a computer. The active choice
is the result of a computational process in neuronal networks (or
even biochemical machines as in microbes). Maybe you meant to say
the human does not make a conscientious choice, i.e., not a choice at
the level of ratio/language that could be verbalized later. Anyway,
a choice is made, be it an unconscious one. But remember that all
the choices an AI based on things like reinforcement learning makes
are "unconscious", below the level of explication.

Sections 1.2 - 1.4 are too short to make own sections. Remove the
subsections from the introductions, use paragraphs instead.
\paragraph{Propose and problem statement} ...

The four components S A P R are nicely explained.

Should it not be

P : S \times A \times S \to [0,1]

because it is probabilities?

"When building upon the work of another creator"

"creator" is probably the wrong word, I have only seen it to denote
"God".

Also, it comes to a surprise that you stress that you are building
on others' work. Of course, everything in computer science builds on
others' work, in the sense of using existing libraries and
frameworks. I though you are just using a framework, but here it
sounds more like you just modified someone else's implementation.

It is not even clear to my why the use of a framework was
necessary. From the description of the project, it did not seem
like there was a substantial implementation task. Neither the
description of the "invasive species" game is very substantial, nor
are the algorithms given in section 3. Neither concurrency,
graphics, network, nor databases are involved in your project, so
this really is a classical programming tasks involving only the core
of a programming environment (language + core libraries).

Somehow the praise in Section 4.5 of RL-Glue is in stark contrast to
your critique in Section 6.2.2. Maybe there is a misunderstanding,
so you have to be more precise what you mean by "work of another
creator".

l 2 "finite number of iterations, if the process" No comma here!

par 2 "Markov secision processes"
Use a spell-checker!! (Such mistakes are truly unnecessary)

"scales polynomially" Actually what polynomial function is it?
linear, quadratic, cubic ...?
There are huge differences between "polynomial" and "polynomial"

"different between" is grammatically wrong
Also, the parameter values are identical, so I do not know why you
distinguish them from the ones in Table A.1.

Equation 3.1 contains a heap of undefined quantities. Please proceed
as in section 2.2.2 where you introduced the quantities S P A R with
their types. Define the relevant quantities before giving equation 3.1.

~{P}
CI
\delta_1
P(s,a)

Continuing with 3.2

N(s,a)
\omega
|| ||_1 The norm you use here is what?

"imrovements"

"Throughout the discussion of this thesis the unfairness...has been
highlighted"
That is a bit exaggerated. You discussed it, yes, but it was not a
recurring issue.

Section 4.2 and 4.3 should be moved at least before section 3.

I think you should move the description of the "Invasive Species"
environment much more to the front of the report. For example, to
the beginning of chapter 3.

Equation 2.4 suffers from similar problems as in issue #16

• Variable "s'" is unbound.
• R(s,s') is ill-typed, should be R(s,a)
  I guess you mean to copy 2.6?

end of first par "psectionMarkov decision process" Looks like a TeX
macro has failed you here.

"exceeds 9. 9 was" Do not start a sentence with a numeral. Try instead
"This number" or "The number 9"

Equation 2.3 is non-sensical.

• Variable "a" is unbound.
• R(s,s') is ill-typed, should be R(s,a)
  I guess the correct equation is 2.5, which looks good.

This appendix needs a serious proof reading an polishing. It looks
like it is in a very raw state.

For instance, in B.2. there is a hole sentence occurring twice.

The sentence in B.3 starting with "By creating classes" lacks a verb.

page 8 l 3 "the different states ... are not very different"
Avoid repetition of "different".

I would not hurt to give the formula for the number of states:
3^(reaches * habitats)

"until V converges" Does this mean I need to compute
V_1, V_2, ... until I have a fixed point of (2.3)?
But then why does (2.4) reference to V_k instead of the fixed point V?
Same question applies to 2.5 / 2.6.

"In (2.2), $\gamma$ is the discount factor ..."
This explanation should be moved closer to equation (2.2)

Equation 2.1
is pure non-sense. Sorry to say. You are using the following
undefined entities and notations:

r
S_t
S,\pi
\sum_{k=0}^{T=k} maybe you meant \sum_{k=0}^{T} ?

Further, the expected value function \mathbb{E} should be applied to
a random variable. I do not see such a beast here.

Remove ", saving a lot of time"

Policy Evaluation "update rule (2.4)" That should be "(2.3)". Move the equation here, instead of putting it way below!

par 3 "what actions lead to success almost always depends..."
Funny sentence construction, hard to read. Try to reformulate

paragraph 2: Use triple-dash --- instead of single dash - to separate
parts of a sentence.
See e.g. http://www.grammarbook.com/punctuation/dashes.asp
(called "em dash" there).

The rest of Section 3.1 is of so bad quality that I gave up on trying
to understand it. Try again, observing the following principles:

1. Define a quantity before you use it.
   When defining it, explain its significance informally.

2. Motivate the formulas. Explain how you arrive at these formulas.

   (For instance 3.7 is completely mystical. There are roots,
   fractions, exponentials, logarithms. Looks totally random to
   me. Could not distinguish it from something out of a random
   formula generator. Please explain how you arrive at this
   formula. If you cannot, then don't bring this formula.)

"paramaters" use a spell-checker!!

In this chapter you also describe variants of the algorithms that you
have implemented, so say that in the beginning.

l4 "were" should be "where"

"Environment parameters" Move the appendix A here! Without a list of
the parameters, the game remains completely mysterious to the reader.

Use consistent notation. Before it was P(s, a s'), and s_t was the state
at time t...

par 2 "number of times a specific state-action pair has been visited has doubled" hard to read, try
"number of visits to a specific state-action pair has doubled"

No results presented from gridworld and network simulator?

just do it!

I think you should move the description of the "Invasive Species"
environment much more to the front of the report. For example, to
the beginning of chapter 3.

While there is a good introduction into the subject, the mathematical
description of the background and implemented algorithms has serious
shortcomings (Section 3.1 is unreadable). Also, the "invasive
species" game should be described earlier to give the reader a feel
what kind of problems should be solved by the proposed algorithms.