It seems that the importing of demo2.py shouldn't create output of "b".

http://scipy-lectures.github.io/intro/language/reusing_code.html#main-and-module-loading

Shall we prep a release in time for EuroScipy?

The new IPython seems to no longer offer double underscore methods when pressing tab. The existing examples in the intro language chapter:

http://scipy-lectures.github.com/intro/language/basic_types.html

http://scipy-lectures.github.com/intro/language/reusing_code.html

And potentially others need to be revisited.

git shortlog -sn reports more people than listed in the AUTHORS.rst file -- sync them!

To avoid that a custom matplotlib_rc file will break the plots on
generation, brutally put pylab.rcParams to
pylab.rcParamsDefault before executing each script.

Was cleaning out my INBOX and found this in an old email from oct 2011, not sure if it is still relevant.

I just found these wonderful lecture notes. Thank you!

Even though I think my python software is up to date I do get a few errors and warnings with the sphinx build and thought I'd let you know:

$ make html
mkdir -p build/html build/doctrees
sphinx-build -b html -d build/doctrees -D latex_paper_size=a4  . build/html
Running Sphinx v1.0.7
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/sphinx/application.py:250: FutureWarning: A plot_directive module is also available under matplotlib.sphinxext; expect this numpydoc.plot_directive module to be deprecated after relevant features have been integrated there.
  mod = __import__(extension, None, None, ['setup'])
WARNING: extension 'ipython_console_highlighting' has no setup() function; is it really a Sphinx extension module?
loading translations [en]... locale not available
loading pickled environment... not yet created
loading intersphinx inventory from http://docs.scipy.org/doc/numpy/objects.inv...
loading intersphinx inventory from http://docs.python.org/2.7/objects.inv...
building [html]: targets for 51 source files that are out of date
updating environment: 51 added, 0 changed, 0 removed
<string>:19: RuntimeWarning: overflow encountered in square                                                                                     
<string>:19: RuntimeWarning: invalid value encountered in square
<string>:17: RuntimeWarning: divide by zero encountered in divide
/opt/local/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/numpy/core/numeric.py:235: ComplexWarning: Casting complex values to real discards the imaginary part
  return array(a, dtype, copy=False, order=order)
reading sources... [100%] intro/summary-exercises/stats-interpolate                                                                             
/Users/deil/github/scipy-lecture-notes/advanced/scikit-learn/index.rst:291: (WARNING/2) toctree contains reference to nonexisting document u'advanced/scikit-learn/digits_classification_exercise'

/Users/deil/github/scipy-lecture-notes/advanced/sympy.rst:110: (WARNING/2) Inline substitution_reference start-string without end-string.

/Users/deil/github/scipy-lecture-notes/index.rst:145: WARNING: download file not readable: PythonScientific.pdf
/Users/deil/github/scipy-lecture-notes/index.rst:145: WARNING: download file not readable: PythonScientific-simple.pdf
/Users/deil/github/scipy-lecture-notes/intro/language/basic_types.rst:49: (ERROR/3) Error in "warning" directive:
invalid option block.

.. warning:: Integer division
    ::

        >>> 3 / 2
        1

    **Trick**: use floats::

        >>> 3 / 2.
        1.5

        >>> a = 3
        >>> b = 2
        >>> a / b
        1
        >>> a / float(b)
        1.5



/Users/deil/github/scipy-lecture-notes/intro/numpy/numpy.rst:586: (WARNING/2) Exception occurred in plotting numpy-3: Traceback (most recent call last):
  File "/Users/deil/github/scipy-lecture-notes/sphinxext/plot_directive.py", line 460, in run_code
    exec code in ns
  File "<string>", line 4, in <module>
ImportError: cannot import name imsave


/Users/deil/github/scipy-lecture-notes/intro/numpy/numpy.rst:1868: (WARNING/2) Literal block expected; none found.

/Users/deil/github/scipy-lecture-notes/intro/summary-exercises/image-processing.rst:52: WARNING: duplicate label image-answers, other instance in /Users/deil/github/scipy-lecture-notes/intro/summary-exercises/answers_image_processing.rst
looking for now-outdated files... none found
pickling environment... done
checking consistency... /Users/deil/github/scipy-lecture-notes/README.rst:: WARNING: document isn't included in any toctree
/Users/deil/github/scipy-lecture-notes/intro/solutions.rst:: WARNING: document isn't included in any toctree
done
preparing documents... done
writing output... [100%] intro/summary-exercises/stats-interpolate                                                                              
writing additional files... search
copying images... [100%] advanced/image_processing/diamond_kernel.png                                                                           
copying downloadable files... [100%] advanced/optimizing/demo.py                                                                                
copying static files... done
dumping search index... done
dumping object inventory... done
build succeeded, 11 warnings.
mkdir -p build/html/data
cp data/max-speeds.npy data/sprog-windspeeds.npy  build/html/data

Build finished. The HTML pages are in build/html.

• The chapter is in the 'guide' directory
• It should explain quickly
  • How to build (using sphinx)
  • How to doctest
  • The use of examples to generate figures (ie: figure are not allowed to be checked in)
  • The use of intersphinx to link across projects
• It should stress our style guide:
  • Chapters should be readable / doable as a tutorial in 1.5 hours to 2 hours max (which implies limiting the amount of information)
  • Links (internal, to others projects and to Wikipedia) matter terribly
  • Jargon is to be avoided at all cost
  • The use of restructured text directives, such as topic, note, seealso, tip, sidebar, is strongly encouraged

I'm not an expert, but googling for a while tells me that when you can estimate gradient or if you have tons of training samples (when used in machine learning) then using SGD can be even more efficient than BFGS(-L).

http://metaoptimize.com/qa/questions/785/online-optimization-algorithms-are-there-good-implementations-libraries-to-use

http://www.quora.com/Machine-Learning/Is-the-L-bfgs-always-better-than-stochastic-gradient-descent

It seems to me that it should be somehow represented in docs. :) Thanks.

The red example is false in intro/numpy/numpy_indexing.png
It is not array([2,22,52]).

Thank you for this nice lecture notes. I like it.

It seemed that it has never mentioned anything about ica.py in the index.rst……

In fact, I just tried to import fastica from sklearn.decomposition for I couldn't find out what the ica module is until I found ica.py at scipy-lecture-notes/advanced/optimizing/ica.py.

It should be better if it is explicitly mentioned for beginners like me, I think

I ran into a problem when converting plots to sphinx plot directive. It apparently can't handle nested directories. When used in the sub-directory

intro/summary-exercices/

The generated plot file is not copied over to the build directory, and hence not included in the final build. The following error is produced when running 'make html':

/home/esc/git-working/scipy-lecture-notes/intro/summary-exercices/stats-interpolate.rst:: WARNING: image file not readable: intro/pyplots/cumulative_wind_speed_prediction.png
/home/esc/git-working/scipy-lecture-notes/intro/summary-exercices/stats-interpolate.rst:: WARNING: image file not readable: intro/pyplots/cumulative_wind_speed_prediction.pdf

This may be caused by buggy plot extension code.

To reproduce the error, check:

https://github.com/esc/scipy-lecture-notes/tree/sphinx-plot-extension-buggy

1. This is an important read for anyone writing C-API code:

http://docs.python.org/2/c-api/intro.html#objects-types-and-reference-counts

2. Add a disadvantage to using the C-API directly:

• Reference count bugs are easy to create and very hard to track down.

3. Use a single multi-iterator for the npyiter example

4. Links for the numpy c api section:

http://docs.scipy.org/doc/numpy/reference/c-api.array.html
http://docs.scipy.org/doc/numpy/reference/c-api.iterator.html

Science is largely the process of testing, confirming, and rejecting hypotheses in order to answer questions.

Test-Driven Development is strongly aligned with this objective.

A section on testing could be helpful?

Replace all the IPython style prompts by standard ">>>"

Rational:

• Simplifies build (we can remove ipython_console_highlighting.py)
• Enables doctesting
• The javascript ">>>" thing to remove interactively prompts is very cool.

Should probably document it, like:

1. Check HTML and PDF compiles, warnings are OK
2. Update and finalize changelog
3. Bump version (and copyright) in conf.py
4. Tag (lightweight)
5. do make install
6. Push tag? Push changes?

My wife was doing the tutorial and said that the list examples (§1.2.2.2.1. and others) were a little confusing. Coming from Matlab, a list l = [1,2,3,4,5] where l[0] == 1, l[2] = 3, while highlighting the indexing differences, was still confusing. perhaps it would be better to have the list be ['a', 'b', 'c', 'd', 'e'] or random floats or something similar.

As of Sphinx 1.1, there is the posibility to generate translatable strings (.pot files) from a Sphinx project, like it is done in the Open Data Handbook:

https://github.com/okfn/opendatahandbook#building-the-english-source-for-translation-i18n

which can be later translated, for example, through a crowdsourcing service like Transifex:

https://www.transifex.com/projects/p/opendatahandbook/

(also self-hosted if needed, otherwise the platform is free for open source projects)

Would it be possible to have this for the notes? For the case of Spanish language, we started our own translation:

https://github.com/Pybonacci/scipy-lecture-notes-ES

but it's very difficult to catch up with the changes upstream, and everything gets messed up. I would like to know if there would be the possibility, otherwise translating is probably not worth the effort.

https://pypi.python.org/pypi/xdress/0.3

Newer IPython versions aren't happy with print?

http://scipy-lectures.github.com/intro/intro.html#the-interactive-workflow-ipython-and-a-text-editor

Since I will be using the lecture notes for teaching from 10-15 of Feb, I would like to do a release next week. I have some more material that needs to be merged, but does anyone see any blockers? Also, are there notes about how to release somewhere?

The example with the demo2.py is buggy:

http://scipy-lectures.github.com/intro/language/reusing_code.html#main-and-module-loading

No one knows why demo2.py should ever print b... Maybe it will need to import demo

We can throw away our gen_rst.py and use sphinx gallery.

Once this is done, we can use it's cool features, like the fact that you can have multiple sub gallery, and the notebook-style examples.

I understand this has to be a printed in color cause of the images but there is lot of other colors like every single code has a color background. That is an awful lot of color to print and gain nothing. Can you create a version where the code background isn't colored. Perhaps add a border around the code to differentiate them. Color printing is expensive. I would prefer to use color ONLY where it is absolutely necessary like the color images

First of all, many thanks for this fabulous resource !

It is stated in the first section that a drawback of Python is that Python has a "less pleasant development environment than, for example, Matlab. (More geek-oriented).".

Of course it is very true is one talks about pure python or ipython from the command line. But it exists fancy GUI to develop in Python, close to to the Matlab's user experience (e.g. Spyder, ...). I would not like that a reader believes that there is no such tools.

I understand that naming directly a project and not some other can be problematic, but at least a sentence explaining that there is such GUIs would help new Python users.

Regards,

Some figures are generated by the Makefile. I don't like to rely too much on the Makefile, as it needs to be maintained. I'd prefer if the plots were generated by the matplotlib plot extension.

On Python 2 and python 3: we need to run doc-test on the documentation, elsewhere it ends up broken (and fix them).

We can mimic the way nilearn does it to use miniconda for python 3
https://github.com/nilearn/nilearn

To run examples, we can use circle-ci, as done in nilearn, because it gives you much more computing power, and can cache dowloads in a persistent way across different runs.

Some functions in scikit-image were removed with 0.9.x :

________________________________________________________________________________
plotting plot_filter_coins.py
/data/home/max/workspace/scipy-lecture-notes/ve/local/lib/python2.7/site-packages/skimage/filter/ctmf.py:22: skimage_deprecation: Call to deprecated function ``median_filter``. Use ``filter.r
ank.median`` instead.
  def median_filter(image, radius=2, mask=None, percent=50):
________________________________________________________________________________
plot_filter_coins.py is not compiling:
Traceback (most recent call last):
  File "/data/home/max/workspace/scipy-lecture-notes/sphinxext/gen_rst.py", line 239, in generate_file_rst
    execfile(os.path.basename(src_file), {'pl': plt})
  File "plot_filter_coins.py", line 14, in <module>
    tv_filter_coins = filter.tv_denoise(coins, weight=0.1)
AttributeError: 'module' object has no attribute 'tv_denoise'
________________________________________________________________________________
plotting plot_segmentations.py
________________________________________________________________________________
plot_segmentations.py is not compiling:
Traceback (most recent call last):
  File "/data/home/max/workspace/scipy-lecture-notes/sphinxext/gen_rst.py", line 239, in generate_file_rst
    execfile(os.path.basename(src_file), {'pl': plt})
  File "plot_segmentations.py", line 11, in <module>
    from skimage.morphology import watershed, is_local_maximum
ImportError: cannot import name is_local_maximum
________________________________________________________________________________
plotting plot_lena_tv_denoise.py
________________________________________________________________________________
plot_lena_tv_denoise.py is not compiling:
Traceback (most recent call last):
  File "/data/home/max/workspace/scipy-lecture-notes/sphinxext/gen_rst.py", line 239, in generate_file_rst
    execfile(os.path.basename(src_file), {'pl': plt})
  File "plot_lena_tv_denoise.py", line 4, in <module>
    from skimage.filter import tv_denoise
ImportError: cannot import name tv_denoise
________________________________________________________________________________
plotting plot_watershed_segmentation.py
________________________________________________________________________________
plot_watershed_segmentation.py is not compiling:
Traceback (most recent call last):
  File "/data/home/max/workspace/scipy-lecture-notes/sphinxext/gen_rst.py", line 239, in generate_file_rst
    execfile(os.path.basename(src_file), {'pl': plt})
  File "plot_watershed_segmentation.py", line 2, in <module>
    from skimage.morphology import watershed, is_local_maximum
ImportError: cannot import name is_local_maximum

The docstring example in skimage.morphology seems to suggest that peak_local_max can be used as a drop-in replacement but that doesn't do the trick for me :

Traceback (most recent call last):
  File "plot_watershed_segmentation.py", line 18, in <module>
    local_maxi = peak_local_max(distance, image, np.ones((3, 3)))
  File "/data/home/max/workspace/scipy-lecture-notes/ve/local/lib/python2.7/site-packages/skimage/feature/peak.py", line 128, in peak_local_max
    image_max = ndi.maximum_filter(image, size=size, mode='constant')
  File "/data/home/max/workspace/scipy-lecture-notes/ve/local/lib/python2.7/site-packages/scipy/ndimage/filters.py", line 926, in maximum_filter
    cval, origin, 0)
  File "/data/home/max/workspace/scipy-lecture-notes/ve/local/lib/python2.7/site-packages/scipy/ndimage/filters.py", line 859, in _min_or_max_filter
    sizes = _ni_support._normalize_sequence(size, input.ndim)
  File "/data/home/max/workspace/scipy-lecture-notes/ve/local/lib/python2.7/site-packages/scipy/ndimage/_ni_support.py", line 66, in _normalize_sequence
    raise RuntimeError(err)
RuntimeError: sequence argument must have length equal to input rank

A section on Interfacing with C through CFFI would be useful, both for CPython and for PyPy.

https://github.com/scipy-lectures/scipy-lecture-notes/edit/master/advanced/interfacing_with_c/interfacing_with_c.rst

• Docs: https://cffi.readthedocs.org/en/latest/
• PyPI: https://pypi.python.org/pypi/cffi
• Src: hg+ https://bitbucket.org/cffi/cffi

Hi,
When I try to execute the casting example in https://scipy-lectures.github.io/advanced/advanced_numpy/index.html#casting the y+256 returns a broadcasted array to int16 not int8.

>>> x = np.array([1, 2, 3, 4], dtype=np.float)
>>> y = x.astype(np.int8)
>>> y + 256
array([257, 258, 259, 260], dtype=int16)

I am using Python 3.4 and numpy 1.8.2. (the same happens with Python 2.7)

From a discussion with Gael, it would be nice to add an introductory section on pandas.

On the one page per side version:

page 36, the last line,
"In you own code" should be "In your own code"

page 38, the last but two code box, code out of box.

Users who want to copy and paste from the examples have to click on the >>> button to filter out the >>> symbols. It'd be nice to get rid of the >>> button and just make the >>> text in the textarea unselectable like in this example.

Note JsFiddle doesn't work as a normal website and although the highlight behaviour works for this example, it's actually saving all the text in the clipboard. It'll work properly in "real" websites.

http://common-lisp.net/project/cffi/

There are several, but I will start with one:

plot_gradient_descent.py is not compiling:
Traceback (most recent call last):
File "/home/vahtras/Courses/scipy-lecture-notes/sphinxext/gen_rst.py", line 236, in generate_file_rst
execfile(os.path.basename(src_file), {'pl' : plt})
File "plot_gradient_descent.py", line 175, in
hessian=counting_hessian)
File "plot_gradient_descent.py", line 65, in gradient_descent_adaptative
return gradient_descent(x0, f, f_prime, adaptative=True)
File "plot_gradient_descent.py", line 57, in gradient_descent
x_i += - step*dx_i
TypeError: bad operand type for unary -: 'NoneType'

The error is due to 'step' being None, which is returned from scipy.optimize.line_search when the procedure fails to converge.

Should we fix or drop the example? The initial comment says

A gradient descent algorithm

do not use: its a toy, use scipy's optimize.fmin_cg

Very nice set of documents - thanks for this.

I am planning to teach with Python 3 - do you have any plans / suggestions for what do with Python 3 edits to these notes?

As advertised on:

http://matplotlib.sourceforge.net/sampledoc/extensions.html

In the HTML version of the document, the plot includes links to the original source code, a high-resolution PNG and a PDF. In the PDF version of the document, the plot is included as a scalable PDF.

However when using this directive the links are not generated. Perhaps this is caused since we ship our own copy of the extension.

See also:

https://github.com/esc/scipy-lecture-notes

The section on getting help seems a little out of place as 1.4 sandwiched between the numpy and matplotlib sections . I suggest to move it to the end of the introductory section.

Thoughts?

• In operations.rst, around line 209: the "data statistics" example
  can probably be completely removed in my eyes (it is in the
  exercises.rst file).
• I'd really like the random walk example back in. I use it to teach
  how a problem can be vectorized, rather than relying on for loops.
• The transpose remark should be put back in. It is a very important
  remark, in my experience.
• array_object.rst, line 767: I would keep the simple 'arange(10)'
  example, just for the sake of simplicity, unless I am missing the point.

when doing :mod:numpy.linalg`` we don't get links into the numpy documentation. Not sure what is going wrong, because the intersphinx configuration looks fine.

We should enable testing on travis for Python 3 and then fix the resulting problems with tests.

Maybe after #176 is merged (ie: the corresponding test failures are fixed) and #141 is fixed.

• Check that it renders well on mobile phone
• Use PureCSS for the above, and because it looks cooler
• Add a "table of contents" that (models are http://nilearn.github.io/introduction.html#python-for-neuroimaging-a-quick-start http://bokeh.pydata.org/en/latest/, the design of bokeh being nicer):
  • Doesn't scroll down
  • Highlights the current section
  • Disappears on small screens (mobile friendly + tutorial friendly)

Sphinx has an epub builder. It would be awesome to be able to provide good epub (eg for reading on the kindle). I suspect that there will be work making sure that everything renders well.

The section at:

http://scipy-lectures.github.com/intro/language/control_flow.html#list-comprehensions

is pretty useless. It has a single example of list comprehensions w/o any further explanation. Best it can do, is confuse students.

Are you interested in a section about swig and the interoperability with C/C++ or even CUDA? I could write something about that, if you like.

Using simple CSS plus html (eg as in the sphinx-gallery http://nilearn.github.io/auto_examples/index.html) we can add a tooltip that pops up on hover for the magic ">>>" that goes with the code prompts. That would strongly increase the discoverability of this feature.

We need to move to more doctesting, which will force us to have more testable code.

Steps are

• Run 'make test', fix the failing, skip the rest
• Convert code snippets to use doctest-like syntax (eg those using IPython prompts or chunks of code without prompts
• Fix the resulting problems by adding things like doctest ellipsis, normalise white-space or skipping the test

Hi,

it seems like some of the commits for the 'intro numpy' break the 'image processing'.

Possible dangerous commits are:

• eda3ceb
• 5a387d9
• 90e41b5 (but replaced by eda3ceb)

I presume this was the reason why Emmanuelle had to revert them for the tuorial this morning?

https://github.com/emmanuelle/scipy-lecture-notes/compare/tutorial

The plot_directive and the pngmath settings (5a387d9) may also break other images/plots in other sections.

We should take a close look at this, perhaps when the conference is over, since it might be a bit tricky to solve.

The Sympy section is outdated, if you want to run the examples in the current version (0.7.5) some of them won't work.

… to build the documentation without the plots, mostly for developing purposes.

See slkearn's makefile for an example.