Giter Club home page Giter Club logo

unagi's Introduction

Unagi (ウナギ)


Besides being the most delicious fish in the world and one of the signature Japanese dishes, unagi can also help you navigate through the public or internal data release of the Hyper Suprime-Cam (HSC) Subaru Strategic Program (SSP; also known as the HSC survey) -- an ambitious multi-band deeep photometric survey using the awesome prime-focus camera on the 8.2m Subaru telescope.

Also, unagi does not stand for anything because forced acronym is for psychopath.

If you want to learn more about the real unagi, you can check out this video on Youtube. And here is a vlog about a famous unagi place I have tried: Obana 尾花. It is pretty close to downtown Tokyo and few statioins away from Kavli-IPMU.

If you want to learn more about this amazing food, just try import unagi; unagi.unagi() in your Jupyter Notebook.

Recent Updates

  • 06/03/2019: Support for HSC PDR2 is added (need more tests)
    • The schema of all tables in the PDR2_DUD and PDR2_WIDE reruns are available in unagi/data/ folder as a .json file.
  • 06/04/2019: New features: filters.py and camera.py
    • Help you get important information about HSC camera (e.g. CCD QE, primary mirror reflectivity) and filters (e.g. transmission curves, absolute AB magnitude of Sun in each filter etc.).
  • 06/06/2019: New features: add hsc_check_coverage() method to check if a coordinate is covered by one HSC Patch. See here for example.

Applications

  • Using HSC sky object to characterize residual background

TODO List

  • Directly download data products from HSC pipeline (e.g., coadded Patch or source catalogs)

  • Access to HSC weak-lensing shape catalog

  • Access to HSC random catalog

  • Reproduce the CModel result

Installation

  • python setup.py install or python setup.py develop will do the job.
  • Right now, unagi only supports Python>=3. If you are still using Python 2, you should make the switch.
  • unagi only depends on numpy, scipy, astropy, and matplotlib. All can be installed using pip or conda.

Documents

I promisehope that documents will be available soon...but right now, please take a look at the Jupyter Notebook demos for each functionality.

Acknowledgement

Thanks the HSC collaboration for making this amazing survey happen and make these beautiful data available. Also thank the good people at NAOJ who work tirelessly to prepare the data release.

Reporting bugs

If you notice a bug in unagi (and you will~), please file an detailed issue at:

https://github.com/dr-guangtou/unagi/issues

Requesting features

If you would like to request a new feature, do the same thing.

License

Copyright 2019 Song Huang and contributors.

unagi is free software made available under the MIT License. For details see the LICENSE file.

unagi's People

Contributors

astrojacobli avatar christopher-bradshaw avatar denekow avatar dr-guangtou avatar eiffl avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar

unagi's Issues

A small shift of the center position

Very useful tools!
I found there is a small shift of the center position between the image I get from unagi (upper) and from hsc_map directly (lower):
unagi
hsc_map
They are fed with the same coordinate and the real size of the image is the same (3.6" x 3.6"). What could be a possible reason for this?
Thanks!

Switch to scm to maintain versions and add automated pypi pushes

One easy way to maintain the version of the package is to use git tags or releases along with the setuptools_scm module. Check this out: https://pypi.org/project/setuptools-scm/

We can also setup a Travis CI automated build to automatically push new releases to PyPi

Then all you need to do to update the version is create a new tag on github, everything else is taken care of.

I've set it up for a number of packages before, I can take of it :-)

unagi import forces the setting of plt.rc('text', usetex=True)

I like the default unagi plotting style but unagi may be a little too aggressive in its setting of the rc plot parameters. Simply doing import unagi sets the matloplib environment, and in particular sets plt.rc('text', usetex=True) here:

plt.rc('text', usetex=True)

I'm runing into an issue when trying to run a code that imports unagi internally, in an environment that doesn't have LaTeX installed. My code crashes with:

FileNotFoundError: [Errno 2] No such file or directory: 'latex': 'latex'

(Why not just installing LaTeX? Because I'm running on Google Colab, I have to reinstall the environment every time the notebook is restarted, and installing LaTeX takes a lot of time....)

Here are 2 suggestions:

Thoughts?

[New] Functions to deal with HSC fluxes from image and catalogs

Deal with HSC fluxes from image and catalog.

  • Convert HSC images unit to fluxes in CGS unit, AB magnitude, and nanomaggies.
  • Convert HSC catalog fluxes to AB magnitude, image unit, or naomaggies.
  • Deal with pre-s18a/PDR1 catalog and after-s18a/PDR2` catalogs separately.

Download and generate SDSS-like color picture

Modified based on the script provided by NAOJ people

TODO:

  • Modify the NAOJ script to generate the picture
  • Generate 3-color figure using Lupton SDSS true color
  • Option to add scale bar or compass
  • Write test function

Bulk Dowload Feature

I was looking through the hsc-das-cutout documentation and noticed they have a bulk upload api for space delimited tables (https://hsc-release.mtk.nao.ac.jp/das_cutout/pdr2/manual.html#list-to-upload)

I think this might be a useful addition to your already great project.
Here is an example table from the documentation:

#? rerun      filter    ra       dec       sw     sh  # column descriptor
   pdr2_wide  HSC-G  -1:36:00  00:00:00  2asec  2asec #
   pdr2_wide  HSC-R  -1:36:00  00:00:00  2asec  2asec #
   pdr2_wide  HSC-I  -1:36:00  00:00:00  2asec  2asec # list of coordinates
   pdr2_wide  HSC-Z  -1:36:00  00:00:00  2asec  2asec #
   pdr2_wide  HSC-Y  -1:36:00  00:00:00  2asec  2asec #

Assuming the table above was in a file called sources.txt we could pull them all at the same time with the following code using the requests library:

import requests
from tqdm import tqdm

creds = {
    "user":"your-username-here",
    "secret":"your-password-here"
}

with open("sources.txt", "r") as f:
    # https://stackoverflow.com/a/37573701
    r = requests.post("https://hsc-release.mtk.nao.ac.jp/das_cutout/pdr2/cgi-bin/cutout",
                      files={"list":f}, auth=(creds["user"], creds["secret"]), stream=True)

block_size = 1024
with open("downloaded.tar.gz", "wb") as f:
    for data in tqdm(r.iter_content(block_size), unit='KB', desc=f"Downloading file"):
        f.write(data)

Push unagi to PyPi

Unagi is awesome, let's make it trivially installable with pip by pushing it to PyPi ;-)

@dr-guangtou Happy to help with this, let me know if I can provide any assistance.

SQL doesn't find S20A photo-z catalog

Issue found by Connor Bottrell:

I find that the SQL query tool in unagi doesn’t recognize the s20a_wide.photoz_mizuki table:

import unagi
dr = unagi.hsc.Hsc(dr='dr3', rerun='s20a_wide')
sql_cmd = [
    'SELECT f.object_id, f.ra, f.dec, z.photoz_median',
    'FROM s20a_wide.forced as f',
    'LEFT JOIN s20a_wide.photoz_mizuki as z',
    'ON f.object_id = z.object_id',
    'LIMIT 10'
          ]
sql_cmd = ' '.join(sql_cmd)
print(sql_cmd)
result = dr.sql_query(sql_cmd, from_file=False, verbose=True)

Which gives:

Waiting for query to finish... [Failed]
query error: - Query Error:
relation "s20a_wide.photoz_mizuki" does not exist
LINE 1: ..., z.photoz_median FROM s20a_wide.forced LEFT JOIN s20a_wide....

[New] HSC filters

Add a filters.py module to deal with HSC filters:

  • Convert the HSC .txt format filter response curves to the .par format used by kcorrect and sedpy
  • Get the center wavelength and effective wavelength of each filter
  • Easy to access HSC filter list for both Wide filters and all fitlers
  • Plot the filter response curves.

[New] Method to check whether a coordinate is covered by HSC

  • There are multiple ways to do this using SQL search, but neither is fast enough to check a large number of galaxies.
  • If we could keep a copy of the Tract/Patch information of HSC footprints, there might be faster way to do this.

Will think more on this...

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.