astrometry.net address hardcoded, impossible to use a local running nova server about stellarsolver HOT 67 CLOSED

Ladislav to answer your question, cygwin astrometry is supported just like ANSVR and local astrometry on Macs and Linux. As long as cygwin or ANSVR is running on the same windows machine you are using for Ekos, KStars will send it all appropriate command line arguments you would like to send it using the Ekos options. I believe it now supports every command line option that astrometry.net offers (at least the ones that matter).

from stellarsolver.

I will see if I can get a screenshot for you

from stellarsolver.

For the source extraction method, you have two options available to you, you can use internal SEP, or astrometry.net's built in Star extraction. For the command line options, that is what the "Options profile" is for. There are so so many options. I found it best to try to develop profiles of options for specific tasks. If you don't like them though, you can customize the options or you can make your own set.

from stellarsolver.

Ladislav, so the "scale and position" options are separated from the rest. They are the settings that get changed all the time, they are in another tab. The rest of the settings can be accessed in the profile editor if you need to. But note that the profile that gets used is the one selected in the other screenshot at the far right. You shouldn't need to change these much, my goal is to figure out what the best settings should be for certain situations and make a profile that the user can select. You can make a profile of the "best" settings as well and save it or send it others as a file. It is exportable.

from stellarsolver.

I am not sure that it would matter that you are using Cygwin vs. ANSVR, I did test both a lot when I was working on the project. But its possible that some configuration is different on your system for some reason. Maybe a path is to a different place for example. I could definitely run a Cygwin test again.

from stellarsolver.

You know I think it would be good if Ekos informed you why the scale it is sending to StellarSolver is not the same as what you entered. . . we should add a message saying that.

from stellarsolver.

Let me think about this, because we could either convert all images to FITS first before sending them to StellarSolver, or I could add another option to StellarSolver for always converting to FITS first. We could hard code that as turned on in KStars and then we could remove all references to Python from the Align options. That also would allow me to delete the crazy recipe's in the KStars Mac craft build that require the building of netpbm and the inclusion of python. That might be a nice idea. KStars does not need to be sending JPEGS to astrometry if we an avoid it. Good idea!

from stellarsolver.

Ok I made the python avoidance commands universal and I made an option that would only sent FITS files to astrometry.

The next step will be to add the usage of this feature to Ekos. Then hopefully we can get rid of some unnecessary recipes in craft and remove clutter from KStars. Maybe this will end our astrometry python issues for good. Actually I thought I had done that, but they came back in this thread. Hopefully now. . .

There will be a few steps before it will get into the windows and Mac releases.

from stellarsolver.

Ok I added the --fits-image parameter

from stellarsolver.

Hi there, we can definitely change that to support this. But I must ask, why do you need that? If it is on Mac, you can directly access astrometry.net's solve-field binary. If it is windows, you can directly access the Cygwin or ANSVR solve-field binary. Are you hosting another astrometry server on your local network on a different machine?

from stellarsolver.

I think I found a post relating to that on the INDI forum just now. But I can't seem to log in. From that forum it looks like I found that you are hosting a local astrometry server for multiple imagers imaging nearby. That makes sense. Do you have any details about how you are hosting your astrometry server? Is this an ANSVR server or does it work exactly like nova.astrometry.net? If it is the second is that easy to set up? If I was going to make such a change I would need to be able to test it.

from stellarsolver.

from stellarsolver.

BTW, I had to make a change to my ANSVR server just now, as I discovered the configuration panel from Ekos has an enormous issue: clicking or not on use position and/or use scale does nothing if you're not using the internal solver. Actually, I'm not able to check if it does something with the internal solver, as that doesn't log anything despite me having the verbose option on on the alignment log.
So I made it that my server always blind solves, no matter what the json object contains, it will always discard ra, dec and radius.

from stellarsolver.

Hi there, we can definitely change that to support this. But I must ask, why do you need that? If it is on Mac, you can directly access astrometry.net's solve-field binary. If it is windows, you can directly access the Cygwin or ANSVR solve-field binary. Are you hosting another astrometry server on your local network on a different machine?

Hi @rlancaste

I have my own custom Cygwin build of astrometry.net on Windows. I also have KSTARS for Windows installed.

Can U please show me (when possible via a screenshot), how to access my solve-field binary from KSTARS?
For example when binary is at
C:\astrometry.net\bin\solve-field.exe
how to specify that in KSTARS/EKOS Windows version?

Another question is, whether is it possible also specify custom cmdline arguments to the solve-field (again from KSTARS/EKOS on Windows)?
Or it is calling it using hardcoded set of arguments?

Thanks and regards,

Ladislav

from stellarsolver.

So one thing that I discovered when I was doing all this work on astrometry.net last year was that I could directly control ANSVR and Cygwin based astrometry.net on windows just like astrometry.net on Macs and Linux. This function is built into StellarSolver and also Ekos as a result. This method of access proved to be way way faster than using ANSVR the traditional way. This is why I made the change to just use online for the actual astrometry.net server and to use "local" for ANSVR, since it worked so much better.

from stellarsolver.

I had not considered the use case of having ANSVR set up on one computer and a bunch of other machines accessing it in the field. I have always put either astrometry, ANSVR, or cygwin ANSVR on the same machine I was using to run Ekos since it is orders of magnitude faster than ANSVR. But I could see if you are using a bunch of older raspberry pi's or you were short on space, it might be good to have a local astrometry server. I will look into supporting this use case then later today.

from stellarsolver.

It might take a little bit because while I had a nice virtual machine all set up when I was doing this work last year, I don't think I have that machine anymore. It wouldn't help as much to show you a Mac or Linux screenshot.

from stellarsolver.

Hmm, let's try this, here are some Mac screenshots where I filled in the settings that I think I was using for Cygwin on Windows. Your paths should be similar, but you should check

from stellarsolver.

from stellarsolver.

@rlancaste
Thanks for the screenshots and explanation, I'll try.

from stellarsolver.

papaf76, as I said a couple of minutes ago, I will run some tests today/this weekend to verify how well StellarSolver is playing with ANSVR when used in place of astrometry. I can easily fix the problem you found, but my concern right now is whether some features I am relying on from astrometry.net (such as downloading logs) will be supported the same way on an ANSVR server.

from stellarsolver.

papaf76, I think I have fixed the problem you brought up. I believe the hard coded nova.astrometry.net was just an oversight on my part because just a few lines above where I hard coded nova.astrometry.net, it was using the custom astrometry server path. So probably just an accident. I also made a couple of edits to make the tester work with alternate astrometry servers and made the log output display what server is being accessed in its statements. I was able to solve an image with your remote ansvr server that you posted above after making these changes. Note that this change was made in master on GIT, so if you are using a stellarsolver that is packaged with windows KStars or if you are using a stellarsolver that is installed from a package manager, then there will be a delay before these changes show up there. Let me know if there are more issues.

from stellarsolver.

papaf76, as for the other thing you brought up, that the online solver is not using the scale or position settings. It definitely should be using them since those options are passed by stellarsolver to the online server along with the image and other info. However, I could certainly check that they correctly get passed on, since there are several steps in the process and there could be a bug somewhere.

from stellarsolver.

from stellarsolver.

Ok, did you try this with ekos's load and slew feature, or is this with plate solving? If you are plate solving in the align tab, you can select to have it upload the whole image, or do SEP on it and just upload the star list. You can set that with star extraction method.

Also are you saying that Ekos is passing the position and scale on to the solver separately or that the image is uploaded with position and scale in the headers and then astrometry on the server is using what is in the headers?

from stellarsolver.

from stellarsolver.

from stellarsolver.

Load and slew is different. When I first made my updates to integrate StellarSolver, I made Load and Slew use the "use position" and "use scale" checkboxes in Ekos (actually I think that I first made it blind solve every time, but that was short lived). I think someone might have changed it more recently so that load and slew ignores the boxes, but we can check that.

from stellarsolver.

@rlancaste

I have KSTARS v3.5.4, this version:
https://www.indilib.org/jdownloads/kstars/kstars-3.5.4.exe

Found the settings you had on your screenshots. Here are the settings I used:

As you can see in the first screenshot I am using custom profile called "Lacko". Also on the fourth screenshot there are settings for the "Lacko" profile. Unfortunately I can set only the solving parameters provided by this editor.

Here is the log output from the EKOS Alignment window:

2021-08-20T17:40:20 Solver Failed.
2021-08-20T17:40:20 Base: "C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_13", basefile "externalSextractorSolver_13.fits", basedir "C:/Users/Ladislav Heller/AppData/Local/Temp", suffix "fits"
2021-08-20T17:40:20 Reading input file 1 of 1: "C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_13.fits"...
2021-08-20T17:40:20 Command: C:/cygwin64/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --downsample 3 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -vv --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_13.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_13.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_13.wcs --keep-xylist C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_13.xyls C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_13.fits
2021-08-20T17:40:20 Starting external Astrometry.net solver with the Lacko profile...
2021-08-20T17:40:20 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-20T17:40:20 Configuring external Astrometry.net solver classically: using python and without Sextractor first
2021-08-20T17:40:20 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

From the log it is clear that it uses the "Lacko" profile during solving but it seems to ignore ScaleLow and ScaleHigh parameters.
Also there are a lot of hardcoded parameters, some of them I would completely ignore. But the most important is that there are missing the parameters I really need during solving and I don't know how/where to enter them.

A typical WORKING solver command line in my custom astrometry.net build is following:
solve-field --timestamp -O -l 30 -D /tmp -L 40.0 -H 90.0 -u dw -z 3 -y --no-plots --no-remove-lines --uniformize 0 --fits-image /cygdrive/d/Astro/Sas.fits

I guess it is not possible to call solve-field like the above example, from EKOS, right?
What I really miss in the Alignment options window (in the profile editor) is a text field where we can enter our custom command line parameters for solve-field and when this text field is filled-in, EKOS shall pass them to solve-field and ignore everything else in the profile setting.

BR,

Ladislav

from stellarsolver.

from stellarsolver.

I agree, I made it that way, it wasn't my decision to change it. I think it was changed though because the load and slew is using totally a different position and scale than the current ekos position and scale which is shown next to the "Use Position" and "use scale" checkboxes. I think it was causing some confusion on the part of users, which is why it was changed.

from stellarsolver.

from stellarsolver.

Yeah, there is always room for improvement. I will see what Jasem thinks about it.

from stellarsolver.

@rlancaste @papaf76
Sorry for jumping in to your conversation.
@rlancaste can you please check my last comment above?

from stellarsolver.

Ladislav,

Sorry I missed that comment. I spent hundreds of hours going through all the details of parameters and code in astrometry.net last year, and working with people over the last 5 years who had no end of problems with plate solving. This project was intended to get on top of all the issues and get rid of most of them. I can help you with these details. The options in astrometry.net can get a little crazy. I studied all of them, exposed the ones you need in an interface, and hard coded the ones that shouldn't be turned off or it will fail. There are also some options that I added that make other features work. The custom box was nice, but it caused many many problems. When users entered one too many spaces for instance, or accidentally deleted something that needed to be there. It was bad.

If it failed immediately, please make sure first that all the paths you entered are correct for your application.

You told me that your solver was here:
C:\astrometry.net\bin\solve-field.exe

But you entered this:
C:\cygwin64\bin\solve-field.exe

Please check to make sure that all 3 paths exist and are entered correctly. You can probably not worry too much about the config file, since StellarSolver should make its own and you should use that, but still check all the paths.

To answer your questions about the parameters, I will try to explain here:

Here was the option set you listed:
solve-field --timestamp -O -l 30 -D /tmp -L 40.0 -H 90.0 -u dw -z 3 -y --no-remove-lines --uniformize 0 --fits-image /cygdrive/d/Astro/Sas.fits

The -O and --no-plots options match what was passed exactly.
The -z parameter was passed, it is the same as --downsample
The -L/--scale-low and -H/--scale-high parameters were not passed because you turned off "use scale", however if you check the temporary config file, you will find that it used the minwidth of 40 and maxwidth of 90 parameters in that file since they are settings that are to be used when the scale is not provided.
The -u parameter was not passed because you turned off "use scale". The parameters in the temporary config file are in degrees by default.
The --timestamp parameter is not really needed, but might be nice to add.
The --no-remove-lines and --uniformize 0 options are added to avoid python usage if you decide to use an external star extractor or SEP, you should leave those options out if you want it to use the built in star extraction process.
The -D parameter is not needed here since StellarSolver copies the file into the temp directory first and then all the files end up in there.
The --fits-image parameter is unnecessary since it automatically assumes it is a fits image.
The -l parameter was not passed because it was generated in the temporary config file.

I think that addresses everything that was in your options list. Please let me know if you have more questions.

Thanks,

Rob

from stellarsolver.

Also, you should not have to make a new options profile unless you want to optimize something. If you are using one of the default profiles, they should just work. None of them would cause it to say "solver failed" right away. I would recommend using a default profile until you get it working, and then make a custom one if you like. An immediate "solver failed" message is usually a configuration or paths problem.

from stellarsolver.

@rlancaste

Thanks for replying. Both the following paths are correct since I have two indepenent Cygwin/Astrometry.net instances:

C:/cygwin64/bin/solve-field.exe
C:/astrometry.net/bin/solve-field.exe

I also enabled the Use Scale settings and entered my estimated scale parameters 40deg and 90deg:

The first interesting and for me confusing behaviour is that when I start "Load & Slew..." the ScaleLow and ScaleHigh parameters are missing from the solver command line, no matter whether I enable or disable the "Use Scale" checkbox:

2021-08-21T15:25:37 Solver Failed.
2021-08-21T15:25:37 Checking if file "C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_32.fits" ext 0 is xylist or image: image
2021-08-21T15:25:37 Base: "C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_32", basefile "externalSextractorSolver_32.fits", basedir "C:/Users/cica/AppData/Local/Temp", suffix "fits"
2021-08-21T15:25:36 Reading input file 1 of 1: "C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_32.fits"...
2021-08-21T15:25:36 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --downsample 3 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -vv --backend-config C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_32.cfg --cancel C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_32.cancel -W C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_32.wcs --keep-xylist C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_32.xyls C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_32.fits
2021-08-21T15:25:36 Starting external Astrometry.net solver with the Ladsi profile...
2021-08-21T15:25:36 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-21T15:25:36 Configuring external Astrometry.net solver classically: using python and without Sextractor first
2021-08-21T15:25:36 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

I also wrote a small program which catches and displays the temporary externalSextractorSolver_XX.cfg file before it is deleted. The content is below and seems to be correct:

minwidth 40
maxwidth 90
cpulimit 600
autoindex
add_path D:/data

I think the above scenario does not really work in Windows version of KSTARS/EKOS. Was able to reproduce many times, always the same result: Scaling parameter are not included in the command line.

For "Load & Slew" I use an image which perfectly works from Cygwin via command line solve-field!

Second scenario when I try "Capture & Solve", it also does some weird things:
It sometimes changes/overwrites my scale settings defined in the "Scale & Position" options even when the "Auto Update" option is disabled. The defined scaling options however are appearing in the logged command line but they are definitely not the ones I specified:

2021-08-21T16:03:04 Solver Failed.
2021-08-21T16:03:04 (not xyls because: FITS file does not have any extensions)
2021-08-21T16:03:03 Checking if file "C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_34.fits" ext 0 is xylist or image: image
2021-08-21T16:03:03 Base: "C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_34", basefile "externalSextractorSolver_34.fits", basedir "C:/Users/cica/AppData/Local/Temp", suffix "fits"
2021-08-21T16:03:03 Reading input file 1 of 1: "C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_34.fits"...
2021-08-21T16:03:02 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --downsample 3 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 32 -H 108 -u degwidth -3 111.089 -4 90 -5 50 -vv --backend-config C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_34.cfg --cancel C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_34.cancel -W C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_34.wcs --keep-xylist C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_34.xyls C:/Users/cica/AppData/Local/Temp/externalSextractorSolver_34.fits
2021-08-21T16:03:02 Starting external Astrometry.net solver with the Ladsi profile...
2021-08-21T16:03:02 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-21T16:03:02 Configuring external Astrometry.net solver classically: using python and without Sextractor first
2021-08-21T16:03:02 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-21T16:03:01 Image received.
2021-08-21T16:02:49 Capturing image...

As you can see in the above log, it is passing ScaleLow(L)=32deg and ScaleHigh(H)=108deg. I have no idea where those values are coming from. I expected values L=40deg and H=90deg.

I tried a lot of different comibnation of settings for both "Capture&Solve" and "Load&Slew" but it never solved me successfully.
I give up, have no idea what I am doing wrong....

BR,

Ladislav

from stellarsolver.

Ladislav,

For this first issue:
"The first interesting and for me confusing behaviour is that when I start "Load & Slew..." the ScaleLow and ScaleHigh parameters are missing from the solver command line, no matter whether I enable or disable the "Use Scale" checkbox"

Ok, I think the issue here is "load and slew". This is an Ekos issue not a StellarSolver issue. As I was recently discussing with papaf76 both in the discussion above and in the other forum. I personally think that Ekos should respect the Scale and position settings when doing a Load and Slew, but it currently ignores them. Jasem made a change just yesterday I think, that would fix that. But it will take time to get into a windows build release I think.

As for Capture and Solve. I investigated it this afternoon.
I just opened a fresh windows machine about 10 minutes ago and I ran some tests. I downloaded and installed ANSVR, then I downloaded and installed KStars. I did not open or change any configurations in ANSVR at this time. Then I opened up KStars, started up an INDI Server on a Mac machine and connected to the INDI server from the Windows machine. I then went into the settings for the Align module, I clicked to use "Local Astrometry" instead of the internal solver. I went to the Index file downloader and downloaded a bunch of index files (which get placed into ANSVR's index file directory). I turned on "verbose logging" and "alignment logging" in the Ekos logs options. I increased the capture time for solving to 10 seconds to get a decent amount of stars for plate solving, and I changed it to 2x2 binning. I did not change any other settings. Then I slewed to a location in the sky and did a "Capture and Solve". It did the following and solved the image:

Then I changed the star extraction method to the "built in option" and I did another capture and solve, and it solved it again:

In the screenshots, you should see all the details about how it solved the image, what parameters were sent etc.

As for this item you said: "It sometimes changes/overwrites my scale settings defined in the "Scale & Position" options". I think we should definitely look at why Ekos is doing this. I don't think it should do that.

from stellarsolver.

I think this statement "I think the above scenario does not really work in Windows version of KSTARS/EKOS. " is true because Ekos is currently not respecting your choice to use or not use scale. However to say that it currently doesn't work on Windows would not be true because I just tested it on a fresh windows machine and it did work.

from stellarsolver.

@rlancaste
Maybe the EKOS doesn't like my astrometry.net built from latest sources on Cygwin 64bit. Which is interesting because when I start Cygwin terminal and try there any astro image, it just works.
Anyway, I'll try that ANSVR stuff.

from stellarsolver.

The ScaleLow(L)=32deg and ScaleHigh(H)=108deg instead of L=40deg and H=90deg is something that Ekos does I believe to enlarge the scale slightly because often times people specify a scale and the image ends up just outside of their range (due to things like filters, paracorr's, and other things that change the focal length a little. We were getting a lot of people who got failures because their scale was too narrow. So I think it is 40 * 0.8 = 32 and 90 * 1.2 = 108

from stellarsolver.

@rlancaste

Just tried the combination of InternalSolver & BuiltInMethodForSolver and it WORKED for both Capture&Solve and Load&Slew !!! So I am 99% happy and thank you for this StellarSolver stuff, it's really a good tool!

However I stil can't believe that it fails for my local custom build of astrometry.net.
One last idea, if you have time, I can upload somewhere my astrometry.net build and you can download and try with that...

BR,

Ladislav

from stellarsolver.

Ok so then you didn’t decide to try ANSVR?

Yes, for the Cygwin build it should work as well unless there is a configuration issue or I guess if astrometry made some big changes and they changed some parameters on us since last year. Did you just install Cygwin and build astrometry with the latest sources following their instructions? I could try that again and see. Yesterday I only tested ansvr since that is much easier to set up.

from stellarsolver.

@rlancaste

Did not tried yet with ANSVR, since the internal solver works.

Anyway, if you want to check it with my custom build of Cygwin/astrometry.net, you can download it from:
https://drive.google.com/file/d/1oJ6JveXm0nVNiwse4CUmF7ozaVyGlSX8/view?usp=drivesdk

If you decide to try it, just simple extract the downloaded file (and also the TAR inside) using 7-zip utility to any folder at the root drive.

BR,

Ladislav

from stellarsolver.

Ok, so I tried your custom astrometry.net build. I unzipped it as instructed, I changed the paths to match the correct programs:
The solver binary was located at:
C:/astrometry.net/bin/solve-field.exe
The wcsinfo program was located at:
C:/astrometry.net/bin/wcsinfo.exe

Then I tested it using my INDIServer simulator as I did yesterday on ANSVR. I set it to use the following options:

It captured and solved successfully. But then I tried these options:

and it failed. This can be seen in the following output:

To investigate why it failed, I went to the cygwin terminal and I tried to plate solve an image using a terminal command and it said the following:

/bin/sh: C:/astrometry.net/bin/image2pnm: /usr/bin/python3: bad interpreter: No such file or directory

Which leads me to think that the reason it failed for you might be because python3 is not properly installed in your cygwin build.

As an interesting aside, a very similar issue is the very reason that I started to embark to create StellarSolver in the first place. A bunch of people were having all kinds of python issues with astrometry.net (mainly on Macs) and it was driving me crazy. So I made StellarSolver to combine SEP source extraction with astrometry.net solving. Then it evolved from there to become the StellarSolver internal solver and the library.

from stellarsolver.

Let me know if there is something I can do with the python3 issue in your cygwin build and I can try again

from stellarsolver.

Wait, it just got weirder. I think python3 does exist, but maybe it's a broken link.
If you say which python2 you get: "/usr/bin/python2"
If you say which python3 you get: "which: no python3 in" (then a whole lot of paths it isn't in)
But I see the symbolic link in /usr/bin and in /usr.

from stellarsolver.

Just tested that theory, it is even weirder still. its a rabbit hole of links. The link "/usr/bin/python3" links to:
/cygdrive/c/astrometry.net/etc/alternatives/python3
Which is another symbolic link, which leads to:
/cygdrive/c/cygwin64/astrometry.net/usr/bin/python3.8
Which doesn't appear to exist in the files you sent.

from stellarsolver.

Ok so I changed your python3 link by doing a
rm /usr/bin/python3
and replaced it by doing a
ln -s /usr/bin/python3.8.exe /usr/bin/python3

And then I opened KStars again and tried the built in star extraction again, and it solved:

So yes, I think the main problem with your custom astrometry setup was the broken link to python3.

from stellarsolver.

@rlancaste

Nice!

I'll check that once I am back from work.
Thanks for investigating!

BR,

Ladislav

from stellarsolver.

As an interesting aside, a very similar issue is the very reason that I started to embark to create StellarSolver in the first place. A bunch of people were having all kinds of python issues with astrometry.net (mainly on Macs) and it was driving me crazy. So I made StellarSolver to combine SEP source extraction with astrometry.net solving. Then it evolved from there to become the StellarSolver internal solver and the library.

Just out of curiosity, might SEP be the reason why the behaviour in communication with nova changed? I mean specifically the need to pass the image size to the solver. That could explain a thing or two...
Another thing, if you know: with my modifications, ANSVR is now able to properly solve with a recent kstars version. Still, I'm not getting the same FOV dimensions compared to the proper nova server. Do you happen to know how to use stars.wcs to calculate this? Right now I'm multiplying the pixel scale to the image size but as I said, I don't seem to get the same result as nova.

Thanks again, for everything! You've been nothing short of awesome!

from stellarsolver.

For the first item, you will find that if you use internal SEP or external Sextractor for star extraction then a number of parameters that StellarSolver sends will change. Yes image height and width are required by astrometry to solve an xy list of sources. If you switch to the built in method and just send an image, width and height will not be sent.

For the second item, do you mean the final scale result sent back by nova? Did you downsample or bin the image? That can affect the scale astrometry reports. Do you have an example image?

from stellarsolver.

@rlancaste

Just askin', why the python(3) is needed for the solving of an image when there is no plotting and the input image is always converted to FITS before even the solver is started? I mean in StellarSolver.

Using the ProcMon tool for Windows and documentation for astrometry.net I figured out that python is completely skipped in case, when the input to solve-field is a FITS format image and the command line has following parameters:

-p (--no-plots)
-9 (--no-remove-lines)
--uniformize 0

So to summarize there are 4 conditions to skip python. If only one is not met, python is started. But we don't need python. We need only center coordinates of the image, FOV, pixel scale, etc but nothing else.
I checked this behaviour many times using the ProcMon tool and it's valid. Got always the same result.

I would suggest to include these cmdline parameters hardcoded into the command line in case of Cygwin, because I think in the 100% of cases the python is not needed for plate solving.

BR,

Ladislav

from stellarsolver.

One of the main reasons for starting StellarSolver was originally to get rid of python. In fact those exact three parameters are passed to local astrometry and Cygwin and ANSVR most of the time. It is only when the user wants to do the traditional “built in” solver for astrometry.net that astrometry is allowed to solve it how it wants.

from stellarsolver.

@rlancaste

According to astrometry.net documentation python is used only in two cases:

1. When input image is NOT the FITS format => converts it using python scripts to FITS
2. When user wants to plot annotations on the solved input image (other than FITS) => annotations are done by a python script

In our astrophotography scenarios none of the above is the case. We simply use FITS format and do NOT plot.
Next to just start another new python3 process by solve-field, without using its result is simply wasting of resources and it just slows down the overall plate solving.

So as you wrote:

One of the main reasons for starting StellarSolver was originally to get rid of python.

If that's the main reason, then why in some cases python is allowed when it's anyway not needed? :-)

BR,

Ladislav

from stellarsolver.

To be honest, the main reason I left astrometry.net to its own devices in that case was so that we could have a basis for comparison. The main goal was to use Sextractor or SEP combined with astrometry.net to avoid needing any python or netpbm at all. And when I did my first experiments, that method was giving far superior results to the astrometry internal star extraction and solving. I definitely see it as the preferred method in almost all cases, including using online astrometry.net or remote ANSVR, because then you are uploading a text file instead of an image file, its WAY smaller. The only reasons I left in the ability to solve using an image and astrometry's built in methods was so that we could compare the performance to what astrometry naturally wants to do and also so that people would still be able to use astrometry.net if for some reason SEP or Sextractor didn't work for them and astrometry.net's method worked better. But so far, SEP and sextractor still work much better almost universally.

Actually one of the best abilities of StellarSolver is its ability to plate solve jpeg images very very fast. To compare its performance to using astrometry.net natively, we would need to allow it to plate solve using python to do the conversion which is what it naturally does. If we are converting it to FITS first, that isn't normal astrometry behavior, so it isn't really a good comparison. Now I could see adding an option that allows the user to force all images to be converted to FITS before being sent to astrometry.net, but that is adding complexity.

It is true that many times we are working with FITS images in KStars, but the "load and slew" feature often works with Jpeg images. Those Jpegs would need to be converted before being sent.

from stellarsolver.

@rlancaste

Just made a short test, how python affects the overall plate solving performance.
I used the free tool called ProcMon, which is able to monitor running processes on Windows.

Configured its filter to allow monitor only solve-field, shell processes and ProcessCreate operation:

First I executed a plate-solving (in the Cygwin terminal) of a PNG image, where also python is involved at least by converting PNG to FITS via netpbm. I also let solve-field to plot annotations.
Command line in Cygwin was:

solve-field -O -l 30 -D ./Solve -L 40 -H 90 -u dw -z 3 ./M31.png

In ProcMon we can see, that solve-field creates a lot of subprocesses including among others python:

As we can see in the monitor, python was 3 times called.

In the second case I used again a PNG image, but this time with following command line:

solve-field -O -l 30 -D ./Solve -L 40 -H 90 -u dw -z 3 -9 --uniformize 0 -p ./M31.png

From the output we can clearly see that python was called only once to do PNG->FITS conversion:

Finally in the third case I used FITS input image with additional command line parameters to completely skip python:

solve-field -O -l 30 -D ./Solve -L 40 -H 90 -u dw -z 3 -9 --uniformize 0 -p -y --fits-image ./M31.fits

Please note that the --fits-image parameter is also required to reach our goal! When this parameter is not specified for a FITS input file, python is still called! Just checked your already pushed commit and saw that this parameter is mising ;-)

As we expected, the python was not called by solver and the overall plate-solving was much faster:

Performance and the fact that we don't (really) need python are the only reason why I suggest to improve StellarSolve by addig the above command line parameters!

BR,

Ladislav

from stellarsolver.

@rlancaste

In the meantime I tried again with my local custom built Cygwin/astrometry.net using this combination:

Means:
Solving method = Local Astrometry
Source Extraction Method = Internal SEP

Using the above combination plate-solving works well, here is the output:

2021-08-24T21:06:14 Solver RA (296.14873) DEC (9.83861) Orientation (-178.26570) Pixel Scale (124.45108) Parity (neg)
2021-08-24T21:06:14 Solver completed after 1.43 seconds.
2021-08-24T21:06:14 Shutting down other child solvers
2021-08-24T21:06:14 Successfully solved with child solver: 3
2021-08-24T21:06:14 Finished Loading WCS...
2021-08-24T21:06:14 Loading WCS from file...
2021-08-24T21:06:14 Field parity: neg

2021-08-24T21:06:14 Field rotation angle: up is -178.266 degrees E of N
2021-08-24T21:06:14 Pixel Scale: 124.451"
2021-08-24T21:06:14 Field size: 63.31 x 44.84 arcminutes
2021-08-24T21:06:14 Field center: (RA H:M:S, Dec D:M:S) = (19:44:35.694, +09:50:18.994).
2021-08-24T21:06:14 Field center: (RA,Dec) = (296.149, 9.83861) deg.
2021-08-24T21:06:14 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 20.625 -H 52.5 -u degwidth --width 2048 --height 1364 --x-column X_IMAGE --y-column Y_IMAGE --sort-column MAG_AUTO --sort-ascending --no-remove-lines --uniformize 0 --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.wcs C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.xyls
2021-08-24T21:06:13 Starting external Astrometry.net solver with the 3-LargeScaleSolving profile...
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 105.625 -H 180 -u degwidth --width 2048 --height 1364 --x-column X_IMAGE --y-column Y_IMAGE --sort-column MAG_AUTO --sort-ascending --no-remove-lines --uniformize 0 --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.wcs C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.xyls
2021-08-24T21:06:13 Starting external Astrometry.net solver with the 3-LargeScaleSolving profile...
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 52.5 -H 105.625 -u degwidth --width 2048 --height 1364 --x-column X_IMAGE --y-column Y_IMAGE --sort-column MAG_AUTO --sort-ascending --no-remove-lines --uniformize 0 --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.wcs C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.xyls
2021-08-24T21:06:13 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 10 -H 20.625 -u degwidth --width 2048 --height 1364 --x-column X_IMAGE --y-column Y_IMAGE --sort-column MAG_AUTO --sort-ascending --no-remove-lines --uniformize 0 --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.wcs C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.xyls
2021-08-24T21:06:13 Starting external Astrometry.net solver with the 3-LargeScaleSolving profile...
2021-08-24T21:06:13 Starting external Astrometry.net solver with the 3-LargeScaleSolving profile...
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 Configuring external Astrometry.net solver using an xylist input
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 Configuring external Astrometry.net solver using an xylist input
2021-08-24T21:06:13 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T21:06:13 Configuring external Astrometry.net solver using an xylist input
2021-08-24T21:06:13 Configuring external Astrometry.net solver using an xylist input
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:13 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T21:06:13 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T21:06:13 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T21:06:12 Solver # 4, Low 105.625, High 180 arcminwidth
2021-08-24T21:06:12 Solver # 3, Low 52.5, High 105.625 arcminwidth
2021-08-24T21:06:12 Solver # 2, Low 20.625, High 52.5 arcminwidth
2021-08-24T21:06:12 Solver # 1, Low 10, High 20.625 arcminwidth
2021-08-24T21:06:12 Starting 4 threads to solve on multiple scales
2021-08-24T21:06:12 Stars Found after Filtering: 50
2021-08-24T21:06:12 Keeping just the 50 brightest stars
2021-08-24T21:06:12 Removing the stars with a/b ratios greater than 1.5
2021-08-24T21:06:12 Stars Found before Filtering: 497
2021-08-24T21:06:12 Starting Internal StellarSolver Sextractor with the 3-LargeScaleSolving profile . . .
2021-08-24T21:06:12 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T21:06:12 There should be enough RAM to load the indexes in parallel.
2021-08-24T21:06:12 Evaluating Installed RAM for inParallel Option.  Total Size of Index files: 0.634597 GB, Installed RAM: 15.9458 GB, Free RAM: 10.352 GB

.

Tried again also the combination:
Solving method = Local Astrometry
Source Extraction Method = BuiltIn method for solver
It again failed like many times before, but this time I took the command line from log, as you did and put it to the Cygwin terminal and started and guess what happend: It works from Cygwin! Can't believe, but the same command line works from Cygwin terminal!!!

I will do further investingating, but now it is clear that my local custom Cywin/astrometry.net is fully working, but somehow when started from StellarSolver, it fails for the above combination.

BR,

Ladislav

from stellarsolver.

if you think the --fits-image parameter is really needed when a user wants to use the "built in" star extraction in order to avoid python. I am not sure why it would need that when its being given a FITS file already? maybe it isn't sure it is a FITS file till it tries?

from stellarsolver.

For your experiments, I don't think you want the "large-scale solving" profile, which I see selected in your screenshot. I designed that one to work best for DSLR lens scale images. You probably want "Parallel small scale"

from stellarsolver.

When I checked your cygwin build it had a broken link as I said, when I fixed that it worked fine.

from stellarsolver.

if you think the --fits-image parameter is really needed when a user wants to use the "built in" star extraction in order to avoid python. I am not sure why it would need that when its being given a FITS file already? maybe it isn't sure it is a FITS file till it tries?

@rlancaste
Just try it with and without this parameter, while watching subprocesses in ProcMon tool, if you don't trust me :)
In my environment python was only skipped when input file was a FITS and following parameters were passed to solve-field:

• -p (--no-plots)
• -9 ( --no-remove-lines)
• --uniformize 0
• --fits-image

Any other combination of input format and command line parameters involved python.

BR,

Ladislav

from stellarsolver.

When I checked your cygwin build it had a broken link as I said, when I fixed that it worked fine.

@rlancaste
Yes I know, but in my cygwin build that symbolic link is correct! Anyway, plate-solve failed.

BR,

Ladislav

from stellarsolver.

For your experiments, I don't think you want the "large-scale solving" profile, which I see selected in your screenshot. I designed that one to work best for DSLR lens scale images. You probably want "Parallel small scale"

@rlancaste
Tried all 4 default profiles and the builtin extractor but always the same error:

2021-08-24T22:57:33 Solver Failed.
2021-08-24T22:57:33 Child solver: 2 did not solve or was aborted
2021-08-24T22:57:33 Child solver: 4 did not solve or was aborted
2021-08-24T22:57:33 Child solver: 1 did not solve or was aborted
2021-08-24T22:57:33 Child solver: 3 did not solve or was aborted
2021-08-24T22:57:32 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --downsample 2 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 0.71875 -H 2.575 -u degwidth --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.wcs --keep-xylist C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.xyls C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_1.fits
2021-08-24T22:57:32 Starting external Astrometry.net solver with the 4-SmallScaleSolving profile...
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --downsample 2 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 2.575 -H 5.66875 -u degwidth --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.wcs --keep-xylist C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.xyls C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_2.fits
2021-08-24T22:57:32 Starting external Astrometry.net solver with the 4-SmallScaleSolving profile...
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --downsample 2 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 0.1 -H 0.71875 -u degwidth --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.wcs --keep-xylist C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.xyls C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_0.fits
2021-08-24T22:57:32 Command: C:/astrometry.net/bin/solve-field.exe -O --no-plots --no-verify --crpix-center --match none --corr none --new-fits none --rdls none --resort --objs 50 --downsample 2 --odds-to-solve 1e+09 --odds-to-tune-up 1e+06 -L 5.66875 -H 10 -u degwidth --backend-config C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.cfg --cancel C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.cancel -W C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.wcs --keep-xylist C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.xyls C:/Users/Ladislav Heller/AppData/Local/Temp/externalSextractorSolver_1_3.fits
2021-08-24T22:57:32 Starting external Astrometry.net solver with the 4-SmallScaleSolving profile...
2021-08-24T22:57:32 Starting external Astrometry.net solver with the 4-SmallScaleSolving profile...
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 Configuring external Astrometry.net solver classically: using python and without Sextractor first
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T22:57:32 Configuring external Astrometry.net solver classically: using python and without Sextractor first
2021-08-24T22:57:32 Configuring external Astrometry.net solver classically: using python and without Sextractor first
2021-08-24T22:57:32 Configuring external Astrometry.net solver classically: using python and without Sextractor first
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
2021-08-24T22:57:32 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T22:57:32 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T22:57:32 The external ANSVR solver on windows does not handle the inparallel option well, disabling it for this run.
2021-08-24T22:57:32 Solver # 4, Low 5.66875, High 10 arcminwidth
2021-08-24T22:57:32 Solver # 3, Low 2.575, High 5.66875 arcminwidth
2021-08-24T22:57:32 Solver # 2, Low 0.71875, High 2.575 arcminwidth
2021-08-24T22:57:32 Solver # 1, Low 0.1, High 0.71875 arcminwidth
2021-08-24T22:57:32 Starting 4 threads to solve on multiple scales
2021-08-24T22:57:32 There should be enough RAM to load the indexes in parallel.
2021-08-24T22:57:32 Evaluating Installed RAM for inParallel Option.  Total Size of Index files: 0.634597 GB, Installed RAM: 15.9458 GB, Free RAM: 9.86173 GB
2021-08-24T22:57:32 Automatically downsampling the image by 2

BR,

Ladislav

from stellarsolver.

Ok I think I should close this issue here now since we pretty much resolved both issues in the thread.

from stellarsolver.

The next release of KStars should get these changes in it.

from stellarsolver.

Ok I think I should close this issue here now since we pretty much resolved both issues in the thread.

OK for me.

The next release of KStars should get these changes in it.

Great!

from stellarsolver.