kunzmi / imagestackalignator Goto Github PK

There's less noise, but it looks like everything has been smoothed out. It is significantly harder to make out fine detail such as pavement lines, borders between windows, etc. Is this just a case where there aren't enough frames, or the SNR is simply too low(in the paper, figure 21 is the only direct comparison between classic demosaicing and their method, but it's quite a bit brighter than the "New York Scene")? Of course, it's unlikely this algorithm is better in all cases, but I'm still curious as to why.

On a side note, can you explain more about Dth, Dtr, kDetail, kDenoise? I actually don't see it in the paper...

Hi @kunzmi
Do you think this project has a reasonable chance to run in mono in linux?
One obvious requirement is that all directX stuff is stripped off.
How about managedCuda and other dependencies? Would they survive and stay functional in *nix (say Ubuntu)?

I am able to compile, open the program, load the sample image files, but the images do not display in the GUI even afte double clicking on the loaded file name. (Tried Cuda 10.1 and 10.2 and Visual Studio 2019 and 2022 with the same result).

Hovering the mouse over the area where the image should be suggest that the image information is retained as the box on top by the image coordinates changes colors with different coordinates, but yet no image is displayed.

Any suggestions? Thanks!

Hi Michael, thanks for your code.
I want to run the quality analysis in Section 6.1 of Google's paper with your code, but the photo datasets they use (Kodak and McMaster) seems only offer tiff and png pictures.
Is there any way to accomplish the analysis, for example transforming TIFF to DNG or something?

In RobustnessModell.cu line 67 to 70，

			maxShift.x = fmaxf(s.x, shiftf.x);
			maxShift.y = fmaxf(s.y, shiftf.y);
			minShift.x = fminf(s.x, shiftf.x);
			minShift.y = fminf(s.y, shiftf.y);

It seems that you are finding minimum and maximum number of s, but it doesn't.
maybe the code below is your original idea

			maxShift.x = fmaxf(s.x, maxShift.x);
			maxShift.y = fmaxf(s.y, maxShift.y);
			minShift.x = fminf(s.x, minshift.x);
			minShift.y = fminf(s.y, minshift.y);

Hi and thanks for the implementation and the detailed explanations on the original article - you definitely helped me a lot.

At the cross correlation validation there is a minor bug that returns a shift check of 5 pixels as true even when sometimes the result is false.
ImageStachAlignatorController.cs - TestCC():
if (sx1-sx2 != 5 && sy1-sy2 != 5) -> should be || instead of &&.

regards, Eyal

Hi! Thank you so much for your sharing of the implementation and correction of Google's Handheld Multi-Frame Super-Resolution algorithm. I struggled with this paper for a long time......

I was going to try this algorithm on my computer, but unfortunately, I don't have Google Pixel or SLR to get a burst of the RAW picture. And I found that the paper part 6.1 mentioned that they created synthetic image bursts by:
generate a set of random offsets (Bivariate Gaussian distribution with a standard deviation of two pixels)
resample the image using nearest-neighbor interpolation
create a Bayer mosaic (discard 2/3 color channels)

But I failed to get the expected results... I just wonder whether you create these synthetic image bursts and if you'd like to post the code for creating synthetic image bursts on your Github? ：）~

Kernels\Kernels.vcxproj: The application which this project type is based on was not found. Please try this link for further information: http://go.microsoft.com/fwlink/?LinkId=441332&projecttype=8BC9CEB8-8B4A-11D0-8D11-00A0C91BC942

Hi, thanks for sharing this code!

I'm trying to run the code from either the New York images or the TestMode. But it always generates the exceptions. The system is Windows 10 and the CUDA version is 11.2. The GPU is GTX 1060 Super (6Gb).

If you are convenient, could you please give some advice? Thanks a lot!

ManagedCuda.CudaException
  HResult=0x80131500
  Message=ErrorInvalidHandle: This indicates that a resource handle passed to the API call was not valid. Resource handles are opaque types like CUstream and CUevent.
  Source=ManagedCuda
  StackTrace:
   at ManagedCuda.CudaKernel.Run(Object[] parameters)
   at PEFStudioDX.ComputeDerivatives2Kernel.RunSafe(NPPImage_32fC1 imgSource, NPPImage_32fC1 Ix, NPPImage_32fC1 Iy) in F:\wy\ImageStackAlignator\PEFStudioDX\OpticalFlowKernels.cs:line 139
   at PEFStudioDX.ImageStackAlignatorController.PrepareAccumulation() in F:\wy\ImageStackAlignator\PEFStudioDX\ImageStackAlignatorController.cs:line 1852
   at PEFStudioDX.MainWindow.PrepareAccumulationBtn_Click(Object sender, RoutedEventArgs e) in F:\wy\ImageStackAlignator\PEFStudioDX\MainWindow.xaml.cs:line 265
   at System.Windows.EventRoute.InvokeHandlersImpl(Object source, RoutedEventArgs args, Boolean reRaised)
   at System.Windows.UIElement.RaiseEventImpl(DependencyObject sender, RoutedEventArgs args)
   at System.Windows.Controls.Primitives.ButtonBase.OnClick()
   at System.Windows.Controls.Button.OnClick()
   at System.Windows.Controls.Primitives.ButtonBase.OnMouseLeftButtonUp(MouseButtonEventArgs e)
   at System.Windows.RoutedEventArgs.InvokeHandler(Delegate handler, Object target)
   at System.Windows.RoutedEventHandlerInfo.InvokeHandler(Object target, RoutedEventArgs routedEventArgs)
   at System.Windows.EventRoute.InvokeHandlersImpl(Object source, RoutedEventArgs args, Boolean reRaised)
   at System.Windows.UIElement.ReRaiseEventAs(DependencyObject sender, RoutedEventArgs args, RoutedEvent newEvent)
   at System.Windows.UIElement.OnMouseUpThunk(Object sender, MouseButtonEventArgs e)
   at System.Windows.RoutedEventArgs.InvokeHandler(Delegate handler, Object target)
   at System.Windows.RoutedEventHandlerInfo.InvokeHandler(Object target, RoutedEventArgs routedEventArgs)
   at System.Windows.EventRoute.InvokeHandlersImpl(Object source, RoutedEventArgs args, Boolean reRaised)
   at System.Windows.UIElement.RaiseEventImpl(DependencyObject sender, RoutedEventArgs args)
   at System.Windows.UIElement.RaiseTrustedEvent(RoutedEventArgs args)
   at System.Windows.Input.InputManager.ProcessStagingArea()
   at System.Windows.Input.InputManager.ProcessInput(InputEventArgs input)
   at System.Windows.Input.InputProviderSite.ReportInput(InputReport inputReport)
   at System.Windows.Interop.HwndMouseInputProvider.ReportInput(IntPtr hwnd, InputMode mode, Int32 timestamp, RawMouseActions actions, Int32 x, Int32 y, Int32 wheel)
   at System.Windows.Interop.HwndMouseInputProvider.FilterMessage(IntPtr hwnd, WindowMessage msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at System.Windows.Interop.HwndSource.InputFilterMessage(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndWrapper.WndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam, Boolean& handled)
   at MS.Win32.HwndSubclass.DispatcherCallbackOperation(Object o)
   at System.Windows.Threading.ExceptionWrapper.InternalRealCall(Delegate callback, Object args, Int32 numArgs)
   at System.Windows.Threading.ExceptionWrapper.TryCatchWhen(Object source, Delegate callback, Object args, Int32 numArgs, Delegate catchHandler)
   at System.Windows.Threading.Dispatcher.LegacyInvokeImpl(DispatcherPriority priority, TimeSpan timeout, Delegate method, Object args, Int32 numArgs)
   at MS.Win32.HwndSubclass.SubclassWndProc(IntPtr hwnd, Int32 msg, IntPtr wParam, IntPtr lParam)
   at MS.Win32.UnsafeNativeMethods.DispatchMessage(MSG& msg)
   at System.Windows.Threading.Dispatcher.PushFrameImpl(DispatcherFrame frame)
   at System.Windows.Application.RunDispatcher(Object ignore)
   at System.Windows.Application.RunInternal(Window window)
   at PEFStudioDX.App.Main()

Hello, could you help me with this exception?

ManagedCuda.CudaException: 'ErrorInvalidHandle: This indicates that a resource handle passed to the API call was not valid. Resource handles are opaque types like CUstream and CUevent.'

It was raised here:
public float RunSafe(NPPImage_32fC1 imgIn, NPPImage_32fC3 imgOut, float3 blackPoint, float3 scale)

I'm running using a NVIDIA RTX A3000 GPU with 6GB of memory.

Hello!
First of all I want to say great work, very thorough. If you ever get an explanaition to why the eigen value "division" is implemented in the way described in the report, I am interested to know the reasoning.

But now to my question. I have never worked with either C# or CUDA, so perhaps this is a beginners mistake. I have installed CUDA 10.2 and if I open the project in VS2019 and try to build, I get 2 succesful and one skipped is that the way it should be? If I try to run it after I've built the solution, I get an error message that says "Unable to start program ...\PEFStudioDX\Kernels the system cannot find the file specified.".

Hi,

Is it possible to run this code on device in C++ on ios RAW DNG files? Does it have any specific hardware limitations that prevents it from being run on other platform? I don't have GPU since i use an old Macbbook so i could not try the demo.

Hello Michael.
I am reading your code recently and get confused about how the super resolution works.
In file DeBayerKernels.cu,
line 398 to 402 writes

float posX = ((float)x + 0.5f + dimX / 2) / 2.0f / dimX;
float posY = ((float)y + 0.5f + dimY / 2) / 2.0f / dimY;

float4 kernel = tex2D<float4>(kernelParam, posX, posY);// *(((const float3*)((const char*)kernelParam + (y / 2 + dimY / 4) * strideKernelParam)) + (x / 2 + dimX / 4));
float2 shift = tex2D<float2>(shifts, posX, posY);// *(((const float2*)((const char*)shifts + (y / 2 + dimY / 4) * strideShift)) + (x / 2 + dimX / 4));

I think this will result in pixels in neighborhood of 2*2 area of the final high resolution image get the same kernel and shift.

Similarly, line 414 to 423 writes

int ppsx = x + px + sx + dimX / 2;
int ppsy = y + py + sy + dimY / 2;
int ppx = x + px + dimX / 2;
int ppy = y + py + dimY / 2;

ppsx = min(max(ppsx/2, 0 + dimX / 4), dimX/2 - 1 + dimX / 4);
ppsy = min(max(ppsy/2, 0 + dimY / 4), dimY/2 - 1 + dimY / 4);

ppx = min(max(ppx / 2, 0 + dimX / 4), dimX / 2 - 1 + dimX / 4);
ppy = min(max(ppy / 2, 0 + dimY / 4), dimY / 2 - 1 + dimY / 4);

Making four pairs of (x, y) get the same (ppsx, ppsy) and (ppx, ppy). I think this will result in four pixels in the final image point to the same pixel in the input low resolution image.

As four pixels in neighbourhood of 2*2 area get the same kernel, shift, uncertaintymask and raw img data, I think this will make them have same value in the final image.

What point do I miss in your code to achieve the high resolution?

Besides, it seems values of the 4th channel of the variable _structureTensor4 are always zero, which instantiated at line 1839 in file ImageStackAlignatorController.cs. What is the difference between _structureTensor4 and _structureTensor ?

Is it possible to use .tiff images as inputs?

Can you update solution for DirectX12 and Visual Studio 2022 ?
I try to make it work at this but still have problems with CUDA, SlimDX etc....

Hi,

1. I'm curious that in the paper k2h = kDetail / (kShrink * A), but in your implementation k2h = kDetail / kShrink * A, why did you modified it?
2. From formula derivation is [c,s]T corresponding to k1, and [s,-c]T corresponding to k2? Also, in your implementation they were reversed.

regards max

The paper said that the gradient is computed at a quater of the full resolution(In section 5.1.2, we create a single pixel from a 2 × 2 Bayer quad by combining four different color channels together. This way, we can operate on single channel luminance images and perform the computation at a quarter of the full resolution cost and with improved signal-to-noise ratio), but the code allocation size is still the RawWidth*Rawheight(the snippet in class prepareAccumulation : _structureTensor4=new NPPImage_32fC4(_pefFiles[refImage].RawWidth,_pefFiles[refImage].Height)). Can you tell me what is the difference between your relization and the description in the paper, thanks!

Hi

How do I adjust the parameter to remove haze with super-resolution enabled?
I follow the steps as you mentioned with the default setting. But the result has some artifacts and purple haze after pressing the "accumulate".

Super-resolution
https://imgur.com/a/4OqWKAR

Single frame zoom-in
https://imgur.com/a/7CcUfca

You can download my dataset from here(https://reurl.cc/j77mNL).

Thank you.

Hi Michael, Thanks for your helpful job.
My tutor gave me a 3060, but it does not support cuda10. So the error shown in the title appeared under cuda11. How can I update ImageStackAlignator with your latest version of ManagedCuda?
Thanks again！

I converted Fuji RAF files to DNG using Adobe DNG Converter.
The program crashes when trying to load them.
How hard would it be to add support for Fuji's non-standard 6x6 sensor patter?

It's a great work! I'm a little confused, I Check the box of Super resolution and also re-clicked Prepare accumulation, but the output image still keep the same size as the original image, without a two fold pixels. Thank you very much for your answer！

Hi @kunzmi , your great code was implemented using C#. I found that Microsoft has provided the C# or .NET runtime library running in Ubuntu.

My question is, can the ManagedCUDA and the code in this repository be run in Ubuntu with that .NET SDKs? Have you tried to run the code in Ubuntu? Would it be slow to run the code using .NET framework?

Your ManagedCUDA has packaged many low-level CUDA functions. Is there an alternative using C++?
I found that, if I need to convert the ManagedCUDA to its C++ version, that would be a huge project for me.

If convenient, could you please give some advice? Thanks.

Hello, I am a beginner of CUDA. I downloaded your code and tried to run it, but I met some problems.

My laptop environment is VS2017(professional)+ GTX960M+CUDA10.0+DirectX12.

When I start the “TestGoogleSuperRes.sln” file, it reported an error " ManagedCuda.CudaException:“ErrorInvalidPtx: This indicates that a PTX JIT compilation failed.” (located at “CUmodule modDebayer = _ctx.LoadModulePTX("DeBayerKernels.ptx");“) . CTX cannot be evaluated correctly.

Then I tried to debug each project. When I started a new instance on the ‘Kernels“ project alone, I got an error “fail to find the file Kernels in PEFStudioDX”. Is this matter? And is the ”PTX JIT compilation failed“ related to this situation?

I would be grateful if you could let me know what caused the above two situations and help me to solve them..

Best wishes.

“System.IO.FileNotFoundException”( ManagedCuda.dll )
“ManagedCuda.CudaException”(ManagedCuda.dll )
ErrorInvalidPtx: This indicates that a PTX JIT compilation failed.

Hi Michael @kunzmi,
I've tried to test run this project on a set of converted DNG raw files (from canon's CR2).
I'm getting a null exception right after loading a few images: basically at the point of getting a value from the null profile (absent in ExtraCameraProfiles.xml):
DNGfile.cs L306:

ExtraCameraProfile profile = profiles.GetProfile(make, uniqueModel); 
// profile is null for everything besides PENTAX K-3

Question: what is the suggested way to generate or get one for my camera?

Thank you very much!

Hi Michael, Thanks for the helpful job.
I meet an error and have not got the final results. I have finished step1 and 2 and 3 successfully, but when clicking Prepare accumulation, it shows the exception: ManagedCuda.NPP.NPPException:“Device allocation error” .
Could you please help me to settle this problem?
Thank you so much~

How do you open the main window what file do I run or open?

Quick context:
This project requires 10.1 for building cuda kernels
Dependencies in Managed CUDA (NPP dlls specifically) rely on 10.0.
Latest published Managed CUDA is 10.0.

@kunzmi any update on bumping all of that to single CUDA version? Let's say 10.2? :)

Thanks for open source implementation, and I also have my own implementation under development hasn't been public.
May I ask some questions about detail?
After kernel reconstruction, I add one more aligned respect to base frame, but it doesn't show in the pipeline of paper, which cause some information lost because of alignment or registration. Do you keep this?

Have you tried compare with other methods of demosaicing, e.g. VNG etc. I created some synthetic data from Kodak and McMaster, but the performance is not good as they said in paper, e.g. SSIM or PSNR.

Thanks, Hao!

Are there tests that verify functional correctness overall and of individual steps?

In other words, does it work? :)

Hi,

I build the three projects successfully and generate each ptx file in the correct location but it shows exception thrown as follows when I execute it

System.NullReferenceException: 'Object reference not set to an instance of an object.'

at ImageStackAlignatorController.cs, line 1004 CUmodule modDebayer = _ctx.LoadModulePTX("DeBayerKernels.ptx");

and output:

Exception thrown: 'ManagedCuda.CudaException' in ManagedCuda.dll
'PEFStudioDX.exe' (CLR v4.0.30319: PEFStudioDX.exe): Loaded 'C:\Program Files (x86)\Microsoft Visual Studio\2019\Community\Common7\IDE\Remote Debugger\x64\Runtime\Microsoft.VisualStudio.Debugger.Runtime.dll'. 
Exception thrown: 'System.NullReferenceException' in PEFStudioDX.exe
An unhandled exception of type 'System.NullReferenceException' occurred in PEFStudioDX.exe
Object reference not set to an instance of an object. 

is it seems load ptx file failed?

(I'm developing in visual studio 2019 and CUDA toolkit 10.1 with .NET framework 4.7.2)

Hi @kunzmi , I found that in your code, there is a file named ExtraCameraProfiles.xml to get the camera profiles.
Is this a necessity for this approach?

If we just have several images without the camera parameters, is this method still suitable for their multi-frame super-resolution?

There is no color points in the original night image. After reading it into the project, it is found that there are color points. Have you ever been in this situation?

HI,
I'm curious whether you were able to get the original code from the Google authors as a starting point, or did you develop everything from scratch? Do you know if the Google authors released any code and if so, where to find it?

I will compare your analysis with their paper and see if I can shed further light on what you've discovered.

Thanks,
Max Buchheit

Would it be possible for you to share some sample RAW images, e.g. the 5 New York images shown in the README? Thanks!

Hello,

When I open the program using dng files, the program crash.
Here is the dng files : https://drive.google.com/drive/folders/105Vzur0sDWTvamjL43Ampbcg4InNT7EQ?usp=sharing

Liza