Suggestion for an alternative implementation for --no-dlopen option about implib.so HOT 15 CLOSED

First ruff test worked, sadly I have no time to look closer at it today but will check it again next week.

Thanks a lot !!!

from implib.so.

Yes, I was able to successfully test the new version.

Thank you again for your effort!

from implib.so.

I see, so you need to load library manually with some custom logic. Currently we support the following use-cases:

• implement a callback function to load library (and initialize handle) and call it automatically at program start:

implib-gen.py --dlopen-callback=mycallback libxyz.so

• same but load library on first call to any of its functions:

implib-gen.py --dlopen-callback=mycallback --lazy-load libxyz.so

Do you think any of them suits your needs?

from implib.so.

My use case is to reopen and close the library many times like that:

• open library
• load all symbols
• run some code with the lib
• close the lib

I need it to "fake" the initialization of some c software to test it. This is needed because static variables in modules and functions keep their value.
First I used the approach with the --dlopen-callback and --no-lazy-load flags and it worked. Yesterday I thought it is only working because the library gets loaded to the exact same memory address every time but I just realized this is wrong. If I store the pointer I return with the callback function I can let it point the reopened libraries and _libDut_so_tramp_resolve() should load the symbols from the reloaded library. Example Code:

implib-gen.py --dlopen-callback=mycallback --no-lazy-load libxyz.so

static void *h;

void *mycallback(const char *lib_name) {
  lib_name = lib_name;  // Please the compiler

  h = dlopen("libxyz.so", RTLD_NOW);
  if (h)
    return h;
  exit(1);
}

setup_beforeFirstTest(){
    dlclose(h);
}


// Application Code
test_runner(void){
    //setup
    h = dlopen("libxyz.so", RTLD_NOW);
    _libxyz_tramp_resolve_all();

    //test
    useFunctionFromLib();


    //teardown
    dlclose(h);
}

The only thing that is a bit ugly with this approach is that I have to close the library manually after the startup (with setup_beforeFirstTest()).
If I use --lazy-load at the second run a segmentation fault occurs. I assume this happens because the _libDut_so_tramp_table entry is not cleared after I close the library manually and therefore the library is not loaded before the function is called. Might be something that could be improved in your script. For example by providing a function to clear the _libDut_so_tramp_table or a function to close the library manually from outside.

from implib.so.

I was wrong again with my last comment. I think it is still only working because the library gets loaded to the exact same space in memory. I just realized that you copy with the lib_handle the address of the loaded library. In my head I was thinking its a pointer to a pointer (hope you get it, don't know how to explain it with more details).

Do you think it is an option to make your script support reloading libraries? Would be happy to support you if you like

from implib.so.

Do you think it is an option to make your script support reloading libraries? Would be happy to support you if you like

Yes, I think it's possible. What if I provide callbacks to load library and reset tables/cached handles? The code will then look like

// Application Code
test_runner(void){
    //setup
    _libxyz_tramp_load_library(); // Will call dlopen callback
    _libxyz_tramp_resolve_all(); // Note that this is not necessary, each symbol will be resolved on first call

    //test
    useFunctionFromLib();


    //teardown
    _libxyz_tramp_reset_tables();
    dlclose(h);
}

from implib.so.

Cool, yes I think that should work. Some additional thoughts:

• If you already provide a load library function, a close or unload function would be more consistent (more intuitive from user perspective). But I am fine with calling those two functions.
• I assume you already have that in mind but you have to make sure that lib_handle gets updated after i use the _libxyz_tramp_load_library()

Thanks a lot for your fast responses and your effort

from implib.so.

Could you check the variant at branch gh-14 ? It should work with --no-dlopen and

extern void _libxyz_so_tramp_reset_tables(void);

test_runner(void){
    //setup
    h = dlopen("libxyz.so", RTLD_NOW);

    //test
    useFunctionFromLib();

    //teardown
    _libxyz_so_tramp_reset_tables();
    dlclose(h);
}

I've tried to add a good testcase but your code may still differ.

from implib.so.

Finally I was able to test your solution again. Today the test with your new version did fail. Sadly I can't reproduce what I tested last week to make sure if I tested the right thing last week.
The problem I came across today is, that the symbol useFunctionFromLib can't be resolved by dlsym with RTLD_NEXT as argument. It seams that it is not looking in for the symbol in my library. Do you know a way to make sure RTLD_NEXT can find my library? Or would it be an option to provide the handler to my library from outside like I suggested in the opening question:

As you mention with the following FIXME comment you have to somehow find the loaded lib_handle if you don't want your script to load the library by itself:

// FIXME: instead of RTLD_NEXT we should search for loaded lib_handle
// as in https://github.com/jethrogb/ssltrace/blob/bf17c150a7/ssltrace.cpp#L74-L112

To be honest I don't really understand what the approach from the ssltrace project does. For my use case it would be okay to provide the handler myself to the *_tramp_resolve_all(void) function. Therefor I modified your script and generated an other function *_tramp_resolve_all_fromHandler(void *handler)

void _${lib_suffix}_tramp_resolve(int i, void *h) {
  assert((unsigned)i + 1 < sizeof(sym_names) / sizeof(sym_names[0]));

  CHECK(!is_lib_loading, "library function '%s' called during library load", sym_names[i]);

#if NO_DLOPEN
  // FIXME: instead of RTLD_NEXT we should search for loaded lib_handle
  // as in https://github.com/jethrogb/ssltrace/blob/bf17c150a7/ssltrace.cpp#L74-L112
  //h = RTLD_NEXT;
#elif LAZY_LOAD
  h = load_library();
#else
  h = lib_handle;
  CHECK(h, "failed to resolve symbol '%s', library failed to load", sym_names[i]);
#endif

  // Dlsym is thread-safe so don't need to protect it.
  _${lib_suffix}_tramp_table[i] = dlsym(h, sym_names[i]);
  CHECK(_${lib_suffix}_tramp_table[i], "failed to resolve symbol '%s'", sym_names[i]);
}

// Helper for user to resolve all symbols
void _${lib_suffix}_tramp_resolve_all(void) {
  size_t i;
  for(i = 0; i + 1 < sizeof(sym_names) / sizeof(sym_names[0]); ++i)
    _${lib_suffix}_tramp_resolve(i,0);
}

void _${lib_suffix}_tramp_resolve_all_fromHandler(void *handler) {
  CHECK(handler, "no valid handler provided");
  size_t i;
  for(i = 0; i + 1 < sizeof(sym_names) / sizeof(sym_names[0]); ++i)
    _${lib_suffix}_tramp_resolve(i,handler);
}

If no handler is provided you could still use the "ssltrace approach" to search for the lib

What do you think about this approach? Do you like to add it to your project? If you can explain me how the ssltrace approach works I might try to implement it too.

from implib.so.

Do you know a way to make sure RTLD_NEXT can find my library?

Hm, I see, perhaps the decision to use RTLD_NEXT was not a good one after all. I assume if you change it to RTLD_DEFAULT the symbols will be resolved successfully?

from implib.so.

No, with RTLD_DEFAULT it can't find the library aswell. If I debug and change h manually to the handler I receive from the dlopen call it is working.
I noticed aswell that the symbol is found if I open the library with the RTLD_GLOBAL | RTLD_LAZY flags both with RTLD_NEXT and RTLD_DEFAULT. That's probably the reason why you tests are working. RTLD_GLOBAL seems to be the reason but this is no option for my use-case because I want to reset the symbols.

from implib.so.

I noticed aswell that the symbol is found if I open the library with the RTLD_GLOBAL | RTLD_LAZY flags both with RTLD_NEXT and RTLD_DEFAULT. That's probably the reason why you tests are working. RTLD_GLOBAL seems to be the reason but this is no option for my use-case because I want to reset the symbols.

Right, --no-dlopen will not work with RTLD_LOCAL because it tries to find symbols in global namespace.

I've pushed another example to gh-14 which shows how RTLD_LOCAL could be supported. But I think your idea to add an API to explicitly specify handle (something like _${lib_suffix}_tramp_set_handle(handle)) is the most straightforward and I'll do that.

from implib.so.

Your new example with the callback approach is working with my tests too. I don't like that it opens the library automatically at startup but it's not a problem. Therefore I prefer the solution with the extra API _${lib_suffix}_tramp_set_handle(handle) too

from implib.so.

Therefore I prefer the solution with the extra API _${lib_suffix}_tramp_set_handle(handle) too

Could you check the updated gh-14 branch (it's been force-pushed)? I believe the tests/multiple-dlopens-3 test matches your use-case.

from implib.so.

Pushed fix into trunk.

from implib.so.